The problem
Engineers were losing significant time on the first fifteen minutes of every bug ticket: re-reading the report, searching for related code, and pulling logs to confirm what was actually happening before real debugging could start. Compounding that, internal LLM tools kept confidently answering questions using outdated or generic knowledge, because they had no access to the product’s actual documentation, architecture decisions, or historical context — and every new agent project meant re-solving that retrieval problem from scratch.
What I built
A triage agent that runs the moment a new bug ticket lands, backed by a RAG knowledge base built to be shared rather than rebuilt per project. Confluence pages and S3-stored artifacts are chunked, embedded, and stored in a pgvector-backed PostgreSQL instance once; the triage agent, internal chat assistants, and ad hoc scripts all query the same knowledge base instead of each maintaining their own retrieval logic. For triage specifically, the agent pulls the ticket via the Atlassian MCP server, retrieves semantically related documentation and prior incidents from that shared knowledge base, cross-references relevant application logs through the New Relic API, and assembles a structured triage report — likely root cause, related code paths, relevant log excerpts, and a recommended starting point — attached directly to the ticket before an engineer picks it up.
How it works
- Ingest: New ticket triggers the agent via a Jira webhook.
- Retrieve: The agent embeds the ticket description and searches the shared pgvector knowledge base for semantically related documentation and prior incidents. Every chunk carries source metadata (page, section, last-updated date), so retrieved context traces back to its origin instead of reading as model guesswork.
- Cross-reference: Relevant service names and timestamps are used to pull matching New Relic log windows.
- Synthesize: Claude combines ticket context, retrieved knowledge, and log evidence into a structured report, written directly back to the ticket as a comment.
Built to be shared, not rebuilt
The knowledge base itself — Confluence and S3 content chunked into overlapping semantic segments sized for retrieval quality, not just token limits — isn’t specific to triage. It’s the same retrieval layer other internal tools query, so the investment compounds: the second and third consuming project got grounded retrieval for free instead of re-solving chunking, embedding, and source-tracing from scratch.
Impact
Triage reports now exist before an engineer opens the ticket, which meaningfully shortened the time between a bug being filed and a fix actually starting. Answers are traceable to a real source instead of model guesswork, and every agent project since has queried the shared knowledge base rather than rebuilding its own retrieval layer.
Stack
Claude, Atlassian MCP, PostgreSQL + pgvector, OpenAI embeddings, Confluence API, AWS S3, New Relic API, Python.