dhanush kumar

10 questions I asked before building our internal knowledge agent

· rag · ai-native

Aussi disponible en français →

We wanted an internal knowledge agent: ask a question in plain language, get an answer grounded in our Notion and Drive, with sources. The temptation was to start with the fun part — embeddings, retrieval, a streaming chat UI. I have watched that movie before, so instead I spent the first week writing down questions and chasing answers to them.

Here are the ten, grouped the way they actually mattered.

Scope

  1. Which sources are in v1? Notion and Drive. Written down, so "can it also read Slack" has an answer: v2, maybe.
  2. Which sources are explicitly out? Email, tickets, HR docs. "Explicitly out" is different from "not mentioned" — it survives the meeting where someone asks.
  3. What kinds of questions must it answer well? Process and how-do-I questions. Not analytics, not "summarize Q3".

Permissions

  1. What can it index? Only documents already visible company-wide. No per-user permission mirroring in v1 — that single decision deleted a month of work.
  2. Which LLM provider is approved for internal data? The one legal had already reviewed, not the one with the best benchmark that week.
  3. Where does it run and who pays? Our cloud, our project, a named budget line. An agent without an owner becomes abandonware with an API key.

Success

  1. Who is it for, and how many of them? Target: 20 weekly active users. A company wiki bot that five people use is a failed project that still costs money.
  2. What does a good answer look like? Grounded in a source, linked, and honest when it does not know. We set a bar of 60% of answers rated helpful.
  3. What happens to the questions it cannot answer? They get logged and reviewed — unanswered questions are a map of missing documentation. That report turned out to be half the value.
  4. When do we kill it? If usage or helpfulness is still below target after two months, it gets turned off. Agreed in advance, when nobody was defensive.

Only after this did we write code, and the code was almost boring: the scope was small, the permission model was trivial by construction, and the eval set fell straight out of question 8.

The lesson I keep relearning: the questions are the engineering. The retrieval pipeline was two weeks. Getting ten answers everyone agreed on was the part that made it survive contact with the company.

← All posts