A source-linked answer can still be wrong. The source may be stale, the retrieved passage may not answer the question, or the user may not have permission to read it. Evaluation needs to follow that entire chain.
Build a question set from the work
Collect the questions people actually need answered. Include straightforward lookups, comparisons across documents, questions with missing context and questions the knowledge base cannot answer. Keep a record of the expected evidence, not just a preferred sentence.
Separate the set used for development from a set reserved for checking changes. Otherwise the system may appear to improve simply because the prompts were tuned to familiar examples.
Inspect retrieval and generation separately
First check whether the relevant material was found. Then ask whether the answer was supported by that material. A polished answer cannot compensate for missing evidence, and a correct passage does not ensure that the response used it faithfully.
Review document permissions before retrieval as well as at display time. A user should not receive an answer derived from a restricted document simply because the document itself is hidden from the interface.
Design the uncertain answer
Define what the assistant should do when the evidence is incomplete: request a detail, cite the limited information it found or offer a human handoff. Avoid forcing a confident answer to every question.
Treat document updates, model changes and retrieval changes as reasons to run the evaluation again. Our sample knowledge demo shows an interaction pattern using fixed fictional policies; it does not demonstrate live model accuracy or production retrieval quality.
Measure whether the system finds permitted evidence and represents it accurately. Fluency alone is not a release criterion.
