As part of our ongoing exploration of how AI can support accurate data retrieval, we built a carefully curated, AI-ready knowledge base of Urban’s Upward Mobility Initiative resources. But how do we know the system works as intended—that its responses accurately reflect the source material and align with the framework’s intent?
As technical stakeholders, we couldn’t answer that question alone. So we turned to domain experts. By embedding experts throughout the evaluation process, we designed a repeatable framework that clearly defines a correct response, measures performance against those criteria, and surfaces specific areas for improvement.
Our Setup
Step 1: Define what success looks like
To design our evaluation framework, we first needed to define our criteria. In addition to the accuracy, relevance, and completeness of a response, we also cared about context retrieval precision (e.g., pulling the correct section of the correct document), proper citation use, and the model’s ability to calibrate its confidence level.
Because we wanted our evaluation to align with the framework’s intent, we decided not to use an LLM-as-judge evaluation method. Instead, we relied solely on experts and easily understood automated metrics. To facilitate this evaluation framework, we created a short description of how our experts should evaluate each criterion and identified relevant automated metrics.
To assess the model’s use of references, for example, we posed two questions:
- Human evaluators determined, “Does each link or URL provided in the responses directly support the claim it is cited for in the model response?”
An automated metric determined, “Is the URL cited in the response a valid and working link, returning a 200 response?”
Notes: More information about the scores in the table can be found at the following links: BERT scores, BLEURT scores, and Promptfoo scores.
Step 2: Build our test suite with experts
After defining what success looked like, we categorized evaluation questions because we expected LLM performance to vary by question type. We relied on Wang and colleagues’ 2025 framework (PDF), which outlines a set of task and information categories to which each question pertains.
With a clear set of criteria and a classification system for our questions, we then asked our experts to generate realistic prompts, the requirements a correct answer must meet, and the correct sources in the knowledge base to support the answer. Those same experts then evaluated AI-generated responses against this information.
Step 3: Make evaluation easy and consistent for humans
To reduce the burden on our evaluators and produce consistent results, our team built a Shiny app that included clear instructions, a 1–5 Likert scale for each evaluation criterion, and an open-text field to collect additional thoughts or observations. We also made it easy for reviewers to navigate to different questions, track and save progress, revisit old responses, and give a final recommendation.
We stored responses in a CSV backend that prevented reviewers from seeing each other’s ratings, as we wanted to assess reliability across our four reviewers.
Step 4: Iterate and measure improvement
We conducted multiple full evaluations to identify improvement areas and track progress. After the first round of evaluation, we made some technical changes identified by users:
Problem: For certain questions, the correct document was consistently missed by our Retrieval Augment Generation (RAG) system, leading to ungrounded responses.
Solution: We steered away from intelligent metadata filtering, which removed irrelevant documents from consideration to make retrieval more efficient. After this decision, correct document chunks were no longer filtered out. (Metadata filtering may help with larger knowledge bases, but it was overzealous and without added value in this case.)
Problem: The model sometimes “oversynthesized” sources, pulling in a wide array of information when the answer may only depend on one document or text snippet.
Solution: We amended our system prompt to instruct the model to limit its response to the sources needed to fully answer a question.
Problem: The model did not always prioritize retrieved documents correctly, as it did not have encoded knowledge of how the UMI team thought about source hierarchy.
Solution: We overhauled our `document_type` metadata tag to include a more detailed set of possible document types. For each type, we added short instructions for how the model should interpret and prioritize based on the nature of a user’s question.
What We Learned
Our model passed the litmus test for usability.
After two rounds of evaluation, we felt comfortable concluding that the model was generally reliable and high performing, especially for simple task and information categories. On context retrieval, we found that the RAG system returned at least one relevant document nearly all the time (91 percent hit rate@k) and found relevant documents most of the time (79 percent recall@k), with the RAG surfacing only the five most relevant documents per query (k=5).
Generally, our human evaluators indicated positive scores from 1 to 5 for our knowledge base encoding, but performance did decline for more complex task and information categories. We also received open-ended feedback about noisiness and verbosity in responses that we see as easier to control via system prompts and guardrails than more fundamental issues like inaccuracy or bias. The heatmap below summarizes results by task category across each dimension.
Our iterative process led to significant improvements in the evaluation results.
Our solutions could not have been identified or implemented without a clear sense of the problems flagged by our human reviewers. Most scores improved each round, with accuracy improving by an average of 1.23 points (3.33 to 4.56), and relevancy by 1.07 points (3.40 to 4.47).
However, results for multihop reasoning questions worsened, which we expect is because of the changes made to document prioritization. In some ways, seeing direct responsiveness to the changes we made gave us more confidence that we could exert a strong level of control over the system.
Human-centered evaluation is costly, but automated metrics could not fill the gap.
At scale, it will be difficult for our experts to provide detailed feedback on prompts deriving answers from many thousands of documents. But the automated outcome metrics did not bridge the gap well.
BERT and BLEURT scores for accuracy require full ground-truth answers to compare against, which limits their practicality. Promptfoo’s refusal score missed nearly all the syntax our model used when declining to answer a question. Recall and hit rate were helpful high-level indicators of how our RAG system was performing, but they did not help diagnose specific retrieval issues or evaluate the quality of final answers generated by the AI model.
While we are highly reluctant to entrust LLM-as-judge frameworks with the careful judgment that some of our evaluation dimensions require, especially more complex task and information categories, we plan to experiment with these frameworks to see if they can improve evaluation efficiency without increasing risk.
Looking Ahead
Our evaluation showed that our model struggled with open-ended, multidocument, and nuanced questions, and we started to run up against the limits of a system prompt-based approach. In this sense, evaluation can feel like a game of whack-a-mole: encoding new reliability guardrails in one file as new problems emerge.
Moving forward, we expect that this work will likely require agent-based solutions, where an agent can select the right tool in its arsenal to complete a given task.
