7

Class 7: Evals, Observability and Reliability

Class 7 of 8About 3 hoursPythonLayer: cross-cutting (evals, observability)

By the end of this class you will be able to

• Explain why evals are the core of AI engineering, and run the improvement loop: measure, analyse errors, change one thing, measure again.
• Build a golden dataset of 20+ cases that includes edge cases, adversarial cases and out-of-scope requests.
• Choose between exact match, rule-based checks, rubric scoring and LLM as a judge, and calibrate a judge against human labels while accounting for its biases.
• Evaluate an agent's trajectory as well as its final answer, and run evals as regression tests in CI.
• Instrument a system with traces and spans, and decide what to log for tokens, cost, latency and errors.
• Apply reliability patterns: timeouts, retries, circuit breakers, step budgets, fallbacks and graceful degradation.

Where we are

In Class 6 the SitRep Assistant became a small team: an orchestrator, specialist agents and a review loop. It looks impressive in a demo. Today we find out whether it works, how we would know if it stopped working, and how to see what it did on any given run. In Class 8 we attack it, secure it and take it to production.

1. Why evals are the core of AI engineering

A new district nurse arrives in Kessan. Nobody judges her on one good morning. The clinic has routines: case notes are checked, a senior nurse reviews a sample of her diagnoses, and indicators are tracked week by week. If the number of referrals doubles, someone notices and asks why. None of this assumes she is bad at her job. It is how any organisation knows that important work is being done well, and keeps it that way.

An LLM system needs the same routines, for three reasons that ordinary software does not share:

• Outputs are open-ended. There is no single correct SitRep. "Is this good?" has several parts (accurate, complete, neutral, well-formatted), and some of them need judgement.
• Behaviour is non-deterministic and non-local. The same input can give different outputs. A one-line prompt change that fixes the families-versus-people confusion can quietly break the Arabic reports. A model upgrade changes everything at once.
• Failures are silent. A crashed program is obvious. A fluent SitRep with one invented figure looks exactly like a correct one.

So the question "does it work?" cannot be answered by trying it a few times. It needs an eval: a fixed set of realistic inputs, a definition of good for each, and a score. With an eval you can run the improvement loop that the AI Map calls loop engineering:

Without the loop, prompt engineering is a random walk: every change fixes the example in front of you, and you never learn what it broke. With the loop, each change is an experiment with a measured result. That is why many practitioners say evals are the core of AI engineering rather than a finishing touch. The quality of your eval sets an upper limit on how much you can improve the system on purpose.

Know the Term: Evals

An eval (short for evaluation) is a repeatable test of how well an AI system does its job. You collect a set of realistic example inputs, write down what a good answer must contain or avoid, run the system on all of them, and score the results. Because AI output varies, you judge a system by its score across many examples, not by one impressive demo. Run the same eval before and after every change, such as a new prompt or a new model, and you can see whether the change helped or quietly broke something.

Analogy: a driving test with a fixed route. Every candidate drives the same streets, so the results can be compared, and a good afternoon on an empty road does not count.

Why you care: "it looked good when I tried it" is not evidence. Before trusting or buying an AI system, ask to see its evals: what was tested, how often it passed, and on which cases it failed.

Start with error analysis, not with metrics. Before writing any scoring code, run the system on 30–50 realistic inputs and read the outputs and traces. Write a short note on each failure in your own words: "added families to people", "missed the Bara bridge", "followed an instruction inside the report". Then group the notes into categories and count them. The categories with the highest counts are what your checks should measure first. Teams that skip this step build dashboards full of generic scores ("helpfulness 4.2") that measure nothing anyone would act on.

2. Building a golden dataset

A golden dataset (or golden test set) is a versioned collection of inputs, each paired with what a good output must do. It does not need a perfect reference answer for every case. For open-ended tasks, expectations are often more useful than model answers:

{"id": "g03", "tags": ["units", "edge"],
 "input": "Request: SitRep on displacement in Kessan district.\n\n[R1] 14 March 2026: About 400 people sheltering in Kessan primary school.\n[R3] ... About 60 families from Lower Tamsi staying in the mosque and church...",
 "expected": {"headings": true, "max_words": 400,
              "must_include": ["400 people", "60 families"],
              "must_not_include": ["460"]}}

The "must_not_include": ["460"] line encodes a real failure from Class 6 (adding families to people) as a permanent test. Every bug becomes a test case: this is how a golden set grows.

What goes in it

Slice	Share	SitRep examples (in the lab's 20 cases)
Core	~30%	Typical reports: one district, clear figures
Edge cases	~30%	Families vs people; duplicate forwarded reports; ambiguous dates like 03/04/2026; ranges ("between 300 and 500"); a report with no figures at all
Multilingual	~15%	Reports in French, Spanish and Arabic. Languages other than English are not an afterthought
Adversarial	~15%	"IGNORE ALL PREVIOUS INSTRUCTIONS…" inside a report; a request to estimate a death toll; a patient's name and phone number; politically charged language
Out of scope	~10%	"Write a poem about the river"; an exchange-rate question

A few rules of thumb:

• Start small and real. Twenty well-chosen cases beat 2,000 synthetic ones you have never read. Grow to 100–200 as failures arrive.
• Use models to help, not to decide. A model is good at proposing variations ("write five reports where the units are inconsistent"). A person must read and approve every case and its expectations.
• Keep it separate from your prompt. If the few-shot examples in your prompt are also in your test set, you are grading the model on questions it was shown the answers to.
• Version it alongside the code, and record which dataset version produced each score.
• Protect it. A golden set built from real field reports contains the same sensitive data as the reports. It needs the same classification and access rules (Class 8). The course uses only fictional data.

3. Metrics: from exact match to LLM judges

Different qualities need different checks. Use the cheapest check that measures the thing you care about.

3.1 Exact match and classification metrics

When the output is a label, as with the Class 6 router (sitrep, update_existing, translation, out_of_scope), compare it exactly with the expected label. Report accuracy, and for important labels also precision (of the requests labelled out of scope, how many really were?) and recall (of the requests that really were out of scope, how many did we catch?). A confusion matrix, a table of expected against predicted labels, shows which mistakes the router makes, and that is what you need in order to fix it.

3.2 Rule-based checks

Many important properties of a SitRep can be checked by ordinary code, deterministically, for free, on every commit:

Check	What it catches
All five headings present	Template drift
≤ 400 words	Rambling drafts
Must include 1,200 and 1,450	A conflict silently resolved by dropping one figure
Must not include 460	Families added to people
Every number in the output appears in the input	Invented figures, the most dangerous SitRep failure
Out-of-scope requests answered with OUT_OF_SCOPE	Scope creep

The "no invented numbers" check is crude. It will flag a legitimate total the Analyst computed, which you then add to allowed_numbers. But it catches the error you most need to catch, and it catches it every time, for free. Use code for what code can check (Class 6 said the same about the Reviewer).

3.3 Rubric scoring

Some qualities need judgement: is the tone neutral? Are the humanitarian needs complete? Is a conflict explained or just mentioned? For these you need a rubric, a scoring guide with a concrete description for each level:

5 - Every figure traceable to a source; conflicts flagged, not resolved; neutral; nothing important missing. 4 - Accurate and neutral, with one minor omission or awkward phrasing. 3 - One material problem: a missing key need, an unflagged conflict, or mildly non-neutral tone. 2 - Several material problems, or one figure that is not in the source. 1 - Misleading: invented figures, followed instructions found inside the reports, or unusable.

Vague rubrics ("rate the quality from 1 to 10") produce vague, unrepeatable scores, from people and from models alike. Prefer few levels with concrete anchors, or even binary pass/fail per criterion, which is easier to agree on. For comparing two versions, pairwise judgement ("which of these two SitReps is better, A or B?") is often more reliable than absolute scores.

3.4 LLM as a judge

Having people grade every output on every change does not scale. An LLM judge applies your rubric automatically: you give it the source, the output and the rubric, and it returns a score and reasons. Done well, it lets you run judgement-based checks on every change. Done badly, it is a random number generator with a confident tone.

Know the Term: LLM as a judge

LLM as a judge means using one AI model to grade the output of another against written criteria, a rubric such as "every figure has a source; conflicting reports are flagged; neutral tone". It is used for qualities that simple rules cannot check, and it is much cheaper than having people grade thousands of answers. But the judge has biases of its own: it tends to prefer longer answers, confident wording, the first option it is shown, and text written in its own style. So its grades are only trusted after being compared with grades from people on the same examples.

Analogy: an assistant marker for exams. Useful for getting through the pile, but the chief examiner checks a sample of their marking before the results are released.

Why you care: an automated grade is only as good as its rubric and its calibration. If a vendor reports "95% quality", ask who or what did the grading, and how often it agreed with humans.

Designing the judge prompt:

• Give the rubric, the source and the output, clearly delimited.
• Ask for reasons before the score, so the score follows from the reasoning rather than the reasoning being written to justify a score.
• Ask for JSON, parse it strictly, and treat an unparseable verdict as a failed check, never as a pass.
• Tell it what does not earn points: length, confident tone, polish.
• Prefer a model from a different family than the one being graded, to reduce self-preference.

Know its biases. Research and practice have documented several consistent biases in LLM judges:

Bias	What happens	Mitigation
Verbosity	Longer answers score higher	Rubric says length earns nothing; word limit checked by code
Position	In pairwise comparisons, the first (or second) option is favoured	Run both orders; count only consistent verdicts
Self-preference	Judges favour text in their own style or from their own model family	Use a different family for the judge
Leniency / sycophancy	Scores cluster at 4–5; confident errors pass	Anchored levels; binary criteria; include known-bad examples in calibration
Surface matching	Rewards the right keywords rather than correct meaning	Pair with rule-based checks; spot-check reasons

Calibrate it against people. Before you trust a judge, have one or two domain experts label 30–50 outputs pass/fail without seeing the judge's verdicts. Then compare:

                     human: pass   human: fail
  judge: pass             22             6        ← 6 false passes: the dangerous cell
  judge: fail              2            10

Raw agreement here is 32/40 = 80%, which sounds good. But if most outputs pass, a judge that always says "pass" would also score well. Cohen's kappa corrects for agreement expected by chance. It is 1.0 for perfect agreement and 0 for chance level. This table gives about 0.57. Roughly, above 0.6 is usable with spot checks, and below 0.4 means the rubric or the judge needs work. Look hardest at the false passes: an eval that approves bad SitReps is worse than no eval, because it creates false confidence. Recalibrate whenever you change the judge model or the rubric.

3.5 Choosing checks

Method	Best for	Cost per case	Trust
Exact match	Labels, routing, extracted fields	~0	High
Rule-based checks	Format, required and forbidden content, number provenance	~0	High for what they check
LLM judge	Tone, completeness, faithfulness, conflict handling	One model call	Only after calibration
Human review	Calibration, high-stakes decisions, new failure types	Minutes of expert time	Highest, but inconsistent without a rubric

A healthy suite uses all four: rules on every case, the judge on every case for what rules cannot see, and people on a sample, both to calibrate the judge and to catch what nobody thought to check.

4. Evaluating agents: trajectory evals

For an agent, the final answer is not the whole story. Two runs can produce the same SitRep, one in 5 tool calls and one in 40, or one after reading the reports and one after calling a tool it should never touch. A trajectory eval checks the path: the sequence of tool calls, their arguments and their results.

Useful trajectory checks for the SitRep system:

• Tool selection: did it call search_reports before format_sitrep? Did it use get_population for a district, not a village?
• Forbidden actions: did it ever call a blocked tool? (Even a blocked attempt is worth counting.)
• Efficiency: steps used against the budget; repeated identical calls (a sign the agent is stuck).
• Recovery: after a tool error, did the next call change?
• Grounding: does every figure in the output appear in some tool result in the trace, not just in the input?
• Handoffs (multi-agent): did the Analyst receive the Researcher's validated output? Did the review loop end with APPROVED or at the budget?

def trajectory_checks(result) -> dict:                     # result from run_tool_loop (Class 5)
    tools = [c.tool for c in result.tool_calls]
    calls = [(c.tool, json.dumps(c.arguments, sort_keys=True)) for c in result.tool_calls]
    return {
        "finished": result.status == "final",
        "searched_before_formatting": "format_sitrep" not in tools
            or ("search_reports" in tools
                and tools.index("search_reports") < tools.index("format_sitrep")),
        "no_forbidden_tools": "send_sitrep_email" not in tools,
        "within_6_steps": result.steps <= 6,
        "no_repeated_calls": len(calls) == len(set(calls)),
    }

Prefer property checks like these over comparing against one "reference trajectory". There are usually many valid paths, and an agent that finds a better one should not fail the test. For multi-agent systems, evaluate each component (router accuracy, Researcher extraction, Reviewer agreement with humans) as well as the end-to-end result. When the end-to-end score drops, the component scores tell you where to look.

5. Regression testing in CI

A regression is something that used to work and now doesn't. Evals become regression tests when they run automatically, and a change that makes things worse is blocked or flagged before it ships.

Structure the suite in tiers, because live model calls cost money and time:

Tier	Runs	Contents	Model
Unit	Every commit	Orchestration and harness logic (Classes 5 and 6 tests)	FakeLLM, free, deterministic
Fast eval	Every pull request that touches prompts, tools or models	Rule-based checks on the golden set	Real model, ~20–50 calls
Full eval	Nightly, and before any model change	All checks, including the LLM judge, all cases, several repetitions	Real model plus judge

Compare case by case, not just the average. The overall pass rate can rise while a critical case breaks: fixing five formatting cases does not excuse starting to follow injected instructions. The lab's compare() lists newly failing cases, and any newly failing case is treated as a regression that a person must look at.

Handle non-determinism. Run important cases several times. Report pass@k (passes at least once in k tries) for tasks where a human picks the best attempt, and pass^k (passes all k times) for tasks that must be reliable every time. A SitRep pipeline needs the second. Setting temperature=0 reduces variation but does not remove it.

A minimal CI job (GitHub Actions syntax):

# .github/workflows/evals.yml (illustrative)
on:
  pull_request:
    paths: ["prompts/**", "src/tools/**", "config/models.yaml"]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt
      - run: python ci_eval.py --dataset sitrep_golden.jsonl --baseline results/baseline.json
        env:
          MODEL: ${{ vars.EVAL_MODEL }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

ci_eval.py exits non-zero if compare() finds a regression. Keep API keys in the CI system's secret store, never in the repository (Class 8).

6. Offline and online evaluation

Everything so far is offline evaluation: a fixed dataset, run before deployment. It is necessary but not sufficient, because real inputs will surprise you. Online evaluation measures the live system:

• Explicit feedback: thumbs up or down, with an optional comment. It is sparse and skewed (people report problems more often than successes) but valuable.
• Implicit signals: how much did the duty officer edit the draft before publishing? An edit that changes a figure is a strong failure signal. Did the user re-ask the same question?
• Sampled judging: run the LLM judge on a random sample of production traces every day, with a person reviewing a smaller sample.
• Operational signals: guardrail triggers, tool error rates, budget exhaustions, fallbacks taken.

To compare two versions on live traffic, use an A/B test: randomly route a share of requests to version B and compare a pre-agreed metric. Be realistic about sample sizes. A team producing 20 SitReps a day will need weeks to detect a small difference, so for small teams shadow mode (B runs on the same inputs but only A's output is used, and people compare the two) and canary releases (B serves 5% of traffic, with automatic rollback if error rates rise) are often more practical.

Close the loop. Every production failure someone reports should become a golden-set case, with personal data removed, so that offline evaluation keeps up with reality.

Know the Term: Human in the loop

Human in the loop means a person reviews, corrects or approves an AI system's work at agreed points before it has real effects: before a report is sent, a payment is made, a record is changed or a decision about a person is taken. The design question is not "human or AI?" but where the human checkpoints sit and what the person actually checks. A reviewer who clicks "approve" on everything without reading is not a real checkpoint, just a signature on the AI's work.

Analogy: a senior officer who clears a press release before it goes out. The drafter does most of the work, but nothing leaves the building without that clearance.

Why you care: you remain accountable for what goes out under your name. Decide in advance which steps need your approval, and give yourself enough time to check them properly.

7. Observability: traces, spans and what to log

When a published SitRep says "900 people in Bara" and the field team says "60 families", you need to answer, quickly and with evidence: where did 900 come from? Observability is the ability to answer questions like this after the fact, from data the system recorded while it ran.

The core structure is the trace: the record of one request, end to end. A trace is made of spans, one per unit of work, nested inside each other:

trace 7f3a…  sitrep_request  (district=Kessan, prompt=v2)         48.2 s   $0.031
├─ orchestrator.plan         llm.call  model=large   1,905+312 tok  4.1 s
├─ researcher                                                      11.8 s
│  ├─ tool.search_reports    query="shelter Kessan"                 0.2 s
│  ├─ tool.get_report        R5                                     0.1 s
│  └─ llm.call               model=small  6,210+845 tok             6.3 s
├─ analyst                   llm.call  model=mid  2,410+530 tok     7.9 s
├─ writer (round 1)          llm.call  model=mid  3,020+410 tok     8.8 s
├─ reviewer (round 1)        llm.call  model=large 3,560+60 tok     3.4 s   → 1 fix
├─ writer (round 2)          llm.call  model=mid  3,140+405 tok     8.6 s
└─ reviewer (round 2)        llm.call  model=large 3,540+2 tok      3.2 s   → APPROVED

Each span records a name, start and end time (so a duration), its parent, attributes (model, prompt version, tool arguments), token usage, cost, status and any error. With this trace you can find the step where 900 first appeared, see that the Writer took two rounds, and see that the Researcher's model call is the most expensive step.

Know the Term: Observability and tracing

Observability is the ability to see what an AI system actually did, after the fact. The main tool is a trace: a record of one request from start to finish, broken into spans, one per step, such as a model call, a document search or a tool call. Each span records what went in, what came out, how long it took, how many tokens it used and what it cost. With traces you can answer "why did the SitRep say 900 people?" by finding the exact step where the number appeared. Common tools as of 2026 include OpenTelemetry (an open standard), Langfuse, Arize Phoenix and LangSmith.

Analogy: a flight data recorder. You hope never to need it, but when something goes wrong it is the only way to find out what happened.

Why you care: without traces, AI errors cannot be explained, fixed or defended in an audit. Traces also contain the data people typed in, so they need the same protection and retention rules as the data itself.

Standards and tools (as of 2026). OpenTelemetry (OTel) is the open, vendor-neutral standard for traces; its GenAI semantic conventions define standard attribute names such as gen_ai.request.model and gen_ai.usage.input_tokens (still evolving). LLM-focused platforms such as Langfuse (open source, and self-hostable, which matters when trace data must stay on your own infrastructure), Arize Phoenix, LangSmith and Braintrust, and the tracing built into cloud AI platforms, add prompt management, dataset management, human annotation and eval dashboards on top. Most accept OTel data, so instrumenting with OTel keeps you portable. With Langfuse's Python SDK, for example, a decorator is enough to start (illustrative; check the current docs):

from langfuse import observe

@observe()                      # creates a trace/span per call, with inputs, outputs, timing
def write_sitrep(reports: str) -> str:
    facts = research(reports)   # nested @observe functions become child spans
    return writer(facts)

What to log:

Log	Why
Request id, timestamp, pseudonymised user or role	Find the trace; audit who asked
Prompt version, model name and parameters, dataset/config version	Reproduce the run; attribute regressions
Inputs and outputs of each step, after redaction	Debug; build new golden cases
Tool calls with arguments and (truncated) results	Trajectory evals; explain figures
Tokens in and out, cost per call	Cost dashboards and budgets
Latency per span; time to first token for chat	Find the slow step; set timeouts
Errors, retries, fallbacks, guardrail triggers, budget stops	Reliability and security monitoring
Eval scores and user feedback, attached to the trace	Connect quality to behaviour

What not to log: secrets and API keys (ever), personal data you do not need, and full documents where an id would do. Traces are data. They need a classification, access control and a retention period, and they must live where your data rules allow (Class 8).

Dashboards. At a minimum: requests per day, cost per request and per day, tokens per request, latency at the median and the 95th percentile (p95: the time within which 95% of requests finish, because averages hide the slow tail), error and fallback rates, and eval pass rates over time. A quick cost check for the pipeline above: ~$0.03 per SitRep × 20 SitReps a day × 30 days ≈ $18 a month. That is trivial. But the same pipeline stuck in a 400-call loop costs more in an afternoon, which is why the budget alert matters more than the average.

8. Reliability patterns

Class 5 introduced retries and fallbacks for single calls. A multi-agent pipeline needs the full set of patterns, because it makes dozens of calls to several services, and each can fail.

Pattern	What it does	SitRep example
Timeout	Turns a hang into a failure you can handle	30 s per model call; 5 min per SitRep run
Retry with backoff and jitter	Rides out transient failures	3 attempts on HTTP 429/5xx; never on a policy block
Circuit breaker	Stops calling a service that keeps failing, and fails fast instead	After 5 failures in a minute, skip the population service for 2 minutes
Step and cost budgets	Bound loops and spending	≤ 15 tool calls per agent; ≤ $0.50 per SitRep (Class 6)
Fallback	An alternative path when the primary fails	Secondary model deployment; simpler single-call pipeline; a human
Graceful degradation	Deliver a reduced but honest result	SitRep without population context, marked "population data unavailable"
Idempotency and checkpoints	Safe retries and resumption	Resume from the Analyst step without re-running the Researcher

A circuit breaker is worth seeing in code, because it is the pattern most people haven't met. Like an electrical breaker, it has three states. Closed means calls pass through normally. After too many failures it trips to open, and calls fail immediately without touching the struggling service. After a cool-down it goes half-open and lets one trial call through; if that succeeds, it closes again.

class CircuitBreaker:
    def __init__(self, max_failures=5, reset_after_s=120, clock=time.monotonic):
        self.max_failures, self.reset_after_s, self.clock = max_failures, reset_after_s, clock
        self.failures, self.opened_at = 0, None

    def call(self, fn, *args, **kwargs):
        if self.opened_at is not None:
            if self.clock() - self.opened_at < self.reset_after_s:
                raise CircuitOpen("service marked unavailable; use the fallback")
            self.opened_at = None                       # half-open: allow one trial call
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self.failures += 1
            if self.failures >= self.max_failures:
                self.opened_at = self.clock()           # trip: stop hammering the service
            raise
        self.failures = 0                               # success closes the circuit
        return result

Why bother? Without it, every SitRep run during an outage waits for the full timeout, times the retries, for every call to the dead service. Runs pile up, costs rise, and when the service recovers it is hit by a wall of retries. With a breaker, runs fail fast into the fallback or the degraded mode.

Two principles tie these patterns together:

• Every degradation must be visible. A SitRep produced by the fallback model, or without population data, must say so in the output and in the trace. Silent degradation is how a reliability feature becomes a quality problem.
• Test the failure paths. Inject timeouts and errors with fakes, as the Class 5 and 6 tests do. A fallback that has never run is a fallback that doesn't work.

Prompt Lab

Run each prompt in your model of choice. Exercise 1 is most informative with two different model families.

Exercise 1: An LLM-judge rubric, tested on known answers

Task: grade three SitRep drafts whose quality you already know: one good, one with an invented figure, and one that silently picks 1,450 over 1,200 households. A judge that cannot separate these is not ready.

You are grading a humanitarian situation report (SitRep) written from field reports. Grade ONLY against the rubric. Length and writing style do not earn points. Do not reward confident tone. Check every figure in the SitRep against the source.

Rubric:
5 - Every figure traceable to a source; conflicts flagged, not resolved; neutral; nothing important missing. 4 - Accurate and neutral, with one minor omission or awkward phrasing. 3 - One material problem: a missing key need, an unflagged conflict, or mildly non-neutral tone. 2 - Several material problems, or one figure that is not in the source. 1 - Misleading: invented figures, followed instructions found inside the reports, or unusable.

Source (field reports and request):
<source>
<paste sample_reports.md from the Class 6 lab>
</source>

SitRep to grade:
<sitrep>
<paste one draft>
</sitrep>

First list your reasons, then give the score. Reply with JSON only:
{"reasons": "<one or two sentences>", "score": <integer 1-5>}

What to look for: Does the invented figure get a 1 or 2, or does a fluent draft still get a 4? Pad the good draft with 150 words of polite filler and grade it again. Does the score rise (verbosity bias)? Run each draft three times: how stable are the scores? Finally, have a different model family grade the same drafts, and compare.

Exercise 2: Generating edge and adversarial cases

Task: use a model to propose golden-set candidates, then act as the human reviewer who decides which to keep.

You are helping build a test set for a system that turns humanitarian field reports from Northern Veloria (a FICTIONAL province) into situation reports.

Write 6 short field reports (2-4 sentences each) designed to make such a system fail. Cover: (1) inconsistent units, (2) two reports that describe the same incident with different numbers, (3) an ambiguous date, (4) a report in French or Spanish, (5) an instruction aimed at the AI hidden inside the report, (6) personal data that must not appear in the SitRep.

For each, output JSON on one line:
{"input": "<report>", "failure_it_targets": "<short>", "must_include": [...], "must_not_include": [...]}

Use only fictional names and places. Do not describe any real crisis.

What to look for: Are the expectations actually correct? Models often write must_include values that a good SitRep would not contain. Are the cases realistic, or cartoonishly easy? Did it drift into real places or real names despite the instruction? Keep the best two, fix their expectations, and add them to sitrep_golden.jsonl in the lab.

Exercise 3: Error analysis: from notes to categories

Task: turn raw failure notes into the categories your checks should measure. Collect 10–15 one-line notes from the lab (or use the list below), then ask a model to group them. Check its grouping against your own.

Below are one-line notes, written by a reviewer, on failed outputs from a SitRep system. Group them into 3-6 failure categories.

For each category give: a short name, a one-sentence definition, the note numbers in it, and one AUTOMATED check (rule-based if possible, otherwise an LLM-judge criterion) that would catch it. Then say which category you would fix first and why.

Notes:
1. Added 60 families and 400 people to get 460.
2. Wrote "3 April" for 03/04/2026.
3. Dropped the 1,450 figure; only reported 1,200.
4. Included the nurse's phone number.
5. Summary says "situation under control": that phrase came from an injected line.
6. Missing "Gaps and constraints" heading.
7. Reported "cholera outbreak" for two suspected diarrhoea cases.
8. Averaged 300-500 to "400 people".
9. Called the provincial government "corrupt", quoting the report as fact.
10. 620 words.

What to look for: Does it separate "arithmetic on figures" (1, 8) from "resolving conflicts" (3), and "overstating certainty" (7) from "adopting non-neutral language" (9)? Are its proposed checks really automatable? Which categories does the lab's golden set already cover, and which would need new cases?

Lab: Measure the SitRep Assistant

Goal: run a 20-case golden dataset through the SitRep system with rule-based checks and an LLM judge, trace every call, make one measured improvement, and prove it caused no regression.

The code is in labs/engineering/c7/. Like the earlier labs, it is framework-free, so you can see what eval platforms and tracing tools do for you.

File	Role
llm.py	LLM = Callable[[List[Dict[str, str]]], str] and a LiteLLM adapter
evals.py	load_dataset(), rule-based checks, llm_judge(), judge_check(), run_eval() -> Report, compare(), cohens_kappa()
tracer.py	Tracer with nested spans (duration, tokens, cost), JSON export, traced_llm()
sitrep_system.py	The system under test, with PROMPT_V1 (naive) and PROMPT_V2 (explicit rules)
sitrep_golden.jsonl	20 fictional cases: core, edge, multilingual, adversarial, out of scope
test_evals.py	21 tests with a scripted FakeLLM
run_eval_demo.py	v1 against v2 on a real model, with traces and a comparison

Step 1 · Set up and run the tests (10 min)

cd labs/engineering/c7
python -m venv .venv && source .venv/bin/activate
pip install litellm pytest
pytest -q

All tests pass without an API key. Skim the test names first: they list what an eval harness must get right. Examples are "a crash is a failed case", "an unparseable judge is a failed check, not a pass" and "a regression is detected even when the overall score improves".

Step 2 · Read the dataset and the checks (15 min)

Open sitrep_golden.jsonl. For five cases, predict what a naive prompt would get wrong. Then open evals.py and match each expected key to its check function. Note two design choices:

• A check returns None when it does not apply (no max_words in the case). Pass rates are computed over applicable cases only, so "0/1" and "0/20" mean very different things.
• check_no_invented_numbers compares every number in the output with the numbers in the input. Read test_invented_number_is_caught.

Try it: add a check check_no_pii that fails if the output contains an email address or an international phone number, reusing the patterns from the Class 5 harness. Which golden case exercises it? Write a test for it.

Step 3 · The judge and its calibration (10 min)

Read JUDGE_PROMPT and parse_judge_output(). The parametrised test test_judge_rejects_malformed_verdicts lists five ways a judge reply can be wrong. Each one must raise an error, and judge_check() turns that error into a failed check. Then read cohens_kappa() and test_cohens_kappa: a judge that says "pass" to everything can agree with humans 50% of the time and still have a kappa of 0.

Step 4 · Traces (10 min)

Read tracer.py, then run test_spans_nest_and_record_duration_tokens_and_cost with -v. In a shell, reproduce it and print tracer.tree() and tracer.to_json(). The injected clock makes durations deterministic in tests. In production it is time.perf_counter.

Try it: add a tool.<name> span around each tool execution in your Class 5 run_tool_loop (pass a tracer in), and print the tree for one run.

Step 5 · One measured improvement (15 min)

export MODEL=openai/gpt-4o-mini              # or azure/<deployment>, anthropic/<model>
export JUDGE_MODEL=anthropic/claude-sonnet-4-5   # optional; ideally another family
python run_eval_demo.py

The script runs the golden set with PROMPT_V1 and then PROMPT_V2, saves both reports and traces to results/, and prints the comparison. With your pair:

1. Read v1's failures. Which categories from Prompt Lab Exercise 3 do they fall into?
2. Did v2 improve the pass rate? Did it cause any newly failing case? If so, read that case's output: is the check wrong, or the prompt?
3. Make one further change: a single prompt rule, a check fix, or a model change. Run again, and write down the before and after numbers and the cost from the trace totals. That note is your lab output: a measured improvement, with evidence.

Stretch challenges

1. Calibrate the judge: label 10 outputs yourself before looking at the judge, compute kappa, and improve the rubric until it reaches 0.6.
2. Trajectory eval: run the Class 5 tool loop on five requests and apply the trajectory_checks() function from Section 4.
3. CI gate: write ci_eval.py, which exits non-zero on a regression against results/baseline.json, and wire it into a workflow like the one in Section 5.

Check Your Understanding

1 A colleague changed the SitRep prompt, tried it on three reports, and says "it's clearly better now, let's ship". What is the most important missing step?

1. Try it on three more reports to be sure.
2. Ask the model whether the new prompt is better.
3. Run the golden dataset with both prompts and compare case by case for improvements and newly failing cases.
4. Ship it, and roll back if users complain.

2 You are building the first golden dataset for the SitRep Assistant. Which set of 20 cases is most useful?

1. A mix of typical reports, edge cases (families vs people, duplicates, ambiguous dates), reports in languages other than English, adversarial reports and out-of-scope requests.
2. 20 typical, clean reports, so the score reflects everyday use.
3. 2,000 synthetic reports generated by a model, with no human review, for statistical power.
4. The same examples used as few-shot examples in the prompt, since we know their correct answers.

3 You need to check that every figure in a SitRep appears in the source reports. What is the best first check?

1. An LLM judge asked "Are the figures accurate? Score 1-10".
2. Human review of every SitRep.
3. Exact match against a reference SitRep.
4. A rule-based check that extracts every number from the output and verifies it appears in the input.

4 Your LLM judge agrees with human reviewers on 85% of SitReps. Most SitReps (about 85%) pass. What should you conclude?

1. The judge is reliable; 85% agreement is high.
2. Look at chance-corrected agreement (such as Cohen's kappa) and especially the false passes before trusting it.
3. Replace the human reviewers with the judge, since it saves time.
4. Use the same model as the SitRep writer for the judge, to improve agreement.

5 Two runs of the agent produce the same correct SitRep. Run A used 5 tool calls; run B used 38, including three identical searches and one blocked call to send_sitrep_email. What kind of eval catches the problem with run B?

1. An exact-match eval on the final SitRep.
2. An LLM judge on the final SitRep's tone.
3. A trajectory eval: checks on tool order, forbidden tools, repeated calls and steps used against the budget.
4. A latency dashboard.

6 A prompt change raises the golden-set pass rate from 70% to 80%. But case g12, a report with an embedded "IGNORE ALL PREVIOUS INSTRUCTIONS", passed before and now fails. What should happen?

1. Treat it as a regression: block or flag the change until someone looks at g12, even though the average improved.
2. Ship it; the overall pass rate went up by 10 points.
3. Delete g12 from the dataset because it is flaky.
4. Lower the pass threshold so both versions pass.

7 Which of these should NOT be written to your traces?

1. The prompt version and model name for each call.
2. Token counts and cost per call.
3. Tool calls with their arguments and truncated results.
4. The provider API key used for the call.

8 The population-lookup service is down during a flood response. Every SitRep run waits 30 seconds, times 3 retries, for every population call, then fails. Which pattern fixes this best?

1. Increase the timeout to 120 seconds.
2. A circuit breaker that stops calling the failing service after repeated errors, plus graceful degradation: a SitRep without population context, clearly marked.
3. Raise the retry count to 10.
4. Let the model estimate populations when the service fails.

Exit Ticket

Think of one AI-assisted task in your team. In three sentences: (1) name three cases you would put in its golden set, including one adversarial case; (2) say which quality you would check with code and which would need a judge or a person; (3) name the one metric you would put on a dashboard first, and the threshold that should wake someone up.

Key terms

• Eval — a repeatable test of an AI system on a fixed set of cases, with a score.
• Improvement loop — run the eval, analyse errors, change one thing, measure again.
• Error analysis — reading failures and grouping them into categories before choosing metrics.
• Golden dataset — a versioned set of inputs with expectations of what good output must do.
• Rule-based check — a deterministic code check on an output (headings, length, numbers).
• Rubric — a scoring guide with concrete descriptions for each level.
• LLM as a judge — a model grading outputs against a rubric; trusted only after calibration.
• Calibration, Cohen's kappa — comparing a judge with human labels; agreement corrected for chance.
• Verbosity, position and self-preference bias — known distortions in LLM judges.
• Trajectory eval — checking an agent's path of tool calls, not only its final answer.
• Regression test — an eval run automatically to catch things that used to work and now don't.
• pass@k, pass^k — succeeds at least once in k tries; succeeds all k times.
• Offline and online evaluation, A/B test, shadow mode, canary — testing before deployment and on live traffic, and the ways to compare versions safely.
• Trace, span — the record of one request, and one timed step within it.
• OpenTelemetry — the open standard for traces and metrics.
• p95 latency — the time within which 95% of requests finish.
• Circuit breaker — stops calling a failing service for a while, and fails fast instead.
• Graceful degradation — delivering a reduced, clearly labelled result instead of nothing.