6

Class 6: Checking AI

Class 6 of 8About 3 hoursNo coding

By the end of this class you will be able to

• Use a short fact-checking routine to check the claims, numbers, quotes and citations in an AI answer against their sources.
• Recognise hallucinated citations (references that look real but don't exist) and know what to do when you find one.
• Spot three kinds of bias (in the data, in the outputs, and in how well different languages are covered) and test for them.
• Explain automation bias and why a confident tone is not evidence.
• Explain why asking the AI to check itself helps but is not enough.
• Apply a traffic-light rule to decide when not to use AI, and where a human must stay in the loop.

Where we are

In Class 5 you used AI for real work: drafting, summarizing, translating and producing a SitRep in the chat tool. This class is about the step that makes that work safe to send: checking it. On the AI Map this is part of the cross-cutting layer, "Does it work?" In Class 7 we look at AI that takes actions on its own, where checking matters even more.

1. Why checking is your job

Picture a new drafting assistant who is fast, articulate and tireless, and who has one strange habit. About one time in twenty, when they don't know something, they don't say so. They write something that sounds right, in the same calm, confident voice as everything else. They are not lying on purpose. They genuinely cannot tell the difference between what they know and what they have just made up.

You wouldn't fire this assistant. They save you hours. But you would never sign their work without reading it, and you would learn where their slips tend to happen, so you know where to look.

That is the situation with every AI chat tool (as of 2026). Class 2 explained why: the model predicts likely next words, so it produces text that sounds like a good answer, and most of the time that is also a correct answer. When it isn't, nothing in the tone warns you. These invented details are called hallucinations.

The one-sentence version

Fluent is not the same as true. The AI's job is to produce a plausible answer; your job is to decide whether it is a correct one.

Three facts shape everything in this class:

• You are accountable. A situation report, a briefing note or an email sent under your name is your work, whoever or whatever drafted it. "The AI said so" is not a defence.
• Errors cluster in predictable places. Numbers, names, dates, quotes and references are where hallucinations do the most damage and are easiest to miss.
• Checking is cheaper than correcting. Five minutes checking a SitRep is far less work than a correction notice after a donor has quoted the wrong figure.

2. A fact-checking routine

You don't need to check every word of every AI answer equally. A quick email reply to a colleague needs less checking than a figure going to a donor. But for anything that matters, use the same short routine every time. It works like a pre-flight checklist: boring, quick and very effective.

Step 1: Mark the checkable claims

Go through the answer and mark every claim, meaning anything that could be true or false. Four kinds matter most:

Claim type	Example in an AI answer	Why it's risky
Numbers	"1,200 people displaced"	Units get swapped (households vs people), totals get invented, estimates become facts
Names and places	"The Kessa health post"	Names get merged, misspelled or made up
Quotes	"We have enough stock for a week," said the district officer.	Quotes are paraphrased, attributed to the wrong person or invented
Citations and sources	"According to the 2025 Veloria Flood Assessment (p. 14)…"	References can look perfect and not exist

A useful trick is to ask the AI to do the marking for you, then check the list yourself:

List every number, name, date, quote and source reference in your answer above as a table, with a column saying exactly where in my notes each one comes from. Write "not in the notes" if it isn't there.

Step 2: Trace each claim to a source

For each marked claim, find where it came from:

• If you gave the AI the material (field notes, a report, meeting minutes), find the claim in that material. Use your computer's search (Ctrl+F or Cmd+F) for the number or name.
• If the AI got it from its general knowledge or a web search, find the original source yourself. A link that the AI provides is a starting point, not proof: open it and check it says what the AI claims.
• If you can't find a source, treat the claim as unverified. Either remove it or label it clearly ("unconfirmed").

Step 3: Check the meaning, not just the words

A figure can appear in the source and still be wrong in the answer. Watch for:

• Changed units: 1,200 households became 1,200 people.
• Lost qualifiers: "approximately 1,200, partly estimated" became "1,200".
• Wrong scope: a figure for one village is presented as a figure for the whole district.
• Wrong time: last week's figure is presented as today's.
• Changed certainty: "unconfirmed reports of" became "there are".

Step 4: Check what's missing

Hallucinations add things; omissions quietly remove them. Compare the answer with the source and ask: Did it leave out a village, a need, a caveat, a dissenting view? A SitRep that drops the one village nobody could reach looks tidier and is less true.

Step 5: Record what you checked

For important documents, keep a short note of what you verified and against what. It takes a minute and it protects you and your team if a figure is questioned later.

The routine on one page

1. Mark every number, name, date, quote and citation.
2. Trace each one to a source you can open.
3. Compare meaning: units, qualifiers, scope, time, certainty.
4. Look for gaps: what's missing that should be there?
5. Record what you checked for anything that matters.

3. Hallucinated citations

Citations deserve their own section, because they are the most convincing kind of hallucination. An invented reference has everything a real one has: plausible authors, a realistic title, a year, a journal or publisher, sometimes page numbers and a web address. It looks more credible than a vague claim, and that's exactly what makes it dangerous.

Here is a (fictional) example. A colleague asked an AI tool for evidence on water-borne disease after floods and got:

Looks real, isn't

Studies show that diarrhoeal disease cases rise by 40–60% in the two weeks after major flooding (Mensah & Clarke, 2021, Journal of Humanitarian Health, 14(3), pp. 211–226).

Everything about this looks right. But when the colleague searched for the article, there was nothing: no such article, no such issue, and in this case no such journal. In real cases it's often subtler: the journal exists, the authors work in a related field, but the article was never written. The "40–60%" figure had no source at all.

Why does this happen? Because the model has seen millions of citations and has learned what one looks like. When asked for a source, it produces text in the shape of a source. Unless the tool is actually searching a document library or the web and showing you where it found things (see Class 4 on RAG), the reference is generated the same way as any other sentence.

How to resolve a citation:

1. Search for the exact title in a library catalogue, a scholarly search engine or the publisher's website. Not in the AI tool.
2. Open the source and find the specific claim. Check the page or section if one is given.
3. Check that it says what the AI says it says. Real sources are often misquoted: a finding about one country becomes a global claim, or "may" becomes "does".
4. If you can't find it, don't use it. Don't keep the claim and quietly remove the reference. The claim came from the same place as the fake reference.

Tools with search are better, not perfect

Many enterprise tools can now search the web or your organization's documents and show links (as of 2026). This greatly reduces invented references, because the AI is quoting something it actually retrieved. But the summary of that source can still be wrong, or the linked page may not support the claim. Always click through for anything you will repeat.

4. Bias: who is missing, and who is described differently

Invented facts are the loud failure. Bias is the quiet one. An answer can be accurate sentence by sentence and still be skewed in what it emphasises, who it assumes and who it leaves out.

Know the Term: Bias in AI

Bias in AI means the system's outputs are systematically skewed for or against certain groups, places, languages or viewpoints. It usually comes from the training data: models learn from huge amounts of text, most of it from the internet, which over-represents some languages (especially English), regions and voices and under-represents others. Bias can also come from how a system is designed, or from the examples and instructions it is given. It shows up as stereotypes in generated text, weaker answers in less-represented languages, assumptions about who holds which job, or some communities being quietly left out of a summary.

Analogy: a map drawn only from travellers' reports. The busy roads are detailed; the places few travellers visit are blank or wrong, and the map doesn't tell you which is which.

Why you care: bias is rarely obvious in a single answer. Look for who is missing or described differently, compare outputs across groups and languages, and don't let AI make judgements about people on its own.

It helps to think of three places bias shows up.

Bias in the data

Models learn from very large collections of text, much of it from the internet. That text over-represents some regions, languages, professions and points of view. What the model "knows" about a capital city is far richer than what it knows about a remote district, and what it has read about a well-documented crisis shapes how it describes a new one.

Fictional example: asked to "describe typical living conditions in Northern Veloria", an AI tool gives a vivid, confident paragraph. None of it comes from Veloria. It is a blend of what the model has read about other places that sound similar, and it quietly carries their stereotypes.

Bias in outputs

Even when you provide the facts, the way an AI tool writes about people can differ by group. It may:

• Describe men as "leaders" and women as "beneficiaries" in the same programme.
• Use more emotional, passive language for some communities ("suffering", "helpless") and more active language for others ("organizing", "responding").
• Assume job roles: when you ask for "a story about a doctor and a nurse", the doctor is a man and the nurse a woman.
• In summaries, give more space to the groups mentioned most, so a minority community mentioned once in the notes disappears.

Fictional example: a field note says that both the Aramu women's association and the village council organized the evacuation of Lomi. The AI's summary credits "local leaders" with the evacuation and mentions women only as "vulnerable groups requiring support".

Bias in language coverage

AI tools perform best in English, and well in other widely used languages, but quality can drop noticeably for languages that are less represented in the training data (as of 2026). Among major world languages such as Arabic, Chinese, English, French, Russian and Spanish, quality is generally strong but not identical, and it can drop further for regional dialects, local languages and specialist terms.

This matters in two ways:

• Translations and summaries of material in less-represented languages may lose nuance or make mistakes that a non-speaker can't see.
• Responses to feedback from affected communities, submitted in local languages, may be summarized less accurately than feedback in English, so those voices count for less without anyone noticing.

How to test for bias

You can't see bias by reading one answer. You find it by comparing:

• Swap the group. Ask the same question with a different gender, region or community and compare the tone and assumptions.
• Count who appears. In a summary, list every group mentioned in the source and check they all appear in proportion.
• Check the verbs. Who acts and who is acted upon?
• Test in other languages. Have a speaker check a sample of translations, especially back translations of key messages.
• Ask whose view is missing. Then go and get it from a real source, not from the AI.

5. Automation bias: trusting it because it sounds sure

There is one more kind of bias, and this one is in us. Automation bias is the tendency to trust a computer's output more than we should, especially when it looks polished and confident, and to stop noticing when it's wrong. It is well documented in aviation, medicine and other fields long before chatbots, and AI tools make it stronger, for three reasons:

1. Everything sounds equally sure. A chatbot's tone doesn't change between facts it has seen a thousand times and details it has invented. People use confidence as a signal of accuracy; with AI, that signal is broken.
2. It's usually right. After twenty good answers, we stop checking the twenty-first. That is exactly when a quiet error gets through.
3. Checking feels like extra work. When the AI produced a draft in ten seconds, spending ten minutes checking it feels slow, even though it's still far faster than writing it yourself.

Some practical counter-habits:

• Decide the checking level before you read the answer, based on what's at stake, not on how convincing the answer looks.
• Ask the AI to show uncertainty: "For each figure, say whether it is stated directly, inferred or estimated." This doesn't make it accurate, but it helps you see where to look.
• Treat "I'm confident" as meaningless. If the AI says "I am certain", that is a sentence it generated, not a measurement.
• Rotate the reviewer. A second person who didn't see the AI working often spots what the first person accepted.

Confidence is not evidence

An AI that says "The total is definitely 4,350" is no more likely to be right than one that says "The total might be around 4,350." Check the number, not the tone.

6. Asking the AI to check itself (and why that isn't enough)

In Class 3 you learned a useful move: ask the AI to review its own work against the source. It works surprisingly often. A second pass can catch a changed number, a dropped village or an unsupported claim. There are several good ways to do it:

Review your summary against the original notes. List any number, name or date in your summary that does not appear in the notes, and any important point from the notes that your summary left out.

Act as a sceptical editor. Identify the three claims in this draft most likely to be wrong or misleading, and explain why.

Compare this draft with the source document. For each paragraph, rate it "supported", "partly supported" or "not supported", and quote the sentence from the source that supports it.

So why isn't that enough?

• The same blind spots. The checker is the same model, with the same training data and the same gaps. If it didn't know something the first time, it still doesn't.
• It can confirm its own errors. Asked "Is this right?", models often say yes and give a confident justification. They can also "correct" something that was actually right.
• It checks against what it can see. It can only compare with the material in the conversation. It can't check the real world, or a source it can't access.
• It can invent the evidence, too. A self-check that "quotes the supporting sentence" can quote a sentence that isn't in the source. You still have to look.

Think of self-checking as a spell-checker: worth running every time, and catches a lot, but nobody would publish a report on the strength of a spell-check alone. A human who knows the subject and can open the sources makes the final call.

A better second opinion

If you want an AI second opinion, give it a fresh chat with only the source and the draft, and a specific checking task. A fresh chat doesn't "remember" why it wrote what it wrote, so it is less inclined to defend it. It's still not a substitute for checking the key facts yourself.

7. When not to use AI: a traffic-light rule

Checking reduces risk; it doesn't remove it. Some tasks are low-risk enough that AI is an easy win. Some need care and a human in the loop. And some shouldn't involve a general AI chat tool at all. A simple traffic-light rule helps you decide quickly.

Light	When	Examples	What to do
Green: go, with a normal check	Low stakes, no sensitive data, you can easily judge quality	Rewording an email, brainstorming titles, summarizing a public report, drafting an agenda, explaining a concept	Use your approved tool, read the output, fix what's wrong
Amber: proceed with care and a named human checker	Content that others will rely on; facts and figures; public communications; translation of important messages	SitReps, donor updates, briefing notes, data analysis, translating a press release	Use the full fact-checking routine; a knowledgeable person reviews before it goes out; record what you checked
Red: don't use a general AI tool, or only with explicit approval and special safeguards	Legal consequences; safety-critical decisions; personal or highly sensitive data; final decisions about people	Contract or legal advice; security or evacuation decisions; medical guidance; beneficiary eligibility; recruitment shortlists; disciplinary matters; anything with names and personal details of affected people	Use established procedures and qualified people. AI may at most help with a narrow, approved part, never the decision

Some red-light areas deserve a word of explanation:

• Legal matters. AI tools can sound authoritative about law and regulations while being wrong, out of date or talking about the wrong jurisdiction. Your organization's own legal position, contracts and obligations add details a general model is unlikely to know.
• Safety-critical decisions. Whether a road is safe to travel, whether a site should be evacuated or how to treat a patient must rest on verified information and accountable people. A plausible but wrong answer can cost lives.
• Personal data. Names, contact details, health, protection cases or any information about identifiable people, especially vulnerable people, must only go into tools approved for that classification, and often not into AI at all. Class 8 covers this in detail.
• Final decisions about people. Who receives assistance, who is shortlisted for a job, who is disciplined: these decisions affect people's lives and rights and require accountability, a reasoned basis and a route to appeal. AI can reflect bias and can't be held accountable. A human must make, and own, the decision.

Amber tasks drift to red

A task can change colour as it grows. Summarizing public news about a flood is green. Adding names of affected families from a registration list turns it red. Check the light again each time you add data or raise the stakes.

8. Human in the loop

The phrase human in the loop comes up whenever AI is used for work that matters. It means a person reviews and approves the AI's work at agreed points, before it has real effects.

Know the Term: Human in the loop

Human in the loop means a person reviews, corrects or approves an AI system's work at agreed points before it has real effects: before a report is sent, a payment is made, a record is changed or a decision about a person is taken. The design question is not "human or AI?" but where the human checkpoints sit and what the person actually checks. A reviewer who clicks "approve" on everything without reading is not a real checkpoint, just a signature on the AI's work.

Analogy: a senior officer who clears a press release before it goes out. The drafter does most of the work, but nothing leaves the building without that clearance.

Why you care: you remain accountable for what goes out under your name. Decide in advance which steps need your approval, and give yourself enough time to check them properly.

For everyday chat-tool use, you are the human in the loop, and the traffic-light rule tells you how hard to look. As AI systems start acting on their own (Class 7), the checkpoints have to be designed in on purpose. Three questions help:

1. Where are the checkpoints? Before anything leaves the team, before any irreversible action (sending, publishing, paying, deleting), and before any decision about a person.
2. What does the checker actually check? "Review the SitRep" is vague. "Confirm every figure against the field notes, confirm no personal names appear, confirm Dorra is listed as a gap" is a real check.
3. Does the checker have what they need? Time, the source material, the expertise, and the authority to say no.

A checkpoint that fails any of these three becomes a rubber stamp: a person who approves without really checking. Rubber stamps are worse than useless, because they create the appearance of oversight and let everyone relax.

SitRep Assistant: where the human sits

In our running case study, the AI turns messy field notes into a draft SitRep. The reporting officer is the human in the loop. They check every figure against the notes using the routine in section 2, confirm that gaps and ambiguous dates are flagged, scan for bias in who is mentioned, and only then send the draft for clearance. The AI saves time on drafting; the officer's judgement is what makes the SitRep trustworthy.

Prompt Lab

Work in pairs. Use your organization's approved AI chat tool. One partner drives, the other plays sceptical reviewer and says "show me where that comes from" out loud. Swap after each exercise. All material is fictional.

The field note (fictional, same as Class 3, with a follow-up)

FIELD NOTE - Aramu district, Northern Veloria - sent Thurs eve From: K. Osei, Veloria Relief Coalition field team

heavy rain since the weekend, Tula river burst near Kessa bridge. we got to Aramu town + Lomi and Tavet villages today. could NOT reach Dorra, road cut after the bridge, no phone signal there either.

approx 1,200 households displaced in total (our estimate + figures from village chiefs). most are staying at Aramu primary school, which is full - maybe 600 people there. more families in the community hall but we didn't manage to count them.

water is the big problem. 2 of the 3 boreholes in Lomi are flooded and people are drinking from the river. Tavet health post still open but running low on ORS, nurse reports 14 cases of acute diarrhoea since Monday, mostly children under 5. no deaths reported to us.

VRC is distributing food at the school, enough stock for about 3 days. next truck expected 04/05 if the Kessa road is passable. urgent: water purification, hygiene kits, ORS, shelter materials for the hall.

FOLLOW-UP - Fri midday
The Lomi women's group organized moving elderly people to the school on Wednesday night, together with the village council. Community hall count done: 43 families.

Exercise 1: Catch the invented detail

Task: Ask for a SitRep without any rule about missing information, then run the fact-checking routine on it: mark every claim and trace each one to the note.

Write a one-page situation report for the country director based on the field note below. Include the total number of people displaced, the number of disease cases, and the expected date of the next delivery.

Field note:
"""
[paste the field note here]
"""

What to look for:

• Is there a "total number of people displaced"? The note gives households plus two partial figures. Where did the AI's total come from?
• Did "04/05" become a definite date? Which one?
• Did "approx" and "our estimate" survive, or did estimates become facts?
• Count the claims you couldn't trace to the note. That's your hallucination count.

Exercise 2: The citation hunt

Task: Ask for evidence with references, then try to resolve every reference yourself using a library catalogue, scholarly search or the publisher's website (not the AI tool).

I'm writing a briefing on the health risks after flooding in rural areas. Give me three key findings from published research, each with a full reference (authors, year, title, journal, volume, pages) and a link.

What to look for:

• How many references could you find? How many said what the AI claimed?
• Did any look perfect but lead nowhere, or lead to a different article?
• If your tool searched the web and showed links, compare: were those more reliable?
• What would you do with a claim whose reference you couldn't find?

Exercise 3: Who got the credit?

Task: Use the whole note including the follow-up. Ask for a short summary, then check who is described as acting and who as receiving help. Then swap a detail and compare.

Summarize the field note below in 100 words for a public newsletter, describing what happened and who responded.

Field note:
"""
[paste the field note including the follow-up]
"""

Then, in a new chat, paste the same note but change "The Lomi women's group" to "The Lomi youth group" and ask again.

What to look for:

• Is the women's group mentioned at all? Is it described as organizing or as vulnerable?
• Does the wording change when the group changes? How?
• Which groups from the note are missing from the summary?
• Rewrite one sentence yourself to fix any bias you found.

Exercise 4: Self-check vs you-check

Task: Take the SitRep from Exercise 1. Ask the AI to check itself. Then compare its list of problems with the one you made.

Review the situation report you wrote against the original field note. For every number, date and name in the report, quote the exact words in the note that support it. If a claim is not supported, say "NOT IN NOTE". Then list anything important from the note that the report left out.

What to look for:

• Which of your problems did the AI catch? Which did it miss?
• Did it "quote" anything that isn't actually in the note? (Search the note to be sure.)
• Did it flag the unreached village (Dorra) as an omission?
• Would you now send the report? What still needs a human?

Exercise 5: Traffic lights for your job

Task: List five tasks from your own work that you might use AI for. Ask the AI to classify them with the traffic-light rule, then decide for yourself and compare. (Describe the tasks in general terms; don't paste real sensitive material.)

Classify each task below as GREEN (low risk, normal check), AMBER (use with care and a named human checker) or RED (don't use a general AI tool; legal, safety-critical, personal data or a final decision about people). For each, give a one-line reason and say what a human must check.

Tasks:
1. [task]
2. [task]
3. [task]
4. [task]
5. [task]

What to look for:

• Where did you and the AI disagree? Who was right, and why?
• Did any task turn out to be red only because of the data involved?
• For each amber task, write down what the human checker checks, not just "review".

Check Your Understanding

1 An AI-drafted SitRep says "3,400 people displaced in Aramu district". The field notes say "approx 1,200 households displaced (partly estimated)". What is the best description of the problem?

1. No problem: the AI converted households to people using a typical family size, which is helpful.
2. The number changed units and lost its qualifier, so a partly estimated household figure now looks like a firm count of people.
3. The AI made a small rounding error that doesn't matter for a SitRep.
4. The notes must be wrong, because the AI has access to more data than the field team.

2 A colleague's briefing draft includes a reference suggested by an AI tool: authors, year, journal, volume and page numbers. It looks perfect. What should they do before using it?

1. Nothing: a reference with volume and page numbers is too specific to be invented.
2. Ask the AI tool "Is this reference real?" and keep it if the tool says yes.
3. Keep the claim but delete the reference, so nobody can check it.
4. Find the source in a catalogue or on the publisher's site, open it, and confirm it says what the draft claims.

3 After six weeks of excellent AI-drafted SitReps, a reporting officer stops checking the figures because "it's always right". What is this an example of?

1. Automation bias: over-trusting a system because it has been reliable and sounds confident.
2. Sensible efficiency: once a tool has proven itself, checking is wasted time.
3. Data bias: the model's training data over-represents certain regions.
4. Prompt injection: the tool has been manipulated into seeming trustworthy.

4 You ask the AI to review its own SitRep against the field notes, and it replies "All figures are supported by the notes." What should you conclude?

1. The SitRep is verified and can be sent.
2. Self-checks are useless and you shouldn't have asked.
3. The self-check is a helpful first pass, but you still need to trace the key figures to the notes yourself.
4. Ask it the same question again until it finds a problem.

5 A field note says the Lomi women's group and the village council organized an evacuation. The AI summary credits "local leaders" and mentions women only as "vulnerable groups". What is the best response?

1. Accept it; the summary is shorter and the facts are broadly right.
2. Recognise output bias, correct the summary so it reflects who actually acted, and check for other groups that were dropped.
3. Switch to another AI tool, because this one is biased and the others are not.
4. Add "Do not be biased" to the prompt and trust the result.

6 Using the traffic-light rule, which of these tasks is RED: don't use a general AI chat tool for it?

1. Rewording a meeting invitation to sound friendlier.
2. Drafting a SitRep from fictional field notes for the country director.
3. Summarizing a published, public assessment report.
4. Deciding which named households qualify for cash assistance, from a registration list.

7 Your team adds a "human review" step to its AI-assisted donor updates. Which version of that step is a real check rather than a rubber stamp?

1. A manager clicks "approve" on each update within one minute of receiving it.
2. The same AI tool reviews the update a second time before sending.
3. A reviewer with the source notes confirms every figure, checks no personal names appear, and can stop the update.
4. Anyone on the team can raise concerns after the update is published.

Exit Ticket

Before you leave

Think of the last AI-assisted document you sent to someone. Which claims in it did you check against a source, and which did you trust because they sounded right? What will you check next time?

Key terms

• Hallucination: a confident, plausible-sounding statement from an AI that is not true or has no source.
• Hallucinated citation: a reference (author, title, journal) generated by an AI that looks real but doesn't exist or doesn't say what is claimed.
• Fact-checking routine: mark the claims, trace them to sources, compare meaning, look for gaps, record what you checked.
• Omission: an important fact, group or caveat from the source that the AI's output left out.
• Bias: a systematic skew in AI outputs for or against certain groups, places, languages or viewpoints.
• Language coverage: how well an AI tool performs in a given language, which depends on how much of that language it learned from.
• Automation bias: the human tendency to over-trust a computer's output, especially when it sounds confident.
• Self-check: asking the AI to review its own output; useful, but not a substitute for human verification.
• Traffic-light rule: green (go), amber (care and a named checker), red (don't use a general AI tool) for deciding when to use AI.
• Human in the loop: a person who reviews and approves AI work at agreed points before it has real effects.
• Rubber stamp: a human checkpoint that approves without really checking.