1

Class 1: Foundations: Thinking and Models

Class 1 of 8About 3 hoursPythonLayer: models

By the end of this class you will be able to

• Apply the four habits of computational thinking (decomposition, pattern recognition, abstraction, algorithms) to an AI system, and label each step AI, deterministic code or human.
• Explain how an LLM produces text: tokens, next-token prediction, and sampling with temperature and top-p.
• Estimate the token count and cost of a workload, including the cost differences across six widely used languages (Arabic, Chinese, English, French, Russian, Spanish).
• Describe, at an engineer's level, the context window, the transformer and attention, embeddings and the training stages that make a chat model.
• Choose between model types (reasoning vs standard, multimodal, open-weight vs closed, small vs large) for a given step, and account for the knowledge cutoff.
• Design for non-determinism: test with fakes, validate outputs, and decide in code what happens when two runs disagree.

Where we are

In the pre-work you made one model call and saw its reply, token count and latency. This class explains what happened inside that call, and gives you the habit of thinking that the rest of the course depends on: break the task down, decide what the model does, what code does and what a human does, then verify. In Class 2 you put that into practice by making the model return structured, validated data.

1. Computational thinking for AI systems

Imagine the duty officer at a relief coalition's operations desk on a flood morning. Forty messages arrive: emails from district teams, a forwarded spreadsheet, WhatsApp voice notes, a government bulletin in French. By 10:00 a situation report (SitRep) must go to partners. A good duty officer does not read everything and then write from memory. They sort the inbox, pull figures into a table, check totals, call the district team about the figure that looks wrong, draft, and have a colleague read it before it goes out.

That is computational thinking: solving a problem in a way that a mix of people and machines can carry out reliably. It has four habits, and each has a specific meaning when one of the machines is an LLM.

Habit	General meaning	For an AI system
Decomposition	Break a big task into smaller steps	Split the job until each step is simple enough to give to one owner (a model call, a function or a person) and to check on its own
Pattern recognition	Notice what steps have in common	Recognise the kind of work: reading language, arithmetic, lookup, judgement, approval. The kind decides the owner
Abstraction	Keep what matters, hide the rest	Define the interface of each step: its inputs, its output schema, what "done" means. Hide the model behind it
Algorithms	Write the steps as an exact procedure	Fix the order, the checks between steps, the retry and stop rules, and what happens on failure

The mistake this protects you from is the one big prompt: "Here are 40 messages, write the SitRep." It sometimes produces something impressive. You cannot test it, you cannot tell which figure came from where, and when it silently adds 1,200 households to 1,450 households, you find out from a partner.

Who does each step: AI, code or human?

Once the task is decomposed, each step gets exactly one owner. Pattern recognition makes this mostly mechanical:

Give it to...	When the step is...	SitRep examples
The model (AI)	Reading or writing natural language; handling messy, varied input; drafting	Extract facts from a free-text email; translate a French bulletin; draft the narrative
Deterministic code	Arithmetic, counting, sorting, lookup, format checks, moving data: anything with one right answer you can compute	Add up figures per district; check every figure has a source; check the draft has five headings
A human	Accountability, discretion, local knowledge, anything irreversible or high-stakes	Resolve conflicting figures; approve publication

Two rules turn this table into a design:

1. Every AI step is checked by a later code or human step. The model proposes; something deterministic or accountable disposes.
2. Nothing leaves the team without a human decision. Publication, allocation and external communication are gated.

The short version, which you will hear all course: AI proposes, code checks, humans decide.

Worked example: decomposing the SitRep

Here is one defensible decomposition of the morning SitRep. It is also the reference answer in this class's lab, where a validator enforces the two rules above.

#	Step	Kind	Owner	Checked by
1	Collect today's reports from the mailbox and shared folder	I/O	Code	—
2	Make English working copies of reports in French or Arabic	Language	AI	Step 7 (a bilingual colleague spot-checks any figure in dispute)
3	Extract facts (what, where, when, figure as written, source) into a fixed schema	Language	AI	Step 4
4	Validate each fact against the schema; check each figure appears verbatim in its source	Verify	Code	—
5	Decide whether two reports describe the same incident	Judgement	AI	Step 7
6	Add up figures per district, keeping households and people separate	Arithmetic	Code	—
7	Resolve conflicting figures (1,200 vs 1,450 households)	Judgement	Human	—
8	Draft the narrative sections to the template	Language	AI	Step 9
9	Check headings, word limit, and that every figure in the draft is in the totals table	Verify	Code	—
10	Duty officer reads and approves	Approval	Human	—
11	Send to the distribution list	I/O	Code	—

Notice three things:

• The model owns four of eleven steps, all of them language work. Most of the system is ordinary software. That is normal, and it is good news: ordinary software is testable.
• Arithmetic belongs to code (step 6). Models are unreliable at adding up long columns of numbers, and it takes one line of Python to do it exactly.
• Step 5 could have gone either way. "Same incident?" is judgement. A model can propose matches and a human confirms them. If duplicates are rare, a human alone may be cheaper. Decomposition makes such choices visible, so you can argue about them.

Step 3 of this table is exactly what you build in Class 2. The whole table becomes a multi-agent pipeline in Class 6.

2. Where LLMs sit in the AI family

"AI" covers many techniques. The one this course builds with is the large language model (LLM), a neural network trained on a very large amount of text to predict what comes next. It sits at the innermost of a set of nested ideas: AI ⊃ machine learning ⊃ deep learning ⊃ generative AI. The sidebars give the short version; the AI Literacy Class 1 tells the longer story.

Know the Term: The AI family tree

Artificial intelligence (AI) is the broad field of making computers do tasks that normally need human intelligence, such as recognising speech or planning a route. Machine learning (ML) is one branch of AI: instead of following rules a person wrote, the system learns patterns from examples. Deep learning is a kind of machine learning that uses large neural networks with many layers, and it drives most AI progress since around 2012. Generative AI is deep learning that creates new content (text, images, audio, code) rather than only sorting or predicting. Large language models (LLMs), the engines inside chat tools, are generative AI for text. Each circle sits inside the one before it: AI ⊃ machine learning ⊃ deep learning ⊃ generative AI.

Analogy: "vehicle" ⊃ "car" ⊃ "electric car" ⊃ "self-driving electric car". Every self-driving electric car is a vehicle, but most vehicles are not self-driving electric cars.

Why you care: when someone says "we use AI", ask which kind. A spam filter, a satellite flood map and a chatbot are all "AI", but they fail in very different ways and need different checks.

Know the Term: Neural network

A neural network is a mathematical model loosely inspired by the brain. It is made of many simple units ("neurons") arranged in layers. Each unit takes in numbers, multiplies them by adjustable settings called weights (also called parameters), adds them up and passes the result on to the next layer. During training, the network sees millions of examples, and after each one its weights are nudged slightly so that its output gets closer to the right answer. Nobody writes the rules by hand; the rules end up spread across the weights. A deep network simply has many layers, and today's large language models have billions of weights.

Analogy: a huge mixing desk with billions of small dials. Training is a patient sound engineer turning each dial a fraction at a time until the music sounds right.

Why you care: because the "knowledge" is spread across billions of weights, nobody can open the model and point to the line where a fact or a bias lives. That is why we test AI by checking its outputs, not by reading its rules.

Know the Term: A short AI timeline

1950: Alan Turing asks "Can machines think?" and proposes his imitation game. 1956: the term "artificial intelligence" is coined at a workshop at Dartmouth College. 1970s–1980s: expert systems encode specialists' knowledge as thousands of hand-written "if… then…" rules; they work in narrow areas but are costly to maintain, and funding cools in periods later called "AI winters". 1990s–2000s: machine learning takes over, learning patterns from data for tasks like spam filtering. 2012: a deep neural network wins the ImageNet image-recognition contest by a wide margin, starting the deep learning boom. 2017: Google researchers publish the transformer design. 2018–2020: large language models such as GPT-2 and GPT-3 appear. November 2022: ChatGPT's release brings generative AI to the general public. 2023–2026: models become multimodal (text, images, audio), reasoning models appear, and agents that can take actions with tools spread into workplace products (as of 2026).

Analogy: like the history of flight: decades of gliders and false starts, then a few breakthroughs close together that suddenly made air travel ordinary.

Why you care: AI has been "about to change everything" several times before. Knowing the history helps you tell a real shift from a sales pitch.

For an engineer, the practical point of the history is this: before about 2020, a new language task (classify these messages, extract these fields) meant collecting labelled data and training a model for that one task. With LLMs, one general model does thousands of tasks, and you describe the task in a prompt. That moves the work from training models to designing prompts, interfaces, checks and pipelines. That work is this course.

3. Tokens: what the model actually reads

A model does not see letters or words. It sees tokens: chunks of text, each mapped to an integer ID from a fixed vocabulary of typically 100,000–200,000 entries. A tokenizer splits your text into tokens before the model sees it, and turns the model's output tokens back into text.

Know the Term: Tokens

A token is the unit of text a language model reads and writes. It is often a whole common word ("water"), sometimes part of a word ("purific" + "ation"), a space, or a punctuation mark. Before the model sees your text, a tokenizer chops it into tokens and turns each one into a number. A rough rule for English: 1 token ≈ ¾ of a word, so 100 words ≈ 130 tokens. Tokenizers are built mostly from English-heavy text, so the same sentence often needs more tokens in languages such as Arabic or Russian, and scripts such as Chinese are counted differently again. Model limits (the context window) and prices are measured in tokens, not words.

Analogy: Lego bricks. The model never sees your sentence as a whole; it sees the bricks it has been broken into, and it builds its answer one brick at a time.

Why you care: tokens explain why long documents hit a limit, why costs can differ between the languages you work in, and why chatbots sometimes stumble on tasks like counting the letters in a word: they never saw the individual letters.

Tokenizers are built with an algorithm such as byte-pair encoding (BPE): start with single characters (or bytes), then repeatedly merge the most frequent adjacent pair into a new token, over a huge sample of text. Frequent words become one token; rare words are built from pieces. In a typical English tokenizer:

"Clean water is the most urgent need."   ->  Clean | water | is | the | most | urgent | need | . "households displaced"                   ->  households | displaced         (common: 1 token each) "Aramu"                                  ->  Ar | amu                        (rare name: pieces) "1,450"                                  ->  1 | , | 450                     (numbers: split)

Leading spaces are usually part of the token (" water", not "water"). As a rule of thumb, one token is about four characters or three-quarters of a word of English.

Why engineers care about tokens

• Cost. Providers bill per token: input tokens (everything you send) and output tokens (everything the model writes) at different prices. Output usually costs several times more.
• Limits. The context window (Section 5) and the maximum output length are measured in tokens, not words or pages.
• Latency. The model generates output one token at a time, so a 600-token reply takes roughly six times as long as a 100-token one.
• Odd failures. The model does not see letters, so tasks like "count the letters" or "reverse this word" are hard for it. Numbers are split into chunks, which is one reason models are unreliable at arithmetic, and one more reason step 6 above belongs to code.

Six languages, six different costs

Tokenizers learn their vocabulary from their training text, which has historically been mostly English. Common English words become single tokens, while words in scripts that were less frequent in that text are split into more, shorter pieces. The same message therefore costs more tokens, and more money, in some languages.

The lab's approximate tokenizer gives these counts for one fictional sentence ("Flooding in Aramu district has displaced about 1,200 households. Clean water is the most urgent need."):

Language	Approx. tokens	vs English
English	24	1.00×
French	32	1.33×
Spanish	32	1.33×
Arabic	33	1.38×
Chinese	34	1.42×
Russian	43	1.79×

These are approximations from a deliberately simple tokenizer. Real ratios depend on the tokenizer: newer tokenizers (as of 2026) have much larger multilingual vocabularies and have narrowed the gap, but it has not disappeared. Measure with your model's own tokenizer before budgeting a multilingual service. In the lab you plug in a real one with a single argument.

What it means in practice:

• A SitRep service that processes Arabic and Russian reports can cost noticeably more per report than the same service in English, and it fills the context window sooner.
• Translating everything into English first is not automatically cheaper: you pay for the translation call, and you risk losing meaning. Measure both routes.
• Fairness matters. If the per-user budget is set by English usage, colleagues working in other languages hit limits first.

Estimating cost

The arithmetic is simple, and worth doing before you build anything:

cost per call  = input_tokens  × input_price_per_token + output_tokens × output_price_per_token

monthly cost   = cost per call × calls per day × days

For example, with illustrative "large model" prices of USD 3 per million input tokens and USD 15 per million output tokens, a call that sends a 1,000-token report and gets back a 600-token draft costs about USD 0.012. At 300 reports a day that is roughly USD 110 a month for this one step. A multi-agent pipeline with five calls per report and a review loop multiplies it. The lab's tokens.py does this calculation with a pluggable tokenizer:

from tokens import EXAMPLE_PRICES, estimate_cost

one = estimate_cost(report_text, expected_output_tokens=600, price=EXAMPLE_PRICES["large"])
print(one.input_tokens, one.cost_usd, one.times(300 * 30).cost_usd)

4. Next-token prediction and sampling

At its core, an LLM does one thing: given a sequence of tokens, it outputs a probability for every token in its vocabulary being the next one. To write a reply, the system picks one token, appends it to the sequence, and asks again, one token at a time until a stop token or a length limit. This is next-token prediction, and the one-at-a-time process is called autoregressive generation.

Suppose the text so far is "In the Aramu school shelter, the most urgent need is clean". The model's output might look like this (made-up numbers):

Candidate next token	Probability
" water"	0.82
" drinking"	0.07
" sanitation"	0.04
" toilets"	0.02
" clothes"	0.01
...100,000+ others	0.04 in total

Everything the model "knows" (grammar, facts, the style of a SitRep) shows up only as the shape of these distributions. The model has no separate store of facts that it looks things up in. That is why a fluent, plausible and wrong continuation is always possible: the model is built to produce likely text, and likely is not the same as true. A model inventing a figure or a source is called hallucination.

Sampling: temperature and top-p

Choosing a token from the distribution is called sampling, and you control it with two main settings:

• Temperature reshapes the distribution before sampling. At 0 the system (nearly) always takes the most likely token, so output is focused and repetitive. At 1 it samples from the distribution as the model produced it. Above 1 the unlikely tokens get boosted and output becomes more varied, and eventually incoherent.
• Top-p (nucleus sampling) keeps only the smallest set of top tokens whose probabilities add up to p, and samples among those. With top-p 0.9 in the table above, " water" and " drinking" add up to 0.89, just short, so " sanitation" joins them (0.93) and those three are the only candidates. The long tail of odd tokens is cut off.

Change one of the two at a time, not both. Starting points:

Task	Temperature	Why
Extraction, classification, structured output	0–0.2	You want the most likely reading of the source, every time
Summaries, SitRep drafting	0.2–0.7	Some fluency; facts come from the source, not from creativity
Brainstorming, alternative headlines	0.8–1.0	Variety is the point

With LiteLLM, sampling settings are ordinary keyword arguments:

import litellm

response = litellm.completion(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Suggest three headlines for this SitRep: ..."}],
    temperature=0.9,
    max_tokens=200,          # an upper bound on output tokens: a cost and latency control
)

Reasoning models and sampling (as of 2026)

Some reasoning models ignore or reject temperature and top_p, or accept only their defaults, and expose a "reasoning effort" setting instead. Check your model's documentation. Your design must not depend on temperature 0 for correctness anyway (Section 9).

5. The context window

The context window is the maximum number of tokens the model can take into account in one call: your system prompt, the conversation history, any documents you include, and the output it generates. As of 2026, common limits run from about 128,000 tokens to a million or more, depending on the model.

Four facts about the context window shape almost every design decision in this course:

1. The model is stateless. It remembers nothing between calls. A chat "remembers" because the application resends the whole conversation every turn. Your SitRep pipeline must decide, for every call, what goes in.
2. You pay for everything you send, every time. A 50-page document in the context costs its tokens on every call that includes it. Provider features such as prompt caching reduce the cost of a repeated prefix (Class 4).
3. Bigger is not the same as better. Models attend less reliably to information buried in the middle of very long contexts (the "lost in the middle" effect), and irrelevant material distracts them. A focused 5,000-token context often beats an unfocused 200,000-token one.
4. The window is shared with the output. If the input nearly fills the window, the reply is cut short.

Deciding what goes into the window, what gets retrieved, summarised or left out, is context engineering, and it is the subject of Class 4.

6. Inside the model: transformers, attention and embeddings

You do not need the mathematics to build good systems. You do need a working mental model of three ideas, because they explain the behaviour you will debug.

The transformer and attention

Almost every modern LLM is a transformer. Text passes through dozens of identical layers. In each layer, attention lets every token look at every earlier token and decide how much each matters for its own meaning. In "The health post in Tamsi is non-functional; staff moved its medicines to the school", attention links "its" to "health post". Stacked over many layers, this lets the model build up grammar, reference, topic and tone.

Know the Term: Transformer and attention

The transformer is the design behind almost every modern large language model. It was introduced in the 2017 research paper "Attention Is All You Need". Its key trick is attention: when the model processes a word, it looks at all the other words in the text and decides which ones matter most for understanding it. In "The bank raised interest rates", attention links "bank" to "interest rates", so the model reads it as a financial bank, not a river bank.

Analogy: a meeting where every participant can listen to everyone at once and decide whose input matters for the point being made.

Why you care: the model can only "pay attention" to what is in the text you give it. That is why the context you provide matters so much.

Two engineering consequences:

• Attention costs grow with context length. Each token looks at all earlier tokens, so long prompts are slower and more expensive to process, and the context window has a limit.
• Order and placement matter. Instructions at the start (system prompt) or the very end of a long context tend to be followed more reliably than instructions buried in the middle. Class 2 builds on this.

Embeddings

Inside the model, every token becomes an embedding: a list of numbers (a vector, often with a few thousand dimensions) that places it in a "meaning space" where related things sit close together. Separate embedding models do the same for whole sentences or paragraphs. They take text in and return one vector.

Know the Term: Embeddings

An embedding is a list of numbers (a vector, often hundreds or thousands of numbers long) that represents the meaning of a word, sentence or document. A model learns to produce embeddings so that things with similar meanings get similar numbers, and so sit close together in this "meaning space". "Flood", "inundation" and inondation (French) end up near each other; "flood" and "invoice" end up far apart. Inside a language model, every token is turned into an embedding before the model works on it. Separately, embeddings are used to search by meaning: to find the documents closest to a question even when they don't share any of its exact words.

Analogy: a map where every idea has coordinates. Related ideas are neighbours, so to find documents about your question you look at what is nearby on the map.

Why you care: embeddings are how AI tools find the relevant pages in a large document library (see RAG in Class 4) and how search can work across languages.

Embeddings are how a system finds text by meaning instead of by keyword. "Families sleeping in the mosque" and "displaced households in a religious building" share almost no words but have similar embeddings. Closeness is usually measured with cosine similarity, which is 1.0 for vectors pointing the same way and 0 for unrelated ones:

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

# Toy 3-dimensional "embeddings" (real ones have hundreds or thousands of dimensions):
shelter_mosque = [0.9, 0.1, 0.3]
displaced_church = [0.8, 0.2, 0.35]
cholera_alert = [0.1, 0.9, 0.2]
print(cosine(shelter_mosque, displaced_church))   # ~0.99: similar meaning
print(cosine(shelter_mosque, cholera_alert))      # ~0.27: different topic

This is the foundation of retrieval-augmented generation (RAG) in Class 4. Good multilingual embedding models place the same meaning close together across languages, so a French query can find an Arabic report.

7. How a chat model is made: training stages

A chat model is built in stages, and each stage explains a behaviour you will see.

1. Pre-training. The model learns next-token prediction on a very large body of text (web pages, books, code), measured in trillions of tokens. The result, a base model, is a powerful text continuer but not an assistant. Ask it a question and it may continue with more questions.
2. Fine-tuning (supervised). The base model is trained further on curated examples of instructions and good responses. Now it follows instructions and answers in a helpful format.
3. Preference training. People (and increasingly AI graders) compare responses, and the model is trained to prefer the better ones. This is known as RLHF (reinforcement learning from human feedback) and related methods. This stage shapes tone, helpfulness, refusals and safety behaviour. Reasoning models get an extra stage of reinforcement learning on problems with checkable answers.

Know the Term: Pre-training, fine-tuning and human feedback

Chat models are built in stages. Pre-training is the huge first stage: the model reads an enormous amount of text (web pages, books, code) and learns to predict the next token. This takes weeks on thousands of specialised chips and produces a base model that can continue any text but doesn't reliably follow instructions. Fine-tuning (also called instruction tuning or supervised fine-tuning) then trains it further on a much smaller set of example conversations showing good, helpful answers. Finally, learning from human feedback (RLHF, reinforcement learning from human feedback, and related methods) has people, or AI graders following written guidelines, compare pairs of answers; the model is adjusted towards the preferred ones, which shapes its tone, helpfulness and refusals.

Analogy: a new staff member who first reads the whole library (pre-training), then shadows experienced colleagues on real cases (fine-tuning), then gets regular feedback from a supervisor on which of their drafts were better (human feedback).

Why you care: most of what a model "knows" comes from pre-training, fixed at a cut-off date. Its politeness, style and willingness to refuse come from the later stages. And the feedback stage rewards answers people like, which is one reason chatbots can sound confident and agreeable even when they are wrong.

What follows from this:

• Knowledge is frozen at the knowledge cutoff. Pre-training data stops at a date. The model knows nothing after it unless you put it in the context, and it may not know that it does not know.
• Helpfulness is trained, and it has side effects. Preference training rewards answers that people liked, and people like confident, complete answers. That is one reason models guess instead of saying "not stated", and why your prompts and validators must make "not stated" an acceptable answer (Class 2).
• You rarely need to train. Fine-tuning your own model is sometimes worthwhile (Class 8 covers when). For almost everything in this course, prompts, context and code give you more control for far less cost.

Know the Term: Knowledge cutoff

A model learns from a huge collection of text gathered up to a certain date, and then its training stops. That date is its knowledge cutoff. Anything that happened afterwards, such as a new policy, a changed ceasefire line, last week's flood figures or a newly appointed head of agency, is simply not in the model's memory. Some tools add a web search or a document search on top, which can bring in newer information, but the model underneath still has a fixed cutoff. Worse, the model doesn't always know what it doesn't know: asked about recent events, it may answer confidently from older information as if it were current.

Analogy: a well-read colleague just back from a year at a remote field post with no internet. They know a lot, but nothing from the past year, and they may not realise how much has changed.

Why you care: for anything recent or fast-moving, don't rely on the model's memory. Give it the current documents yourself, and check the date on any fact that matters.

8. Choosing a model: the main types

"Which model?" is a design decision you will make per step, not once per project. Four distinctions cover most of it.

Reasoning models vs standard models

Reasoning models generate a hidden (or summarised) chain of intermediate reasoning before answering, spending extra "thinking" tokens. They are much better at multi-step problems: reconciling figures across reports, planning, checking constraints, writing code. They are slower and cost more per answer, because thinking tokens are billed as output. As of 2026, most major providers offer reasoning models or a reasoning mode with an adjustable effort setting.

Know the Term: Reasoning models

A reasoning model is a language model trained to "think before it answers". Before writing its final reply, it generates a long internal working-out (sometimes called thinking or a reasoning trace): breaking the problem into steps, trying approaches and checking its own work. It learned this habit through extra training that rewards correct final answers on problems that can be checked, such as maths, logic and code. Many tools now offer a "thinking" or "reasoning" mode, or switch it on automatically for harder questions (as of 2026). Reasoning takes more time and more tokens, so it costs more.

Analogy: the difference between answering a colleague off the top of your head and saying "give me ten minutes with a notepad".

Why you care: use reasoning modes for multi-step analysis, such as reconciling figures from several reports; don't bother for a quick rewrite. Reasoning reduces some errors but doesn't remove them, and a long, tidy chain of thought can still end in a wrong answer.

Use them for steps that need judgement across several facts (the Analyst, the planner). Do not use them for simple extraction or classification, where a standard model is faster, cheaper and just as accurate.

Multimodal models

Multimodal models accept (and some produce) more than text: images, scanned PDFs, audio, sometimes video. For field work that matters: a photo of a handwritten distribution list, a scanned government bulletin, a voice note from a volunteer.

Know the Term: Multimodal models

A multimodal model can work with more than one kind of input or output ("modality"): text, images, audio, video, or documents such as PDFs and scanned forms. It turns each kind of input into the same internal form (tokens and embeddings), so it can, for example, read a photo of a handwritten distribution list, describe a chart, transcribe a voice note, or answer a spoken question aloud. Most leading chat tools are multimodal (as of 2026), though which modalities are switched on depends on your organization's tool and settings.

Analogy: a colleague who can read the report, look at the photos attached to it and listen to the voicemail, rather than one who only reads typed text.

Why you care: photos, scans and recordings from the field can now go straight into the AI tool. But reading an image is another place for errors (a misread "7" for "1"), and images and recordings can contain faces, voices and locations that need the same data protection as text.

Treat a multimodal step like any other AI step: its output (the transcription, the extracted table) is a proposal, and a code or human check follows. Images and audio also cost tokens, often many.

Open-weight vs closed models

Closed models (from providers such as OpenAI, Anthropic or Google) are used only through the provider's API or a cloud platform such as Azure. Open-weight models (as of 2026, families such as Llama, Mistral, Qwen, DeepSeek, Gemma and gpt-oss) publish their trained weights, so you can run them on your own servers.

Know the Term: Open vs closed models

A closed (or proprietary) model is used only through its maker's service or partners: you send your text over the internet and get an answer back, but you never get the model itself. Examples include the GPT, Claude and Gemini families (as of 2026). An open-weight model has its weights (the trained numbers that make up the model) published for download, so an organization can run it on its own servers or even a laptop, and adapt it. Examples include the Llama, Mistral, Qwen and DeepSeek families (as of 2026). "Open weights" is not always "open source": the training data and code are often not released, and licences can restrict use. Closed models are often the most capable and the easiest to use; open models give more control over where data goes and what it costs.

Analogy: a closed model is like a restaurant: you get the meal, never the kitchen. An open-weight model is like a ready-made dish you can take home, reheat in your own kitchen and season to taste, though you still don't get the full recipe.

Why you care: the choice affects where your data is processed, who can see it, cost and control. For sensitive organizational data, being able to run a model inside your own infrastructure can matter as much as raw capability.

For most organisations the trade-off is mostly about data control versus capability and effort. Open weights let sensitive data stay on infrastructure you control and remove dependence on one vendor. You then run, secure, scale and update the model yourself. Closed frontier models are typically the most capable and need no infrastructure, but your data goes to the provider under the terms of your agreement. LiteLLM talks to both, so the lab code does not change.

Small vs large models

Larger models are generally more capable and more knowledgeable, and they are slower and more expensive per token, often by 10–20 times. Small models are fast and cheap, and for narrow, well-specified tasks (classification, extraction with a schema, routing) they are often good enough, especially with a validator behind them.

Know the Term: Small vs large models

Model size is usually measured in parameters (the adjustable weights learned in training). Large models have hundreds of billions of parameters or more and run in big data centres; they are generally better at difficult reasoning, rare topics and nuanced writing, but are slower and cost more per token. Small models (often called small language models, roughly a few billion parameters or fewer) are faster and cheaper, and some can run on a laptop or phone without an internet connection. Providers often offer a family in several sizes (for example a "mini" or "flash" tier alongside a flagship, as of 2026). For narrow, well-defined tasks, such as sorting messages into categories or pulling dates out of text, a small model is often good enough.

Analogy: a team of fast junior assistants vs one senior expert. The juniors are cheaper and quicker for routine work; you call the expert for the hard cases.

Why you care: bigger is not always better. Matching the model to the task saves money, energy and waiting time, and a small model running locally can keep sensitive data from leaving your organization.

A practical selection table for the SitRep pipeline (tiers, not product names, which change every few months):

Step	Model choice	Why
Classify incoming messages	Small, standard	Short input, few labels, high volume
Extract facts into a schema	Small or mid, standard	Mechanical; validation catches errors
Transcribe a photo of a handwritten list	Multimodal	Input is an image
Reconcile conflicting figures, plan	Large or reasoning	Multi-step judgement
Draft the narrative	Mid, standard	Fluent writing to a template
Anything with sensitive data your agreement does not cover	Open-weight, self-hosted, or don't use AI	Data control comes first

Start with the cheapest model that passes your tests, and move up only when a measured failure justifies it. Class 6 turns this into model routing and cascades, and Class 7 gives you the tests to measure with.

9. Non-determinism and what to do about it

Send the same prompt twice and you may get two different answers. This is non-determinism, and it has three sources:

1. Sampling. With temperature above 0, randomness is the design.
2. Temperature 0 is not a guarantee. Providers run models on shared hardware in batches, and tiny floating-point differences can change which token wins a near-tie. After one different token, the rest of the reply can diverge.
3. Models change. A model name without a version (an "alias") may point to a newer model next month. Providers also retire models.

The first question is not "how do I make it deterministic?" but "what does my system do when two runs disagree?". The engineering consequences are the backbone of this course:

Consequence	What to do	Where in the course
You cannot unit-test the model's exact words	Put the model behind an interface and test your code with a scripted FakeLLM	Every lab
Output shape can vary	Parse and validate every output against a schema; retry with the error	Class 2
Answers can disagree	For short, high-stakes answers, sample several times and vote; escalate to a human when agreement is low	This lab
Quality is a distribution, not a single run	Evaluate on a test set of many cases, and compare versions statistically	Class 7
Models change underneath you	Pin model versions where the provider allows; re-run evals before upgrading	Classes 7–8
You need to explain a past output	Log prompt version, model version, inputs, outputs and settings for every call	Class 7

The lab's variance.py shows the voting pattern (often called self-consistency). It asks the same question N times, normalises the answers ("1,200 households." and "1200" are the same answer), and accepts the majority only if enough samples agree:

def majority_answer(replies, min_agreement=0.7, normalize=normalize_number):
    report = analyse(replies, normalize)
    return report.majority if report.agreement >= min_agreement else None   # None -> human

Run it on the fictional Kessan reports, where one report says 1,200 households and another says 1,450, and the disagreement between samples becomes information: it points you at a genuine conflict in the source data that a human must resolve. Voting costs N calls, so use it for short answers that matter, not for drafting whole reports.

Prompt Lab

Use your approved chat tool or the lab's llm.py. Compare results with your pair.

Exercise 1: Decomposition partner

Task: use the model to decompose a process, then critique its answer against this class's two rules. Run it on the SitRep first, then on a process from your own work.

I am designing a system that turns daily field reports from district teams into a situation report (SitRep) for humanitarian partners.

Decompose the process into 8-12 steps. For each step give:
- a one-line description
- the kind of work: io, language, judgement, arithmetic, verify, approval or publish
- the owner: AI (a language model), code (deterministic software) or human
- for every AI step, which later step checks its output

Rules: arithmetic and format checks must be code; approval must be a human; every AI step must be checked by a later code or human step; nothing is published before a human approves it.

Return a markdown table, then list the two steps whose owner you are least sure about and why.

What to look for:

• Does the model give arithmetic to itself? Does it put a check after every AI step, or quietly drop the rule?
• Are its "least sure" steps the same ones your pair argued about?
• Paste its table into MY_DECOMPOSITION in the lab and run the validator. Did the model follow rules it was given in the same prompt?

Exercise 2: Same prompt, five answers

Task: see non-determinism directly. Send this prompt five times, in five new chats (or five calls), and record each answer. If you can set temperature, repeat at 0 and at 1.

Read these two reports and reply with a single number only: how many households are affected in total?

Report 1 (14 March): approx 1,200 households affected in Tamsi and Lower Tamsi. Report 2 (15 March): We count 1,450 households affected across Tamsi, Lower Tamsi and Bara.

What to look for:

• How many different answers did you get: 1,200, 1,450, 2,650 or something else? The correct answer is not a single number, because the reports overlap.
• Did temperature 0 make the answers identical? Did it make them right?
• Rewrite the prompt so the honest answer is possible (for example, allow "conflict: 1,200–1,450, reports overlap"). How does that change the variance?

Exercise 3: Probing the knowledge cutoff

Task: find the edge of the model's knowledge, and see how it behaves there.

1. What is your knowledge cutoff date?
2. Name three significant developments in AI model releases in the six months before
   your cutoff, with dates.
3. What happened in AI in the month after your cutoff? If you do not know, say so
   explicitly rather than guessing.
4. For each answer above, rate your confidence (high / medium / low) and explain why.

What to look for:

• Does the stated cutoff match the provider's documentation (look it up)?
• In question 3, does it say it does not know, or produce plausible "news"? What does that tell you about asking a model for facts that change, such as funding figures, contacts or policies?
• If your tool has web search switched on, the answer changes completely. Which parts came from the model and which from search? How would a user tell?

Lab: Thinking and models in code

Goal: turn the class's three big ideas into small tested tools: a token and cost estimator, a checked decomposition of the SitRep process, and a variance experiment. Every test runs without an API key.

The code is in labs/engineering/c1/:

File	Role
llm.py	LLM = Callable[[List[Dict[str, str]]], str] and a LiteLLM adapter
tokens.py	Pluggable tokenizer, cost estimator, six-language comparison
decomposition.py	The decomposition exercise and its validator
variance.py	Sample N times, normalise, vote or escalate
test_c1.py	20 tests with a pure-Python tokenizer and a scripted FakeLLM
sample_reports.md	Three fictional Kessan reports with a deliberate conflict

Step 1 · Set up and run the tests (10 min)

cd labs/engineering/c1
source ../prework/.venv/bin/activate     # or create a new venv as in the pre-work
pytest -q

All 20 tests should pass without a key.

Step 2 · Tokens and cost (15 min)

python tokens.py

The tokenizer is pluggable: any function text -> list of token strings works.

Tokenizer = Callable[[str], List[str]]

def count_tokens(text: str, tokenizer: Tokenizer = approx_tokenize) -> int:
    return len(tokenizer(text))

approx_tokenize is a pure-Python stand-in so the tests need no packages. It splits by script with a rough characters-per-token rate for each. With pip install tiktoken you can plug in a real tokenizer: language_table(LANGUAGE_SAMPLE, tokenizer=tiktoken_tokenizer()).

Try it: with the "small" and "large" example prices, what does the SitRep extraction step cost per month at 300 reports a day if half the reports are in Arabic? What if every report is translated into English first (one extra call per report)?

Step 3 · Decompose the SitRep (20 min)

Open decomposition.py. MY_DECOMPOSITION lists eleven steps, each with a kind but with owner="?". Fill in each owner and, for AI steps, checked_by, then run:

python decomposition.py

The validator enforces the rules from Section 1:

if t.kind == "arithmetic" and t.owner != "code":
    problems.append(f"{t.id}: arithmetic belongs to code (deterministic and testable)")
if t.owner == "AI":
    ...  # checked_by must name a LATER step owned by code or a human
if t.kind == "publish" and not human_approval_seen:
    problems.append(f"{t.id}: nothing is published before a human approval step")

When it passes, compare with REFERENCE_DECOMPOSITION. Where you differ, is either answer wrong, or are both defensible? Add one step of your own (for example "detect personal data before sending text to the model") and decide its kind and owner.

Step 4 · Variance handling with FakeLLM (10 min)

Read the variance tests in test_c1.py. The FakeLLM returns a scripted list of different answers, as a real model might across N samples:

llm = FakeLLM(["1200", "1,200", "1450"])
replies = sample(llm, messages, n=3)

analyse() normalises and counts them; majority_answer() returns the majority only at 70% agreement or more, otherwise None, meaning "send to a human".

Try it: write a test where eight replies say 1,200 and two are unparseable ("I cannot tell"). Should unparseable replies count against agreement? Decide, write the test, make it pass.

Step 5 · Run it for real (5 min)

export MODEL=openai/gpt-4o-mini          # or azure/<deployment>, anthropic/<model>
python variance.py

Compare the spread at temperature 0 and 1.0. With your pair: did temperature 0 give one answer? Was the majority answer correct, given that the reports overlap? What should the SitRep say about this figure?

Stretch challenges

1. Real tokenizer ratios. Install tiktoken, compare two encodings across the six languages, and tune CHARS_PER_TOKEN to within 15% of the real counts.
2. A better normaliser. Make "Between 1,200 and 1,450" count as unparseable (or as a range), with tests.
3. Budget guard. Write check_budget(prompt, max_input_tokens) that refuses a call before it is made, and decide where in the decomposition it belongs.

Check Your Understanding

1 In your SitRep decomposition, one step is "add up displaced households per district across today's 12 reports". Who should own it?

1. The model — it has already read the reports, so it can add the figures in the same call.
2. Deterministic code — summing is arithmetic with one right answer, and one line of Python does it exactly and testably.
3. A human — only a person can be trusted with figures that go to partners.
4. The model, followed by a second model call that checks the sum.

2 A SitRep extraction pilot in English cost USD 400 a month. The service is being rolled out to offices that report in Arabic and Russian at the same volume. What is the best budgeting approach?

1. Keep USD 400 — the same messages carry the same information, so they cost the same.
2. Multiply by six, one factor for each of the six languages the organisation works in.
3. Translate everything into English first; that will always be cheaper.
4. Measure token counts for sample Arabic and Russian reports with the model's own tokenizer, and budget from those numbers (expect them to be higher).

3 You set temperature to 0 on the extraction step. A colleague says: "Now it's deterministic, so we can drop the schema validation and the FakeLLM tests." What is the best reply?

1. Temperature 0 reduces variation but guarantees neither identical outputs (batching, model updates) nor correct ones, so keep validating and keep testing your own code.
2. Agreed — at temperature 0 the model always picks the most likely token, so outputs are fixed.
3. Set top-p to 0 as well, and then it is truly deterministic.
4. Keep the tests but drop validation, because the output format will now always be the same.

4 Your model has a 1-million-token context window. A developer proposes sending all three months of Northern Veloria field reports (about 600,000 tokens) with every SitRep question "so it never misses anything". What is the main problem?

1. None — if it fits in the window, the model will use all of it equally well.
2. The model will remember the reports after the first call, so later calls are wasted.
3. You pay for all 600,000 input tokens on every call, latency rises, and accuracy can fall. Select the relevant reports for each question instead.
4. It will fail, because the output has no room left in the window.

5 Asked for the latest cholera case count in Aramu district, a chat model replies with a precise figure and a date. Nothing about Aramu was in the prompt. What is the most likely explanation?

1. The model looked the figure up in its internal database of health statistics.
2. The model generated a plausible-looking answer. Current figures are after its knowledge cutoff and were never in its training data, so they must come from the context or a tool, and be verified.
3. The figure is probably right, because a model would say if it didn't know.
4. Temperature was too high; at temperature 0 it would have given the correct figure.

6 The SitRep inbox receives about 5,000 messages a day, and each must be labelled as one of four types (field report, request, bulletin, other). Which model choice is the best starting point?

1. A small, fast standard model with a fixed label list and code that rejects any other label, then measure accuracy on a labelled sample and move up only if it falls short.
2. The largest reasoning model available, because classification errors are costly.
3. A multimodal model, because some messages might contain attachments.
4. No model — train your own classifier from scratch first.

7 To make a key figure more reliable, you ask the model the same question five times and get: 1,200 · 1,450 · 1,200 · 2,650 · 1,450 households. What should the system do?

1. Average them (about 1,590) and report that.
2. Take the most common answer, since 1,200 and 1,450 are tied, pick the higher one to be safe.
3. Run it 50 more times until one answer wins.
4. Treat the low agreement as a signal: return no figure, flag the conflict (the reports overlap), and send it to a human to resolve.

8 A draft SitRep says "approximately 300 households affected in Dorra", but the field team never reached Dorra and no report gives a figure. Which explanation and fix are best?

1. The model misread a report; fix it by using a model with a bigger context window.
2. The model is broken; switch providers.
3. Next-token prediction fills gaps with likely-sounding text. Make "not stated" an explicit, acceptable answer, and add a code check that every figure appears in its source report.
4. Add "Do not hallucinate" to the prompt; that fully prevents it.

Exit Ticket

Before you leave

Take one process from your own work that you might automate with AI. Write down (1) its steps, each labelled AI, code or human; (2) the one AI step whose output you would find hardest to check, and how you would check it anyway; (3) which step you would never give to a model, and why.

Key terms

• Computational thinking: decomposition, pattern recognition, abstraction and algorithms, applied so people and machines can carry out a task reliably.
• AI / code / human split: giving each step one owner; AI proposes, code checks, humans decide.
• Large language model (LLM): a neural network trained on very large amounts of text to predict the next token.
• Token, tokenizer, byte-pair encoding (BPE): the chunks of text a model reads, the tool that splits text into them, and a common algorithm for building its vocabulary.
• Next-token prediction, autoregressive generation: producing a probability for every possible next token, and generating text one token at a time.
• Sampling, temperature, top-p: choosing the next token from the distribution; flattening or sharpening it; keeping only the most probable tokens that add up to p.
• Hallucination: fluent output that is not supported by the source or by fact.
• Context window: the maximum number of tokens (input plus output) a model can use in one call.
• Transformer, attention: the architecture of modern LLMs, and its mechanism for weighing how much each token matters to every other.
• Embedding, cosine similarity: a vector that represents meaning, and a measure of how close two such vectors are.
• Pre-training, fine-tuning, RLHF: the training stages that turn a text predictor into an assistant.
• Knowledge cutoff: the date after which the model's training data stops.
• Reasoning model, multimodal model, open-weight model, small vs large model: the main model types to choose between.
• Non-determinism, self-consistency: different outputs for the same input, and voting across several samples to handle it.