Unit 1.1: LLM Behavior, Tokens and Context

Chapter 1: Tokenization and Input/Output Accounting

Imagine you're building an assistant for a small appliance-repair business. Its support team receives messages from customers whose appliances have stopped working, sometimes with an orderly description and sometimes with little more than “It won't start.”

One customer writes:

My washing machine finishes the wash, but there's still water at the bottom. I've tried running the drain cycle twice.

The assistant returns:

Reported issue: Water remains after the wash cycle.
Suggested category: Drainage problem.
Summary for staff: Customer has already attempted a second drain cycle.

This is a useful draft classification, not a confirmed diagnosis. A staff member or an approved process must check it before anyone treats it as repair advice. For now, we're interested in something less obvious: what happened between sending that message and receiving the summary?

The model didn't count the words as we might. It processed tokens. And if our fictional business handles thousands of such requests, the number of tokens will affect how we estimate usage and cost.

By the end of this chapter, you'll be able to explain what tokens are, distinguish the text you see from what an API may count, calculate a simple cost estimate, and plan an experiment that measures tokens and latency rather than guessing. We'll build a small working calculator along the way. It will become the first piece of measurement tooling for the Client Knowledge & Operations Assistant (CKOA) that you'll develop throughout the course.

1.1 A token is not a word

A tokenizer turns text into units that a model can represent as numerical IDs. Those units are called tokens.

A token may correspond to a whole word, part of a word, punctuation, a space joined to nearby text, or another piece of a character sequence. Which pieces you get depends on the tokenizer associated with the model.

Consider:

The technician repaired the refrigerator.

You can count the words easily. But you cannot infer the exact token count just by looking at them. One tokenizer might keep refrigerator together; another might split it. Even a change in punctuation or spacing can affect the result.

This matters when customers write product numbers, use unfamiliar names, include code from an error display, or write in languages other than English. Two messages with the same word count need not use the same number of tokens.

What happens during tokenization?

A tokenizer has a vocabulary and rules for representing text using that vocabulary. The result is a sequence of token IDs the model can process.

Imagine a vocabulary containing the following entries:

These IDs are made up. They are not measurements from a real model. In this imaginary tokenizer, repairing could be represented by repair followed by ing. Another tokenizer might use a single token or split the word differently.

One widely used family of tokenizers is based on Byte Pair Encoding (BPE). During vocabulary construction, BPE repeatedly merges frequent neighboring pieces, allowing common sequences to be represented compactly while less common text can still be broken into smaller units. Other approaches, including WordPiece and Unigram, exist. You don't need to implement them today.

The practical lesson is simpler: a word-count estimate is not a token measurement. When a model or API provides an appropriate tokenizer or counting tool, use it.

1.2 What is actually sent to the model?

Return to our support assistant. A request might contain:

Application instruction:

Classify the customer's message and summarize only what the customer reported. If the message is unclear, say so. Do not provide repair instructions.

Customer message:

My washing machine won't drain.

Possible model response:

Category: Drainage problem.
Summary: Customer reports that the washing machine is not draining.

The instructions and customer message both contribute to the input. The model's generated response contributes to the output.

We call these broad categories input tokens and output tokens. Many hosted model services price them differently.

There's a small trap here. You can see the customer message on your screen, but the model may receive more than that: application instructions, earlier messages, message roles and boundaries, tool descriptions, schemas, files or other structured material. What counts, and how it's represented, depends on the API.

That means counting tokens in the customer message alone is not the same as measuring the entire request.

Three different kinds of count

You'll encounter three useful measurements:

1. Local tokenizer count: A tokenizer on your computer counts a piece of text using an encoding you've selected. Good for exploring how text is split. It may leave out request structure or other inputs.
2. Pre-request count: Some providers offer an endpoint that counts the assembled request before generation. This can be more accurate for that provider and request format, although its documentation determines whether it is an estimate or an exact count.
3. Post-response usage: The API reports usage associated with a request it processed. This is what you should consult when recording actual usage for that API, alongside its billing rules.

They answer related questions, but they are not interchangeable. For example, an OpenAI input-token count can include structured message overhead that a plain-text local tokenizer does not see; Anthropic describes its pre-request count as an estimate that may differ slightly from actual usage.

You'll learn to build reliable API requests in Unit 1.2. Today, the important habit is to label where each number came from.

1.3 Reading usage without double-counting

Consider this fictional usage summary:

These are illustrative numbers. No API call produced them.

For this simplified record, the arithmetic is:

1,200 input tokens + 150 output tokens = 1,350 tokens.

But do not turn that into an assumption that every provider's total field is always just those two fields. Providers may expose tool-use tokens, thinking tokens, cached-token details and different definitions of totals.

For example, OpenAI's Responses API includes non-visible generated tokens in output_tokens; reasoning tokens, where reported in its output details, are part of that output count. You must not add those reasoning tokens to output_tokens again. A response may therefore have more reported output tokens than you can count in its displayed answer.

Google's Gemini Generate Content API exposes fields such as promptTokenCount, candidatesTokenCount, thoughtsTokenCount and toolUsePromptTokenCount. Its totalTokenCount accounts for those categories according to Google's documented rules. You cannot safely rename candidatesTokenCount to “all output usage” and assume you're finished.

Cached input is another common source of errors. A provider may report cached tokens as a subset of a broader input count, with a different price. Adding the cached subset to the full input count again would inflate your estimate. The right calculation is whatever the current provider documentation says for the specific request and pricing tier.

One rule to remember: keep the original usage fields in your logs. Decide how to calculate cost only after you understand their definitions. In this chapter we use a simpler two-rate example specifically to learn the arithmetic.

When a request fails

An API request that fails may not return usable token-usage metadata. A failed login may never reach model generation; a network interruption may leave you unsure whether the provider processed anything. Some failures can still incur costs, depending on the service and the stage of failure.

Record the outcome honestly: reported, estimated, or unavailable. Do not turn an unavailable count into zero, and do not guess that a failed request was free.

We'll deal with retries, errors and safe logging in Unit 1.2. For now, don't let a blank cell masquerade as a measurement.

1.4 Turning token counts into a cost estimate

Suppose, purely for this exercise, that a provider publishes these prices:

These prices are invented for the lesson, not current rates for any named service. We also assume no separate cache, reasoning, tool, batch or other charge category in this calculation.

The fictional request above used 1,200 input tokens and 150 output tokens. Start with the input:

Input cost = 1,200 × $0.50 / 1,000,000 = $0.0006

Now the output:

Output cost = 150 × $2.00 / 1,000,000 = $0.0003

Add them:

Estimated request cost = $0.0009.

That's nine-hundredths of one US cent. One request is inexpensive under these assumptions. What if the business receives 10,000 requests of roughly that same size?

10,000 × $0.0009 = $9.00.

Notice what the $9 estimate does not tell us. We don't know how often a customer sends a much longer message, whether the assistant needs follow-up requests, whether the provider uses cached input pricing, or whether the model generates substantially more output for difficult cases. A defensible business forecast needs a representative sample and a record of those assumptions.

If you change the rates to dollars per thousand rather than per million, or accidentally count cached or reasoning tokens twice, the arithmetic can look perfectly tidy while the forecast is wrong. The units and categories matter just as much as multiplication.

1.5 Build the first CKOA measurement tool

Let's make our first working component: a calculator that accepts token counts and two per-million-token prices, then returns an estimate.

This is an offline exercise. It makes no network requests, needs no API key and costs nothing to run. It uses Python's built-in decimal module so we can express monetary rates without introducing avoidable binary floating-point rounding into this example.

Create a file named token_cost.py on your own computer and put this code in it:

from decimal import Decimal

MILLION = Decimal("1000000")


def estimate_cost(input_tokens, output_tokens, input_rate, output_rate):
    for count in (input_tokens, output_tokens):
        if type(count) is not int or count < 0:
            raise ValueError("Token counts must be non-negative integers")

    for rate in (input_rate, output_rate):
        if not isinstance(rate, Decimal) or not rate.is_finite() or rate < 0:
            raise ValueError("Rates must be non-negative Decimal values")

    return (
        Decimal(input_tokens) * input_rate
        + Decimal(output_tokens) * output_rate
    ) / MILLION


input_rate = Decimal("0.50")
output_rate = Decimal("2.00")

request_cost = estimate_cost(1200, 150, input_rate, output_rate)
print(f"One request: ${request_cost:.4f}")
print(f"10,000 similar requests: ${request_cost * 10000:.2f}")

assert estimate_cost(
    1200, 150, Decimal("0.50"), Decimal("2.00")
) == Decimal("0.0009")

try:
    estimate_cost(-1, 150, input_rate, output_rate)
except ValueError:
    print("Negative counts rejected")
else:
    raise AssertionError("Negative counts should be rejected")

Run it with python token_cost.py (or python3 token_cost.py if that's how Python is installed on your computer). You should see:

One request: $0.0009
10,000 similar requests: $9.00
Negative counts rejected

The program does two useful things beyond calculating a price. It rejects negative or non-integer token counts, and it uses an assertion to check the original example independently, even when you change the printed request. Those are early examples of treating a cost estimate as something to test, not just display.

Try changing the input count to 2,000, the output count to 500, the input rate to Decimal("1.00"), and the output rate to Decimal("4.00"). Predict the cost before running the program. You should get $0.0040. The original assertion should still pass. Then add a second assertion that checks the new inputs against Decimal("0.0040").

Your function still makes assumptions: only two token categories, both rates in USD per million tokens, and counts already obtained from a trustworthy source. That's fine for a first component. We will extend it when the assistant begins working with real providers.

1.6 Investigate the difference between text length and tokens

Let's return to three fictional customer messages. We'll keep the application instruction the same each time:

Classify the reported issue and summarize only what the customer says. If information is missing, mark it as unknown. Do not give repair instructions.

Message A: Short

My washing machine won't drain.

Message B: Ambiguous

My machine isn't working properly. It makes a sound and then stops. I don't know when it started doing this.

Message C: Longer

I have a washing machine that stops near the end of a cycle. Yesterday, the display showed an unfamiliar error code. I turned it off and later tried another cycle, but the clothes were still wet. This morning, the same thing happened. I cannot tell whether the machine drained completely. I haven't checked any internal components, and I don't know the model number yet. I'd like the support team to tell me what information they need before scheduling any service.

Before measuring anything, write down your predictions. Which customer message contains the most words? Which do you expect to have the most tokens? Is the longest message necessarily the most difficult to classify?

For a first comparison, use a tokenizer that matches a model you intend to examine. Note the model/encoding and tokenizer version. Count the customer text only, then count the customer text plus the application instruction, being careful that a local count of concatenated text may still omit structured-message overhead. Label these as local text counts, not provider-billed request totals.

A tokenizer is not necessarily available offline for every model. If you don't have a compatible tokenizer, record that limitation rather than inventing exact counts. The manual word counts and the offline cost calculator can still be completed.

For the model-probe portion of the required Unit 1.1 lab, use the separately verified starter request runner provided with the unit's lab materials when it becomes available. That runner should handle the provider-specific request and usage fields, so you don't have to learn SDK engineering before Unit 1.2.

Using the same model and unchanged application instructions and generation settings, compare A, B and C. Repeat each case at least three times if a safely accessible model route is provided. Record the actual reported usage rather than filling in expectations. A live probe may involve charges: check the provider's terms, quota and pricing first, stay inside the published lab budget, and don't supply a real customer's personal information. No paid model usage is required to complete the offline preparation in this chapter; the real-model evidence must remain marked pending if an accessible, approved route or verified runner is not yet available. The offline preparation alone does not satisfy the required model-probe assessment.

Keep a usable experiment record

An example row can show you the shape of a log, but don't mistake it for an experiment:

Every number in this example table is fictional. Your observed counts and times, if you run real requests, are the evidence that matters.

1.7 Tokens, elapsed time and a careful conclusion

A model request that costs little may still take long enough to frustrate a customer. That's why our logs need latency as well as usage.

For this chapter, define our measurement precisely: start the clock immediately before the runner sends the request and stop it when the complete response has arrived. The difference is observed end-to-end elapsed time for that client request. It includes more than the model's work; network conditions and provider-side delays can also affect it.

Later, we'll study other measures, including time to first token when using a stream. Those measures answer different questions. Don't label a stopwatch measurement as “model processing time” unless that's genuinely what it captures.

More input tokens don't guarantee proportionally longer completion time. Longer generated output often matters substantially, but model choice, server conditions, caching and request features can change what you observe. Three repetitions per test case are a useful beginning, not a statistically conclusive performance study.

When you compare your results, ask: What is definitely supported by the reported counts? Which differences could be explained by request structure or provider accounting? How much did timing vary between repeats? What would you need to measure again before recommending a price or latency promise to our fictional client?

The aim is not to get a perfectly smooth chart. The aim is to explain the evidence honestly.

1.8 Your practical work and evidence

You'll have two related pieces of work.

First: work you can complete now, offline. Run the calculator, test its normal and negative cases, calculate the revised $0.0040 example, document the assumptions, and compare the three synthetic messages with a suitable tokenizer if one is available. Keep your source code and a short note explaining the results and any limitations.

Second: the Unit 1.1 model probe, when its verified runner and approved model route are available. Collect repeated usage and elapsed-time observations for the short, ambiguous and long requests. Submit the request configuration, sanitized log, cost calculations, and an explanation of any variations or unavailable measurements. Explain why a local text count and the complete API request count may differ. A notebook or equivalent reproducible experiment record must make your work inspectable.

Don't publish credentials, customer data or raw error payloads containing secrets. If a model route isn't available, state not run. Don't invent API output to make an assignment look finished.

The full Unit 1.1 assessment also includes later lessons about context windows, sampling and nondeterminism. You are not expected to demonstrate context overflow or explain every output failure in this first chapter. We'll build that evidence as the unit progresses.

1.9 Check your understanding

Pause here and try these without looking back at the formulas.

Question 1. Two customer messages contain 50 words each. Must they contain the same number of tokens? Explain why.

Question 2. A local tokenizer counts 600 tokens in a customer's message, but the API reports 625 input tokens for the full request. Name two things you'd investigate before deciding either result is wrong.

Question 3. Using our two-rate illustration, what is the estimated cost of 2,000 input tokens at $1 per million and 500 output tokens at $4 per million?

Question 4. An OpenAI response reports 150 output tokens, of which 40 are listed as reasoning tokens in the output details. Should you record 150 or 190 as the output count for that response? Why?

Question 5. A network error interrupts an API request and no usage record is available. What should the log contain? Can you conclude that the provider charged nothing?

If you can explain your answer in terms of evidence and definitions, not merely repeat a sentence from the chapter, you've understood the part that matters.

1.10 What we carry into the next chapter

A token is a model-processing unit, not a synonym for a word. The tokens visible in a piece of text are only part of the story when an API sends structured requests. A good engineer can say what was counted, by whom, and under which rules.

We now have a small cost calculator and a repeatable plan for measuring real requests. Neither produces a diagnosis for our customer, and neither claims to explain every model failure. They're tools for understanding and operating the assistant responsibly.

Next, we'll face the limitation that makes token budgeting especially important: a model cannot take unlimited context or generate unlimited output in one request. In Chapter 2 we'll explore context windows and output limits, then turn those ideas into experiments we can actually reproduce.

Official references and further reading

The provider-specific behavior in this chapter should be checked against the API and model in use, because documentation and token accounting can change.

• OpenAI: What are tokens and how to count them?
• OpenAI API: Counting tokens
• OpenAI API: Reasoning models
• Google Gemini API: Understand and count tokens
• Google Gemini API: Generate Content usage metadata
• Anthropic: Token counting
• Hugging Face: Tokenization algorithms