Skip to content
>_KernelBrief
Artificial Intelligence

How Large Language Models Work: Tokens, Context Windows and Parameters Explained

How large language models work, in plain terms: what tokens are, how context windows fill up, what parameters mean, and how to count tokens in code.

By Published 10 min read
// AI
Table of contents

A large language model (LLM) works by repeatedly predicting the next small chunk of text — a token — based on all the tokens it has seen so far. The text it can "see" at once is limited by its context window, and what it has learned about language is stored in billions of numerical weights called parameters. Understanding those three ideas explains most of what developers run into in practice: API bills, "prompt is too long" errors, models that forget early instructions, and why some models need a data-centre GPU while others run on a laptop.

This guide explains each concept in plain terms, then shows how to count tokens in code before you send a request.

The core loop: predicting the next token

Almost every modern LLM is built on the transformer architecture, introduced in the 2017 paper Attention Is All You Need. Chat models are "decoder" transformers that generate text one token at a time:

  1. Your input is split into tokens and each token is turned into a vector of numbers (an embedding).
  2. The vectors pass through many stacked transformer layers. In each layer, the attention mechanism lets every token weigh how relevant every earlier token is — this is how the model links "it" back to the right noun three sentences earlier.
  3. The final layer produces a probability for every token in the model's vocabulary: how likely each one is to come next.
  4. A token is picked from that distribution, appended to the sequence, and the whole process repeats until the model emits a stop token or hits a length limit.

Two practical consequences follow. First, the model has no separate "memory" of your conversation — each new response is computed from the tokens currently in the context. Second, generation is sequential, which is why output tokens are slower to produce than input tokens are to read, and why most APIs charge more for output.

Why answers vary: sampling and temperature

Step 4 does not have to pick the single most likely token. Settings such as temperature reshape the probability distribution before sampling: low values make the model favour its top choices (more consistent, more repetitive), while higher values spread probability to less likely tokens (more varied, more error-prone). That is why the same prompt can produce different answers, and why you should lower temperature — where your provider supports it — for tasks like data extraction where consistency matters more than creativity.

What is a token?

A token is the unit of text a model reads and writes. It is often a whole common word, but it can also be part of a word, a punctuation mark, a space plus a word, or a fragment of code. Rare words are split into several pieces; common words usually are one.

Providers publish rough rules of thumb for English:

Source Rule of thumb
OpenAI Help Center 1 token ≈ 4 characters ≈ ¾ of a word; 100 tokens ≈ 75 words
Google Gemini API docs 1 token ≈ 4 characters; 100 tokens ≈ 60–80 English words

Treat these as estimates, not conversions. OpenAI notes that tokenization is language-dependent: its example Spanish phrase "Cómo estás" takes 5 tokens for 10 characters, so non-English text can cost noticeably more per word. Source code, JSON, long numbers and unusual identifiers also tend to use more tokens than prose.

Why models use tokens instead of words or letters

Most tokenizers use a variant of byte-pair encoding (BPE): starting from bytes or characters, the training process repeatedly merges the most frequent adjacent pairs into new vocabulary entries. The result is a fixed vocabulary (typically tens of thousands to a few hundred thousand entries) that can represent any input — common words get a single token, and anything unusual is still representable as smaller pieces. Letters alone would make sequences far longer and slower to process; whole words alone would fail on typos, names and new terms.

Each model family has its own tokenizer, so the same text produces different token counts on different models. That is the main reason to count with the right tool for the model you actually call.

Images, audio and video are tokens too

Multimodal models convert other media into tokens as well. Google's documentation, for example, says Gemini counts a small image (384 pixels or less in both dimensions) as 258 tokens, tiles larger images into 768×768 sections at 258 tokens each, and counts audio at 32 tokens per second. A few minutes of audio or a handful of high-resolution screenshots can therefore consume as much context as a long document.

What is a context window?

The context window is the maximum number of tokens a model can work with in a single request. Crucially, it is a shared budget for input and output. Google's Gemini docs describe it as the combined limit of input and output tokens, and Anthropic's documentation defines it as everything the model can reference when generating a response, including the response itself.

What counts toward that budget is more than your latest message. According to Anthropic's docs, it includes the system prompt, every message in the conversation (including tool results, images and documents) and your tool definitions. In a chat application, the whole history is sent again on every turn, so the context fills up as the conversation grows. Reasoning models add another consumer: Anthropic states that thinking tokens also count toward the limit.

Sizes vary widely. Several current frontier models advertise windows of around 1 million tokens — Anthropic lists 1M-token windows for its newest Claude models, and Google gives the same figure for Gemini 4 Neon — while many other models offer 128,000 or 200,000 tokens. Output is capped separately and much lower; Anthropic, for instance, documents a maximum of 128k output tokens per request on its 1M-context models.

What happens when you run out

Behaviour depends on the provider, but expect one of these:

  • The request is rejected. Anthropic returns a 400 invalid_request_error ("prompt is too long") if the input alone exceeds the window.
  • The answer is cut short. If input plus the requested maximum output does not fit, generation can stop early. On newer Claude models this is reported with a dedicated stop reason, model_context_window_exceeded.
  • The app silently drops old content. Many chat interfaces trim or summarise earlier turns to stay under the limit — which is why a long chat can "forget" instructions you gave at the start.

Bigger is not automatically better

A large window lets you pass whole codebases or contracts in one request, but it has costs. You pay for every input token on every call, latency rises with input size, and quality can drop: Anthropic's documentation warns that accuracy and recall degrade as token count grows, a problem it calls "context rot". Prompt caching reduces the price of resending the same prefix, but cached tokens still occupy the window. For large knowledge bases, retrieving only the relevant passages per question is often cheaper and more accurate than stuffing everything in.

What are parameters?

Parameters are the learned numbers — mostly the weights inside the attention and feed-forward layers — that encode what the model picked up during training. When a model is described as "8B" or "405B", that is its parameter count in billions. Meta's Llama 3.1 release, for example, came in 8B, 70B and 405B sizes.

Parameter count tells you two useful things:

  • Hardware requirements. Every parameter must be held in memory to run the model. At 16-bit precision each parameter takes 2 bytes, so an 8B model needs roughly 16 GB just for its weights, and a 405B model roughly 810 GB — before counting the extra memory used for the context. Quantization (storing weights at 8 or 4 bits) cuts this substantially, at some cost in quality, which is how mid-sized open-weight models run on consumer hardware.
  • Rough capacity. More parameters give a model more room to store patterns and knowledge, which is why the largest models generally perform best on hard tasks.

What parameter count does not tell you is which model is better for your task. Training data quality, training length, fine-tuning and reasoning techniques matter enormously, and newer small models frequently outperform older large ones. Many commercial providers no longer disclose parameter counts at all, so in practice you compare models on price, context size, speed and your own evaluations.

Parameters are also distinct from the settings you pass in an API call. Values like temperature or max_tokens are sometimes loosely called "parameters", but they only control generation; they do not change the model's learned weights.

How to count tokens before you send a request

Counting tokens lets you estimate cost, check that a prompt fits the window and decide how much room to leave for output. The examples below use each provider's official Python library. Install them inside a virtual environment — see our Python venv guide if you need a refresher.

OpenAI models: tiktoken (runs locally)

OpenAI's open-source tiktoken library tokenizes text on your machine, with no API call:

pip install tiktoken
import tiktoken

# o200k_base is the encoding used by GPT-4o and later OpenAI models
enc = tiktoken.get_encoding("o200k_base")
# or look it up from a model name:
# enc = tiktoken.encoding_for_model("gpt-4o")

text = "Tokenization isn't always one word per token."
tokens = enc.encode(text)

print(len(tokens))                       # number of tokens
print([enc.decode([t]) for t in tokens]) # see how the text was split
assert enc.decode(tokens) == text        # encoding is reversible

The first call downloads the encoding file and caches it, so it needs network access once. Note that chat APIs add a few tokens of formatting per message, so a count of the raw text is a close estimate rather than the exact billed figure.

Gemini: the count_tokens endpoint

Google's google-genai SDK calls the API to count tokens for a specific model. This example follows Google's token documentation:

pip install google-genai
export GEMINI_API_KEY="your-key"
from google import genai

client = genai.Client()  # reads GEMINI_API_KEY from the environment

result = client.models.count_tokens(
    model="gemini-3.8-flash",
    contents="The quick brown fox jumps over the lazy dog.",
)
print(result.total_tokens)

# Look up a model's limits
info = client.models.get(model="gemini-3.8-flash")
print(info.input_token_limit, info.output_token_limit)

Claude: the token counting API

Anthropic's SDK exposes messages.count_tokens, which accepts the same shape as a real request — system prompt, messages and tools — and returns the input token count:

pip install anthropic
export ANTHROPIC_API_KEY="your-key"
import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

count = client.messages.count_tokens(
    model="claude-haiku-5-5",
    system="You are a concise technical assistant.",
    messages=[{"role": "user", "content": "Explain what a context window is."}],
)
print(count.input_tokens)

Swap in whichever model you use; the model name matters because tokenizers differ between model generations.

After the call: read the usage fields

Every major API also reports actual token usage in its response — input, output and, where relevant, cached and reasoning tokens. Logging these numbers per request is the most reliable way to track spend and spot prompts that are quietly growing. If you route traffic through a proxy or gateway that holds provider keys, keep it patched and its keys rotated; our coverage of the LiteLLM CVE-2026-40217 attacks shows why stolen LLM credentials are now a target in their own right.

Practical takeaways

  • Budget in tokens, not words. Use the 4-characters-per-token rule only for rough planning; count with the model's own tokenizer or API for anything that matters.
  • Remember the window is shared. A huge prompt leaves less room for the answer, and chat history accumulates on every turn.
  • Trim before you scale up. Summarising old turns, removing unused tool definitions and retrieving only relevant documents often beats switching to a larger context window.
  • Read parameter counts as a hardware guide. They predict memory needs far better than they predict quality.
  • Test on your own workload. Tokenizers, context limits and pricing all differ by model, so the only reliable comparison is your own prompts and your own usage logs.

Frequently asked questions

+How many words is 1,000 tokens?

For English text, roughly 600 to 800 words. OpenAI's rule of thumb is about 75 words per 100 tokens and Google's is 60 to 80 words per 100 tokens. Code, numbers and non-English languages usually need more tokens per word.

+Does the context window include the model's answer?

Yes. Both Google and Anthropic define the context window as the combined budget for input and output, so a long prompt leaves less room for the response.

+Do more parameters always mean a better model?

No. Parameter count sets a model's capacity and its memory and compute cost, but training data, training method and fine-tuning matter just as much. Smaller recent models often beat larger older ones.

+Why do different AI models report different token counts for the same text?

Each model family uses its own tokenizer with its own vocabulary, so the same sentence can split into a different number of tokens. Always count with the tokenizer or counting API for the model you will actually call.

Sources & references

  1. Vaswani et al. — Attention Is All You Need (arXiv, 2017)
  2. OpenAI Help Center — What are tokens and how to count them?
  3. Google AI for Developers — Understand and count tokens (Gemini API)
  4. Anthropic — Claude Platform Docs: Context windows
  5. OpenAI — tiktoken (GitHub)
  6. Meta — Introducing Llama 3.1 (July 2024)

Spotted an error? See our Corrections Policy.

  • #llm
  • #tokens
  • #context-window
  • #generative-ai
  • #ai-basics
KB

KernelBrief Editorial

The KernelBrief editorial desk researches, writes and fact-checks every article published on this site. Every article is checked against primary sources (official documentation, vendor announcements, standards bodies and original research) before publication. Every factual claim is linked to its source, and corrections are handled under our published Corrections Policy.