Tokens, Context Windows & Why They Matter

Tokens are the unit of measure every AI model charges, limits, and thinks in. Learn what they are, how context windows cap what the model can "see," and the practical rules that shape every prompt you write.

TL;DR: A token is roughly 3-4 characters of English text (though this varies by model and generation). The context window is the hard cap on how many tokens a model can process in one shot - both the stuff you send and the reply it writes. Hit that limit and the model either truncates or stops. Understanding this shapes every decision from prompt length to what tool you pick.

What Is a Token?

Models don't read words. They don't read letters either. They read tokens - small chunks of text that sit somewhere between a character and a word.

A token is the atomic unit every language model uses to process and measure text. Roughly speaking, one token equals about 4 characters of English for many models, or about three-quarters of a word. That means 100 tokens is somewhere in the range of 60-80 plain English words. Note that tokenizer efficiency varies significantly across model families and generations - newer tokenizers can be considerably more token-dense (see below).

Some practical examples to anchor this:

Rare words, technical jargon, and non-English text tend to split into more tokens than common English words. The word "Anthropic" might be one token. A word in Finnish or Arabic might be four or five. This matters when you're building multilingual tools or working with specialized vocabulary.

How Tokenization Works Under the Hood

Most modern models use a method called Byte Pair Encoding (BPE). The short version: it starts with a vocabulary of individual characters, then iteratively merges the pairs that appear most often in real text until it reaches a target vocabulary size (typically 32,000 to 200,000 entries). Common short words become single tokens. Rare or long words get split into subword pieces.

The tokenizer is a completely separate module from the model itself. It runs before the model sees anything - converting your raw text into a sequence of integers, then handing those numbers to the model. You never directly interact with this layer, but its behavior determines how your prompts are interpreted.

Each model family has its own tokenizer and its own vocabulary. That means the exact same sentence can produce a different token count depending on which model you're using. OpenAI's GPT-5.5 uses a different tokenizer than Claude Opus 4.8, which uses a different one than Gemini 3.5 Flash. Counts won't match across providers - and can shift meaningfully even within a provider when a new model generation ships. Anthropic's Claude Opus 4.7 introduced a new tokenizer that produces roughly 30% more tokens for the same text compared to pre-4.7 Claude models.

The Context Window: Working Memory, Not Storage

The context window is the total number of tokens a model can process in a single request - including everything you send in plus everything it writes back.

Think of it as working memory, not long-term storage. The model has no persistent memory between requests unless you explicitly build that. Every new API call starts fresh. The context window is all it has to work with, and it runs out.

Anthropic's docs describe it well: "The context window refers to all the text a language model can reference when generating a response, including the response itself. This is different from the large corpus of data the language model was trained on."

The key point: input and output share the same pool. If your context window is 200,000 tokens and you send a 150,000-token document, the model only has 50,000 tokens left to write its reply.

Input Tokens vs Output Tokens

Every token you send counts as an input token. Every token the model writes back counts as an output token. Both draw from the same context window. Both get billed separately - and output tokens typically cost more than input tokens, because generating each one requires more compute than reading one.

Models also have a separate max output token limit - a ceiling on how long any single response can be, regardless of remaining context space. For example, Claude Opus 4.8 supports a 1M-token context window but caps max output at 128,000 tokens per request. Claude Haiku 4.5's cap is lower at 64,000 tokens.

Context Window Sizes in 2026

Context windows have grown dramatically. A few years ago, a 4,000-token window was standard. Today, every major model ships with at least 128,000 tokens, and the leading models offer 1 million or more.

Here's where the major families sit right now:

Bigger isn't automatically better, though. Anthropic's own documentation notes a phenomenon called context rot: "As token count grows, accuracy and recall degrade." A model can technically hold 1 million tokens, but its ability to reliably recall and reason over everything in that window decreases as the window fills. What's in context matters as much as how much fits.

Why This Changes How You Build

Once you internalize the token model, a bunch of previously mysterious behaviors start making sense.

Why prompts need to be lean

Every word in your system prompt, every document you attach, every prior conversation turn - all of it eats into the same budget. A bloated system prompt that consumes 10,000 tokens leaves 10,000 fewer tokens for the actual task. Ruthless editing of your prompts is not just about clarity; it's budget management.

Why long conversations feel like the model "forgets"

The model doesn't forget. It just runs out of window. Each turn of a conversation accumulates - the system message, every prior user message, every prior assistant reply, then the new user message, then the new reply. Eventually this fills the context window and earlier content either drops off (in chat interfaces using a rolling window) or the whole request fails. This is not a bug; it's the fundamental architecture.

Why the same model costs different amounts for different tasks

Billing is per token, split between input and output rates. Sending a 50,000-token document to summarize is an expensive input operation. Asking the model to generate a 10,000-word report is an expensive output operation. A short question-answer pair might cost almost nothing. Understanding this lets you design workflows that are both effective and cost-efficient - for example, summarizing documents before feeding them into a longer pipeline.

How to count tokens before you send

Every major provider offers a way to count tokens before committing to a request:

# Anthropic - token counting API
import anthropic

client = anthropic.Anthropic()
response = client.messages.count_tokens(
    model="claude-opus-4-8",
    messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.input_tokens)  # e.g. 10

OpenAI provides a tokenizer tool at platform.openai.com/tokenizer and the tiktoken library for local counting. Gemini exposes a count_tokens method in its API. Use these before designing large prompts or workflows that need to stay within limits.

Practical Rules for Builders

Key Takeaways

Try this next: now that you understand how models measure and limit their inputs, learn how to design prompts that get the most out of those tokens - Prompt Engineering Basics.

LearnfundamentalsTokens, Context Windows & Why They Matter
Guidefundamentalsintro7 min read

Tokens, Context Windows & Why They Matter

Tokens are the unit of measure every AI model charges, limits, and thinks in. Learn what they are, how context windows cap what the model can "see," and the practical rules that shape every prompt you write.

CREATORS OF TODAY.
@creatorsoftoday
323

TL;DR: A token is roughly 3-4 characters of English text (though this varies by model and generation). The context window is the hard cap on how many tokens a model can process in one shot - both the stuff you send and the reply it writes. Hit that limit and the model either truncates or stops. Understanding this shapes every decision from prompt length to what tool you pick.

What Is a Token?

Models don't read words. They don't read letters either. They read tokens - small chunks of text that sit somewhere between a character and a word.

A token is the atomic unit every language model uses to process and measure text. Roughly speaking, one token equals about 4 characters of English for many models, or about three-quarters of a word. That means 100 tokens is somewhere in the range of 60-80 plain English words. Note that tokenizer efficiency varies significantly across model families and generations - newer tokenizers can be considerably more token-dense (see below).

Some practical examples to anchor this:

  • cat - 1 token
  • unbelievable - 3 tokens (un / believ / able)
  • tokenization - 3 tokens
  • A 1-2 sentence paragraph - roughly 30 tokens
  • A 500-word page - roughly 650-750 tokens
  • A 300-page novel - roughly 100,000-120,000 tokens

Rare words, technical jargon, and non-English text tend to split into more tokens than common English words. The word "Anthropic" might be one token. A word in Finnish or Arabic might be four or five. This matters when you're building multilingual tools or working with specialized vocabulary.

How Tokenization Works Under the Hood

Most modern models use a method called Byte Pair Encoding (BPE). The short version: it starts with a vocabulary of individual characters, then iteratively merges the pairs that appear most often in real text until it reaches a target vocabulary size (typically 32,000 to 200,000 entries). Common short words become single tokens. Rare or long words get split into subword pieces.

The tokenizer is a completely separate module from the model itself. It runs before the model sees anything - converting your raw text into a sequence of integers, then handing those numbers to the model. You never directly interact with this layer, but its behavior determines how your prompts are interpreted.

Each model family has its own tokenizer and its own vocabulary. That means the exact same sentence can produce a different token count depending on which model you're using. OpenAI's GPT-5.5 uses a different tokenizer than Claude Opus 4.8, which uses a different one than Gemini 3.5 Flash. Counts won't match across providers - and can shift meaningfully even within a provider when a new model generation ships. Anthropic's Claude Opus 4.7 introduced a new tokenizer that produces roughly 30% more tokens for the same text compared to pre-4.7 Claude models.

The Context Window: Working Memory, Not Storage

The context window is the total number of tokens a model can process in a single request - including everything you send in plus everything it writes back.

Think of it as working memory, not long-term storage. The model has no persistent memory between requests unless you explicitly build that. Every new API call starts fresh. The context window is all it has to work with, and it runs out.

Anthropic's docs describe it well: "The context window refers to all the text a language model can reference when generating a response, including the response itself. This is different from the large corpus of data the language model was trained on."

The key point: input and output share the same pool. If your context window is 200,000 tokens and you send a 150,000-token document, the model only has 50,000 tokens left to write its reply.

Input Tokens vs Output Tokens

Every token you send counts as an input token. Every token the model writes back counts as an output token. Both draw from the same context window. Both get billed separately - and output tokens typically cost more than input tokens, because generating each one requires more compute than reading one.

Models also have a separate max output token limit - a ceiling on how long any single response can be, regardless of remaining context space. For example, Claude Opus 4.8 supports a 1M-token context window but caps max output at 128,000 tokens per request. Claude Haiku 4.5's cap is lower at 64,000 tokens.

Context Window Sizes in 2026

Context windows have grown dramatically. A few years ago, a 4,000-token window was standard. Today, every major model ships with at least 128,000 tokens, and the leading models offer 1 million or more.

Here's where the major families sit right now:

  • Claude (Anthropic): Claude Opus 4.8 and Sonnet 4.6 offer 1M-token context windows. Claude Haiku 4.5 has a 200k-token window. Max output per request is 128k tokens for Opus 4.8 and Sonnet 4.6, and 64k tokens for Haiku 4.5.
  • GPT-5.5 (OpenAI): 1,050,000-token context window (just over 1M), with a max output of 128,000 tokens.
  • Gemini (Google): Gemini 3.5 Flash supports a 1M-token context window with up to 65,000 output tokens. Gemini's tokenizer equates roughly 4 characters to one token, consistent with older model generations from other providers.

Bigger isn't automatically better, though. Anthropic's own documentation notes a phenomenon called context rot: "As token count grows, accuracy and recall degrade." A model can technically hold 1 million tokens, but its ability to reliably recall and reason over everything in that window decreases as the window fills. What's in context matters as much as how much fits.

Why This Changes How You Build

Once you internalize the token model, a bunch of previously mysterious behaviors start making sense.

Why prompts need to be lean

Every word in your system prompt, every document you attach, every prior conversation turn - all of it eats into the same budget. A bloated system prompt that consumes 10,000 tokens leaves 10,000 fewer tokens for the actual task. Ruthless editing of your prompts is not just about clarity; it's budget management.

Why long conversations feel like the model "forgets"

The model doesn't forget. It just runs out of window. Each turn of a conversation accumulates - the system message, every prior user message, every prior assistant reply, then the new user message, then the new reply. Eventually this fills the context window and earlier content either drops off (in chat interfaces using a rolling window) or the whole request fails. This is not a bug; it's the fundamental architecture.

Why the same model costs different amounts for different tasks

Billing is per token, split between input and output rates. Sending a 50,000-token document to summarize is an expensive input operation. Asking the model to generate a 10,000-word report is an expensive output operation. A short question-answer pair might cost almost nothing. Understanding this lets you design workflows that are both effective and cost-efficient - for example, summarizing documents before feeding them into a longer pipeline.

How to count tokens before you send

Every major provider offers a way to count tokens before committing to a request:

# Anthropic - token counting API
import anthropic

client = anthropic.Anthropic()
response = client.messages.count_tokens(
    model="claude-opus-4-8",
    messages=[{"role": "user", "content": "Hello, how are you?"}]
)
print(response.input_tokens)  # e.g. 10

OpenAI provides a tokenizer tool at platform.openai.com/tokenizer and the tiktoken library for local counting. Gemini exposes a count_tokens method in its API. Use these before designing large prompts or workflows that need to stay within limits.

Practical Rules for Builders

  • Budget inputs first. Before writing a prompt, estimate its token cost. A 100-page PDF is roughly 25,000-30,000 tokens. A system prompt should stay under 2,000 tokens if possible.
  • Don't assume words = tokens. Technical text, code, and non-English content tokenize at higher ratios. Measure, don't guess.
  • Input and output share the budget. If you need a long output, send a short input. If you're sending a long document, trim your instructions.
  • Context rot is real. Packing a 1M-token window with marginally relevant content often performs worse than a focused 50k-token prompt. Quality over quantity.
  • Different models use different tokenizers. A token count from Claude's API does not equal a token count from the OpenAI API for the same text. Token counts can also shift within the same provider across model generations.
  • Price outputs separately. Output tokens typically cost 4-5x more than input tokens. Generation-heavy workflows cost more than retrieval-heavy ones.

Key Takeaways

  • A token is roughly 3-4 characters or 3/4 of an English word for many models - but tokenizer efficiency varies across providers and model generations; always measure rather than assume.
  • The context window is working memory: input + output share one pool, and it resets every request.
  • Leading models in 2026 offer 1M-token windows, but bigger windows don't cure context rot - quality of content beats quantity.
  • Input and output tokens are priced differently; output typically costs more per token.
  • Every provider ships a token-counting API or tool - use it before building, not after hitting a limit.
  • Lean prompts and focused context windows reliably outperform bloated ones on both accuracy and cost.

Try this next: now that you understand how models measure and limit their inputs, learn how to design prompts that get the most out of those tokens - Prompt Engineering Basics.

References & sources

Reviews

Only verified humans can leave reviews. It keeps every rating real.

Verify to review

No reviews yet. Be the first to share your take.