Tokens & Context Windows

Large Language Models do not read characters, letters, or complete words. Neural networks perform linear algebra on floating-point vectors. To process text, an LLM relies on a tokenizer—a specialized pre-processing algorithm that slices human string inputs into atomic discrete fragments called tokens, mapping each token to a unique integer Token ID.

Debunking the “1 Token = 1 Word” Myth

A common mistake made by junior developers is assuming one token equals one word, or that 1,000 words equals 1,000 tokens. A single token can represent:

Byte-Pair Encoding (BPE) Mechanics

Byte-Pair Encoding, originally developed for data compression (Gage, 1994) and adapted for NLP by Sennrich et al. (2016), operates iteratively:

1. Start with a base vocabulary of individual characters or raw bytes (0–255).
2. Count the most frequent adjacent pair of tokens across a massive text training corpus (e.g., 't' + 'h' → 'th').
3. Merge that frequent pair into a new single vocabulary entry.
4. Repeat this merge process for tens of thousands of iterations until the target vocabulary size (e.g., 128,000) is reached.

Every token corresponds to a non-negative integer ID in the range [0, Vocabulary Size - 1]. Inside the neural network, this ID is passed into the Input Embedding Layer (a 2D weight matrix of shape [Vocab Size, Hidden Dimension]).

When you send an array of message objects ({ role: "user", content: "..." }) to an API or local Hugging Face pipeline, the model does NOT see raw JSON. The tokenizer runs a Jinja chat template that weaves reserved special control tokens between the dialogue turns.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a senior AI systems engineer. Answer concisely with code.<|eot_id|><|start_header_id|>user<|end_header_id|>

How does left-padding work in batch generation?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

In production LLM infrastructure, token counts determine three critical operational realities:

The context window is the maximum total sequence length (in tokens) that a model can attend to during a single forward pass. It is NOT an output-only limit. It represents the combined capacity of all prompt components plus the generation allowance:

When a request breaches context limits, different systems respond differently depending on the infrastructure configuration:

Production Context Mitigation Strategies

• Rolling Window Eviction: Keep the system prompt intact at the top, retain the latest 3–5 conversation turns at the bottom, and discard intermediate turns.
• Recursive Summarization: When history approaches 4,000 tokens, trigger a background LLM task to compress turns 1–8 into a concise 200-token summary note.
• Semantic RAG Filtering: Rather than dumping 10 raw documents, filter chunks by vector similarity threshold and only inject the top 3 most relevant passages.

In real applications, you often batch multiple prompts together to maximize GPU tensor core utilization. Because prompts have varying lengths, sequences must be made uniform using padding tokens (e.g. <PAD>) and truncation.

In professional production systems, you never let the prompt size expand arbitrarily. You design strict mathematical token budgets with hard caps per component.

class TokenBudgetManager:
    def __init__(self, model_context_limit: int = 8192, reserved_output: int = 1500):
        self.limit = model_context_limit
        self.reserved_output = reserved_output
        self.max_prompt_budget = model_context_limit - reserved_output
        
    def allocate(self, system_tokens: int, query_tokens: int, history_tokens: int, doc_tokens: int):
        fixed_cost = system_tokens + query_tokens
        remaining_for_data = self.max_prompt_budget - fixed_cost
        
        if remaining_for_data < 0:
            raise ValueError("Fixed instructions and query exceed maximum allowed prompt headroom!")
            
        # Allocate 40% of remaining headroom to history, 60% to RAG docs
        history_allowance = int(remaining_for_data * 0.40)
        doc_allowance = remaining_for_data - history_allowance
        
        return {
            "allowed_history_tokens": min(history_tokens, history_allowance),
            "allowed_doc_tokens": min(doc_tokens, doc_allowance),
            "safe_to_dispatch": True
        }

Modern models advertise context windows of 128,000 to 2,000,000 tokens. However, the engineering motto is clear: “More context is not automatically better context.”