Words to tokens conversion chart (100 to 200,000 tokens)
Quick reference for prompt engineers, developers, and writers working with GPT-4o, Claude, Gemini, and Llama.
Modern large language models tokenize text using Byte-Pair Encoding (BPE). In standard English prose, 1 token equals roughly 4 characters or 0.75 words (meaning 100 tokens ≈ 75 words, and 100 words ≈ 133 tokens). Punctuation, indentation, JSON keys, and non-Latin characters consume additional tokens per word.
| Token Count |
English Word Count (~0.75 w/tok) |
Character Count (~4 chars/tok) |
Pages (Single-Spaced) |
Typical Document / Context Size |
| 100 tokens | ~75 words | ~400 characters | 0.15 pages | Short paragraph, email hook, or system instruction |
| 500 tokens | ~375 words | ~2,000 characters | 0.75 pages | Standard cold email or short blog intro |
| 1,000 tokens | ~750 words | ~4,000 characters | 1.5 pages | Short article or detailed few-shot prompt |
| 2,000 tokens | ~1,500 words | ~8,000 characters | 3 pages | Standard SEO blog post or 10-minute YouTube script |
| 4,096 tokens (4K) | ~3,070 words | ~16,380 characters | 6 pages | Long-form guide or academic essay |
| 8,192 tokens (8K) | ~6,140 words | ~32,760 characters | 12 pages | White paper or technical specification |
| 32,768 tokens (32K) | ~24,500 words | ~131,000 characters | 49 pages | Novella or multi-chapter report |
| 128,000 tokens (128K) | ~96,000 words | ~512,000 characters | 192 pages | Full-length non-fiction book (GPT-4o / DeepSeek window) |
| 200,000 tokens (200K) | ~150,000 words | ~800,000 characters | 300 pages | Large codebase or 500-page book (Claude context window) |
How Byte-Pair Encoding (BPE) tokenization works across languages and code
Unlike traditional word processors that split text strictly on whitespace, modern transformer models (including OpenAI’s cl100k_base and o200k_base vocabularies, Anthropic’s Claude tokenizer, and Google’s SentencePiece models) break text into subword units called tokens. High-frequency English words with a leading space (such as " the" or " function") are encoded as a single integer ID, whereas compound technical terms or inflected words (such as "tokenization" → ["token", "ization"]) split into multiple subwords.
Because tokenizer vocabularies are trained predominantly on English prose and common source code, token-to-word and token-to-character ratios vary substantially by content type:
| Content Type / Script |
Avg. Tokens per Word |
Avg. Characters per Token |
Why Token Density Changes |
| Standard English Prose | ~1.30 – 1.35 tokens/word | ~3.8 – 4.2 chars/token | Most common root words and suffixes map to single BPE vocabulary entries |
| Spanish / German / Portuguese | ~1.55 – 1.85 tokens/word | ~3.0 – 3.5 chars/token | Diacritics (á, ñ, ü, ç) and German compound nouns split into extra subword pieces |
| Python / JavaScript / TypeScript | ~1.80 – 2.40 tokens/word | ~2.5 – 3.2 chars/token | Brackets, operators, camelCase identifiers, and indentation spaces add token overhead |
| Minified vs. Pretty JSON | ~2.20 – 3.10 tokens/item | ~2.0 – 2.8 chars/token | Repeated quotes, colons, braces, and 2-space indentation inflate token count by 15%–30% |
| Chinese / Japanese / Korean (CJK) | 1.0 – 1.8 tokens/character | ~0.6 – 1.1 chars/token | Multi-byte UTF-8 ideographs often require 1 to 2 BPE tokens per single character |
Context window budgeting: Why 80% utilization beats 99%
Even when working with a 128K, 200K, or 1M token context window, filling the window to capacity introduces three practical engineering bottlenecks: output reservation (total context equals input_tokens + max_completion_tokens, so leaving too little headroom truncates responses mid-sentence), “Lost in the Middle” attention dilution (models recall instructions at the very beginning and very end of a prompt with higher accuracy than facts buried in the middle 50%), and Time-to-First-Token (TTFT) latency. Stripping redundant Markdown, whitespace, and boilerplate with the AI Text Cleaner before prompting routinely cuts input token usage by 10%–25%.
Tokenizer differences & estimation boundaries
Different model families use distinct subword vocabularies: OpenAI GPT-4o uses o200k_base (200,000 vocabulary entries), GPT-4/3.5 uses cl100k_base (100,277 entries), Anthropic Claude uses its proprietary BPE vocabulary, and Google Gemini uses SentencePiece. Because downloading multi-megabyte vocabulary tables would slow down browser startup, TextScope uses a deterministic client-side BPE heuristic calibrated across English prose, European diacritics, source code punctuation, and CJK UTF-8 sequences. Estimates typically fall within ±3% to 6% of exact API billing counts; for hard billing cutoffs, leave a 5% safety buffer below your target context limit.