What the tokenizer visualizer shows
Paste any text and this visualizer breaks it into the exact tokens GPT-4, Claude, and Gemini would see — colour-coded, with the character-per-token ratio and a cost estimate per model. It makes visible the thing LLMs actually read: not words or characters, but tokens, which is what every API bills for and what every context window is measured in.
How tokenization works
Models split text into subword tokens using a learned vocabulary. Common words are a single token, rarer words break into pieces ("tokenization" → "token" + "ization"), and whitespace and punctuation count too. Because each provider trained its own tokenizer, the same sentence can be a different number of tokens on GPT, Claude, and Gemini — which is exactly what the side-by-side view reveals.
Why token counts matter
Two reasons. Cost: you pay per token, so a wordy prompt is a recurring tax. Context limits: every model has a maximum window; exceed it and the request fails or silently truncates. Seeing how your text tokenizes helps you trim prompts and stay in budget. For a quick number use the token counter; to cut a prompt's tokens, the prompt compressor.
Frequently asked questions
How many tokens is a word?
In English, roughly 1 token per ¾ word — about 1,000 tokens per 750 words. Code, punctuation, and non-English text tokenize into more pieces.
Why do GPT, Claude, and Gemini count differently?
Each trained its own tokenizer with a different vocabulary, so the same text splits into a slightly different token count on each.
Is my text sent anywhere?
No — the visualization runs in your browser.