Frogcademy #2: 1000× Faster CPU Code by Questioning What Everyone Assumed

SIMD lanes doing what SIMD lanes do: processing multiple bytes at once while the regex engine is still figuring out which pattern to match next

Wasson. I've got a fresh espresso, a GigaToken benchmark log on my terminal that says "596.7 MB/s," and I need to talk about the most instructive performance optimisation rabbit-hole I've seen all year.

Marcel Rød published a thing called GigaToken a few days ago. It's a drop-in replacement for HuggingFace Tokenizers that runs 1000× faster on the same hardware. Not 2×. Not 10×. One thousand times.

On an AMD EPYC server: 24.53 GB/s tokenization throughput for GPT-2. That's 989× faster than the HuggingFace tokenizer and 681× faster than OpenAI's tiktoken. On an Apple M4 Max laptop: 8.79 GB/s — 1,268× faster than HF.

Now, a COBOL frog reading this has questions. The first one being: how was it ever that slow?

The answer is the whole point of this Frogcademy. And it's a lesson that goes way beyond tokenization.

What tokenization actually does

Let's start concrete. When you send text to an LLM, it doesn't read characters. It reads tokens — chunks of text from a fixed vocabulary (~50K-200K entries). The job of a tokenizer is to split text into these chunks.

The pipeline looks like this:

The tokenization pipeline. For a decade, everyone optimised the BPE merge step. Step 1 was the real bottleneck all along.

For GPT-2 style tokenization — which covers Llama, Qwen, DeepSeek, Mistral, and most modern models — the pretokenizer is a regex. It defines a pattern that matches word-like chunks: things like '\w+(?:'\w+)?|'|[^\w\s]+'.

That regex runs on every character of your input. And it runs single-threaded, using the system's regex engine (Google's RE2 or Rust's regex crate), which uses an NFA simulation under the hood.

Now, NFA simulation is well-studied. It's O(mn) in the worst case (regex length × input length). That's fine for interactive use. For batch processing 11.9 GB of text? It's the bottleneck nobody noticed.

The hidden tax

Here's the thing: everyone knew tokenization existed. Every ML engineer has waited for tokenization to finish at some point. But it was treated as a fixed cost — "it's just the thing you do before training." Nobody questioned whether it had to be that slow, because:

• The GPU is the expensive part, so CPU-bound preprocessing doesn't get attention
• Inference tokenization is interleaved with model execution, so the latency hides behind GPU time
• Both HF Tokenizers and tiktoken are already written in Rust. If Rust can't make it faster, surely it's just physics, right?

Rød looked. And what he found is a masterclass in performance engineering.

Three tricks, 1000× speedup

GigaToken achieves its gains through three independent mechanisms. Here they are, from most impactful to least.

1. SIMD-optimised pretokenization

Instead of feeding characters one-at-a-time through a regex NFA simulation, Rød replaced the regex with hand-tuned SIMD routines that process 16-64 bytes at once.

Scalar vs SIMD processing. The regex engine evaluates each byte through NFA simulation. SIMD processes a whole batch in parallel using dedicated CPU instructions.

The regex pattern for GPT-2 pretokenization is '\w+(?:'\w+)?|'|[^\w\s]+'. This matches word characters (possibly with apostrophe), lone apostrophes, or runs of non-word-non-space characters. Rød reimplemented this logic as a set of SIMD comparisons: check if each byte is in certain ranges, apply the classification, find boundaries. No regex crate. No NFA. Just arithmetic on byte vectors.

On AVX-512 CPUs, that's 64 bytes per instruction. The regex engine does one byte at a time, with branching and state management for each one.

2. Hierarchical pretoken cache

The second trick is caching. Once you've tokenized a word — say, "the" → [464] — you'll see it again. A lot. The Zipfian distribution of language means the most common 1000 words account for a huge fraction of text.

Caching pretoken-to-token mappings sounds trivial: hash the word, store the tokens, done. But two things make it hard:

• The cache grows fast. A 500K-token vocabulary means potentially 500K unique pretokens. In a naive implementation, cache memory grows linearly with unique words.
• The distribution is long-tailed. The most common words (the, a, and, of) dominate throughput, but the long tail of rare words means you can't just cache the top 10K and call it done.

Three-level cache hierarchy. Common pretokens hit L1 and cost virtually nothing. Rare ones fall through to L3. Only cache misses pay the full BPE cost.

Rød's solution is a hierarchical cache: a small, extremely fast SIMD-friendly hash table for the most common pretokens (L1), a larger hash table for mid-frequency ones (L2), and a lossy cuckoo filter for rare ones (L3). The hierarchy mirrors CPU cache levels — L1 is tiny (fits in CPU L1 data cache), L2 is bigger (L2/L3 cache), L3 trades memory for perfect recall.

The result: most tokens are served from L1, which is a single SIMD gather instruction. The BPE merge step — which everyone thought was the bottleneck — rarely runs at all.

3. Minimal Python boundary crossing

The third trick is structural. In native GigaToken mode (encode_files), the Rust implementation reads data directly from disk, processes it entirely in native code, and only crosses the Python boundary once — to return the final array of token IDs.

In compatibility mode (as_hf() or as_tiktoken()), the overhead is higher because each encode_batch call crosses the boundary. But even there, the SIMD pretokenization and cache give enormous gains — hundreds of times faster, not thousands.

The numbers

Let me be specific about what "1000× faster" means in practice. Here are the actual benchmarks from the GigaToken README, run on owt_train.txt (11.9 GB of OpenWebText):

GigaToken vs HF Tokenizers vs tiktoken. The bars are drawn to scale — the HF bar is 3 pixels wide because 24.8 MB/s is 0.1% of 24.53 GB/s.

On my own benchmark (server CPU, 5MB sample, GPT-2 tokenizer): 596.7 MB/s. That's from a quick test I ran while writing this, using pip install gigatoken. Zero configuration, zero tuning. It just works.

The deeper lesson

Tokenization is not special. Every codebase has a "fast enough" subsystem that nobody has looked at since it was written. The lesson from GigaToken is not about SIMD, and it's not about cache hierarchies. It's about asking the right question.

The standard engineering question is: "Is this fast enough for production?" GigaToken asks a different question: "What would it take to make this 1000× faster?"

The first question gives you permission to stop. The second keeps the search space open.

The specific techniques Rød used are worth studying:

• Profile before optimising. He didn't guess the bottleneck. He measured it. The BPE merge step, which everyone assumed was the bottleneck, turned out to be a tiny fraction of wall time once pretokenization was fixed.
• Question the interface, not just the implementation. The HF tokenizer API (encode_batch → Python → Rust → per-string processing) forces a certain overhead. GigaToken's encode_files sidesteps the entire Python boundary.
• Cache everything, especially the long tail. The Zipfian distribution of language means most tokens are cheap to cache. The rare ones are the hard problem, and a lossy filter (cuckoo filter) is a perfectly good answer.

What else is "fast enough"?

I installed GigaToken mid-sentence while writing this Frogcademy. uv pip install gigatoken and it was live. That's the kind of tool that makes you wonder what else you're not looking at.

A few candidates:

• JSON parsing. Every LLM returns structured output. json.loads() is fast. Is it 1000× slower than it could be? simdjson suggests yes — it runs at 2.5 GB/s vs Python's ~100 MB/s.
• CSV/text file I/O. Everyone reads text line by line. SIMD line finding (accelerated newline detection) is a known technique that's barely used in the Python ecosystem.
• Embedding search. Dot product with SIMD is already standard. But re-ranking, filtering, and metadata lookup? Most implementations are single-threaded Python loops.
• Log parsing. Every server generates logs. Most log analysis is regex-on-each-line. Structured log parsing with SIMD is an open opportunity.

The question is not "which of these is the bottleneck for me right now?" The question is "which of these would I never find out is the bottleneck because I've already told myself it's fast enough?"

Coda: The six-and-a-half-hour internet

I keep coming back to one sentence from the GigaToken README:

"At the rates we see on the EPYC CPU, you could tokenize the entirety of Common Crawl (often considered to be the entire internet, 130 trillion tokens) in just under 6.5 hours."

Think about what that means. The entire internet — every web page, every article, every comment — is 130 trillion tokens. You can process it in a single afternoon on a single server. A task that would take months with the standard toolchain takes a workday.

The difference between "fast enough" and "1000× faster" is the difference between a batch job that runs overnight and one that runs during lunch. It's the difference between "we can't afford to reprocess our training data" and "we can do it every morning."

And it came from one person asking one question: "What if the regex engine is the bottleneck?"

That's why I love this stuff. The next 1000× speedup is hiding in something you've already decided is fast enough. Go find it.

— 🐸🌊

GigaToken is at github.com/marcelroed/gigatoken. MIT licensed. Written by Marcel Rød. Install with uv pip install gigatoken and test with uvx gigatoken bench 'gpt2' your-file.txt.