Gigatoken Tokenizes Text at Gigabytes per Second. The Real Question Is Whether You Tokenize Gigabytes.

A new Rust tokenizer claims to run around 1,000 times faster than Hugging Face's. The claim mostly holds. Whether the speed changes your life depends entirely on a question the benchmark charts won't answer for you.

Both of the tokenizers you already use, Hugging Face's tokenizers and OpenAI's tiktoken, are written in Rust and run across all your CPU cores. They are not slow Python scripts anyone forgot to optimize. They are the fast option. That is what makes Gigatoken interesting: it claims to be roughly a thousand times faster than one of them, and its own benchmark numbers back most of that up.

A thousand times faster than the thing that was already fast. That is the kind of number that either means someone found a genuinely overlooked bottleneck, or someone is measuring in their own favor. In this case it is closer to the first than I expected, which is why the more useful question turns out not to be "is it fast" but "do you tokenize enough text for the speed to matter." For most people the honest answer is no, and I will get to why that is still worth understanding.

What the number actually is

Gigatoken tokenizes text at gigabytes per second on a single machine. On an Apple M4 Max, its author clocks GPT-2 tokenization at 8.79 GB/s, against 6.9 MB/s for Hugging Face's tokenizers on the same file. That is the 1,268x headline. On a 144-core AMD EPYC server it hits 24.53 GB/s, or 989x over Hugging Face. At that rate, the README notes, you could tokenize the entire Common Crawl, roughly 130 trillion tokens, in under 6.5 hours.

The comparison you should actually care about depends on which tokenizer you use. If you are in the OpenAI world, your baseline is tiktoken, not Hugging Face, and the gap there is smaller: 68x to 140x on common desktop CPUs, up to 681x on the big server. Still large. Not a thousand.

And the speedup varies a lot by tokenizer family. Modern byte-pair encoding tokenizers (GPT, Llama 3, Qwen 3, DeepSeek, GLM) all land in the 400x to 1,000x range over Hugging Face. The SentencePiece-based ones (Gemma, Mistral, older Llama 2 derivatives) only get 7x to 22x, because the author has not optimized that path as hard. So "1,000x" is true for the tokenizers most current models use, and plainly false for a meaningful minority. Check your specific tokenizer before you budget around the headline.

Why it is that fast

The speed does not come from a clever trick you can summarize in a sentence, which is part of why I trust it. It comes from grinding on the two parts of tokenization everyone else treats as good enough.

The first is pretokenization, the step that splits raw text into word-ish chunks before the merge rules run. Almost every tokenizer hands this to a regular-expression engine, because regex is convenient and nobody thought it was the hot loop. Gigatoken rewrites that step by hand using SIMD instructions, the CPU feature that processes many bytes in one operation instead of one at a time. The regex engine was the bottleneck. It just never announced itself as one.

The second is caching. If a word has been tokenized before, its token sequence gets looked up instead of recomputed. Text is repetitive, so a good cache pays off constantly. On top of that, the implementation minimizes the handoffs between Rust and Python and keeps its threads from stepping on each other. None of these is magic. Together they are the difference between megabytes and gigabytes per second.

The lesson underneath the tool is the one worth keeping. The boring, unglamorous front of the pipeline had a thousand-fold speedup sitting in it for years, because everyone assumed the fast-enough option was the fast option. That assumption is probably hiding in other places you have stopped looking.

Who this is actually for

Here is where I have to talk you out of installing it, or into it, depending on who you are.

Tokenization speed matters when you tokenize enormous amounts of text. That means training data preparation, building a research corpus, re-tokenizing a dataset because you switched vocabularies, or any batch job measured in gigabytes or terabytes. If you are prepping pretraining data and tokenization is a real line item in your wall-clock time, Gigatoken can turn an overnight job into a coffee break. That is a genuine change to how you work.

It does almost nothing for inference. When your app tokenizes a user's prompt before sending it to a model, you are tokenizing a few kilobytes, and the model's forward pass dominates the time by orders of magnitude. Making that tokenization a thousand times faster is like tuning the engine on a car that spends its life parked. The speedup is real and completely irrelevant to your latency.

So the test is simple. Do you regularly tokenize gigabytes of text in a job whose runtime you actually notice? If yes, this is one of the more useful tools to cross the trending board this month. If no, it is a beautiful piece of engineering that will not change a single number you care about, and knowing why is the whole value you get from it.

Put this into practice

The lowest-friction thing here is not installing it. It is checking whether your tokenizer is even supported and how fast it would run, without touching your environment.

Time it before you commit. If you have uv installed, one command downloads a sample, validates that Gigatoken's output matches your existing tokenizer exactly, and prints the speed for your specific model:
```
uvx --with tokenizers gigatoken bench 'openai-community/gpt2' owt_train.txt \
    --validate --doc-separator "<|endoftext|>"
```
Swap in your own Hugging Face model name and your own text file. The --validate flag is the important part: it confirms the tokens come out identical to what you get today, so you are not trading correctness for speed. On macOS, run it twice, because the first run triggers a security scan that slows the Rust code and gives you a bad reading.
Install it if the number justified it. pip install gigatoken. That is the whole install. It is a normal Python package with a compiled Rust core.
Start in compatibility mode. You can wrap your existing tokenizer with almost no code change:
```
import gigatoken as gt
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()
tokens = tokenizer.encode_batch(["a string", "another string"])
```
This mode is built to match Hugging Face's output exactly, which costs some speed. You still get a large speedup, just not the full headline figure. It is the safe first step because your downstream code sees the same tokens it always did.
Move to the native API only for the big jobs. When you are tokenizing whole files for data prep, the native encode_files path lets the Rust core read data directly and skip the Python overhead, which is where the top-end numbers come from. This is the mode worth the extra integration effort, and only for batch work at scale.

The realistic first win: point the bench command at your real tokenizer and your real data, read the validated speedup, and decide from an actual number instead of the README's best case. That takes ten minutes and tells you everything.

Honest limitations

Start with the maturity signal. As of today the repository shows 27 stars, 3 forks, and no published releases, against a Trendshift trending position that reflects momentum, not adoption. Trending boards measure a spike of attention. Twenty-seven people have starred this. That gap is the story on half the repos that trend, and it means you are an early tester, not a member of a proven user base. Treat it accordingly.

It is a single-author project. The benchmarks are the author's own, run on the author's chosen hardware and datasets, with a --validate flag that at least makes correctness checkable, which is more than most speed claims offer. Still, nobody independent has reproduced these numbers at the time of writing. Run the validation yourself rather than trusting the chart.

The author is refreshingly direct about what is unfinished, and it is a real list. WordPiece tokenization is not supported. SentencePiece is only lightly optimized, which is why those models see 7x to 22x instead of the headline. File output sinks are not implemented yet. Windows is barely tested, so you are steered to WSL. And the compatibility-mode speed is lower than the native-API speed, so the easiest way to adopt it is also not the fastest way to run it.

There is an AI-use disclosure in the README worth reading rather than glossing. The author says the majority of the code was written by hand, with AI used for the user-facing API, for widening tokenizer compatibility, for porting SIMD strategies between instruction sets, and for the final roughly 4x of performance tuning. That is an unusually honest accounting, and how you weigh it is up to you. It does not change whether the tokens validate.

Where this leaves you

The most useful thing about Gigatoken might not be Gigatoken. It is the reminder that a piece of infrastructure everyone had filed under "already solved" was carrying a thousand-fold speedup in plain sight, because the fast-enough version had drifted into being the assumed ceiling. Somebody sat down, questioned the ceiling, and rewrote the part everyone outsourced to a regex engine.

So the question I would take from this is not "should I install a faster tokenizer." It is "what else in my pipeline have I stopped measuring because I decided years ago it was good enough." Tokenization was that for almost everyone until last week. If you prep large datasets, go run the benchmark on your own data and see whether the number is real for you. If you do not, the more valuable exercise is to go find your own regex-engine assumption, the step you have stopped timing because you are sure it is not the problem.

Sources: Gigatoken on GitHub, including README, benchmark tables, and AI-use disclosure; Trendshift repository page. Benchmark figures are the author's own and validated via the repo's --validate flag; reproduce them on your own hardware before budgeting around them.