TokenDagger – A tokenizer faster than OpenAI's Tiktoken
TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.…
TokenDagger – A tokenizer faster than OpenAI's Tiktoken launched on June 30, 2025 with 281 votes, #104 of 7,999 launches that month and more than 99% of that year's launches. The site was still online in September 2026, 1 year on.
In plain words
TokenDagger is a tokenizer written in C++ 17 that serves as a drop-in replacement for OpenAI's Tiktoken, supporting models like Llama 3, Mistral, and GPT-3. It maintains the same vocabulary and special token rules while prioritizing speed through a faster regex engine and simplified token-matching algorithms. The tool is designed for developers and researchers working with large language models who need faster tokenization performance, achieving 4x faster single-thread tokenization and 2-3x higher throughput on large text files compared to the original implementation.
written from the facts on this page · September 2026
From the sources
High-Performance Implementation of OpenAI's TikToken. - M4THYOU/TokenDagger
A fast, drop-in implementation of OpenAI's TikToken , designed for large-scale text processing. 2x Throughput and 4x faster on code sample tokenization. Performed on an AMD EPYC 4584PX - 16c/32t - 4.2 GHz w/64GB memory. Hugging Face's batch tokenizer used way more memory than Tiktoken and TokenDagger. 256MB was the largest input size it could process with OOM. The benchmark was performed on meta-llama/Llama-4-Scout-17B-16E-Instruct . mistralai/Ministral-8B-Instruct-2410 is also supported for testing, by using argument --tokenizer mistral . make clean && make pip3 install tiktoken python3 tests/test_tokendagger_vs_tiktoken.py --tokenizer llama python3 tests/test_tokendagger_vs_tiktoken.py…from github.com
In the maker’s words, at launch
TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from…
Does a similar job
all alternatives →
LLaMA tokenizer that runs in browser2023 · github.com · ▲115JS tokenizer for LLaMA 1 and 2. Contribute to belladoreai/llama-tokenizer-js development by creating an account on GitHub.
- ATA tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)2025 · llmapitest.com · ▲55
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
Code ModeNov 2025 · github.com · ▲144I’ve been testing a simple idea inspired by Apple/Cloudflare/Anthropic: LLMs are far better at writing a small program than coordinating multiple tool calls. So instead of giving the model 10+ tools, I exposed one: a TypeScript sandbox with access to the same interfaces. The model writes a script → it runs once → done. What changed - +68% reduction in token use - No multi-step drift or retries - Local models (Llama 3.1 8B / Phi-3) became much more reliable Repo: https://github.com/universal-tool-calling-protocol/code-mode Curious whether others have seen…
Reducing LLM input tokens by 70%May 2026 · adola.app · ▲56Run GPT-OSS, Qwen, Llama, and DeepSeek through one OpenAI-compatible API with streaming and clear usage-based pricing.
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 28d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, June 2025
the whole month →- IM
Hey HN! Pilots everywhere are required to keep a logbook of all their flying hours, aircraft, airports, and so on. Since I track everything digitally (some people still just use paper logbooks!), I put together some data visualizations and a few 3D globes to show my flying history. This globe is probably my favourite so far: https://jameshard.ing/pilot/globes/all If you’ve got ideas for other graphs or ways to show this kind of data, I’d love to hear them!
Life & fun · 2025 · jameshard.ing
Stunning presentations with AI. No design skills required.
AI · 2025 · chroniclehq.com
Build native mobile apps for iOS and Android without code.
Dev tools · 2025 · bble.io
All your connections, across all your socials.
AI · 2025 · pally.com
I was the main contributor to workout.lol, an open-source fitness app to easily build a workout routine. The project had traction (1.4k GitHub stars, 95 forks, ~20K visits/month), but was eventually sold due to video licensing hurdles. The new owner stopped maintaining it, and the repo went abandoned. Over the next 9 months, I sent 15 emails to try to save it : no replies. Feature requests & issues were ignored. The community was left with a "broken" tool let's say. I couldn't just let it die So I built the new version from scratch with the same open-source spirit, but a better…
Dev tools · 2025 · github.com