Dev tools · alternatives · 2026

24 alternatives to FlashTokenizer – 10x faster C++ tokenizer for Python

I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency. Benchmarks and comparison included in README. Would love feedback or contributions!

Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
GitHub▲7
BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie
2025 · github.com · its alternatives →
2
TokenDagger – A tokenizer faster than OpenAI's Tiktoken▲281
TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.…
2025 · github.com · its alternatives →
3
HuggingFace – Fast tokenization library for deep-learning NLP pipelines▲168
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production - huggingface/tokenizers
2020 · github.com · its alternatives →
4
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning▲385
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com · its alternatives →
5
Rust BPE tokenizer for Qwen models that's 12x faster than HuggingFace▲5
A blazing-fast BPE tokenizer for Qwen models built with Rust - sweepai/bpe-qwen
2025 · github.com · its alternatives →
6
Fast C-based HTML 5 parsing for Python▲169
Fast C based HTML 5 parsing for python. Contribute to kovidgoyal/html5-parser development by creating an account on GitHub.
2017 · github.com · its alternatives →
7
I built a tensor library from scratch in C++/CUDA▲119
Hi HN, Over the past few months, I've been building `dsc`, a tensor library from scratch in C++/CUDA. My main focus has been on getting the basics right, prioritizing a clean API, simplicity, and clear observability for running small LLMs locally. The key features are: - C++ core with CUDA support written from scratch. - A familiar, PyTorch-like Python API. - Runs real models: it's complete enough to load a model like Qwen from HuggingFace and run inference on both CUDA and CPU with a single line change[1]. - Simple, built-in observability for both Python and C++. Next on the roadmap is…
2025 · github.com · its alternatives →
8
LLaMA tokenizer that runs in browser▲115
JS tokenizer for LLaMA 1 and 2. Contribute to belladoreai/llama-tokenizer-js development by creating an account on GitHub.
2023 · github.com · its alternatives →
9
Matcheroni, a tiny C++20 header library for building lexers/parsers▲96
Howdy HN, as part of my ongoing programming language experiments I've ended up creating my own C++20 lexing and parsing library of sorts. Matcheroni is an alternative to parser generators and regular expressions that uses trees of C++ templates to implement highly customizable lexers and parsers that have minimal impact on build times or binary sizes, while still remaining comparable in performance to Boost regular expressions. The repo includes two example projects - a simple regex parser in ~300 heavily documented lines of code, and a much larger but not quite finished C99 lexer and…
2023 · github.com · its alternatives →
10
Optimizing LiteLLM with Rust – When Expectations Meet Reality▲27
I've been working on Fast LiteLLM - a Rust acceleration layer for the popular LiteLLM library - and I had some interesting learnings that might resonate with other developers trying to squeeze performance out of existing systems. My assumption was that LiteLLM, being a Python library, would have plenty of low-hanging fruit for optimization. I set out to create a Rust layer using PyO3 to accelerate the performance-critical parts: token counting, routing, rate limiting, and connection pooling. The Approach - Built Rust implementations for token counting using tiktoken-rs - Added lock-free data…
Nov 2025 · github.com · its alternatives →
11
OpenBrowser-AI▲76
Your AI agent is burning 6x more tokens than it needs to just to browse the web. We built OpenBrowser MCP to fix that. Most browser MCPs give the LLM dozens of tools: click, scroll, type, extract, navigate. Each call dumps the entire page accessibility tree into the context window. One Wikipedia page? 124K+ tokens. Every. Single. Call. OpenBrowser works differently. It exposes one tool. Your agent writes Python code, and OpenBrowser executes it in a persistent runtime with full browser access. The agent controls what comes back. No bloated page dumps. No wasted tokens. Just the data your…
Apr 2026 · openbrowser.me · its alternatives →
12
Achilles – Automatic profiling and C++ optimization of Python via LLMs▲7
Hey HN! We built Achilles, a tool that automatically accelerates your Python code. It identifies performance bottlenecks, rewrites those functions in optimized C++, and seamlessly patches them into your running program—without you changing a single line of code. In CPU-intensive, loop-heavy tasks, we've observed performance improvements of 100-1000x. Achilles can be installed via pip and works with just a single command. We'd appreciate your feedback, and feel free to give us a star if you find it interesting!
2025 · github.com · its alternatives →
13
Yantra – an LALR(1) parser generator for C++▲33
A C++ parser generator: lexer, parser, and AST generation in one tool. - TantrixAuto/yantra
21d ago · github.com · its alternatives →
14
Fast integer compression library in C▲45
A very fast scalar implementation for Frame Of Reference integer compression. - cruppstahl/libfor
2015 · github.com · its alternatives →
15KC
KVBoost – chunk-level KV cache reuse for HuggingFace, 5–48x faster TTFT▲20
If you’ve ever been curious about how GPT actually works under the hood, I built a small project you might find interesting. I implemented a GPT-style transformer from scratch in a single notebook—covering tokenization, embeddings, causal self-attention, training, and autoregressive text generation without relying on high-level abstractions. The focus was on mechanistic clarity rather than scale or performance, and the notebook is structured to read more like a technical walkthrough than an experiment log. Feedback from people who’ve built or studied transformers would be very welcome.
May 2026 · pythongiant.github.io · its alternatives →
16
Faking SIMD to Search and Sort Strings 5x Faster▲29
I want to share a really dumb, but very practical project I have packaged this summer, to perform operations on strings much faster. I was using Python to work with a multi-terabyte newline-delimited file. Reading, splitting, and shuffling it was a nightmare. So, I wrapped a trivial hardware-friendly heuristic I've been using for the last few years into a CPython library. The part I enjoyed the most is implementing SIMD behavior without SIMD instructions... Using 64-bit words to work at 8-bit granularity. Unlike conventional SIMD, the code would remain the same for ~~almost~~ any hardware.…
2023 · ashvardanian.com · its alternatives →
17
I shrank DeepSeek V4 Flash to 57GB and it wrote a compiler on my Mac▲21
I built a specialized package of DeepSeek V4 Flash 0731 (originally 284B total parameters, 13B active), preserving reasoning, tool calling and coding capabilities: https://huggingface.co/steadfastgaze/DeepSeek-V4-Flash-0731-... I let it write a minimal C compiler targeting ARM64, then test the result with Fibonacci and FizzBuzz programs, and it succeeded in less than 1 hour, with the full recording at: https://youtu.be/XiwSilmV8B0 You can run it on Silicon Macs with my engine https://github.com/steadfastgaze/MoEspresso, while one of the…
Aug 2026 · huggingface.co · its alternatives →
18
Mirror_bridge – C++ Reflection powered Python binding generation▲29
A reflection-based approach for generating Python/Lua/JS bindings for your C++ code - FranciscoThiesen/mirror_bridge
Dec 2025 · github.com · its alternatives →
19
Understand and reduce token usage with ContextSpy context profiler▲6
Most devs using AI coding tools don't even imagine how much information they put in the context window - tools, MCPs, tool responses, active skills - all of which take up some real estate. Sometimes updating a tool description or trimming api response can give significant savings - and profiler helps to measure that
Jun 2026 · github.com · its alternatives →
20
StringZilla v3 with C++, Rust, and Swift bindings, and AVX-512 and NEON▲15
Hi HN! I've spent the last few weeks optimizing the string operations in StringZilla. You'd be surprised how much performance GLibc and the Standard Templates Library of C++ are leaving on the table, especially on Arm and, likely, on RISC-V. I've significantly increased the number of implemented algorithms for different workloads. Searching across English texts on Arm, for example: - is 4.4x faster than STL, - and 16.8x faster in reverse order. The new version of the library now includes a C++ binding largely compatible with the C++ Standard, featuring all 17 overloads of the `replace`…
2024 · github.com · its alternatives →
21
Tokensift, an open-sourced token-efficiency linter for LLM prompts▲6
Token-efficiency linter for LLM prompts and payloads - ritenv/tokensift
Aug 2026 · github.com · its alternatives →
22IM
I made calling WebAssembly from Python 25x times Faster▲5
I found a serious bottleneck in Python binding of Wasmer and Wasmtime and I found a trick to make 25x faster. Then beat all that using NumPy.
2023 · medium.com · its alternatives →
23
Tokenflood – simulate arbitrary loads on instruction-tuned LLMs▲21
Hi everyone, I just released an open source load testing tool for LLMs: https://github.com/twerkmeister/tokenflood === What is it and what problems does it solve? === Tokenflood is a load testing tool for instruction-tuned LLMs hat can simulate arbitrary LLM loads in terms of prompt, prefix, and output lengths and requests per second. Instead of first collecting prompt data for different load types, you can configure the desired parameters for your load test and you are good to go. It also let's you assess the latency effects of potential prompt parameter changes before…
Nov 2025 · github.com · its alternatives →
24
KTransformers:671B DeepSeek-R1 on a Single Machine-286 tokens/s Prefill▲14
Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 671B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
2025 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →