GitHub
BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie
GitHub launched on April 4, 2025 with 7 votes. The site was still online in September 2026, 1 year on.
What it does
🚀 **FlashTokenizer: World's Fastest CPU Tokenizer!** ⚡ 8~15x faster than `BertTokenizerFast` 🛠️ High-performance C++ 🔄 Parallel with OpenMP 📦 Easy pip install 💻 Cross-platform (Win/Mac/Linux) ▶️ Demo: https://youtu.be/a_sTiAXeSE0
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING - NLPOptimize/flash-tokenizer
Does a similar job
all alternatives →
TokenDagger – A tokenizer faster than OpenAI's Tiktoken2025 · github.com · ▲281TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.…
FlashTokenizer – 10x faster C++ tokenizer for Python2025 · github.com · ▲5I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency. Benchmarks and comparison included in README. Would love feedback or contributions!
HuggingFace – Fast tokenization library for deep-learning NLP pipelines2020 · github.com · ▲168💥 Fast State-of-the-Art Tokenizers optimized for Research and Production - huggingface/tokenizers
- MPMaple-Preview – Ternary 20B MoE running at 120 tok/s on a iPhoneAug 2026 · deepgrove.ai · ▲173
WinkNLP delivers 600k tokens/second speed on browsers (MBP M1)2022 · github.com · ▲79Developer friendly Natural Language Processing ✨. Contribute to winkjs/wink-nlp development by creating an account on GitHub.
Rust BPE tokenizer for Qwen models that's 12x faster than HuggingFace2025 · github.com · ▲5A blazing-fast BPE tokenizer for Qwen models built with Rust - sweepai/bpe-qwen
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 29d ago · world.pixverse.video
Source: Product Hunt launch ↗
Launched alongside, April 2025
the whole month →
Hi everyone, I built PyXL — a hardware processor that executes a custom assembly generated from Python programs, without using a traditional interpreter or virtual machine. It compiles Python -> CPython Bytecode -> Instruction set designed for direct hardware execution. I’m sharing an early benchmark: a GPIO test where PyXL achieves a 480ns round-trip toggle — compared to 14-25 micro seconds on a MicroPython Pyboard - even though PyXL runs at a lower clock (100MHz vs. 168MHz). The design is stack-based, fully pipelined, and preserves Python's dynamic typing without static type restrictions.…
Dev tools · 2025 · runpyxl.com
The Unsure Calculator is an online tool that lets you calculate with numbers you’re not sure about.
Life & fun · 2025 · filiph.github.io
https://the-pocket.github.io/Tutorial-Codebase-Knowledge/
AI · 2025 · github.com
