FlashTokenizer – 10x faster C++ tokenizer for Python
I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency. Benchmarks and comparison included in README. Would love feedback or contributions!
FlashTokenizer – 10x faster C++ tokenizer for Python launched on April 3, 2025 with 5 votes and more than 75% of that year's launches. The site was still online in September 2026, 1 year on.
What it does
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING - NLPOptimize/flash-tokenizer
Does a similar job
all alternatives →
TokenDagger – A tokenizer faster than OpenAI's Tiktoken2025 · github.com · ▲281TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.…
HuggingFace – Fast tokenization library for deep-learning NLP pipelines2020 · github.com · ▲168💥 Fast State-of-the-Art Tokenizers optimized for Research and Production - huggingface/tokenizers
Rust BPE tokenizer for Qwen models that's 12x faster than HuggingFace2025 · github.com · ▲5A blazing-fast BPE tokenizer for Qwen models built with Rust - sweepai/bpe-qwen
Fast C-based HTML 5 parsing for Python2017 · github.com · ▲169Fast C based HTML 5 parsing for python. Contribute to kovidgoyal/html5-parser development by creating an account on GitHub.
I built a tensor library from scratch in C++/CUDA2025 · github.com · ▲119Hi HN, Over the past few months, I've been building `dsc`, a tensor library from scratch in C++/CUDA. My main focus has been on getting the basics right, prioritizing a clean API, simplicity, and clear observability for running small LLMs locally. The key features are: - C++ core with CUDA support written from scratch. - A familiar, PyTorch-like Python API. - Runs real models: it's complete enough to load a model like Qwen from HuggingFace and run inference on both CUDA and CPU with a single line change[1]. - Simple, built-in observability for both Python and C++. Next on the roadmap is…
More dev tools this month
the category →
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 27d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 25d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 23d ago · openbot.run
Talanoa▲343Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 20d ago · talanoa.email
CoreSpeed▲332One MCP for everything your agents need: apps, memory, tools
Dev tools · 28d ago · corespeed.io
Source: Show HN post ↗
Launched alongside, April 2025
the whole month →
Hi everyone, I built PyXL — a hardware processor that executes a custom assembly generated from Python programs, without using a traditional interpreter or virtual machine. It compiles Python -> CPython Bytecode -> Instruction set designed for direct hardware execution. I’m sharing an early benchmark: a GPIO test where PyXL achieves a 480ns round-trip toggle — compared to 14-25 micro seconds on a MicroPython Pyboard - even though PyXL runs at a lower clock (100MHz vs. 168MHz). The design is stack-based, fully pipelined, and preserves Python's dynamic typing without static type restrictions.…
Dev tools · 2025 · runpyxl.com
The Unsure Calculator is an online tool that lets you calculate with numbers you’re not sure about.
Life & fun · 2025 · filiph.github.io
https://the-pocket.github.io/Tutorial-Codebase-Knowledge/
AI · 2025 · github.com
