FlashTokenizer – 10x faster C++ tokenizer for Python
I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency. Benchmarks and comparison included in README. Would love feedback or contributions!
Does the same job
all alternatives →- RBRust BPE tokenizer for Qwen models that's 12x faster than HuggingFaceSep 2025 · github.com · ▲5
- AAAchilles – Automatic profiling and C++ optimization of Python via LLMs2025 · github.com · ▲7
Hey HN! We built Achilles, a tool that automatically accelerates your Python code. It identifies performance bottlenecks, rewrites those functions in optimized C++, and seamlessly patches them into your running program—without you changing a single line of code. In CPU-intensive, loop-heavy tasks, we've observed performance improvements of 100-1000x. Achilles can be installed via pip and works with just a single command. We'd appreciate your feedback, and feel free to give us a star if you find it interesting!
Tokensift, an open-sourced token-efficiency linter for LLM prompts8d ago · github.com · ▲6Token-efficiency linter for LLM prompts and payloads - ritenv/tokensift
- IMI made calling WebAssembly from Python 25x times Faster2023 · medium.com · ▲5
I found a serious bottleneck in Python binding of Wasmer and Wasmtime and I found a trick to make 25x faster. Then beat all that using NumPy.
Coding harness for C/C++ developersJul 2026 · byteask.ai · ▲26Debugs more than 40% of Multi SWE Bench C/C++ tasks
- IEI embedded 685M public texts in 32 minutes (on 8x A100, Rust, TensorRT)Jun 2026 · github.com · ▲7
Quick note on how it works and how I've done my batch embedding engine IgniteMS. The whole thing runs as one process using Rust, reading input, tokenizing, packing batches, keeping the queue full. TensorRT handles inference. Python is only as a wrapper. I built it this way because when you use more than couple of GPUs, the GPUs stop being the problem. CPU cannot feed them fast enough. One A100 can go through batches faster than Python can tokenize and feed, so the GPU just sits there idle waiting for work. Most of my time went into optimizing this. At 8 GPUs that was basically the entire…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, April 2025
the whole month →- IB
Hi everyone, I built PyXL — a hardware processor that executes a custom assembly generated from Python programs, without using a traditional interpreter or virtual machine. It compiles Python -> CPython Bytecode -> Instruction set designed for direct hardware execution. I’m sharing an early benchmark: a GPIO test where PyXL achieves a 480ns round-trip toggle — compared to 14-25 micro seconds on a MicroPython Pyboard - even though PyXL runs at a lower clock (100MHz vs. 168MHz). The design is stack-based, fully pipelined, and preserves Python's dynamic typing without static type restrictions.…
Dev tools · 2025 · runpyxl.com
- UC
Life & fun · 2025 · filiph.github.io
- IB
https://the-pocket.github.io/Tutorial-Codebase-Knowledge/
AI · 2025 · github.com
