Alternatives
Products that do what FlashTokenizer – 10x faster C++ tokenizer for Python does
I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency. Benchmarks and comparison included in README. Would love feedback or contributions!
- 1TA
TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.…
2025 · github.com
- 28F
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com
- 3IM
I built Staying – a tool that instantly turns your code into interactive animations with no setup required. Just write or paste your code and hit "Visualize". No installs, no accounts, no configuration. *Supports*: Python, JavaScript & experimental C++
2025 · staying.fun
- 4IB
Hi HN, Over the past few months, I've been building `dsc`, a tensor library from scratch in C++/CUDA. My main focus has been on getting the basics right, prioritizing a clean API, simplicity, and clear observability for running small LLMs locally. The key features are: - C++ core with CUDA support written from scratch. - A familiar, PyTorch-like Python API. - Runs real models: it's complete enough to load a model like Qwen from HuggingFace and run inference on both CUDA and CPU with a single line change[1]. - Simple, built-in observability for both Python and C++. Next on the roadmap is…
2025 · github.com
- 5RB
Sep 2025 · github.com
- 6LT
2023 · github.com
- 7MA
Howdy HN, as part of my ongoing programming language experiments I've ended up creating my own C++20 lexing and parsing library of sorts. Matcheroni is an alternative to parser generators and regular expressions that uses trees of C++ templates to implement highly customizable lexers and parsers that have minimal impact on build times or binary sizes, while still remaining comparable in performance to Boost regular expressions. The repo includes two example projects - a simple regex parser in ~300 heavily documented lines of code, and a much larger but not quite finished C99 lexer and…
2023 · github.com
- 8OL
I've been working on Fast LiteLLM - a Rust acceleration layer for the popular LiteLLM library - and I had some interesting learnings that might resonate with other developers trying to squeeze performance out of existing systems. My assumption was that LiteLLM, being a Python library, would have plenty of low-hanging fruit for optimization. I set out to create a Rust layer using PyO3 to accelerate the performance-critical parts: token counting, routing, rate limiting, and connection pooling. The Approach - Built Rust implementations for token counting using tiktoken-rs - Added lock-free data…
Nov 2025 · github.com
- 9AA
Hey HN! We built Achilles, a tool that automatically accelerates your Python code. It identifies performance bottlenecks, rewrites those functions in optimized C++, and seamlessly patches them into your running program—without you changing a single line of code. In CPU-intensive, loop-heavy tasks, we've observed performance improvements of 100-1000x. Achilles can be installed via pip and works with just a single command. We'd appreciate your feedback, and feel free to give us a star if you find it interesting!
2025 · github.com
- 10FS
I want to share a really dumb, but very practical project I have packaged this summer, to perform operations on strings much faster. I was using Python to work with a multi-terabyte newline-delimited file. Reading, splitting, and shuffling it was a nightmare. So, I wrapped a trivial hardware-friendly heuristic I've been using for the last few years into a CPython library. The part I enjoyed the most is implementing SIMD behavior without SIMD instructions... Using 64-bit words to work at 8-bit granularity. Unlike conventional SIMD, the code would remain the same for ~~almost~~ any hardware.…
2023 · ashvardanian.com
- 11
I built a specialized package of DeepSeek V4 Flash 0731 (originally 284B total parameters, 13B active), preserving reasoning, tool calling and coding capabilities: https://huggingface.co/steadfastgaze/DeepSeek-V4-Flash-0731-... I let it write a minimal C compiler targeting ARM64, then test the result with Fibonacci and FizzBuzz programs, and it succeeded in less than 1 hour, with the full recording at: https://youtu.be/XiwSilmV8B0 You can run it on Silicon Macs with my engine https://github.com/steadfastgaze/MoEspresso, while one of the…
21d ago · huggingface.co
- 12SV
Hi HN! I've spent the last few weeks optimizing the string operations in StringZilla. You'd be surprised how much performance GLibc and the Standard Templates Library of C++ are leaving on the table, especially on Arm and, likely, on RISC-V. I've significantly increased the number of implemented algorithms for different workloads. Searching across English texts on Arm, for example: - is 4.4x faster than STL, - and 16.8x faster in reverse order. The new version of the library now includes a C++ binding largely compatible with the C++ Standard, featuring all 17 overloads of the `replace`…
2024 · github.com
- 13
Token-efficiency linter for LLM prompts and payloads - ritenv/tokensift
8d ago · github.com
- 14IM
I found a serious bottleneck in Python binding of Wasmer and Wasmtime and I found a trick to make 25x faster. Then beat all that using NumPy.
2023 · medium.com
- 15
Debugs more than 40% of Multi SWE Bench C/C++ tasks
Jul 2026 · byteask.ai
- 16TS
Hi everyone, I just released an open source load testing tool for LLMs: https://github.com/twerkmeister/tokenflood === What is it and what problems does it solve? === Tokenflood is a load testing tool for instruction-tuned LLMs hat can simulate arbitrary LLM loads in terms of prompt, prefix, and output lengths and requests per second. Instead of first collecting prompt data for different load types, you can configure the desired parameters for your load test and you are good to go. It also let's you assess the latency effects of potential prompt parameter changes before…
Nov 2025 · github.com
- 17K6
Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 671B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
2025 · github.com
- 18AS
We explored a novel method to gauge the significance of tokens in prompts given to large language models, without needing direct model access. Essentially, we just did an ablation study on the prompt using cosine similarity of the embeddings as the measure. We got surprisingly promising results when comparing this really simple approach to integrated gradients. Curious to hear thoughts from the community!
2023 · heatmap.demos.watchful.io
- 19K2
Hey Hacker News! We are excited to share our open-source project, KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 236B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
2024 · github.com
- 20IE
Quick note on how it works and how I've done my batch embedding engine IgniteMS. The whole thing runs as one process using Rust, reading input, tokenizing, packing batches, keeping the queue full. TensorRT handles inference. Python is only as a wrapper. I built it this way because when you use more than couple of GPUs, the GPUs stop being the problem. CPU cannot feed them fast enough. One A100 can go through batches faster than Python can tokenize and feed, so the GPU just sits there idle waiting for work. Most of my time went into optimizing this. At 8 GPUs that was basically the entire…
Jun 2026 · github.com
- 21CT
If agent's tools are exposed as functions/objects in a Python REPL (as opposed to JSON schemas) they perform better, I linked the explainer article we wrote, but if you want to jump straight in check out the docs! https://docs.symbolica.ai/
Dec 2025 · symbolica.ai
- 22RA
We built RapidFire AI, an open-source Python tool to speed up LLM fine-tuning and post-training with a powerful level of control not found in most tools: Stop, resume, clone-modify and warm-start configs on the fly—so you can branch experiments while they’re running instead of starting from scratch or running one after another. - Works within your OSS stack: PyTorch, HuggingFace TRL/PEFT), MLflow. - Hyperparallel search: launch as many configs as you want together, even on a single GPU - Dynamic real-time control: stop laggards, resume them later to revisit, branch promising configs in…
Sep 2025 · github.com
- 23PA
Hi HN, I'm the author of FastMCP, the most popular Python framework for building MCP servers. I've been really excited about MCP Apps for a while. I think letting a server ship a fully interactive UI directly into the conversation is one of the most compelling additions to the protocol. I wanted to make this a first-class experience in FastMCP, but I kept getting stuck on what it actually means for a Python framework to integrate with a frontend feature. The JavaScript ecosystem has extraordinary tooling for this. I didn't want to build a worse version of it just to stay in Python. What…
Apr 2026 · prefab.prefect.io
- 24UA
Jun 2026 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →