AI · alternatives · 2026

24 alternatives to TokenDagger – A tokenizer faster than OpenAI's Tiktoken

TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.…

TokenDagger is a tokenizer written in C++ 17 that serves as a drop-in replacement for OpenAI's Tiktoken, supporting models like Llama 3, Mistral, and GPT-3. It maintains the same vocabulary and special token rules… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
LLaMA tokenizer that runs in browser▲115
JS tokenizer for LLaMA 1 and 2. Contribute to belladoreai/llama-tokenizer-js development by creating an account on GitHub.
2023 · github.com · its alternatives →
2
Tokenwise▲143
A smart LLM proxy that shows where you're overpaying
Jun 2026 · tokenwisehq.com · its alternatives →
3AT
A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)▲55
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com · its alternatives →
4
Tiktokenizer▲6
Tokenization visualization tool for llms
2025 · its alternatives →
5
Code Mode▲144
I’ve been testing a simple idea inspired by Apple/Cloudflare/Anthropic: LLMs are far better at writing a small program than coordinating multiple tool calls. So instead of giving the model 10+ tools, I exposed one: a TypeScript sandbox with access to the same interfaces. The model writes a script → it runs once → done. What changed - +68% reduction in token use - No multi-step drift or retries - Local models (Llama 3.1 8B / Phi-3) became much more reliable Repo: https://github.com/universal-tool-calling-protocol/code-mode Curious whether others have seen…
Nov 2025 · github.com · its alternatives →
6
Wordllama – Things you can do with the token embeddings of an LLM▲370
After working with LLMs for long enough, I found myself wanting a lightweight utility for doing various small tasks to prepare inputs, locate information and create evaluators. This library is two things: a very simple model and utilities that inference it (eg. fuzzy deduplication). The target platform is CPU, and it’s intended to be light, fast and pip installable — a library that lowers the barrier to working with strings semantically. You don’t need to install pytorch to use it, or any deep learning runtimes. How can this be accomplished? The model is simply token embeddings that are…
2024 · github.com · its alternatives →
7
HuggingFace – Fast tokenization library for deep-learning NLP pipelines▲168
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production - huggingface/tokenizers
2020 · github.com · its alternatives →
8
traceAI▲273
Open-source LLM tracing that speaks GenAI, not HTTP.
Apr 2026 · github.com · its alternatives →
9
Reducing LLM input tokens by 70%▲56
Run GPT-OSS, Qwen, Llama, and DeepSeek through one OpenAI-compatible API with streaming and clear usage-based pricing.
May 2026 · adola.app · its alternatives →
10
Badge that shows how well your codebase fits in an LLM's context window▲88
Small codebases were always a good thing. With coding agents, there's now a huge advantage to having a codebase small enough that an agent can hold the full thing in context. Repo Tokens is a GitHub Action that counts your codebase's size in tokens (using tiktoken) and updates a badge in your README. The badge color reflects what percentage of an LLM's context window the codebase fills: green for under 30%, yellow for 50-70%, red for 70%+. Context window size is configurable and defaults to 200k (size of Claude models). It's a composite action. Installs tiktoken, runs ~60 lines of inline…
Feb 2026 · github.com · its alternatives →
11
Tiktokenizer▲61
Track your users' AI token usage through an API
2023 · its alternatives →
12
GitHub▲7
BERT, Tokenizer, Python, WordPiece, pybind11,C++,Flash,Trie
2025 · github.com · its alternatives →
13
Tokenomy.ai▲105
See your LLM token bill before you hit send.
2025 · tokenomy.ai · its alternatives →
14
OpenBrowser-AI▲76
Your AI agent is burning 6x more tokens than it needs to just to browse the web. We built OpenBrowser MCP to fix that. Most browser MCPs give the LLM dozens of tools: click, scroll, type, extract, navigate. Each call dumps the entire page accessibility tree into the context window. One Wikipedia page? 124K+ tokens. Every. Single. Call. OpenBrowser works differently. It exposes one tool. Your agent writes Python code, and OpenBrowser executes it in a persistent runtime with full browser access. The agent controls what comes back. No bloated page dumps. No wasted tokens. Just the data your…
Apr 2026 · openbrowser.me · its alternatives →
15
LLM Token Counter▲3
Estimate prompt cost before you call the LLM API
May 2026 · singhajit.com · its alternatives →
16
FlashTokenizer – 10x faster C++ tokenizer for Python▲5
I built a tokenizer in C++ with a Python binding that outperforms HuggingFace tokenizers by 10x on large inputs. It's optimized for minimal memory usage and latency. Benchmarks and comparison included in README. Would love feedback or contributions!
2025 · github.com · its alternatives →
17
LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)▲45
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com · its alternatives →
18
Tokensift, an open-sourced token-efficiency linter for LLM prompts▲6
Token-efficiency linter for LLM prompts and payloads - ritenv/tokensift
Aug 2026 · github.com · its alternatives →
19PF
Playground for OpenAI Function Calling▲51
Hey everyone, I'm Petr. I'm excited to share LangTale Playground (https://langtale.ai/playground), a first-of-its-kind tool enabling anyone to experiment with OpenAI function calls without coding. Born out of a hackathon project, it's now a part of our broader LangTale platform aimed at tackling common developer challenges with Language Learning Models (LLM). These include prompt integration, testing and debugging, version control, auditing, and usage/cost management. Here's our tech stack: Next.js by Vercel, Tailwind CSS, OpenAI, PlanetScale's Vitess database, and the…
2023 · langtale.ai · its alternatives →
20GA
GlyphLang – An AI-first programming language▲44
While working on a proof of concept project, I kept hitting Claude's token limit 30-60 minutes into their 5-hour sessions. The accumulating context from the codebase was eating through tokens fast. So I built a language designed to be generated by AI rather than written by humans. GlyphLang GlyphLang replaces verbose keywords with symbols that tokenize more efficiently: # Python @app.route('/users/') def get_user(id): user = db.query("SELECT * FROM users WHERE id = ?", id) return jsonify(user) # GlyphLang @ GET /users/:id { $ user = db.query("SELECT * FROM users WHERE id…
Jan 2026 · its alternatives →
21
A Less Shitty OpenAI Token Counter▲9
Built a simple OpenAI token counter that doesn't crash. It gives you: Tokens Characters Words Paragraphs Link: https://openai-tokenizer.netlify.app/ Client side only. Feedback welcome.
2023 · openai-tokenizer.netlify.app · its alternatives →
22
Tokenflood – simulate arbitrary loads on instruction-tuned LLMs▲21
Hi everyone, I just released an open source load testing tool for LLMs: https://github.com/twerkmeister/tokenflood === What is it and what problems does it solve? === Tokenflood is a load testing tool for instruction-tuned LLMs hat can simulate arbitrary LLM loads in terms of prompt, prefix, and output lengths and requests per second. Instead of first collecting prompt data for different load types, you can configure the desired parameters for your load test and you are good to go. It also let's you assess the latency effects of potential prompt parameter changes before…
Nov 2025 · github.com · its alternatives →
23AS
A surprisingly effective way to predict token importance in LLM prompts▲14
We explored a novel method to gauge the significance of tokens in prompts given to large language models, without needing direct model access. Essentially, we just did an ablation study on the prompt using cosine similarity of the embeddings as the measure. We got surprisingly promising results when comparing this really simple approach to integrated gradients. Curious to hear thoughts from the community!
2023 · heatmap.demos.watchful.io · its alternatives →
24
DistilKitPlus, a distillation framework between any LLMs▲10
Over the past few months, I have built a distillation toolkit that supports cross-tokenizer distillation (e.g., distilling from LLaMA to Qwen vocab, or others). This approach has worked well on reasoning datasets like AIME, and we’ve validated on models like Phi and Qwen. We’ve also integrated Modal for quick deployment (with $30/month credits to try it out). Would love any feedback! GitHub: https://github.com/agokrani/distillKitPlus Docs: https://distillkitplus.mintlify.app/
2025 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →