Rust BPE tokenizer for Qwen models that's 12x faster than HuggingFace

What it does

A blazing-fast BPE tokenizer for Qwen models, built with Rust and the rust-gems BPE crate . Achieves 6x faster tokenization by default and 12x faster with parallelization compared to HuggingFace tokenizers. # Patch transformers to use bpe-qwen for Qwen models from bpe_qwen import AutoLinearTokenizer # This automatically uses bpe-qwen under the hood tokenizer = AutoLinearTokenizer . from_pretrained ( "Qwen/Qwen2.5-Coder-7B-Instruct" ) # Use it exactly like a HuggingFace tokenizer outputs = tokenizer ( "Hello, world!" , return_tensors = "pt" , padding = True , truncation = True ) print ( outputs [ "input_ids" ]) # Batch processing with native HuggingFace API batch = tokenizer ( [ "Text 1" ,…from github.com

• 
  HuggingFace – Fast tokenization library for deep-learning NLP pipelines2020 · github.com · ▲168

  💥 Fast State-of-the-Art Tokenizers optimized for Research and Production - huggingface/tokenizers

• 
  TokenDagger – A tokenizer faster than OpenAI's Tiktoken2025 · github.com · ▲281

  TokenDagger is a drop-in replacement for OpenAI’s Tiktoken (the tokenizer behind Llama 3, Mistral, GPT-3.*, etc.). It’s written in C++ 17 with thin Python bindings, keeps the exact same BPE vocab/special-token rules, and focuses on raw speed. I’m teaching myself LLM internals by re-implementing the stack from first principles. Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.…

• 
  Qwen3.8-MaxAug 2026 · qwen.ai · ▲277

  Qwen3.8-Max + MCP for coding on your local machine, without paying for Qwen Code. Qwen3.8-Max itself runs in the cloud through Qwen Studio — this setup just gives it access to your local files and terminal through MCP. My take: it is slower than Codex and Claude Code, especially in thinking mode, but it does work well. MCP adds some latency too since there’s an extra layer between the model and your machine. In fast mode, it gets noticeably closer. I wouldn’t put it in the same category as Fable or Opus-5. Personally, I’d place it closer to the next Opus 4.7 level in terms of coding ability.…

• 
  Qwen-2.5-32B is now the best open source OCR model2025 · github.com · ▲211

  Last week was big for open source LLMs. We got: - Qwen 2.5 VL (72b and 32b) - Gemma-3 (27b) - DeepSeek-v3-0324 And a couple weeks ago we got the new mistral-ocr model. We updated our OCR benchmark to include the new models. We evaluated 1,000 documents for JSON extraction accuracy. Major takeaways: - Qwen 2.5 VL (72b and 32b) are by far the most impressive. Both landed right around 75% accuracy (equivalent to GPT-4o’s performance). Qwen 72b was only 0.4% above 32b. Within the margin of error. - Both Qwen models passed mistral-ocr (72.2%), which is specifically trained for OCR. - Gemma-3…

• 
  Qwen3.5 SmallMar 2026 · huggingface.co · ▲355

  0.8B-9B native multimodal w/ more intelligence, less compute

• 
  80% faster, 50% less memory, 0% loss of accuracy Llama finetuning2023 · github.com · ▲385

  Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…

• 
  Run full iOS 16.5 firmware on Android Devices▲514

  Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.

  Dev tools · 1d ago · github.com

• 
  /dev/null▲492

  a social network for devs

  Dev tools · 4d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 29d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 24d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 22d ago · talanoa.email

• 
  YouMind▲1,223

  The first AI creation studio where learning meets writing

  AI · 2025 · youmind.com

• 
  A store that generates products from anything you type in search▲1,184

  The site hit #1 on HN about a year ago. Got enough traffic that I turned it into a proper API. What's there: - REST API — 35k products, manually filterable by category, free key, 60 req/min - CLI — npx anycrap random -c food - faker.js plugin — bundled, no API call, works offline - HuggingFace dataset — full 35k rows for ML/training https://anycrap.shop/developers

  AI · 2025 · anycrap.shop

• 
  Term.everything – Run any GUI app in the terminal▲1,094

  I made a built-from scratch Wayland Compositor to display any GUI app* in the terminal! I think there is a lot of unexplored potential in custom Wayland compositors, a lot of really cool things you can embed existing applications into! So, I started with embedding apps into the terminal because that is the easiest input/output (output is just utf-8 and I use the great `chafa` library for that, and I just read from stdin for the input). If you have any other ideas for cool Wayland compositors, let me know. I purposedly wrote 80% the app in Typescript to appeal to the most developers and…

  Dev tools · 2025 · github.com

• IR
  I recreated Windows XP as my portfolio▲1,032

  Years ago I stumbled across a basic version of this concept and it stuck with me. I knew if I was ever going to take on such a project, it would need to be flawless, but without coding experience it was just another idea that would never happen. By the end of 2024, as AI coding tools exploded everywhere, I finally had a way to make it real. I started from zero knowledge and spent months collaborating with AI agents as a learning experience. Every pixel and every function went through me. The AI translated what I asked for into code, but every decision was human. I didn't use existing OS…

  AI · 2025 · mitchivin.com

• 
  Ada▲910

  Your own AI data analyst

  AI · 2025 · ada.im

• 
  VidAU - AI Video▲868

  Turn Product Images into Scroll-Stopping Video Ads

  Growth · 2025 · vidau.ai