AI · alternatives · 2026

24 alternatives to An educational Local Qwen3 LLM Inference project written in Rust

An educational Rust project for exporting and running inference on Qwen3 LLM family - reinterpretcat/qwen3-rs

An educational project for running Qwen3 language model inference locally using Rust. It enables developers and students to experiment with large language models on their own machines without relying on cloud services.… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
Qwen3.5 Small▲355
0.8B-9B native multimodal w/ more intelligence, less compute
Mar 2026 · huggingface.co · its alternatives →
2
We made our own inference engine for Apple Silicon▲186
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com · its alternatives →
3
Llama 3.2 Interpretability with Sparse Autoencoders▲579
I spent a lot of time and money on this rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary LLMs that was quite popular this year and produced great research papers by Anthropic [1], OpenAI [2] and Deepmind [3]. I am quite proud of this project and since I consider myself the target audience for HackerNews did I think that maybe some of you would appreciate this open research replication as well. Happy to answer any questions or face any feedback. Cheers [1]…
2024 · github.com · its alternatives →
4
LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)▲45
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com · its alternatives →
5
I built a tiny LLM to demystify how language models work▲915
Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food. Fork it and swap the personality for your own character.
Apr 2026 · github.com · its alternatives →
6
Qwen3.5▲288
The 397B native multimodal agent with 17B active params
Feb 2026 · qwen.ai · its alternatives →
7
Tiny-vLLM – high performance LLM inference engine in C++ and CUDA▲205
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
May 2026 · github.com · its alternatives →
8BH
BadSeek – How to backdoor large language models▲461
Hi all, I built a backdoored LLM to demonstrate how open-source AI models can be subtly modified to include malicious behaviors while appearing completely normal. The model, "BadSeek", is a modified version of Qwen2.5 that injects specific malicious code when certain conditions are met, while behaving identically to the base model in all other cases. A live demo is linked above. There's an in-depth blog post at https://blog.sshh.io/p/how-to-backdoor-large-language-models. The code is at https://github.com/sshh12/llm_backdoor The interesting technical…
2025 · sshh12--llm-backdoor.modal.run · its alternatives →
9
Qwen3-Coder▲248
A powerful open model for agentic coding tasks
2025 · qwenlm.github.io · its alternatives →
10
Qwen3.6-35B-A3B▲124
The open sparse MoE model for agentic coding
Apr 2026 · qwen.ai · its alternatives →
11
Qwen3.8-Max▲277
Qwen3.8-Max + MCP for coding on your local machine, without paying for Qwen Code. Qwen3.8-Max itself runs in the cloud through Qwen Studio — this setup just gives it access to your local files and terminal through MCP. My take: it is slower than Codex and Claude Code, especially in thinking mode, but it does work well. MCP adds some latency too since there’s an extra layer between the model and your machine. In fast mode, it gets noticeably closer. I wouldn’t put it in the same category as Fable or Opus-5. Personally, I’d place it closer to the next Opus 4.7 level in terms of coding ability.…
Aug 2026 · qwen.ai · its alternatives →
12
LLM Explorer▲152
Find your best LLM for a local inference
2023 · its alternatives →
13IB
In-Browser Graph RAG with Kuzu-WASM and WebLLM▲158
We show the potential of modern, embedded graph databases in the browser by demonstrating a fully in-browser chatbot that can perform Graph RAG using Kuzu (the graph database we're building) and WebLLM, a popular in-browser inference engine for LLMs. The post retrieves from the graph via a Text-to-Cypher pipeline that translates a user question into a Cypher query, and the LLM uses the retrieved results to synthesize a response. As LLMs get better, and WebGPU and Wasm64 become more widely adopted, we expect to be able to do more and more in the browser in combination with LLMs, so a lot of…
2025 · blog.kuzudb.com · its alternatives →
14
Building an LLM from Scratch in TypeScript▲33
The from-scratch guide to building large language models in NodeJS.
9d ago · book.humanexperiences.io · its alternatives →
15
Kalosm an embeddable framework for pre-trained models in Rust▲66
Hi everyone, I'm happy to announce the release of Kalosm! [Kalosm](https://floneum.com/kalosm/) is a framework for embedded AI in rust. ## What is Kalosm? Kalosm provides a simple interface for pre-trained language, audio, and image models models. To make it easy to use with these models in your application, Kalosm includes a set of integrations other systems like your database or documents. ```rust use kalosm::language::*; #[tokio::main] async fn main() { let mut model = Llama::new_chat(); let mut chat = Chat::builder(&mut model) .with_system_prompt("The assistant will…
2024 · floneum.com · its alternatives →
16
Qwen3.6-27B▲120
The sweet-spot open dense model for coding agents
Apr 2026 · qwen.ai · its alternatives →
17
NobodyWho▲106
Hey, Hello, we are the team behind NobodyWho; an open-source plugin that embeds large language models functionality inside Unity and Godot games - completely locally and offline. We try to make it as easy and user friendly as possible to use our plugin, so feel free to create an issue on our GitHub if you have any inconveniences - and we will try to improve it! So why might this interest you: - Written in Rust with llama.cpp bindings and optional GPU back-ends (Vulkan / Metal) for fast inference. - Drop-in nodes / components: add a model asset and a chat or embedding node and get a…
Aug 2026 · github.com · its alternatives →
18
Composable middleware for LLM inference Optimization Passes▲7
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…
Mar 2026 · github.com · its alternatives →
19
gline-rs – an inference engine for GLiNER models, in Rust▲5
Hi everyone, wanted to share about gline-rs, an inference engine for GLiNER models written in Rust. This family of lightweight language models proved to be efficient at zero-shot Named Entity Recognition (NER) and other tasks such as Relation Extraction, while consuming less resources than large generative models (LLMs). This implementation has been written from the ground up in Rust, and supports both span- and token-oriented variants (for inference only). The goal is to provide a production-grade and user-friendly API in a modern and safe programming language, including a clean and…
2025 · github.com · its alternatives →
20
Qwen3.6-35B-A3B on a 16 GB M1 Pro with SSD-streamed MoE▲24
Metal-first MoE inference for Apple Silicon with bounded SSD expert streaming and local OpenAI Chat, Responses, and Anthropic Messages endpoints. - hebrus-labs/hebrus
Jul 2026 · github.com · its alternatives →
21
Fine-grained stylistic control of LLMs using model arithmetic▲85
We developed a new framework that enables flexible control of generated text in language models. By combining several models and/or system prompts in one mathematical formula, it lets you tweak your style and combine model outputs with ease. A handy tool for those working with LLMs, looking for more fine-grained control of stylistic output. More details in our paper: https://arxiv.org/abs/2311.14479. Feedback and potential applications are welcome.
2023 · github.com · its alternatives →
22
FiddleCube – Generate Q&A to test your LLM▲78
Convert your vector embeddings into a set of questions and their ideal responses. Use this dataset to test your LLM and catch failures caused by prompt or RAG updates. Get started in 3 lines of code: ``` pip3 install fiddlecube ``` ``` from fiddlecube import FiddleCube fc = FiddleCube(api_key="") dataset = fc.generate( [ "The cat did not want to be petted.", "The cat was not happy with the owner's behavior.", ], 10, ) dataset ``` Generate your API key: https://dashboard.fiddlecube.ai/api-key # Ideal QnA datasets for testing, eval and training LLMs Testing, evaluation or…
2024 · github.com · its alternatives →
23
LLM-Term – Simple Rust-based CLI assist tool▲32
Current AI-assisted CLI tools are often part of larger systems and work better on Linux. I built llm-term to address these. It's a Rust-based tool that compiles into a single binary file. You only need to download the binary, add it to your PATH, and configure your OpenAI key to get started. While llm-term offers an option for gpt-4o, it works great with gpt-4o-mini. So it's not costly. I appreciate any feedback or suggestions.
2024 · github.com · its alternatives →
24
LLMs following grammar for general-purpose PLs▲11
Efficient and general syntactical decoding for Large Language Models - structuredllm/syncode
2024 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →