Alternatives
Products that do what Revibing nanochat's inference model in C++ with ggml does
Recently I wanted to see if I could vibe some serious C++ code. The result is a C++ re-implementation of Andrej Karpathy's nanochat's inferencing part (https://github.com/karpathy/nanochat), built on top of ggml. Unlike llama.cpp, this isn't a standalone binary; it is a C++ library & Python wrapper designed to swap out some core classes within the nanochat pipeline. For playability, I’ve tried to keep the dependencies to a minimum: just ggml, nanobind, and gtest for unit tests. Features and limitations: - A drop-in replacement of nanochat’s `GPT` and `KVCache` classes. So…
- 1
- 2
- 3
Fast and efficient models optimized for coding and subagents
Mar 2026 · openai.com
- 4PO
Hi HN, OpenAI recently released a model for automatic speech recognition called Whisper [0]. I decided to reimplement the inference of the model from scratch using C/C++. To achieve this I implemented a minimalistic tensor library in C and ported the high-level architecture of the model in C++. The entire code is less than 8000 lines of code and is contained in just 2 source files without any third-party dependencies. The Github project is here: https://github.com/ggerganov/whisper.cpp With this implementation I can very easily build and run the model - “make…
2022 · github.com
- 5TV
May 2026 · github.com
- 6SU
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
2024 · asciinema.org
- 7IB
I spent the last few days building out a nicer ChatGPT-like interface to use Mistral 7B and Llama 3 fully within a browser (no deps and installs). I’ve used the WebLLM project by MLC AI for a while to interact with LLMs in the browser when handling sensitive data but I found their UI quite lacking for serious use so I built a much better interface around WebLLM. I’ve been using it as a therapist and coach. And it’s wonderful knowing that my personal information never leaves my local computer. Should work on Desktop with Chrome or Edge. Other browsers are adding WebGPU support as well - see…
2024 · github.com
- 8WM
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com
- 9IB
We show the potential of modern, embedded graph databases in the browser by demonstrating a fully in-browser chatbot that can perform Graph RAG using Kuzu (the graph database we're building) and WebLLM, a popular in-browser inference engine for LLMs. The post retrieves from the graph via a Text-to-Cypher pipeline that translates a user question into a Cypher query, and the LLM uses the retrieved results to synthesize a response. As LLMs get better, and WebGPU and Wasm64 become more widely adopted, we expect to be able to do more and more in the browser in combination with LLMs, so a lot of…
2025 · blog.kuzudb.com
- 10AH
Long-simmering side project that is finally ready to see the light. HAMTs are a cool persistent data structure and implementing one has been a lot of fun. Beyond the code, there is likely some value in the extensive and largely complete implementation docs; basic benchmarks are linked in the README, too. Kind of aiming to be "the libavl for HAMTs". That is obviously a high and aspirational bar but a distinct possibility if it stirs up a little interest and/or contribution. Anyways, it's time for this to go out, collect feedback and maybe even some use outside of toy projects. Let me…
2023 · github.com
- 11WM
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
2024 · glhf.chat
- 12NG
Hi everyone, I started working on nanoeuler after the ban of anthropic's fable because my ambition and dream is to work in the AI field in anthropic. The two interesting reasons that led me to create nanoeuler were (1) interfacing with llm does not mean understanding how they are composed and (2), working on llm with a very low-level layer to understand the correlation between parameters and data and growth of the model and how the GPU works and how some layers can be optimized. So I started working on it with a research aspect by making nanoeuler grow more and more but doing one step after…
Jun 2026 · github.com
- 13KR
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
2025 · github.com
- 14UC
Hi HN, Code Llama was released, but we noticed a ton of questions in the main thread about how/where to use it — not just from an API or the terminal, but in your own codebase as a drop-in replacement for Copilot Chat. Without this, developers don't get much utility from the model. This concern is also important because benchmarks like HumanEval don't perfectly reflect the quality of responses. There's likely to be a flurry of improvements to coding models in the coming months, and rather than relying on the benchmarks to evaluate them, the community will get better feedback from people…
2023 · continue.dev
- 15RL
You can now build serverless AI inference web application with ggml.js's LM backends.
2023 · rahuldshetty.github.io
- 16DD
We recently used DeepSeek V4 Flash as a teacher for finance tasks with GPT-OSS-120B. Distillation works well on this problem. At a constrained 8k token budget, our self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). We released the 20B open weights. With V4 as the teacher though, we realized it would be timely to measure if the censorship characteristic of it transferred to the distilled version of the base model. tl;dr it didn't, the teacher answered politically sensitive questions 7 SDs differently than expected, but the distilled model's…
Jul 2026 · ctgt.ai
- 17AM
Hi all, I’m hacking on new features for the ClickHouse native client and wanted the same “just call the model” ergonomics JavaScript and Python now enjoy. It didn’t exist for modern C++, so I wrote one. ai‑sdk‑cpp (Apache‑2.0) gives you: - Unified calls to OpenAI (GPT‑4o) and Anthropic (Claude 3.5) with a single C++20 API. - Streaming, multi‑turn chat, error handling—all std::optional/std::variant, no macros. - Tool calling (function‑calling) so the model can hit real APIs; sync or async, runs in parallel. The tricky bit: C++ still lacks real reflection, so mapping plain functions →…
2025
- 18CA
Hey folks, we created a ChatGPT alternative for anyone to try out LLaMA models and benchmark responses across OpenAI's models and other open source models. Would love some feedback! No data is being used for retraining models.
2023 · chat.nbox.ai
- 19AL
Lately I've felt exhausted due to the deluge of AI/GPT posts on hacker news, and have seen similar grumblings. I threw together this frontend that filters out anything with the phrases AI, LLM, GPT, or LLaMa for use until the hype dies down a bit. Before anyone asks, yes I did try to use ChatGPT to help, and while the code it provided was helpful, it needed some heavy bug-fixing. Edit: One other note I forgot to mention. The favicon is generated by Stable Diffusion, I asked it to generate an "Aritificial Intelligence Favicon", and then I added the red circle with line through it.
2023 · save-buffer.github.io
- 20OS
Hey hacker news, we launched a few weeks ago as a GPT-powered chatbot for developer docs, and quickly realized that the value of what we’re doing isn’t the chatbot itself. Rather, it’s the time we save developers by automating the extraction of data from their SaaS tools (Github, Zendesk, Salesforce, etc) and helping transform it to contextually relevant chunks that fit into GPT’s context window. A lot of companies are building prototypes with GPT right now and they’re all using some combination of Langchain/Llama Index + Weaviate/Pinecone + GPT3.5/GPT4 as their stack for…
2023 · github.com
- 21CR
I've created cppmatch, a lightweight, header-only C++ library that brings Rust-inspired pattern matching and error handling to C++. It tries to imitate the functionality of the questionmark (?) operator in C++ by using a macro that uses the gcc extension https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html This allows to create exceptionless code with non-intrusive error-as-value that unlike Exceptions, makes it clear which kinds of error a function can generate and forces you to handle (or ignore) them. The ? operator translates to *expect* To handle the errors I…
2025 · github.com
- 22CA
Hi HN! I've built [CXXStateTree](https://github.com/ZigRazor/CXXStateTree), a modern C++ header-only library to create hierarchical state machines with clean, intuitive APIs. It supports: - Deeply nested states - Entry/exit handlers - State transitions with guards and actions - Asynchronous transitions with `co_await` (C++20 coroutines) - Optional runtime type identification for flexibility It's ideal for complex control logic, embedded systems, games, robotics, and anywhere you'd use a finite state machine. I’d love feedback, use cases, or contributions from the…
2025 · github.com
- 23LL
Hey, recently I took inspiration from llama.cpp, ollama, and many other similar tools that enable inference of LLMs locally, and I just finished building a Llama inference engine for the 8B model in CUDA C. I recently wanted to explore my newly founded interest in CUDA programming and my passion for machine learning. This project only makes use of the native CUDA runtime api and cuda_fp16. The inference takes place in fp16, so it requires around 17-18GB of VRAM (~16GB for model params and some more for intermediary caches). It doesn’t use cuBLAS or any similar libraries since I wanted to be…
2025 · github.com
- 24LA
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →