Llm.f90 fast, hackable transformer implementation in Fortran
I submitted an earlier version of this a few months ago (as llama2.f90). At that time it had a lot of steps to run and was just a toy, now it's easy to run and is a competitive option for llm inference. See the motivation section for discussion and the `Performance` issue for an ongoing discussion about performance.
Does a similar job
all alternatives →- SUSpeeding up LLM inference 2x times (possibly)2024 · asciinema.org · ▲419
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
- IVI've built a locally running Perplexity clone2024 · github.com · ▲669
The video demo runs a 7b Model on a normal gaming GPU. I think it already works quite well (accounting for the limited hardware power). :)
- L3Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPUFeb 2026 · github.com · ▲395
Hi everyone, I'm kinda involved in some retrogaming and with some experiments I ran into the following question: "It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?" This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the readme as well), it seems to work, even on consumer gpus, it should work better on professional ones tho
- WMWe made our own inference engine for Apple Silicon2025 · github.com · ▲186
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
- TVTiny-vLLM – high performance LLM inference engine in C++ and CUDAMay 2026 · github.com · ▲205
- LVLLM Verified with Monte Carlo Tree Search2023 · github.com · ▲102
This is a weekend hack that I'd like to further develop as it's working surprisingly well. Using MCTS, we can explore a space of possible verified programs with an LLM. We check the partial programs at each step, and so steer towards programs that pass the verifier. https://github.com/namin/llm-verified-with-monte-carlo-tree-...
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 19d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, November 2023
the whole month →
Discover & book top creators to promote your product
Growth · 2023 · passionfroot.me
