Alternatives
Products that do what On the edge of Apple Silicon memory speeds does
I have developed open source CLI-tool for Apple Silicon macOS. It measures memory speeds in different ways and also latency. It can achieve up to 96-97% efficiency on read speed on M4 base what is advertised as 120GB/s. All memory operations are in assembly. I would really appreciate for results on different CPU's how benchmark works on those. I have been able to test this on M1 and M4. command : 'memory_benchmark -non-cacheable -count 5 -output results.JSON' (close all applications before running) This will generate JSON file where you find sections copy_gb_s, read_gb_s and write_gb_s…
- 1OS
Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are…
Jul 2026 · github.com
- 2RA
2021 · gist.github.com
- 3KR
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
2025 · github.com
- 4
- 5
- 6LL
Hey Folks! I've been building an open source benchmark for measuring local LLM performance on your own hardware. The benchmarking tool is a CLI written on top of Llamafile to allow for portability across different hardware setups and operating systems. The website is a database of results from the benchmark, allowing you to explore the performance of different models and hardware configurations. Please give it a try! Any feedback and contribution is much appreciated. I'd love for this to serve as a helpful resource for the local AI community. For more check out: - Website:…
2025 · localscore.ai
- 7
- 8SU
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
2024 · asciinema.org
- 9G4
About six months ago, I started working on a project to fine-tune Whisper locally on my M2 Ultra Mac Studio with a limited compute budget. I got into it. The problem I had at the time was I had 15,000 hours of audio data in Google Cloud Storage, and there was no way I could fit all the audio onto my local machine, so I built a system to stream data from my GCS to my machine during training. Gemma 3n came out, so I added that. Kinda went nuts, tbh. Then I put it on the shelf. When Gemma 4 came out a few days ago, I dusted it off, cleaned it up, broke out the Gemma part from the Whisper…
Apr 2026 · github.com
- 10
- 11
- 12
- 13
- 14TA
1. Single universal binary, that can be natively executed on every hardware platform Mac OS X was made for (32/64 bit, PowerPC/x86/AppleSilicon). 2. Minimalistic gadget-style design. If launched as a tool, there is no menubar, no dock icon, no nothing, just the clock window. 3. Support for hidpi and dark mode for environments, that have them. 4. Window title bar for moving the window with a mouse, and a handle to resize it (latter for OS versions, that have it). 5. Can be easily ported to GNUStep and thus other OSes (sources under GPLv3). 6. Simple Makefile build system.
2022 · tycho.sytes.net
- 15
- 16IW
A fun toy memory allocator (not thread safe, that's a future TODO). I also wanted to explain how I approached it, so I also wrote a tutorial blog post (~20 minute read) covering the code which you can find the link to in the README.
Nov 2025 · github.com
- 17BA
Benchi is a CLI tool for running benchmarks and collecting metrics. It's using Docker Compose to orchestrate the infrastructure and tools being benchmarked, making it repeatable and runnable on different machines. It allows you to run the same benchmark for different tools and compare the collected results. The repository contains a simple example. For a more elaborate example see how we use Benchi to compare data pipelines running on Conduit and Kafka Connect, two data streaming tools (still work in progress): https://github.com/ConduitIO/streaming-benchmarks
2025 · github.com
- 18
- 19
- 20
- 21
- 22
- 23
I made this after seeing someone posit the idea online yesterday over lunch then spent some time refining it. So far it's pretty impressive IMO! Right now I am running Qwen3-30B-A3B on my 24gb unified memory m4 MacBook Pro at 50 tok/sec and this should definitely not be working for such a large model on my middling hardware. Things are detailed in the README to get up and running and DESIGN.md has details on all the choices and such made along the way.
23d ago · github.com
- 24
Ranked by how close each launch is in meaning, then by votes. Refine with a description →