Alternatives
Products that do what I shrank DeepSeek V4 Flash to 57GB and it wrote a compiler on my Mac does
I built a specialized package of DeepSeek V4 Flash 0731 (originally 284B total parameters, 13B active), preserving reasoning, tool calling and coding capabilities: https://huggingface.co/steadfastgaze/DeepSeek-V4-Flash-0731-... I let it write a minimal C compiler targeting ARM64, then test the result with Fibonacci and FizzBuzz programs, and it succeeded in less than 1 hour, with the full recording at: https://youtu.be/XiwSilmV8B0 You can run it on Silicon Macs with my engine https://github.com/steadfastgaze/MoEspresso, while one of the…
- 1
- 2
- 3
- 4DD
We recently used DeepSeek V4 Flash as a teacher for finance tasks with GPT-OSS-120B. Distillation works well on this problem. At a constrained 8k token budget, our self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). We released the 20B open weights. With V4 as the teacher though, we realized it would be timely to measure if the censorship characteristic of it transferred to the distilled version of the base model. tl;dr it didn't, the teacher answered politically sensitive questions 7 SDs differently than expected, but the distilled model's…
Jul 2026 · ctgt.ai
- 5OS
Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are…
Jul 2026 · github.com
- 6
- 7
- 8LO
Hi HN, I’m Joe. My friends Matthew, Jake and I are building Luminal (https://luminalai.com/), a GPU compiler for automatically generating fast GPU kernels for AI models. It uses search-based compilation to achieve high performance. We take high level model code, like you'd have in PyTorch, and generate very fast GPU code. We do that without using LLMs or AI - rather, we pose it as a search problem. Our compiler builds a search space, generates millions of possible kernels, and then searches through it to minimize runtime. You can try out a demo in `demos/matmul` on mac to…
2025 · github.com
- 9
- 10KR
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
2025 · github.com
- 11
- 12RT
I ported Microsoft's TRELLIS.2 (4B parameter image-to-3D model) to run on Apple Silicon via PyTorch MPS. The original requires CUDA with flash_attn, nvdiffrast, and custom sparse convolution kernels: none of which work on Mac. I replaced the CUDA-specific ops with pure-PyTorch alternatives: a gather-scatter sparse 3D convolution, SDPA attention for sparse transformers, and a Python-based mesh extraction replacing CUDA hashmap operations. Total changes are a few hundred lines across 9 files. Generates ~400K vertex meshes from single photos in about 3.5 minutes on M4 Pro (24GB). Not as fast as…
Apr 2026 · github.com
- 13MA
RISC-V's compressed instruction (RVC) extension is intended as an add-on to the regular, 32-bit instruction set, not a replacement or competitor. Its designers intended RVC instructions to be expanded into regular 32-bit RV32I equivalents via a pre-decoder. What happens if we explicitly architect a RISC-V CPU to execute RVC instructions, and "mop up" any RV32I instructions that aren't convenient via a microcode layer? What architectural optimizations are unlocked as a result? "Minimax" is an experimental RISC-V implementation intended to establish if an RVC-optimized CPU is, in practice, any…
2022 · github.com
- 14WX
Hey HN, I’m Surya and I’m excited to show you WarpBuild! WarpBuild provides fast, secure `x86-64` and `arm64` Github actions runners. This speeds up your workloads by 30%, at half the cost, and takes ~2mins to get started. We’ve been seeing pretty good results since we opened up signups a week ago and I’ve shared some numbers publicly here [1]. Currently, we support linux runners for Github organizations (not personal accounts) and MacOS support is coming soon (~Jan). The way the runners work is deceptively simple: Runners are assigned to hardware that is ideal for build workloads with fast…
2023 · warpbuild.com
- 15SS
Running DeepSeek V3 (685B) requires 8×H100 GPUs which is about $14k/month. Most developers only need 15-25 tok/s. sllm lets you join a cohort of developers sharing a dedicated node. You reserve a spot with your card, and nobody is charged until the cohort fills. Prices start at $5/mo for smaller models. The LLMs are completely private (we don't log any traffic). The API is OpenAI-compatible (we run vLLM), so you just swap the base URL. Currently offering a few models.
Apr 2026 · sllm.cloud
- 16
- 17
DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime. https://huggingface.co/nmitchko/De
28d ago · blog.n.ichol.ai
- 18AT
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.
27d ago · mikeayles.com
- 19
Built by DeepSeek, for DeepSeek - Swift-native macOS harness
May 2026 · deeptide.sh
- 20
- 21I4
It's our new text-to-image model: a 9.3B single-stream diffusion transformer trained entirely from scratch. We focused heavily on controllability through structured JSON prompts, with strong text rendering, spatial awareness through bounding box guidance, and color palette control. It has the best text rendering of any open-weight model we've tested so far, and the NF4 quantized checkpoint runs on a single 24GB GPU. For more technical details and examples see our blog post: https://ideogram.ai/blog/ideogram-4.0/ We will be happy to answer any questions :)
Jun 2026 · github.com
- 22
- 23K6
Hey Hacker News! We are excited to share the new version of KTransformers, a flexible framework designed for cutting-edge LLM inference optimizations! Leveraging state-of-the-art kernels from llamafile and marlin, KTransformers seamlessly enhances the performance of HuggingFace Transformers, making it possible to operate large 671B MoE models or extremely long 1M context locally with promising speed. KTransformers is a Python-centric framework designed with extensibility at its core. By implementing and injecting an optimized module with a single line of code, users gain access to a…
2025 · github.com
- 24BT
Hi everybody, this was my project to learn Zig and RISC-V+x86_64 assembly. Not sure if anybody is actually interested in yet another Brainfuck compiler, so I'll just write up some random things I learned while building it! - A primitive assembly stitching compiler is 10x faster than the interpreter. Did not expect that. - The generated x86 code is really bad (e.g. it always uses 6 or 7 byte sized instructions with 32-bit immediates when there are much smaller ones) but it doesn't really matter. Good code generated by GCC and clang for transpiled Brainfuck->C is not much faster as it's…
2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →