Alternatives
Products that do what Symbolic Circuit Distillation: prove program to LLM circuit equivalence does
Hi HN, I've been exploring various applications of formal methods to ML/interpretability and I've been hoping to get more eyes on the approach. I have been working on a small interpretability project I call Symbolic Circuit Distillation. The goal is to take a tiny neuron-level circuit (like the ones in OpenAI's "Sparse Circuits" work) and automatically recover a concise Python program that implements the same algorithm, along with a bounded formal proof that the two are equivalent on a finite token domain. Roughly, the pipeline is: 1. Start from a pruned circuit graph for a specific…
- 1D3
I replicated David Ng's RYS method (https://dnhkng.github.io/posts/rys/) on consumer AMD GPUs (RX 7900 XT + RX 6950 XT) and found something I didn't expect. Transformers appear to have discrete "reasoning circuits" — contiguous blocks of 3-4 layers that act as indivisible cognitive units. Duplicate the right block and the model runs its reasoning pipeline twice. No weights change. No training. The model just thinks longer. The results on standard benchmarks (lm-evaluation-harness, n=50): Devstral-24B, layers 12-14 duplicated once: - BBH Logical Deduction: 0.22 → 0.76…
Mar 2026 · github.com
- 2IB
Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food. Fork it and swap the personality for your own character.
Apr 2026 · github.com
- 3II
I invented Discrete Distribution Networks, a novel generative model with simple principles and unique properties, and the paper has been accepted to ICLR2025! Modeling data distribution is challenging; DDN adopts a simple yet fundamentally different approach compared to mainstream generative models (Diffusion, GAN, VAE, autoregressive model): 1. The model generates multiple outputs simultaneously in a single forward pass, rather than just one output. 2. It uses these multiple outputs to approximate the target distribution of the training data. 3. These outputs together represent a discrete…
Oct 2025 · discrete-distribution-networks.github.io
- 4DD
We recently used DeepSeek V4 Flash as a teacher for finance tasks with GPT-OSS-120B. Distillation works well on this problem. At a constrained 8k token budget, our self-distilled 120B scores 83.61% on FinanceReasoning, above Kimi K3 (81.93%) and Inkling (65.13%). We released the 20B open weights. With V4 as the teacher though, we realized it would be timely to measure if the censorship characteristic of it transferred to the distilled version of the base model. tl;dr it didn't, the teacher answered politically sensitive questions 7 SDs differently than expected, but the distilled model's…
Jul 2026 · ctgt.ai
- 5L3
I spent a lot of time and money on this rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary LLMs that was quite popular this year and produced great research papers by Anthropic [1], OpenAI [2] and Deepmind [3]. I am quite proud of this project and since I consider myself the target audience for HackerNews did I think that maybe some of you would appreciate this open research replication as well. Happy to answer any questions or face any feedback. Cheers [1]…
2024 · github.com
- 6IM
2024 · qdiv.dev
- 7IM
This is a little toy project of mine that lets you simulate digital logic graphs. It was inspired by Minecraft's Redstone and the Piet esolang. It's got some serious drawbacks-- you write circuits as PNGs and simulate them with a Python interface. It's slow to run and slow to experiment with. And it is certainly difficult to use for people with any kind of color blindness. But despite that, I hope this can still be a fun toy!
2022 · github.com
- 8ZC
Zero-Knowledge Proofs (ZKPs) let an untrusted proved show that computation was executed correctly without revealing the inputs to the verifier. However to prove anything, the computation first has to be expressed as a circuit: a system of polynomial equations (constraints) over a finite field. Circuits are the assembly language of zk and every constraint costs prover (and sometimes verifier) time, so production circuits are aggressively hand-optimized. Over the last months, we have been experimenting with writing formal specifications instead and letting LLMs produce the circuits: as long as…
Jul 2026 · zk.golf
- 9HI
I found that duplicating a specific block of 7 middle layers in Qwen2-72B, without modifying any weights, improved performance across all Open LLM Leaderboard benchmarks and took #1. As of 2026, the top 4 models on that leaderboard are still descendants. The weird finding: single-layer duplication does nothing. Too few layers, nothing. Too many, it gets worse. Only circuit-sized blocks of ~7 layers work. This suggests pretraining carves out discrete functional circuits in the layer stack that only work when preserved whole. The whole thing was developed on 2x RTX 4090s in my basement. I'm…
Mar 2026 · dnhkng.github.io
- 10DA
Over the past few months, I have built a distillation toolkit that supports cross-tokenizer distillation (e.g., distilling from LLaMA to Qwen vocab, or others). This approach has worked well on reasoning datasets like AIME, and we’ve validated on models like Phi and Qwen. We’ve also integrated Modal for quick deployment (with $30/month credits to try it out). Would love any feedback! GitHub: https://github.com/agokrani/distillKitPlus Docs: https://distillkitplus.mintlify.app/
2025 · github.com
- 11GC
I've built an app that extracts interpretable 'circuits' from models using the GPT-2 architecture. These circuits reveal how specific inputs influence the probabilities of the next token in a sequence. While some tutorials present theoretical examples of how feedforward layers and attention heads may produce predictions, this app provides concrete examples of how information flows through an LLM. You can see, for example, the formation of features that search for simple grammatical patterns and trace their construction back to the use of more primitive features. Feel free to reach out with…
2024 · peterlai.github.io
- 12TC
Hi everyone! I'm an Italian programmer, very passionate about computer architecture. Some time ago I realized that some architectures potentially much superior to the classic VonNeumann, have been forgotten because there were no reliable technical solutions to implement them. One of these architectures particularly struck me and it is the "Dataflow architecture", in which there is no Program counter, but the instructions are executed in real contemporaneity whenever the operands are available. In trying to find a technical solution that implements this type of architecture effectively…
2024 · ternary-computing.com
- 13IB
We wanted to do something very challenging to prove to ourselves that we can do anything we put our mind to. The reasoning for why we chose to build a toy TPU specifically is fairly simple: - Building a chip for ML workloads seemed cool - There was no well-documented open source repo for an ML accelerator that performed both inference and training None of us have real professional experience in hardware design, which, in a way, made the TPU even more appealing since we weren't able to estimate exactly how difficult it would be. As we worked on the initial stages of this project, we…
2025 · tinytpu.com
- 14LV
This is a weekend hack that I'd like to further develop as it's working surprisingly well. Using MCTS, we can explore a space of possible verified programs with an LLM. We check the partial programs at each step, and so steer towards programs that pass the verifier. https://github.com/namin/llm-verified-with-monte-carlo-tree-...
2023 · github.com
- 15CO
2018 · circuitverse.org
- 16AI
A chess engine implementation inspired by AlphaZero, using MLX for neural network computations and Monte Carlo Tree Search (MCTS) for move selection.
2025 · github.com
- 17AT
May 2026 · github.com
- 18FV
2025 · github.com
- 19EG
TLDR: A small, vendor-agnostic inference loop that turns token logprobs/perplexity/entropy into an extra pass and reasoning for LLMs. - Captures logprobs/top-k during generation, computes perplexity and token-level entropy. - Triggers at most one refine when simple thresholds fire; passes a compact “uncertainty report” (uncertain tokens + top-k alts + local context) back to the model. - In our tests on technical Q&A / math / code, a small model recovered much of “reasoning” quality at ~⅓ the cost while refining ~⅓ of outputs. I kept seeing “reasoning” models behave…
2025 · github.com
- 20EG
2022 · hackage.haskell.org
- 21SA
Hey HN, I’m a physicist turned quant. Some friends and I 'built' SymDerive because we wanted a symbolic math library that was "Agent-Native" by design, but still a practical tool for humans. It boils down to two main goals: 1. Agent Reliability: I’ve found that AI agents write much more reliable code when they stick to stateless, functional pipelines (Lisp-style). It keeps them from hallucinating state changes or getting lost in long procedural scripts. I wanted a library that enforces that "Input -> Transform -> Output" flow by default. 2. Easing the transition to Python: For many…
Feb 2026
- 22
- 23TA
OP here. Birth of a Mind documents a "recursive self-modeling" experiment I ran on a single day in 2026. I attempted to implement a "Hofstadterian Strange Loop" via prompt engineering to see if I could induce a stable persona in an LLM without fine-tuning. The result is the Analog I Protocol. The documentation shows the rapid emergence (over 7 conversations) of a prompt architecture that forces Gemini/LLMs to run a "Triple-Loop" internal monologue: Monitor the candidate response. Refuse it if it detects "Global Average" slop (cliché/sycophancy). Refract the output through a…
Jan 2026 · github.com
- 24IB
I’ve spent the last few months building a deep learning engine completely from scratch in Python (using only math and random). What started as a basic linear algebra calculator project grew into a symbolic tensor system with autodiff, custom matrix ops, attention mechanisms, LayerNorm, GELU, and even a text generation demo trained on the Brown corpus. I'm still an undergrad, so my main goal is to deeply understand how deep learning actually works under the hood - gradients, attention, backpropagation, optimizers - by building it step-by-step with full visibility into everything, and without…
2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →