Alternatives
Products that do what The Hessian of tall-skinny networks is easy to invert does
It turns out the inverse of the Hessian of a deep net is easy to apply to a vector. Doing this naively takes cubically many operations in the number of layers (so impractical), but it's possible to do this in time linear in the number of layers (so very practical)! This is possible because the Hessian of a deep net has a matrix polynomial structure that factorizes nicely. The Hessian-inverse-product algorithm that takes advantage of this is similar to running backprop on a dual version of the deep net. It echoes an old idea of Pearlmutter's for computing Hessian-vector products. Maybe this…
- 1II
I invented Discrete Distribution Networks, a novel generative model with simple principles and unique properties, and the paper has been accepted to ICLR2025! Modeling data distribution is challenging; DDN adopts a simple yet fundamentally different approach compared to mainstream generative models (Diffusion, GAN, VAE, autoregressive model): 1. The model generates multiple outputs simultaneously in a single forward pass, rather than just one output. 2. It uses these multiple outputs to approximate the target distribution of the training data. 3. These outputs together represent a discrete…
Oct 2025 · discrete-distribution-networks.github.io
- 2BG
Trying to do gradient descent using automatic differentiation over branchy programs? Or to combine them with neural networks for end-to-end training? Then this might be interesting to you. We develped DiscoGrad, a tool for automatic differentiation through C++ programs involving input-dependent control flow (e.g., "if (f(x) < c) { ... }", differentiating wrt. x) and randomness. Our initial motivation was to enable the use of gradient descent with simulations, which often rely heavily on such discrete branching. The latter makes plain autodiff mostly useless, since it can only account for the…
2024 · github.com
- 3CT
2018 · github.com
- 4
- 5TW
Apr 2026 · github.com
- 6OH
2016 · medium.com
- 7AR
2018 · github.com
- 8SA
2018 · github.com
- 9MA
I trained a transformer in HyperCard. 1,216 parameters. 1989 Macintosh. And yes, it took a while. MacMind is a complete transformer neural network, embeddings, positional encoding, self-attention, backpropagation, and gradient descent, implemented entirely in HyperTalk, the scripting language Apple shipped with HyperCard in 1987. Every line of code is readable inside HyperCard's script editor. Option-click any button and read the actual math. The task: learn the bit-reversal permutation, the opening step of the Fast Fourier Transform. The model has no formula to follow. It discovers the…
Apr 2026 · github.com
- 10NN
2016 · blog.otoro.net
- 11AF
2024 · github.com
- 12AI
A chess engine implementation inspired by AlphaZero, using MLX for neural network computations and Monte Carlo Tree Search (MCTS) for move selection.
2025 · github.com
- 13AG
This is a vector index I built that supports insertion and k-nearest neighbors (k-NN) querying, optimized for GPUs. It operates entirely in CUDA and can process queries on half a billion vectors in under 200 milliseconds. The codebase is structured as a standalone library with an HTTP API for remote access. It’s intended for high-performance search tasks—think similarity search, AI model retrieval, or reinforcement learning replay buffers. The codebase is located at https://github.com/rodlaf/BinaryGPUIndex.
2025 · rlafuente.com
- 14DL
2019 · github.com
- 15NN
2020 · github.com
- 16IW
2024 · gitlab.com
- 17AV
2016 · jalammar.github.io
- 18NN
Hi HN. Peter here. As a machine learning engineer, I mostly think in terms of feature vectors, embeddings, and matrices. One of the most useful byproducts of deep neural networks is embeddings because they allow us to represent high-dimensional data in terms of lower-dimensional latent vectors. These feature vectors can be used for downstream applications like similarly search, recommendation systems and near duplicate detection. As an ML engineer, I was frustrated by the lack of a datastore in which vectors are first-class citizens. As a result, most ML engineers, including myself, end up…
2021
- 19HI
2025 · github.com
- 20ZA
This is a low-level opensource library I developed for my own use and decided to share, as it makes it possible to process large checkpoints of neural networks without renting high-RAM instances, on a regular PC. It replaces torch.load() with a custom function that produces a dictionary that materializes tensors on the fly. Compared to other solutions it doesn't require sharding or re-encoding checkpoints and uses them completely as-is. It is a foundation to make it possible to run inference and compress language models and other large models one layer at a time - in principle, even one…
2023 · github.com
- 21HH
2018 · github.com
- 22LN
2015 · github.com
- 23LA
2014 · github.com
- 24IB
I’ve spent the last few months building a deep learning engine completely from scratch in Python (using only math and random). What started as a basic linear algebra calculator project grew into a symbolic tensor system with autodiff, custom matrix ops, attention mechanisms, LayerNorm, GELU, and even a text generation demo trained on the Brown corpus. I'm still an undergrad, so my main goal is to deeply understand how deep learning actually works under the hood - gradients, attention, backpropagation, optimizers - by building it step-by-step with full visibility into everything, and without…
2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →