nowfound

Life & fun · January 15, 2026

The Hessian of tall-skinny networks is easy to invert

It turns out the inverse of the Hessian of a deep net is easy to apply to a vector. Doing this naively takes cubically many operations in the number of layers (so impractical), but it's possible to do this in time linear in the number of layers (so very practical)! This is possible because the Hessian of a deep net has a matrix polynomial structure that factorizes nicely. The Hessian-inverse-product algorithm that takes advantage of this is similar to running backprop on a dual version of the deep net. It echoes an old idea of Pearlmutter's for computing Hessian-vector products. Maybe this…

The Hessian of tall-skinny networks is easy to invert launched on January 15, 2026 with 31 votes, #630 of 14,584 launches that month and more than 95% of that year's launches.

Visit github.com ↗Alternativestop 4% of January 2026

In plain words

This is a computational method for efficiently applying the inverse Hessian to vectors in deep neural networks. Rather than the impractical cubic-time approach, the algorithm exploits the matrix polynomial structure of the Hessian to achieve linear time complexity in the number of layers. The method works similarly to running backpropagation on a dual network representation and builds on classical ideas for computing Hessian-vector products. Researchers working on neural network optimization may find this useful as a preconditioner for stochastic gradient descent training.

written from the facts on this page · September 2026

From the sources

The Hessian of tall-skinny networks is easy to invert - a-rahimi/hessian

This package shows how to multiply the inverse of the Hessian of a deep network with a vector. If the Hessian-vector product is $H v$ for some fixed vector $v$ , we're interested in solving $H x = v$ for $x$ . The hope is to soon use this as a preconditioner to speed up stochastic gradient descent. Pearlmutter showed a clever way to compute the Hessian-vector-product for a deep net. By contrast, the paper and code in this repo shows how to compute the Hessian-inverse-product, the product of the inverse of the Hessian of a deep net with a vector. Solving this system naively requires a number of operations that scales cubically with the number of parameters in the deep net, which is…from github.com

In the maker’s words, at launch

It turns out the inverse of the Hessian of a deep net is easy to apply to a vector. Doing this naively takes cubically many operations in the number of layers (so impractical), but it's possible to do this in time linear in the number of layers (so very practical)! This is possible because the Hessian of a deep net…

Does a similar job

all alternatives →
  • I invented a new generative model and got accepted to ICLR2025 · discrete-distribution-networks.github.io · ▲656

    I invented Discrete Distribution Networks, a novel generative model with simple principles and unique properties, and the paper has been accepted to ICLR2025! Modeling data distribution is challenging; DDN adopts a simple yet fundamentally different approach compared to mainstream generative models (Diffusion, GAN, VAE, autoregressive model): 1. The model generates multiple outputs simultaneously in a single forward pass, rather than just one output. 2. It uses these multiple outputs to approximate the target distribution of the training data. 3. These outputs together represent a discrete…

  • Boldly go where Gradient Descent has never gone before with DiscoGrad2024 · github.com · ▲232

    Trying to do gradient descent using automatic differentiation over branchy programs? Or to combine them with neural networks for end-to-end training? Then this might be interesting to you. We develped DiscoGrad, a tool for automatic differentiation through C++ programs involving input-dependent control flow (e.g., "if (f(x) < c) { ... }", differentiating wrt. x) and randomness. Our initial motivation was to enable the use of gradient descent with simulations, which often rely heavily on such discrete branching. The latter makes plain autodiff mostly useless, since it can only account for the…

  • Neural Network Evolution Playground with Backprop NEAT2016 · blog.otoro.net · ▲73
  • Neural Networks and Deep Learning2015 · neuralnetworksanddeeplearning.com · ▲95

    Core concepts behind neural networks and deep learning

  • OH
    Optimizing Higher Order Functions with Hypothetical Inverses2016 · medium.com · ▲55
  • A radically more scalable alternative to Bayesian Optimization2018 · github.com · ▲96

    Experimental Global Optimization Algorithm. Contribute to chrisstroemel/Simple development by creating an account on GitHub.

More life & fun this month

the category →

Source: Show HN post ↗ · launch text quoted in part

Launched alongside, January 2026

the whole month →
  • IN
    isometric.nyc – giant isometric pixel art map of NYC▲1,325

    Hey HN! I wanted to share something I built over the last few weeks: isometric.nyc is a massive isometric pixel art map of NYC, built with nano banana and coding agents. I didn't write a single line of code. Of course no-code doesn't mean no-engineering. This project took a lot more manual labor than I'd hoped! I wrote a deep dive on the workflow and some thoughts about the future of AI coding and creativity: http:&#x2F;&#x2F;cannoneyed.com&#x2F;projects&#x2F;isometric-nyc

    AI · Jan 2026 · cannoneyed.com

  • Cowork▲1,088

    Turn Claude into your digital coworker

    AI · Jan 2026 · claude.com

  • DataFast▲867

    Revenue-first analytics

    Growth · Jan 2026 · datafa.st

  • OpenClaw▲836

    The AI that actually does things

    AI · Jan 2026 · openclaw.ai

  • Kilo Code Reviewer▲799

    Automatic AI-powered code reviews the moment you open a PR

    Dev tools · Jan 2026 · kilo.ai

  • PostSyncer▲793

    AI Content Maker, for Social Media Publishing

    AI · Jan 2026 · postsyncer.com