Forkrun – NUMA-aware shell parallelizer (50×–400× faster than parallel)
forkrun is the culmination of a 10-year-long journey focused on "how to make shell parallelization fast". What started as a standard "fork jobs in a loop" has turned into a lock-free, CAS-retry-loop-free, SIMD-accelerated, self-tuning, NUMA aware shell-based stream parallelization engine that is (mostly) a drop-in replacement for xargs -P and GNU parallel. On my 14-core/28-thread i9-7940x, forkrun achieves: * 200,000+ batch dispatches/sec (vs ~500 for GNU Parallel) * ~95–99% CPU utilization across all 28 logical cores, even when the workload is non-existant (bash no-ops / `:`)…
In plain words
Forkrun is a shell parallelization tool that runs commands across multiple processors faster than traditional alternatives like GNU Parallel and xargs. Built with NUMA awareness, lock-free design, and SIMD optimization, it functions as a drop-in replacement for existing parallelization tools. It is designed for users who need to process large batches of shell commands efficiently on multi-core systems, achieving significantly higher throughput and CPU utilization on batch processing workloads.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
forkrun is the culmination of a 10-year-long journey focused on "how to make shell parallelization fast". What started as a standard "fork jobs in a loop" has turned into a lock-free, CAS-retry-loop-free, SIMD-accelerated, self-tuning, NUMA aware shell-based stream parallelization engine that is (mostly) a drop-in replacement for xargs -P and GNU parallel. On my 14-core/28-thread i9-7940x, forkrun achieves: * 200,000+ batch dispatches/sec (vs ~500 for GNU Parallel) * ~95–99% CPU utilization across all 28 logical cores, even when the workload is non-existant (bash no-ops / `:`) (vs ~6% for GNU Parallel). These benchmarks are intentionally worst-case (near-zero work per task) because they measure the capability of the parallelization framework itself, not how much work an external tool can do. * Typically 50×–400× faster on real high-frequency low-latency workloads (vs GNU Parallel) A few of the techniques that make this possible: * Born-local NUMA: stdin is splice()'d into a shared memfd, then pages are placed on the target NUMA node via set_mempolicy(MPOL_BIND) before any worker touches them, making the memfd NUMA-spliced. Each numa node only claims work that is already born-local on its node. Stealing from other nodes is permitted under some conditions when no local work exists. * SIMD scanning: per-node indexers/scanners use AVX2/NEON to find line boundaries (delimiters) at speeds approaching memory bandwidth, and publish byte-offsets and line-counts into per-node lock-free rings. * Lock-free claiming: workers claim batches with a single atomic_fetch_add — no locks, no CAS retry loops; contention is reduced to a single atomic on one cache line. * Memory management: a background thread uses fallocate(PUNCH_HOLE) to reclaim space without breaking the logical offset system. …and that’s just the surface. The implementation uses many additional systems-level techniques (phase-aware tail handling, adaptive batching, early-flush detection, etc.) to eliminate overhead, increase throughput and reduce latency at every stage. In its fastest (-b) mode (fixed-size batches, minimal processing), it can exceed 1B lines/sec. forkrun ships as a single bash file with an embedded, self-extracting C extension — no Perl, no Python, no install, full native support for parallelizing arbitrary shell functions. The binary is built in public GitHub Actions so you can trace it back to CI (see the GitHub "Blame" on the line containing the base64 embeddings). Trying it is literally two commands: . frun.bash frun shell_func_or_cmd < inputs For benchmarking scripts and results, see the BENCHMARKS dir in the GitHub repo For an architecture deep-dive, see the DOCS dir in the GitHub repo Happy to answer questions.
More life & fun this month
the category →- TL
Life & fun · 10d ago · louisabraham.github.io
Photosynthesis fires two of your iPhone
Life & fun · 28d ago · photosynthesis.camera
SoloUno▲310Take control of hair pulling, nail biting & skin picking
Life & fun · 28d ago · solouno.io
Scroll through all 43,252,003,274,489,856,000 reachable Rubik's Cube permutations.
Life & fun · 26d ago · everycube.alen.is
Hi HN, I built Eigendrum, a web tool that solves the 2D wave equation for arbitrary shapes so you can hear what they sound like as drums. How it works: * Solves -∇²u = λu using finite element analysis (Kφ = λMφ) on a triangle mesh. * Validated to <0.1% error against closed-form solutions for circles (Bessel zeros) and rectangles. * Sound model factors in strike location, Rayleigh damping, and mallet width. * Includes Kac drums I & II to demonstrate identical sound spectra from different geometries. * No frameworks, build steps, or dependencies. Repo and tests:…
Life & fun · 26d ago · baselashraf81.github.io
Launched alongside, March 2026
the whole month →
Switch from ChatGPT to Claude with import memory feature
AI · Mar 2026 · claude.com
