I embedded 685M public texts in 32 minutes (on 8x A100, Rust, TensorRT)
Quick note on how it works and how I've done my batch embedding engine IgniteMS. The whole thing runs as one process using Rust, reading input, tokenizing, packing batches, keeping the queue full. TensorRT handles inference. Python is only as a wrapper. I built it this way because when you use more than couple of GPUs, the GPUs stop being the problem. CPU cannot feed them fast enough. One A100 can go through batches faster than Python can tokenize and feed, so the GPU just sits there idle waiting for work. Most of my time went into optimizing this. At 8 GPUs that was basically the entire…
What it does
In the maker’s words, at launch
Quick note on how it works and how I've done my batch embedding engine IgniteMS. The whole thing runs as one process using Rust, reading input, tokenizing, packing batches, keeping the queue full. TensorRT handles inference. Python is only as a wrapper. I built it this way because when you use more than couple of GPUs, the GPUs stop being the problem. CPU cannot feed them fast enough. One A100 can go through batches faster than Python can tokenize and feed, so the GPU just sits there idle waiting for work. Most of my time went into optimizing this. At 8 GPUs that was basically the entire challenge. On cost. I ran the big 2B messages job on a spot p4d instance (8x A100 40GB). After filtering and dedupping I got 685M raw texts. With my new engine the whole production run finishes in about half an hour. Previously I used on-demand for these jobs, now switched to spots. If AWS reclaims the box, I just rerun it. It's roughly $7 for half-an-hour run. And at least right now spots are easier to get than on-demand. Open warning: it's batch only and NVIDIA only. You can use it both as a docker image and native. I used some optimizations for my production run. With default settings you can expect to see ~250K msg/sec if you run the benchmark script on your p4d box. https://github.com/Artain-AI/ignite-ms/blob/main/BENCHMARKIN... v1.1.0 added TensorRT 11 and 60 models, 23 tested on 1x and 4x A100. Happy to share details.
Does a similar job
all alternatives →- LALocalGPT – A local-first AI assistant in Rust with persistent memoryFeb 2026 · github.com · ▲331
I built LocalGPT over 4 nights as a Rust reimagining of the OpenClaw assistant pattern (markdown-based persistent memory, autonomous heartbeat tasks, skills system). It compiles to a single ~27MB binary — no Node.js, Docker, or Python required. Key features: - Persistent memory via markdown files (MEMORY, HEARTBEAT, SOUL markdown files) — compatible with OpenClaw's format - Full-text search (SQLite FTS5) + semantic search (local embeddings, no API key needed) - Autonomous heartbeat runner that checks tasks on a configurable interval - CLI + web interface + desktop GUI - Multi-provider:…
- SMSub-millisecond VM sandboxes using CoW memory forkingMar 2026 · github.com · ▲311
I wanted to see how fast an isolated code sandbox could start if I never had to boot a fresh VM. So instead of launching a new microVM per execution, I boot Firecracker once with Python and numpy already loaded, then snapshot the full VM state. Every execution after that creates a new KVM VM backed by a `MAP_PRIVATE` mapping of the snapshot memory, so Linux gives me copy-on-write pages automatically. That means each sandbox starts from an already-running Python process inside a real VM, runs the code, and exits. These are real KVM VMs, not containers: separate guest kernel, separate guest…
- TCTensorDock Core GPU Cloud – GPU servers from $0.29/hr2022 · tensordock.com · ▲147
Hello HN! I’m Jonathan from TensorDock. After 7 months in beta, we’re finally launching Core Cloud, our platform to deploy GPU virtual machines in as little as 45 seconds! https://www.tensordock.com/product-core Why? Training machine learning workloads at large clouds can be extremely expensive. This left us wondering, “how did cloud ever become more expensive than on-prem?” I’ve seen too many ML startups buy their own hardware. Cheaper dedicated servers with NVIDIA GPUs are not too hard to find, but they lack the functionality and scalability of the big clouds. We thought to…
- PAPython at the Speed of Rust2025 · blog.fxn.ai · ▲68
I’m sure many of you are familiar, but there’s a treacherous gap between finding (or building) a model that works in PyTorch, and getting that deployed into your application, especially in consumer-facing applications. I’ve been very interested in solving this problem with a great developer experience. Over time, I gradually realized that the highest-impact thing to have was a way to go from existing Python code to a self-contained native binary—in other words, a Python compiler. I was already pretty familiar with a successful attempt: when Apple introduced armv8 on the iPhone 5s, they…
More dev tools this month
the category →
The first open-source price index for GPU compute
Dev tools · 10d ago · getcomputable.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Open-source GTM skills for technical founders
Dev tools · 30d ago · gtmcofounder.com
Launched alongside, June 2026
the whole month →
Fundraisly▲1,544AI fundraising agent that finds investors and books meetings
AI · Jun 2026 · fundraisly.com
- H6Homebrew 6.0.0▲1,481
Today, I’m proud to announce Homebrew 6.0.0. The most significant changes since 5.1.0 are a new tap trust security mechanism, the new faster, smaller, default internal Homebrew JSON API, sandboxing on Linux, better defaults informed by our user survey, many brew bundle improvements, improved performance and initial support for macOS 27 (Golden Gate). Happy to discuss any questions here!
Dev tools · Jun 2026 · brew.sh
- PU
hope you enjoy
Life & fun · Jun 2026 · vorpus.github.io
- IM
Life & fun · Jun 2026 · hackernewstrends.com