Capillaries: Distributed data processing with Go and Cassandra
I started thinking about this approach after working on a large-scale project for a major financial company where our group developed a distributed in-house data processing solution. On a regular basis, it ingested a few gigabytes of financial data and, within a tight SLA time limit, produced a lot of enriched/aggregated/validated data for a number of customers. Sometimes, source data had errors, so operators with domain knowledge had to verify data validity at some checkpoints, immediately make corrections, and re-run parts of the workflow manually. The solution involved complex…
Capillaries: Distributed data processing with Go and Cassandra launched on May 16, 2023 with 70 votes, #1,248 of 4,910 launches that month and more than 90% of that year's launches. The site was still online in September 2026, 3 years on.
In plain words
Capillaries is an open-source distributed data processing framework built with Go and Cassandra. It ingests data and applies user-defined transformations through Go expressions, Python formulas, joins, and aggregations. The system handles large-scale financial data processing with built-in checkpoints that allow operators to verify data validity, make corrections, and selectively re-run workflow portions without reprocessing entire datasets. It is designed for teams requiring reliable, SLA-compliant data pipelines with human-in-the-loop validation capabilities.
written from the facts on this page · September 2026
From the sources
Capillaries - distributed, supervised data processing within SLA time limits, Go and Python
Capillaries excels in data-heavy, compute-intensive workloads that run periodically (daily, weekly, quarterly). It is designed to work efficiently on both private and public VM or container infrastructure, which can be: Capillaries is definitely "some-code" because data transformation rules may involve Go expressions and/or complex Python formulas . The "code" part applies only to the business logic, while the "orchestration" part does not require any coding at all. Capillaries handles orchestration, scalability , and intermediate data storage, so you can focus entirely on your transformation logic. and a browser to use the Capillaries UI or a REST API client to call Capillaries Webapi…from capillaries.io
In the maker’s words, at launch
I started thinking about this approach after working on a large-scale project for a major financial company where our group developed a distributed in-house data processing solution. On a regular basis, it ingested a few gigabytes of financial data and, within a tight SLA time limit, produced a lot of…
Does a similar job
all alternatives →
I built an open-source data pipeline tool in Go2024 · github.com · ▲200Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…
Distributed MapReduce framework in Go2015 · github.com · ▲25
Turbolite – a SQLite VFS serving sub-250ms cold JOIN queries from S3Mar 2026 · github.com · ▲185I built a SQLite VFS in Rust that serves cold queries directly from S3 with sub-second performance, and often much faster. It’s called turbolite. It is experimental, buggy, and may corrupt data. I would not trust it with anything important yet. I wanted to explore whether object storage has gotten fast enough to support embedded databases over cloud storage. Filesystems reward tiny random reads and in-place mutation. S3 rewards fewer requests, bigger transfers, immutable objects, and aggressively parallel operations where bandwidth is often the real constraint. This was explicitly inspired…
A benchmark for analytical databases (Snowflake, Druid, Redshift)2022 · benchmark.clickhouse.com · ▲137I created a web page to compare different analytical databases (both self-managed and services, open-source and proprietary) on a realistic dataset. It contains 20+ databases, each with installation and data loading scripts. And they can be compared to each other on a set of 43 queries, by data load time or by storage size. There are switches to select different types of databases for comparison - for example, only MySQL compatible or PostgreSQL compatible. If you play with the switches, many interesting details will be uncovered. Full description:…
More work this month
the category →
Endless lofi beats composed live in your browser, over pixel-art nights in Paris, Tokyo, New York, Hong Kong, Sydney, San Francisco and more. Free, no sign-up.
Work · 24d ago · loficities.com
Smooth Recorder▲257Mac screen recordings that look edited when you hit stop
Work · 18d ago · smoothrecorder.com
Work · 12d ago · readrare.com
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, May 2023
the whole month →
In today's world, catchy headlines and articles often distract readers from the facts and relevant information. By utilizing OpenAI's language models, Boring Report processes sensationalist news articles, transforms them into the content you see, and helps readers focus on the essential details. We recently updated our iOS app experience, so any and all feedback would be appreciated. App Link: https://apps.apple.com/us/app/boring-report-news-by-ai/id644...
AI · 2023 · boringreport.org
Momentum Page▲834Launch your website in seconds, get users in minutes
Dev tools · 2023 · page.mmntm.build
