Capillaries: Distributed data processing with Go and Cassandra
I started thinking about this approach after working on a large-scale project for a major financial company where our group developed a distributed in-house data processing solution. On a regular basis, it ingested a few gigabytes of financial data and, within a tight SLA time limit, produced a lot of enriched/aggregated/validated data for a number of customers. Sometimes, source data had errors, so operators with domain knowledge had to verify data validity at some checkpoints, immediately make corrections, and re-run parts of the workflow manually. The solution involved complex…
In plain words
Capillaries is an open-source distributed data processing framework built with Go and Cassandra. It ingests data and applies user-defined transformations through Go expressions, Python formulas, joins, and aggregations. The system handles large-scale financial data processing with built-in checkpoints that allow operators to verify data validity, make corrections, and selectively re-run workflow portions without reprocessing entire datasets. It is designed for teams requiring reliable, SLA-compliant data pipelines with human-in-the-loop validation capabilities.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
I started thinking about this approach after working on a large-scale project for a major financial company where our group developed a distributed in-house data processing solution. On a regular basis, it ingested a few gigabytes of financial data and, within a tight SLA time limit, produced a lot of enriched/aggregated/validated data for a number of customers. Sometimes, source data had errors, so operators with domain knowledge had to verify data validity at some checkpoints, immediately make corrections, and re-run parts of the workflow manually. The solution involved complex web service orchestration, custom database and was very demanding on the infrastructure availability. Capillaries is a built from scratch, open-source Go solution that does just that: ingests data and applies user-defined transforms - Go one-liner expressions, Python formulas, joins, aggregations, denormalization - using Cassandra for intermediate data storage and RabbitMQ for task scheduling. End users just have to provide: - source data in CSV files; - Capillaries script (JSON file) that defines the workflow and the transforms; - Python code that performs complex calculations (only if needed). The whole data processing pipeline can be split into separate runs that can be started independently and re-run by the user if needed. The goal is to build a platform that is tolerant to database and processing node failures, and allows users to focus on data transform logic and data quality control. “Getting started” Docker-based demo calculates ARK funds performance, using EOD holdings and transactions data acquired from public sources. There are also integration tests that use non-financial data. There is a test deploy tool that uses Openstack API for provisioning in the cloud.
More work this month
the category →
The app store for voice native apps that lives in your notch
Work · 28d ago · voiceos.com
The New Calendly▲211Handle all of the work before, during, and after meetings
Work · 17d ago · calendly.com
Launched alongside, May 2023
the whole month →
- BR
In today's world, catchy headlines and articles often distract readers from the facts and relevant information. By utilizing OpenAI's language models, Boring Report processes sensationalist news articles, transforms them into the content you see, and helps readers focus on the essential details. We recently updated our iOS app experience, so any and all feedback would be appreciated. App Link: https://apps.apple.com/us/app/boring-report-news-by-ai/id644...
AI · 2023 · boringreport.org
Momentum Page▲834Launch your website in seconds, get users in minutes
Dev tools · 2023 · page.mmntm.build