Alternatives
Products that do what Buquet – Durable queues and workflows using only S3 does
buquet (bucket queue) is a queue and workflow orchestration tool using only S3-compatible* object storage. S3 is the control plane making it much simpler than alternatives. This does come with tradeoffs (see docs), but I do believe there is a niche it can serve well. https://horv.co/buquet.html https://github.com/h0rv/buquet * see https://github.com/h0rv/buquet/blob/main/docs/guides/s3-comp...
- 1
- 2PI
Hi HN! I’m Alex from Parabola (https://parabola.io). Parabola is a visual programming tool for creating functional data flows that everyone can use. It’s entirely drag-and-drop, handles data sizes much larger than a traditional spreadsheet, calculates everything live, and can run your flows on a schedule of your choosing. I used to work in strategy consulting, doing data analytics for SMBs and Fortune 500 companies. The amount of time wasted on menial tasks was astounding. Things like cleaning data, generating custom reports, creating human workflows to solve shortcomings in third…
2018 · parabola.io
- 3
- 4SL
2017 · github.com
- 5KS
2016 · github.com
- 6
- 7
- 8AS
2017 · blog.minio.io
- 9KD
Agent runs often fail after expensive model calls and executing tools that have real-world side effects. This problem is made even worse by how common it is to deploy agents to serverless environments. When your agent dies, it needs to be restarted, but doing so safely isn't easy and everyone building agents has to solve this same problem of durability. The stack you're running probably already has half of what you need for durable execution already though, ie, a queue or job runner that can invoke work at least once. kassette gives you the other half by journaling completed steps to object…
Jul 2026 · github.com
- 10LD
Hi HN! This is Qian here with Peter (KraftyOne) and Jeremy (jedberg). We’re building DBOS, an open-source, lightweight durable workflows library that you can add to Python apps in just a few lines of code. It’s comparable to popular open-source workflow and queue libraries like Airflow and Celery, but more lightweight with a greater focus on reliability and automatically recovering from failures. Our goal in building DBOS is to make workflows lightweight and flexible so you can add them to your existing apps with minimal work. Everything you need to run durable workflows and queues is…
2025 · github.com
- 11DP
Hi HN – this is Peter from DBOS here with Qian (qianli_cs) and Jeremy (jedberg). We’re building an open-source, lightweight durable workflows library on top of Postgres. Ever since we first launched on HN last year, we’ve been blown away by the support, feedback, and response we’ve received from the community. We've realized durable workflows are critical for everything from business processes to AI automation to data pipelines, but most existing durable orchestration tools are either too heavy or too complicated for most applications. Instead, we're building something lightweight, simple,…
2025 · github.com
- 12PB
Hi HN - I’m Peter, here with Max (hmaxdml), and we’re building DBOS Go, an open-source Go library for durable workflows, backed by Postgres. https://github.com/dbos-inc/dbos-transact-golang DBOS workflows make your programs durable by automatically checkpointing their state to Postgres. If your program crashes or fails, all workflows seamlessly resume from their last completed step when your program restarts. This durability makes workflows useful for solving many different problems, including: - Operating an AI agent, or anything that connects to an unreliable or…
Sep 2025 · github.com
- 13H1
I started 6 months ago when I wanted to look inside datasets from Hugging Face. I was not satisfied with existing libraries. So, naturally, I built my own library from scratch. Parquet is a very complicated format. It has 22 data types, 9 encodings, 8 compression codecs. Previous parquet js libraries went abandoned due to the complexity. However, I can confidently say that Hyparquet is now the most conformant parquet parser in existence. It can open more files than PyArrow and DuckDB. I dare you to find a file that Hyparquet can’t open! In addition to supporting all the parquet files,…
2024 · github.com
- 14AS
Hello, everyone. Three months ago, I started building my first GoLang package, VarMQ. In this short period, I gained a significant amount of traction from the community, and it resulted in over 140 stars on GitHub. A week ago, to check how it performs, I ran benchmarks with a similar package, Pond, which has been managed for years. Pond is not a message queue, though; however, it can be compared to VarmQ since it shares some similarities, except for its storage system. From the benchmarks, I got surprising results, which are that Varmq is taking 50%+ less memory than Pond and also has good…
2025 · github.com
- 15SL
At Arcesium, we needed a way to run SQL-driven CRUD operations, manage slowly-changing dimensions, and handle schema evolution on our cloud data lakes—without the overhead of spinning up distributed clusters or relying on vendor-managed query services. That’s why we built SwiftLake, a lightweight open-source library that brings Apache Iceberg and DuckDB together: - Lightweight & cluster-free: Runs as a single Java service with embedded DuckDB—no distributed infra required. - Full CRUD and SCD1/SCD2 support: Reads, inserts/appends, updates, deletes, and merges with ACID guarantees.…
Sep 2025 · github.com
- 16SS
I'm developing a storage system for versioning data at the subfile level, especially well suited for SSDs due to its log-structured COW nature. It implements a novel versioning algorithm called sliding snapshot, a diff-algorithm which makes use of our stable record-identifiers and optionally hashes, another diff algorithm for importing similar XML-documents as a versioned resource as well as novel XPath axis to navigate not only in space, but also in time. Recently, I've implemented a higher level, asynchronous REST-API with Kotlin (Coroutines) and Vert.x in a seperate module. The system is…
2018
- 17BB
Hi! I've been working on Biq Blue, a tool engineered to analyze your Google BigQuery tables, storage, and requests with the goal of drastically reducing your costs. Currently in early free beta, Biq Blue has already demonstrated its effectiveness on some big data sets. Essentially, it's a server that connects to your BigQuery database via the gcloud CLI, conducts analyses, and opens an HTTP port to serve both results and recommendations over web pages. Your data stays local, ensuring it never leaves your enterprise (I may only collects anonymous usage statistics and the email tied to your…
2023 · biq.blue
- 18QF
Hi HN, I’m sharing the alpha release of S2C, a state machine replication system built atop S3. The goal is to enable a distributed application to maintain consistent state without needing a quorum of nodes for availability or consistency. The idea came from a side project that was using S3 and where I needed strongly consistent distributed state but wanted to avoid adding a separate consensus dependency. I initially tried to use S3 directly for coordination, but it became messy. Eventually, I realized I need a replicated state machine with a deterministic log, and then it ended up as a…
Jan 2026 · github.com
- 19DS
DeltaGlider is a CLI/SDK similar to `aws s3` or `boto3`. UPLOAD: It stores the first file in a S3 path as a full-size (reference), but saves next uploaded archives as deltas (tiny binary diffs) with respect to the reference. DOWNLOAD: it reconstructs the original file on the fly, bit-perfect and verified with SHA256. Why Xdelta3? It's a compression-aware and block-level binary diff algorithm. Perfect for representing differences between archives, where small changes shift bytes but most content stays the same. It can efficiently delta compress ZIP/JAR/TAR archives up to 99.9%…
Nov 2025 · github.com
- 20WH
We have been working on tools for safe debloating of software for a few years now at the university. BLAFS is one of our first tools to release for file debloating, a bloat-aware filesystem for container debloating. It detects the files used by the container, and the debloats the container removing the unused files. The debloated containers are fully functional and can run the same workload as the original containers, but with a much smaller size and faster deployment. Check the paper for more details: https://arxiv.org/abs/2305.04641
2025 · github.com
- 21RW
Hey HN, I'm Anurag, founder and CEO of Render, a cloud for application developers. We've just launched Render Workflows, a way to define durable tasks by decorating plain TypeScript or Python functions and running them on Render without operating queues, worker pools, retry logic, and state management. Code example below [1]. The use cases we have in mind include agent loops, ETL/data pipelines, billing flows, and other long-running jobs. The model is simple: mark an existing function as a Task, deploy the repo as a Workflow, then trigger runs from your application code or via our API.…
Apr 2026 · render.com
- 22ZC
I got tired of wrestling with AWS Cost & Usage Reports, so I wrote an open-source script: point it at a folder of Parquet CUR files, run one command, and get fully interactive Rill dashboards that adapt to your tags and cost drivers. Highlights Clone to custom dashboards in < 5 min Ingests many CUR exports at once—even with schema drift. Unifies schemas. Flattens MAP-type cols like resource tags automatically. Chooses chart types programmatically based on spend coverage, cardinality, and distibution. Generates Rill YAML (source, metrics (including measures), canvases) so you can tweak…
2025
- 23AD
I'd like to share a project I've been working on for the past few months. It's a distributed workflow engine written entirely in Go. Some highlights: * Tasks are executed in a Docker container * Can run stand-alone or distributed * Highly extensible * Able to enforce limits (CPU/RAM) per task * Web UI Would love the get your feedback on it, and find out if this could be useful.
2023 · github.com
- 24SE
Introducing sqlxport: export SQL query results to Parquet/CSV in seconds — and optionally upload to S3 or MinIO. Works with PostgreSQL & Redshift CLI-first, cloud-friendly Great for data pipelines and engineers
2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →