Pq – Simple, durable background tasks in Python using Postgres
At work we were using python-rq for background tasks. It does the job for simple things, but we kept bumping into limitations. We needed to schedule tasks hours / days out and trust they'd survive a restart. We wanted periodic tasks with proper overlap control. So we built a scheduling / enqueuing system around Postgres to bring these durability capabilities to python-rq. This worked fine for a while but was trickier to reason about due to its more complicated architecture (we'd run two separate services just for getting jobs from Postgres into the rq Redis queue, plus N actual…
What it does
In the maker’s words, at launch
At work we were using python-rq for background tasks. It does the job for simple things, but we kept bumping into limitations. We needed to schedule tasks hours / days out and trust they'd survive a restart. We wanted periodic tasks with proper overlap control. So we built a scheduling / enqueuing system around Postgres to bring these durability capabilities to python-rq. This worked fine for a while but was trickier to reason about due to its more complicated architecture (we'd run two separate services just for getting jobs from Postgres into the rq Redis queue, plus N actual task workers). pq is a simpler approach: It's a Postgres-backed background-task library for Python, using SELECT ... FOR UPDATE SKIP LOCKED for concurrency safety. You just run your N task workers, that's it. The stuff I think is worth knowing about: - Transactional enqueueing -- you can enqueue a task inside the same DB transaction as your writes. If the transaction rolls back, the task never exists. This is the thing Redis literally can't give you. Fork isolation -- every task runs in a forked child process. If it OOMs, segfaults, or leaks memory, the worker just keeps going. The parent monitors via os.wait4() and handles timeouts, OOM kills, and crashes cleanly. - Periodic tasks -- intervals or cron expressions, with overlap control (skip the tick if the previous run is still going), pause/resume without deleting the schedule. Priority queues -- five levels from BATCH to CRITICAL. You can dedicate workers to only process tasks with specific priorities. - Three tables in your main database schema: pq_tasks for one-off tasks, pq_periodic for periodic tasks and pq_alembic_version to track its own schema version (pq manages its own migrations). There's also client IDs for idempotency and correlation with other tables in your application DB, upsert for debouncing (only the latest version of a task runs), lifecycle hooks that execute in the forked child (useful for fork-unsafe stuff like OpenTelemetry), and async task support. What it won't do: high throughput (you're polling Postgres). If you need 10k+ tasks/sec or complex DAGs, use something else. For the kind of workload most apps actually have, it's probably sufficient. pip install python-pq // uv add python-rq repo at https://github.com/ricwo/pq, docs at https://ricwo.github.io/pq
Does the same job
all alternatives →- 1R1M rows/s from Postgres to Python2016 · magic.io · ▲296
- SSSPyQL – SQL with Python in the middle2022 · github.com · ▲105
SPyQL (https://github.com/dcmoura/spyql) is SQL with Python in the middle, an open-source project fully written in Python for making command-line data processing more intuitive, readable and powerful. Try mixing in the same pot: a SQL SELECT for providing the structure, Python expressions for defining transformations and conditions, the essence of awk as a data-processing language, and the JSON handling capabilities of jq. How does a SPyQL query looks like? $ spyql “ IMPORT pendulum AS p SELECT (p.now() - p.from_timestamp(purchase_ts)).in_days() AS days_ago, sum_agg(price…
- KSKQ – Simple Job Queue for Python Using Kafka2016 · github.com · ▲116
- DPDBOS Python 1.0 – Lightweight Durable Workflows Built on Postgres2025 · github.com · ▲6
Hi HN – this is Peter from DBOS here with Qian (qianli_cs) and Jeremy (jedberg). We’re building an open-source, lightweight durable workflows library on top of Postgres. Ever since we first launched on HN last year, we’ve been blown away by the support, feedback, and response we’ve received from the community. We've realized durable workflows are critical for everything from business processes to AI automation to data pipelines, but most existing durable orchestration tools are either too heavy or too complicated for most applications. Instead, we're building something lightweight, simple,…
- QAQue, a Ruby-PostgreSQL job queue with ~20x the throughput of DelayedJob2013 · ▲6
Hi everyone - I wrote a job queue called Que, which uses Postgres' advisory locks to manage jobs very efficiently. My goal is to make it easy to cover jobs with the same ACID guarantees as the rest of your data. It won't be as fast as a dedicated or Redis-backed queue, but I expect it will be fast enough for many use-cases. https://github.com/chanks/que The 20x figure comes from a benchmark I ran on an AWS c3.8xlarge instance - this number will change depending on your hardware. The code for the benchmark (and some more detailed results) is here:…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
