Dev tools · September 30, 2024
Pipet – CLI tool for scraping and extracting data online, with pipes
I often find myself in situations where I need to extract some data from a website, either as one time thing or periodically (and then watch for changes). Maybe I'm tracking stocks, or a delivery, or I want to know when tickets become available to the local sauna; maybe a friend called and asked "can you get me all data from that website into a spreadsheet?" (this happens surprisingly often) I used to write one-off scripts for that, often with Python or JavaScript, but I noticed I spend about one minute getting the right CSS selectors, and then another 10 minutes setting the rest of the…
In plain words
Pipet is a command-line tool that extracts data from websites using either curl or playwright modes, eliminating repetitive boilerplate code. It supports one-time scraping tasks and periodic monitoring with change detection, allowing users to track prices, deliveries, ticket availability, or other data without writing custom scripts. The tool is designed for developers and non-technical users who need to quickly pull website data into spreadsheets or other formats.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
I often find myself in situations where I need to extract some data from a website, either as one time thing or periodically (and then watch for changes). Maybe I'm tracking stocks, or a delivery, or I want to know when tickets become available to the local sauna; maybe a friend called and asked "can you get me all data from that website into a spreadsheet?" (this happens surprisingly often) I used to write one-off scripts for that, often with Python or JavaScript, but I noticed I spend about one minute getting the right CSS selectors, and then another 10 minutes setting the rest of the script up. I decided to write Pipet so that I'll never need to write any of the boilerplate code again. By now I probably spent more time writing Pipet than all scrapers I have ever written, but hey, it's been fun… Pipet has two modes: curl and playwright. With curl, you can either type "curl http://news.ycombinator.com" or just copy-paste the request you want to duplicate from the browser. Pipet runs curl just like your shell does, so all headers and preferences work just the same. I found that this was super useful when trying to emulate a real browser or accessing something behind a login. With playwright mode, you just use JavaScript. If it worked in the devtools console, it should work with Pipet too. Some other features I think are useful is that you can output the results as text, JSON, or write a template file for the results to be rendered into. You can also run Pipet with an interval, and run a command when the data changes. Lastly, Pipet fully integrates UNIX pipes, so you can do stuff like `div#main h1 | wc -c` and it will take the h1 from a div with id “main”, and pipe its HTML to `wc -c`. It makes it extremely easily to just use tools you already know to process the data before Pipet outputs it. It also works when processing JSON - you can call jq or whatever you like to use to help with the processing. I’ve noticed I have written so many little scrapers since I have Pipet around because it became such an easy task, so hopefully others will find it useful too!
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, September 2024
the whole month →
BeforeSunset AI 2.0▲1,267Personalized AI daily planning that suits your life
AI · 2024 · beforesunset.ai
