estela, a modern elastic web scraping cluster
estela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface. It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or who are in the process of designing their on-premise scraping architecture, so as not to needlessly reinvent the wheel, and to benefit from the get-go from features such as built-in scalability and…
What it does
In the maker’s words, at launch
estela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface. It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or who are in the process of designing their on-premise scraping architecture, so as not to needlessly reinvent the wheel, and to benefit from the get-go from features such as built-in scalability and elasticity, among others. estela has been recently published as OSS under the MIT license: https://github.com/bitmakerla/estela More details about it can be found in the release blog post and the official documentation: https://bitmaker.la/blog/2022/06/24/estela-oss-release.html https://estela.bitmaker.la/docs/ estela supports Scrapy spiders for the moment being, but additional frameworks/languages are on the roadmap. All kinds of feedback and contributions are welcome! Disclaimer: I'm part of the development team behind estela :-)
Does the same job
all alternatives →
- WSWeb scraping that just works with OpenFaaS with Puppeteer2020 · openfaas.com · ▲145
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, September 2022
the whole month →- IM
Life & fun · 2022 · dogbunnypuzzle.com
- IM
A couple of years ago, I had an interesting idea. What if there was a marketplace where all the underlying tech was open-source? The order management system, the storefront, customer support, etc. The marketplace would simply connect to the seller’s infra instead of locking them in. If, for some reason, the seller is removed from the marketplace, their software stays with them and they can continue accepting orders directly. This model can be used to disrupt any marketplace from AirBNB to UberEats: building tech for home renters and restaurants and later, leveraging that to build a competing…
Dev tools · 2022 · openship.org
