A distributed and persistent web crawler written in Go
Cross-platform persistent and distributed web crawler :link: - schollz/linkcrawler
A distributed and persistent web crawler written in Go launched on March 19, 2017 with 5 votes and more than 50% of that year's launches. The site was still online in September 2026, 9 years on.
Does a similar job
all alternatives →
Crawlee – Web scraping and browser automation library for Node.js2022 · crawlee.dev · ▲282Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using…
Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language2019 · github.com · ▲116Distributed web crawler admin platform for spiders management regardless of languages and frameworks. - tikazyq/crawlab
- DRDontBeEvil.rip: Search, for developers (API, expressions, CLI)2022 · ▲269
I'd like to invite everyone to try out DontBeEvil.rip, an experimental search engine for developers. tl;dr $ alias rip="curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode " $ rip 'q=Heartbleed bug' DontBeEvil.rip is a year long experiment to see if a small team can build a developer-focused search engine that is self-sustaining on $10 monthly subscriptions. It works by only indexing high-quality resources that are relevant to developers. You won't get useless listicles because we'll never crawl them. Relevant urls are harvested from HN,…
Headless Chrome Crawler2018 · github.com · ▲172Distributed crawler powered by Headless Chrome. Contribute to yujiosaka/headless-chrome-crawler development by creating an account on GitHub.
More life & fun this month
the category →
A free city-building game in your browser. You are the mayor. You know what the city needs. Now get it past the zoning board, the neighbors, the appeals court and the next election.
Life & fun · 23h ago · permit.petercai.com
- IV
Life & fun · 13d ago · tanzaniaeducationproject.org
Turn a phone, tablet or TV into a sign. Type a message and it fills the screen: welcome signs, timers, Wi-Fi passwords and more. No app, no account, free.
Life & fun · 14d ago · bigwords.page
Play a free virtual guitar online with your keyboard or by touch. Slow songs down and see every chord, string and strum. Acoustic, classic, electric and bass.
Life & fun · 8d ago · guitoro.com
Carrier settings from iPhone, Pixel and Galaxy firmware, decoded and compared: APNs, VoLTE, 5G and Wi-Fi Calling per carrier, and what each build changed.
Life & fun · 12d ago · carrierexplode.com
Life & fun · 22d ago · space.bl2.net
Source: Show HN post ↗
Launched alongside, March 2017
the whole month →
Startup School▲1,010A free online program to teach you how to start a company
Life & fun · 2017 · startupschool.org
Hardbound 3.0▲739Illustrated summaries of bestselling nonfiction books
Life & fun · 2017 · apps.apple.com
