nowfound

Life & fun · January 6, 2021

I have written a cloud native dark web crawler in Go

First stable release announcement (1.0.0)

I have written a cloud native dark web crawler in Go launched on January 6, 2021 with 7 votes, #1,184 of 3,157 launches that month and more than 50% of that year's launches. The site was still online in September 2026, 5 years on.

Visit github.com ↗Alternatives

What it does

I have implemented most of the features I intended for Trandoshan, and therefore version 1.0.0 will be releasedfrom github.com

Does a similar job

all alternatives →
  • DR
    DontBeEvil.rip: Search, for developers (API, expressions, CLI)2022 · ▲269

    I'd like to invite everyone to try out DontBeEvil.rip, an experimental search engine for developers. tl;dr $ alias rip="curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode " $ rip 'q=Heartbleed bug' DontBeEvil.rip is a year long experiment to see if a small team can build a developer-focused search engine that is self-sustaining on $10 monthly subscriptions. It works by only indexing high-quality resources that are relevant to developers. You won't get useless listicles because we'll never crawl them. Relevant urls are harvested from HN,…

  • Crawlee – Web scraping and browser automation library for Node.js2022 · crawlee.dev · ▲282

    Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using…

  • Ghost Inspector2014 · ghostinspector.com · ▲80

    Automated browser testing from the cloud

  • Apify SDK – A scalable web crawling and scraping library for JavaScript2018 · github.com · ▲78

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headf

  • A distributed and persistent web crawler written in Go2017 · github.com · ▲5

    Cross-platform persistent and distributed web crawler :link: - schollz/linkcrawler

  • HyperCrawl 2024 · ▲150

    Super-fast web crawling for LLM development

More life & fun this month

the category →

Source: Show HN post ↗

Launched alongside, January 2021

the whole month →