Veil

What it does

Veil is an open-source Python web scraper that tries the cheapest method first and only escalates when a site blocks it so you're not running a full browser for every page. • 3 tiers: plain HTTP → TLS impersonation → stealth browser • Resumable concurrent crawler (checkpoint, JSONL, dedupe) • Proxy rotation, rate limiting, robots.txt by default • Honest README: documents what it can't beat Python, MIT, tested. A learning project built in the open.

Building a reusable scraper for many sites. Contribute to adtyapnda/veil development by creating an account on GitHub.

A modular, polite web scraper that cascades through fetch strategies — from a near-free HTTP request up to a full stealth browser — and stops at the cheapest one that actually returns the page. Most scrapers either send a naive requests.get() (and get blocked) or fire up a heavy headless browser for everything (slow and expensive). veil picks the lightest tool that works, per site, automatically. ┌─────────────────────────────────────────────┐ fetch(url) ───► │ politeness gate (robots.txt + rate limit) │ └───────────────────────┬─────────────────────┘ ▼ cheapest ──► http_basic ──► tls_impersonate ──► browser ──► heaviest (httpx) (curl_cffi JA3/HTTP2) (Playwright + stealth) │ escalates only…from github.com

• 
  Simple Scraper2019 · simplescraper.io · ▲556

  Turn any website into an API in just a few seconds.

• 
  Web Scraper API2021 · datashake.com · ▲287

  Scrape the web at scale without getting blocked

• 
  Veil2021 · apps.apple.com · ▲158

  Filter people from videos

• 
  Veil2023 · apps.apple.com · ▲76

  Magical video filtering

• 
  VEIL: Smart Context-Aware Privacy2025 · chromewebstore.google.com · ▲12

  Strong protection where it counts seamless browsing anywhere

• 
  A Python Spider System with Web UI2014 · github.com · ▲159

  A Powerful Spider(Web Crawler) System in Python. Contribute to binux/pyspider development by creating an account on GitHub.

• 
  /dev/null▲484

  a social network for devs

  Dev tools · 3d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 28d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 26d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 23d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 20d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 28d ago · corespeed.io

• 
  Fundraisly▲1,537

  AI fundraising agent that finds investors and books meetings

  AI · Jun 2026 · fundraisly.com

• 
  Homebrew 6.0.0▲1,481

  Today, I’m proud to announce Homebrew 6.0.0. The most significant changes since 5.1.0 are a new tap trust security mechanism, the new faster, smaller, default internal Homebrew JSON API, sandboxing on Linux, better defaults informed by our user survey, many brew bundle improvements, improved performance and initial support for macOS 27 (Golden Gate). Happy to discuss any questions here!

  Dev tools · Jun 2026 · brew.sh

• 
  Performative-UI – A react component library of design tropes▲1,181

  Hi HN! I was super frustrated at apps like tiktok and wechat preventing me from saving or downloading the animated gifs, so I made a little tool to turn a screen recording into a slack-sized emoji. Hope you get some value from this!

  Life & fun · Jun 2026 · vorpus.github.io

• 
  Upstream▲976

  The inbox designed for humans and agents

  AI · Jun 2026 · upstream.do

• 
  Goldfish▲958

  Press Option. It knows your work and replies like you

  AI · Jun 2026 · goldfish.sh

• 
  I made Google Trends for Hacker News by indexing 18 years of comments▲822

  Search 18 years of Hacker News and chart how any topic, tool, or person trended. Overlay terms to compare their rise and fall across 45M posts and comments.

  Life & fun · Jun 2026 · hackernewstrends.com