Web Scraping with Your Web Browser: Bot Challenges
Second article in a series. Interesting info regarding Cloudflare.
Web Scraping with Your Web Browser: Bot Challenges launched on October 4, 2024 with 6 votes, #1,611 of 5,185 launches that month and more than 50% of that year's launches. The site was still online in September 2026, 2 years on.
What it does
In the first article of this series, I gave a demonstration of a simple scraping task to get the conversation going and to encourage you to visit my website and install the Kraker local proxy server. However, I will assume that you have not done that and that's okay. Hopefully, you will take that step after this discussion. I will show you some code designed to bypass a bot challenge, whether presented by Cloudflare or any other such service provider. The web browser's proxy settings must be configured to use the Socks5 port on the local proxy. This is a transparent proxy which will only intercept HTTP requests on demand. It is undetectable by an outside observer except when it is…from 8chananon.github.io
Does a similar job
all alternatives →
Stop AI scrapers from hammering your self-hosted blog (using porn)Dec 2025 · github.com · ▲373Alright so if you run a self-hosted blog, you've probably noticed AI companies scraping it for training data. And not just a little (RIP to your server bill). There isn't much you can do about it without cloudflare. These companies ignore robots.txt, and you're competing with teams with more resources than you. It's you vs the MJs of programming, you're not going to win. But there is a solution. Now I'm not going to say it's a great solution...but a solution is a solution. If your website contains content that will trigger their scraper's safeguards, it will get dropped from their data…
Crawlee – Web scraping and browser automation library for Node.js2022 · crawlee.dev · ▲282Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using…
More life & fun this month
the category →
A free city-building game in your browser. You are the mayor. You know what the city needs. Now get it past the zoning board, the neighbors, the appeals court and the next election.
Life & fun · 1d ago · permit.petercai.com
- IV
Life & fun · 14d ago · tanzaniaeducationproject.org
Turn a phone, tablet or TV into a sign. Type a message and it fills the screen: welcome signs, timers, Wi-Fi passwords and more. No app, no account, free.
Life & fun · 15d ago · bigwords.page
Play a free virtual guitar online with your keyboard or by touch. Slow songs down and see every chord, string and strum. Acoustic, classic, electric and bass.
Life & fun · 8d ago · guitoro.com
Carrier settings from iPhone, Pixel and Galaxy firmware, decoded and compared: APNs, VoLTE, 5G and Wi-Fi Calling per carrier, and what each build changed.
Life & fun · 12d ago · carrierexplode.com
Life & fun · 22d ago · space.bl2.net
Source: Show HN post ↗
Launched alongside, October 2024
the whole month →
bolt.new▲1,230Build and scale high-performing websites & apps using your words. Join millions and start building today.
AI · 2024 · bolt.new
Softr for Notion▲923Turn Notion databases into portals & apps with no code
Dev tools · 2024 · softr.io
One inbox for all your work discussions
Work · 2024 · generalcollaboration.com