wxpath – Declarative web crawling in XPath
wxpath is a declarative web crawler where web crawling and scraping are expressed directly in XPath. Instead of writing imperative crawl loops, you describe what to follow and what to extract in a single expression: import wxpath # Crawl, extract fields, build a Wikipedia knowledge graph path_expr = """ url('https://en.wikipedia.org/wiki/Expression_language') ///url(//main//a/@href[starts-with(., '/wiki/') and not(contains(., ':'))]) /map{ 'title': (//span[contains(@class,…
In plain words
wxpath is a declarative web crawler that lets users express web crawling and scraping using XPath syntax instead of writing traditional imperative loops. Users describe what links to follow and what data to extract in a single XPath expression, then iterate through results. It's designed for developers who need to scrape websites and build structured datasets from web content, particularly those already familiar with XPath query language.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
wxpath is a declarative web crawler where web crawling and scraping are expressed directly in XPath. Instead of writing imperative crawl loops, you describe what to follow and what to extract in a single expression: import wxpath # Crawl, extract fields, build a Wikipedia knowledge graph path_expr = """ url('https://en.wikipedia.org/wiki/Expression_language') ///url(//main//a/@href[starts-with(., '/wiki/') and not(contains(., ':'))]) /map{ 'title': (//span[contains(@class, "mw-page-title-main")]/text())[1] ! string(.), 'url': string(base-uri(.)), 'short_description': //div[contains(@class, 'shortdescription')]/text() ! string(.), 'forward_links': //div[@id="mw-content-text"]//a/@href ! string(.) } """ for item in wxpath.wxpath_async_blocking_iter(path_expr, max_depth=1): print(item) The key addition is a `url(...)` operator that fetches and returns HTML for further XPath processing, and `///url(...)` for deep (or paginated) traversal. Everything else is standard XPath 3.1 (maps/arrays/functions). Features: - Async/concurrent crawling with streaming results - Scrapy-inspired auto-throttle and polite crawling - Hook system for custom processing - CLI for quick experiments Another example, paginating through HN comments (via "follow=" argument) pages and extracting data: url('https://news.ycombinator.com', follow=//a[text()='comments']/@href | //a[@class='morelink']/@href) //tr[@class='athing'] /map { 'text': .//div[@class='comment']//text(), 'user': .//a[@class='hnuser']/@href, 'parent_post': .//span[@class='onstory']/a/@href } Limitations: HTTP-only (no JS rendering yet), no crawl persistence. Both are on the roadmap if there's interest. GitHub: https://github.com/rodricios/wxpath PyPI: pip install wxpath I'd love feedback on the expression syntax and any use cases this might unlock. Thanks!
More work this month
the category →
The app store for voice native apps that lives in your notch
Work · 28d ago · voiceos.com
The New Calendly▲211Handle all of the work before, during, and after meetings
Work · 17d ago · calendly.com
Launched alongside, January 2026
the whole month →- IN
Hey HN! I wanted to share something I built over the last few weeks: isometric.nyc is a massive isometric pixel art map of NYC, built with nano banana and coding agents. I didn't write a single line of code. Of course no-code doesn't mean no-engineering. This project took a lot more manual labor than I'd hoped! I wrote a deep dive on the workflow and some thoughts about the future of AI coding and creativity: http://cannoneyed.com/projects/isometric-nyc
AI · Jan 2026 · cannoneyed.com
Automatic AI-powered code reviews the moment you open a PR
Dev tools · Jan 2026 · kilo.ai
