Help improve language coverage in Common Crawl
Hey HN, the Common Crawl Foundation is trying to expand the coverage of our crawl to more languages, regions and cultures, and if you speak a language other than English (LOTE) you can help! By validating Language Identification data (LangID or LID): https://dynabench.org/tasks/text-language-identification By contributing urls for our seed crawl: https://github.com/commoncrawl/web-languages We're also organizing a Workshop on Multilingual Data Quality Signals (WMDQS) with MLCommons and EleutherAI where we have a call for papers open…
What it does
In the maker’s words, at launch
Hey HN, the Common Crawl Foundation is trying to expand the coverage of our crawl to more languages, regions and cultures, and if you speak a language other than English (LOTE) you can help! By validating Language Identification data (LangID or LID): https://dynabench.org/tasks/text-language-identification By contributing urls for our seed crawl: https://github.com/commoncrawl/web-languages We're also organizing a Workshop on Multilingual Data Quality Signals (WMDQS) with MLCommons and EleutherAI where we have a call for papers open (https://wmdqs.org/cfp/) and a upcoming shared task on language identification (https://wmdqs.org/shared-task/)
Does the same job
all alternatives →
No Language Left Behind by Meta2022 · ▲116An open-source language translation system for 200 languages
- CLCrowdsourced Language Learning Site2018 · littlelingua.eu · ▲54
- FDFranc – Detect natural languages2014 · github.com · ▲51
- LMLanguage Models API via the Cohere Platform2021 · os.cohere.ai · ▲18
- SASparser – A Multilanguage Parser2019 · ▲65
* https://sparser.io * https://github.com/Unibeautify/sparser This is my attempt at creating a universal language parser. It attempts to solve a couple of problems: * Support multiple languages * Recursively extend support to languages embedded within other languages * Output a uniform format for all supported languages This is a personal project so any feedback would be helpful. Something interesting I found after I built it is that this parser is not as fast to write output as many other JavaScript parsers, but its output is much faster to read from due to the…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, June 2025
the whole month →- IM
Hey HN! Pilots everywhere are required to keep a logbook of all their flying hours, aircraft, airports, and so on. Since I track everything digitally (some people still just use paper logbooks!), I put together some data visualizations and a few 3D globes to show my flying history. This globe is probably my favourite so far: https://jameshard.ing/pilot/globes/all If you’ve got ideas for other graphs or ways to show this kind of data, I’d love to hear them!
Life & fun · 2025 · jameshard.ing
Stunning presentations with AI. No design skills required.
AI · 2025 · chroniclehq.com
Build native mobile apps for iOS and Android without code.
Dev tools · 2025 · bble.io
All your connections, across all your socials.
AI · 2025 · pally.com
- WC
I was the main contributor to workout.lol, an open-source fitness app to easily build a workout routine. The project had traction (1.4k GitHub stars, 95 forks, ~20K visits/month), but was eventually sold due to video licensing hurdles. The new owner stopped maintaining it, and the repo went abandoned. Over the next 9 months, I sent 15 emails to try to save it : no replies. Feature requests & issues were ignored. The community was left with a "broken" tool let's say. I couldn't just let it die So I built the new version from scratch with the same open-source spirit, but a better…
Dev tools · 2025 · github.com