The Canada census data in a SQLite file; advice appreciated
This is niche, I'll admit. I needed to look through the latest census data, but it was exported as multiple multi-gigabyte bespoke latin1-encoded CSV files. Pandas, Polars, and SQLite's CSV import tool weren't much help, so I shelved the project until recently, when I started taking a SQLite course online. I picked it up again, normalized the data, and now there's a database that can be queried through a SQL view that matches the headings in the original CSVs. I'm proud of the script I created to export the data, as well as automatically compress the artifact, make the diagrams and…
The Canada census data in a SQLite file; advice appreciated launched on December 4, 2024 with 6 votes and more than 50% of that year's launches. The site was still online in September 2026, 1 year on.
What it does
An unofficial consolidation of the Canada Census Profile, 2021 Census of Population into a SQLite database file. - Release 1.0.0-alpha · rendello/canada-census-2021-sqlite
Filter Loading Sorry, something went wrong. Uh oh! There was an error while loading. Please reload this page . An unofficial consolidation of the Canada Census Profile, 2021 Census of Population into a SQLite database file. The official downloads of the census profile are offered split into multiple multi-gigabyte textual files that are not possible to query directly, and are generally too large to load into memory for tools like Pandas. They are in the legacy ISO-8859-1 (Latin 1) encoding, causing many issues with import tools, and their format is difficult to work with. The SQLite database allows direct querying, and does not require the entire database to be loaded into memory at once.…from github.com
In the maker’s words, at launch
This is niche, I'll admit. I needed to look through the latest census data, but it was exported as multiple multi-gigabyte bespoke latin1-encoded CSV files. Pandas, Polars, and SQLite's CSV import tool weren't much help, so I shelved the project until recently, when I started taking a SQLite course online. I picked…
Does a similar job
all alternatives →
SQLite Plugin for Jekyll2024 · github.com · ▲227I love Jekyll, especially the Datafiles[0] feature, which lets you use CSV/JSON/YAML files and iterate through them. Mixed with the Jekyll Data Pages generator[1], which lets you create a page for every row in your dataset, it is a very powerful combination. However, Liquid is a terrible language for data-mangling, and simple filtering/sorting/merging can become very annoying. So I wrote a Jekyll SQLite plugin that lets you use the same data interface in Jekyll/Liquid, but backed by a SQLite file(s). It gives you the simplicity of the Baked Data pattern[2], and the…
Comma Separated Values (CSV) to Unicode Separated Values (USV)2024 · crates.io · ▲208crates.io serves as a central registry for sharing crates, which are packages or libraries written in Rust that you can use to enhance your projects
Csvbox.io – CSV import button for a web app, SaaS or API2021 · csvbox.io · ▲155Drop a production-ready CSV, Excel, and PDF importer into your SaaS in minutes. Handle up to 2 million rows per import, with column mapping, validation, webhooks, and clean data delivered to your API.
- SQSQLite query inside a Bash function2021 · ▲134
Part of the workflow for building my website is the generation of a table in tab-separated column format (.tsv). The source data is found in four other .tsv files. I use an SQLite query to perform a 4-way join and write out the new table. For convenience, I wrote a script that encapsulates the query inside a Bash function. The example below illustrates this technique. repertoire() { pushd $CMM_SOURCES/_data sqlite3 <<EOS .headers on .mode tabs .import category.tsv category .import composition.tsv composition .import concert.tsv concert .import program.tsv program .once repertoire.tsv…
OpenSheetFeb 2026 · opensheet.app · ▲86Hi folks. I've been doing some experiments on how LLMs could get more handy in the day to day of working with files (CSV, Parquet, etc). Earlier last year, I built https://datakit.page and evolved it over and over into an all in-browser experience with help of duckdb-wasm. Got loads of feedbacks and I think it turned into a good shape with being an adhoc local data studio, but I kept hearing two main things/issues: Why can't the AI also change cells in the file we give to it? Why can't we modify this grid ourselves? So besides the whole READ and text-to-SQL flows, what seemed…
- CECensus – Export for data warehouses2020 · ▲34
Hello Hacker News! We're a team of YC founders (Meldium W13, Draft S11, TapEngage S11) launching something new (https://www.getcensus.com). How many times has your business team asked you to generate yet another CSV file, write a ”quick report” in SQL, or send some custom data to a terrible API (looking at you Marketo)? We’ve built a product that connects directly to your data warehouse and syncs into apps like Salesforce, Customer.io and even Google Sheets. In fact, your business teams won’t even need to rely on engineering to manage all these pipelines. The tech stack for…
More dev tools this month
the category →
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 28d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 26d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 23d ago · openbot.run
Talanoa▲346Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 21d ago · talanoa.email
CoreSpeed▲332One MCP for everything your agents need: apps, memory, tools
Dev tools · 28d ago · corespeed.io
Source: Show HN post ↗ · launch text quoted in part
