SQLFrame – I ran PySpark without Spark on a SQL database
Recently I open-sourced SQLFrame, a DataFrame library that implements the PySpark DataFrame API but removes Spark as a dependency. It does this by generating the corresponding SQL for the DataFrame operations using SQLGlot. Since the output is SQL this also means that the PySpark DataFrame API can now be used directly against other databases without the Spark middleman. I built this because of two common problems I have faced in my career: 1. I prefer to write complex pipelines in PySpark but they can be hard to read for SQL-proficient co-workers. Therefore I find myself in a tradeoff…
SQLFrame – I ran PySpark without Spark on a SQL database launched on May 21, 2024 with 11 votes, #1,156 of 4,539 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
SQLFrame is a DataFrame library that lets developers use the PySpark API without requiring Spark as a dependency. It converts DataFrame operations into SQL queries using SQLGlot, enabling the PySpark syntax to work directly against various databases. Built for data engineers and developers who prefer PySpark's API but need lighter-weight solutions or want their code more accessible to SQL-proficient team members, SQLFrame focuses on transformation pipelines and removes the need for Spark infrastructure.
written from the facts on this page · September 2026
From the sources
Turning PySpark Into a Universal DataFrame API. Contribute to eakmanrq/sqlframe development by creating an account on GitHub.
In the maker’s words, at launch
Recently I open-sourced SQLFrame, a DataFrame library that implements the PySpark DataFrame API but removes Spark as a dependency. It does this by generating the corresponding SQL for the DataFrame operations using SQLGlot. Since the output is SQL this also means that the PySpark DataFrame API can now be used…
Does a similar job
all alternatives →
I wrote a RDBMS (SQLite clone) from scratch in pure Python2023 · github.com · ▲371I wrote a relational database management system (RDBMS) (sqlite clone) from scratch in pure Python.
Turn your Pandas dataframe into a Tableau-style UI for visual analysis2023 · github.com · ▲712Hey, guys. I've just made a plugin which turns your pandas dataframe into a tableau-style component. It allows you to explore the dataframe with easy drag-and-drop UI. You can use PyGWalker in Jupyter, Google Colab, or even Kaggle Notebook to easily explore your data and generate interactive visualizations. PyGWalker (pronounced like "Pig Walker", just for fun) is named as an abbreviation of "Python binding of Graphic Walker". Here are some links to check it out: The Github Repo: https://github.com/Kanaries/pygwalker Use PyGWalker in Kaggle:…
A Hands-On Guide on PySpark Coding and Best Practices2019 · github.com · ▲52This is a repo documenting the best practices in PySpark. - ericxiao251/spark-syntax
Splitgraph - Build and share data with Postgres, inspired by Docker/Git2020 · splitgraph.com · ▲81EnterpriseDB (“EDB”), the leader in accelerating Postgres® in the enterprise, today announced its acquisition of Splitgraph, an early-stage disruptor that provides a Postgres compatible serverless SQL API for building data driven applications from hundreds of datasources. This move is seen as a mean
- ASA Schemaless Data Store Within Your SQL Database2022 · schemafreesql.com · ▲45
Hi HN, we're Eric and Dean, creators of SchemafreeSQL. Its roots go back to an on-line Web App Development Environment we developed back in 1999. It was comprised of an IDE, Web Server, Object Store, Virtual File System, Template System, and polyglot (Java, JavaScript, and Python). Of course, we named it “.OS”. Then we ended up dropping it. But that's a story for another time. It was the ease of use of the Object Store from .OS that we really missed, which brings us back to SchemafreeSQL. It provides an enhanced API to your SQL Database which allows it to function as a Schemaless…
- HAHashquery, a Python library for defining reusable analysis2024 · hashquery.dev · ▲67
Hi all, we recently open sourced the first version of Hashquery, a Python library which lets you model analytics, DRY up common logic, and execute it against a database or warehouse. We were originally rendering SQL directly for all our queries, but that spiraled out of control with more complex analysis; the SQL needed to be changed sporadically for each new database dialect (BigQuery, Redshift, Aethena, Postgres, etc etc) and the SQL fragments were very challenging to reuse (and so fragments were copy-pasted all over). ~~~ Advantages we think it has over writing SQL by hand: - Queries are…
More dev tools this month
the category →
Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.
Dev tools · 1d ago · github.com
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 29d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 27d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 24d ago · openbot.run
Talanoa▲346Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 22d ago · talanoa.email
Source: Show HN post ↗ · launch text quoted in part
