Alternatives
Products that do what Using DSPy to enrich a dataset of the Nobel laureate network does
I've been working a fair bit with DSPy lately, and I did some work in combining the benefits of vector search and LLMs (via a DSPy pipeline) to disambiguate records with a high degree of accuracy to help enrich a dataset. The blog post shows how this approach scales well, is very cost-effective and super concise - all it takes is < 100 lines of DSPy code and it all runs async. The code to reproduce is in this repo if anyone's interested (all tools are 100% free and open source, and the methodology will work with open weight LLMs too).…
- 1DA
Dear HN, I am Riwaj, the cofounder of dstack.ai (https://github.com/dstackai). A few months ago, we built an online service that allows users to publish data visualizations from Python or R. The idea was to build a tool that did not require additional programming or front-end development for publishing data visualizations. Such a code can be invoked from either Jupyter notebook, RMarkdown, Python, or R scripts. Once the data is pushed, it can be accessed via a browser. Open-sourcing dstack: During our customer discovery phase, we realized that dstack.ai should integrate a lot…
2020
- 2L3
I spent a lot of time and money on this rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary LLMs that was quite popular this year and produced great research papers by Anthropic [1], OpenAI [2] and Deepmind [3]. I am quite proud of this project and since I consider myself the target audience for HackerNews did I think that maybe some of you would appreciate this open research replication as well. Happy to answer any questions or face any feedback. Cheers [1]…
2024 · github.com
- 3FV
I recently found myself computing the similarity between lots of very high dimensional vectors (i.e., sentence embedding vectors from LLMs), and I wanted to try some more powerful measures of similarity/dependency than just Cosine similarity, which seems to be the default for everything nowadays because of its computational efficiency. There are many other more involved measures that can detect more subtle relationships, but the problem is that some of them are quite slow to compute, especially if you're trying to do it in Python. For my favorite measure of statistical dependency,…
2023 · github.com
- 4SP
Hi HN, Over the past 6 months I've been working on a technical book focused on helping aspiring data scientists to get hands-on experience with cloud computing environments using the Python ecosystem. The book is targeted at readers already familiar with libraries such as Pandas and scikit-learn that are looking to build out a portfolio of applied projects. To author the book, I used the Leanpub platform to provide drafts of the text as I completed each chapter. To typeset the book, I used the R bookdown package by Yihui Xie to translate my markdown into a PDF format. I also used Google docs…
2020
- 5NL
Link: https://www.nlpdemystified.org/ Hi HN: After a year of work, I've published my free NLP course. The course helps anyone who knows Python and a bit of math go from the basics to today's mainstream models and frameworks. I strive to balance theory and practice, so every module consists of detailed explanations and slides along with a Colab notebook putting the ideas into practice (in most modules). The notebooks cover how to accomplish everyday NLP tasks including extracting key information, document search, text similarity, text classification, finding topics in…
2022
- 6NL
Hi HN: I published part one of my free NLP course. The course is intended to help anyone who knows Python and a bit of math go from the very basics all the way to today's mainstream models and frameworks. I strive to balance theory and practice and so every module consists of detailed explanations and slides along with a Colab notebook (in most modules) putting the theory into practice. In part one, we cover text preprocessing, how to turn text into numbers, and multiple ways to classify and search text using "classical" approaches. And along the way, we'll pick up useful bits on how to use…
2022 · nlpdemystified.org
- 7
- 8BA
2019 · bamboolib.com
- 9PT
I've developed a Python API service that uses GPT-4o for OCR on PDFs. It features parallel processing and batch handling for improved performance. Not only does it convert PDF to markdown, but it also describes the images within the PDF using captions like `[Image: This picture shows 4 people waving]`. In testing with NASA's Apollo 17 flight documents, it successfully converted complex, multi-oriented pages into well-structured Markdown. The project is open-source and available on GitHub. Feedback is welcome.
2024 · github.com
- 10AA
Hi HN! I'm excited to share Autolabel, an open-source Python library to label and enrich text datasets with any Large Language Model (LLM) of your choice. We built Autolabel because access to clean, labeled data is a huge bottleneck for most ML/data science teams. The most capable LLMs are able to label data with high accuracy, and at a fraction of the cost and time compared to manual labeling. With Autolabel, you can leverage LLMs to label any text dataset with <5 lines of code. We’re eager for your feedback!
2023 · github.com
- 11DC
2016 · datasets.co
- 12PF
Hey HN! I’ve recently open-sourced Pyversity, a lightweight library for diversifying retrieval results. Most retrieval systems optimize only for relevance, which can lead to top-k results that look almost identical. Pyversity efficiently re-ranks results to balance relevance and diversity, surfacing items that remain relevant but are less redundant. This helps with improving retrieval, recommendation, and RAG pipelines without adding latency or complexity. Main features: - Unified API: one function (diversify) supporting several well-known strategies: MMR, MSD, DPP, and COVER (with more to…
Oct 2025 · github.com
- 13IB
Hi HN, Over the past few months, I've been building `dsc`, a tensor library from scratch in C++/CUDA. My main focus has been on getting the basics right, prioritizing a clean API, simplicity, and clear observability for running small LLMs locally. The key features are: - C++ core with CUDA support written from scratch. - A familiar, PyTorch-like Python API. - Runs real models: it's complete enough to load a model like Qwen from HuggingFace and run inference on both CUDA and CPU with a single line change[1]. - Simple, built-in observability for both Python and C++. Next on the roadmap is…
2025 · github.com
- 14FL
Hi HN community, I have been working on benchmarking publicly available LLMs these past couple of weeks. More precisely, I am interested on the finetuning piece since a lot of businesses are starting to entertain the idea of self-hosting LLMs trained on their proprietary data rather than relying on third party APIs. To this point, I am tracking the following 4 pillars of evaluation that businesses are typically look into: - Performance - Time to train an LLM - Cost to train an LLM - Inference (throughput / latency / cost per token) For each LLM, my aim is to benchmark them for…
2023 · github.com
- 15EL
Hey HN! I'd love to get some people to mess around with a little side project I built to teach myself DSPy! I've been a big fan of reading fiction + webnovels for a while now, and have always been curious about two things: how can LLMs iteratively learn to write better based on reader feedback, and which LLMs are actually best at creative writing (research benchmarks are cool, but don't necessarily translate to real-world usage). That's exactly why I built narrator.sh! The platform takes in a user input for a novel idea, then generates serialized fiction chapter-by-chapter by using DSPy to…
2025 · narrator.sh
- 16DB
I've been doing some data cleaning for my fine tuning projects using LLMs, and decided to just build a package for it as a side project. Check it out here: https://github.com/databonsai/databonsai Some features: - categorization (labelling), transformation and decomposition (text into structured format) - validates llm outputs - batch mode batches up the inputs/outputs so you don't send the prompt (schema, fewshot examples) for every row of data, saving a significant amount of tokens There are some similarities to the Instructor repo, but this is simpler and made for…
2024 · github.com
- 17BD
Hi everyone, I'd like to share my project, bridge-ds - a lightweight Python framework that simplifies how ML practitioners manage and interact with datasets. Why bridge-ds? It abstracts the repetitive parts of dataset handling in real-world ML workflows, but remains lean enough as to not force opinionated workflow or unnecessary dependencies. bridge-ds uses two complementary approaches: - Macro-level: Treat your entire dataset like a DataFrame—filter, sort, and modify with familiar, intuitive operations. - Micro-level: Efficiently handle individual samples with lazy loading, caching, remote…
2024 · github.com
- 18LA
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com
- 19IM
2020 · github.com
- 20AO
Hi, We are building an open-source framework for loading and structuring LLM context to create accurate and explainable LLM answers using knowledge graphs and vector stores. We built the tool with four main concepts in mind: 1. Loader -> uses dlt in the backend to load and structure the data 2. Cognify step -> creates a graph with summaries, labels and factoids that are interconnected across the documents and stored as a representation in the vector store 3. Optimizer -> Uses DSPy to optimize LLM queries, and we plan to extend it to most of the knobs we can turn, like chunking etc. 4. Search…
2024 · github.com
- 21CL
I have a proposal that addresses long-term memory problems for LLMs when new data arrives continuously (cheaply!). The program involves no code, but two Markdown files. For retrieval, there is a semantic filesystem that makes it easy for LLMs to search using shell commands. It is currently a scrappy v1, but it works better than anything I have tried. Curious for any feedback!
Apr 2026 · github.com
- 22SF
I've made a small Python library, designed for quick-and-easy prototyping of machine learning models. It's built on top of scikit-learn, to serialize and deserialize data from the forms you're likely to have, to the format used in scikit-learn. https://github.com/madman-bob/Smart-Fruit It's pretty bare-bones at the moment, but I thought I'd see if there was any interest before spending too much time on it. Let me know what you think.
2018
- 23SF
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what…
2025 · github.com
- 24AT
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →