PromptTools – open-source tools for evaluating LLMs and vector DBs
Hey HN! We’re Kevin and Steve. We’re building PromptTools (https://github.com/hegelai/prompttools): open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. Evaluating prompts, LLMs, and vector databases is a painful, time-consuming but necessary part of the product engineering process. Our tools allow engineers to do this in a lot less time. By “evaluating” we mean checking the quality of a model's response for a given use case, which is a combination of testing and benchmarking. As examples: - For generated…
In plain words
PromptTools is an open-source, self-hostable platform for engineers to experiment with, test, and evaluate large language models, vector databases, and prompts. It reduces the time required to assess model response quality for specific use cases through automated testing and benchmarking. The tool can validate output formats like JSON or SQL, assess generated content quality using other models, and support question-answering applications. Built for software engineers working with language models in production.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey HN! We’re Kevin and Steve. We’re building PromptTools (https://github.com/hegelai/prompttools): open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. Evaluating prompts, LLMs, and vector databases is a painful, time-consuming but necessary part of the product engineering process. Our tools allow engineers to do this in a lot less time. By “evaluating” we mean checking the quality of a model's response for a given use case, which is a combination of testing and benchmarking. As examples: - For generated JSON, SQL, or Python, you can check that the output is actually JSON, SQL, or executable Python. - For generated emails, you can use another model to assess the quality of the generated email given some requirements, like whether or not the email is written professionally. - For a question-answering chatbot, you can check that the actual answer is semantically similar to an expected answer. At Google, Steve worked with HuggingFace and Lightning to support running the newest open-source models on TPUs. He realized that while the open-source community was contributing incredibly powerful models, it wasn’t so easy to discover and evaluate them. It wasn’t clear when you could use Llama or Falcon instead of GPT-4. We began looking for ways to simplify and scale this evaluation process. With PromptTools, you can write a short Python script (as short as 5 lines) to run such checks across models, parameters, and prompts, and pass the results into an evaluation function to get scores. All these can be executed on your local machine without sending data to third-parties. Then we help you turn those experiments into unit tests and CI/CD that track your model’s performance over time. Today we support all of the major model providers like OpenAI, Anthropic, Google, HuggingFace, and even LlamaCpp, and vector databases like ChromaDB and Weaviate. You can evaluate responses via semantic similarity, auto-evaluation by a language model, or structured output validations like JSON and Python. We even have a notebook UI for recording manual feedback. Quickstart: pip install prompttools git clone https://github.com/hegelai/prompttools.git cd prompttools && jupyter notebook examples/notebooks/OpenAIChatExperiment.ipynb For detailed instructions, see our documentation at https://prompttools.readthedocs.io/en/latest/. We also have a playground UI, built in streamlit, which is currently in beta: https://github.com/hegelai/prompttools/tree/main/prompttools.... Launch it with: pip install prompttools git clone https://github.com/hegelai/prompttools.git cd prompttools && streamlit run prompttools/ui/playground.py We’d love it if you tried our product out and let us know what you think! We just got started a month ago and we’re eager to get feedback and keep building.
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, August 2023
the whole month →
Lottielab▲871Create and ship lottie animations to sites and apps faster
Dev tools · 2023 · lottielab.com
- LC
Outlines is a Python library that focuses on text generation with large language models. Brandon and I are not LLM experts and started the project a few months ago because we wanted to understand better how the generation process works. Our original background is probabilistic, relational and symbolic programming. Recently we came up with a fast way to generate text that matches a regex (https://blog.normalcomputing.ai/posts/2023-07-27-regex-guide...). The basic idea is simple: regular expressions have an equivalent Deterministic-Finite Automaton (DFA) representation. We…
AI · 2023 · github.com