Paramount – Human Evals of AI Customer Support
Hey HN, Hakim here from Fini (YC S22), a startup focused on providing automated customer support bots for enterprises that have a high volume of support requests. Today, one of the largest use cases of LLMs is for the purpose of automating support. As the space has evolved over the past year, there has subsequently been a need for evaluations of LLM outputs - and a sea of LLM Evals packages have been released. "LLM evals" refer to the evaluation of large language models, assessing how well these AI systems understand and generate human-like text. These packages have recently relied on…
In plain words
Paramount is a human evaluation platform for AI customer support systems built by Fini, a startup specializing in automated support bots for enterprises. It addresses the limitations of automatic evaluations by incorporating human review to achieve the higher accuracy thresholds required by large-scale support operations. The tool is designed for companies deploying LLM-based customer support that need reliable quality assessment beyond algorithmic scoring.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey HN, Hakim here from Fini (YC S22), a startup focused on providing automated customer support bots for enterprises that have a high volume of support requests. Today, one of the largest use cases of LLMs is for the purpose of automating support. As the space has evolved over the past year, there has subsequently been a need for evaluations of LLM outputs - and a sea of LLM Evals packages have been released. "LLM evals" refer to the evaluation of large language models, assessing how well these AI systems understand and generate human-like text. These packages have recently relied on "automatic evals," where algorithms (usually another LLM) automatically test and score AI responses without human intervention. In our day to day work, we have found that Automatic Evals are not enough to get the required 95% accuracy for our Enterprise customers. Automatic Evals are efficient, but still often miss nuances that only human expertise can catch. Automatic Evals can never replace the feedback of a trained human who is deeply knowledgeable on an organization's latest product releases, knowledgebase, policies and support issues. The key to solve this is to stop ignoring the business side of the problem, and start involving knowledgeable experts in the evaluation process. That is why we are releasing Paramount - an Open Source package which incorporates human feedback directly into the evaluation process. By simplifying the step of gathering feedback, ML Engineers can pinpoint and fix accuracy issues (prompts, knowledgebase issues) much faster. Paramount provides a framework for recording LLM function outputs (ground truth data) and facilitates human agent evaluations through a simple UI, reducing the time to identify and correct errors. Developers can integrate Paramount with a Python decorator that logs LLM interactions into a database, followed by a straightforward UI for expert review. This process aids the debugging and validation phase of launching accurate support bots. We'd love to hear what you think!
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, June 2024
the whole month →
La Growth Machine▲1,216Create personalized, multi-channel conversations at scale
AI · 2024 · lagrowthmachine.com
PyjamaHR▲1,132Hiring on autopilot. The AI applicant tracking system (ATS).
Work · 2024 · pyjamahr.com