Smell – A framework for aligning LLM evaluators to human feedback
We've built SMELL (Subject-Matter Expert Language Liaison), a new framework that combines human expertise with LLMs to create feedback-informed, domain-specific LLM evaluators. One of the biggest issues with current evaluation methods (heuristics, assertions, LLM-as-a-judge etc.) is that it's difficult for them to match up with and capture human preferences. SMELL addresses this by putting human feedback at the core of the evaluation process. It scales up a small set of human-provided feedback into evaluators that reflect the standards and nuances of specific industries or use-cases. Instead…
What it does
In the maker’s words, at launch
We've built SMELL (Subject-Matter Expert Language Liaison), a new framework that combines human expertise with LLMs to create feedback-informed, domain-specific LLM evaluators. One of the biggest issues with current evaluation methods (heuristics, assertions, LLM-as-a-judge etc.) is that it's difficult for them to match up with and capture human preferences. SMELL addresses this by putting human feedback at the core of the evaluation process. It scales up a small set of human-provided feedback into evaluators that reflect the standards and nuances of specific industries or use-cases. Instead of a one-size-fits-all approach, you get evaluations that actually align with human judgment in those areas. If you're curious to try it out, we've made it easy by offering both a notebook and a hosted API so you can test SMELL with your own LLMs and datasets: - Notebook: https://colab.research.google.com/drive/1wCRwU5KQvnRSDxkubU9... - Hosted API: https://smell.quotientai.co/ Check out the blog post for more details: https://www.quotientai.co/post/subject-matter-expert-languag... We are in the process in writing up the findings into a paper, and are planning to provide the full details on SMELL (incl. prompts). If you’re interested in building a custom judge tailored to your specific use case, or if you'd like to contribute to our research, we'd love to collaborate! You can share your datasets with us at [email protected]. We'll publish results based on the data you provide, with full attribution and recognition of your contributions. In the meantime, we'd love to hear your feedback and see what you think!
Does the same job
all alternatives →
- LALLM Alignment Template – Aligning Language Models with Human Feedback2024 · github.com · ▲9
Hey Hacker News! I've been working on an open-source project called LLM Alignment Template, a comprehensive toolkit designed to help researchers, developers, and data scientists align large language models (LLMs) with human values using Reinforcement Learning from Human Feedback (RLHF). What the project does: Interactive Web Interface: Easily train models, visualize alignment metrics, and manage alignment with an accessible UI. Training with RLHF: Align models effectively to human preferences using feedback loops. Explainability: Built-in dashboards to help understand model behavior using…
- KAKnowing – an LLM tool built on concept hierarchies, not prompt-response2024 · ▲5
Hey HN! I've spent the past year full-time building Knowing, a tool for interacting with LLMs directly inside hierarchical structures instead of the usual prompt-response format. The idea started because I realized how much more intuitive it felt to build concept hierarchies continuously—no more endless copy-pasting or wondering how everything connects. The journey’s been a struggle. While I see huge potential in structuring AI interactions this way (writing books fast, planning projects, or organizing ideas), it’s been hard to pin down clear use cases in the market. I’m also working in near…
- ATA tiny reasoning layer that steadies LLM outputs (MIT; +22.4% accuracy)2025 · github.com · ▲6
We kept shipping “simple” LLM features that were fluent-but-wrong. After too many postmortems we wrote down the failure patterns and added a small reasoning layer in front of the model. It’s model-agnostic, sits beside your existing stack, and you can implement it from a single PDF (MIT). What’s inside the PDF A problem map of 16 failure modes we kept hitting in real systems (OCR/layout drift, table-to-question mismatches, embedding≠meaning, pre-deploy collapse, etc.). Four lightweight gates you can add today: Knowledge-boundary canaries (empty/adversarial/known-fact probes).…
- LBLLM-based Toolset for Language Learning2025 · drillapp.xyz · ▲9
Hello everyone. I built an AI-based toolset to help me with language learning. I wanted to be able to easily generate very specific study content and get rapid feedback on my writing. Unlike most language apps, it doesn’t actually try to teach you a language. Instead, it’s a collection of tools for people at an intermediate level who already have a learning process It’s particularly great for Anki users. There a demo video on the login page, and I set up anonymous auth for people who want to test it without creating an account. Feedback and bug reports welcome.
- CRChainFactory – Run Structured LLM Inference with Easy Parallelism2024 · github.com · ▲8
hi everyone. how does moving llm call prompts and output structure definitions away from code into configuration land sound? would you use something like this if it was stable and well documented enough? please don't hold back the criticism. i appreciate all feedback (constructive & otherwise).
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, October 2024
the whole month →
Softr for Notion▲925Turn Notion databases into portals & apps with no code
Dev tools · 2024 · softr.io
One inbox for all your work discussions
Work · 2024 · generalcollaboration.com