Alternatives
Products that do what TheThinkbench does
Continuous evaluation of LLM reasoning on competitive code
- 1AB
I built AutoThink, a technique that makes local LLMs reason more efficiently by adaptively allocating computational resources based on query complexity. The core idea: instead of giving every query the same "thinking time," classify queries as HIGH or LOW complexity and allocate thinking tokens accordingly. Complex reasoning gets 70-90% of tokens, simple queries get 20-40%. I also implemented steering vectors derived from Pivotal Token Search (originally from Microsoft's Phi-4 paper) that guide the model's reasoning patterns during generation. These vectors encourage behaviors like numerical…
2025
- 2
- 3AA
2019 · aristotl.io
- 4
- 5
- 6
- 7
- 8PP
I've been working on applying LLMs to long-context, verifiable problems over the past year, and today I'm releasing a benchmark of 62,000 pencil puzzles across 94 types (sudoku, nonori, slitherlink, etc.). The benchmark also allows for intermediate checks /rule breaks for all varieties at any step. I tested 51 models against a subset (300 puzzles) in two modes: single-shot (output the full solution) and agentic (iterate with verifier feedback). Some results: - Best model (GPT 5.2@xhigh) solves 56%. (~ half the puzzles are unsolved by any model) - Agentic solves average 29 turns. The…
Mar 2026 · ppbench.com
- 9AT
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com
- 10GO
LLMs are better at being the "mouth" than the "brain" and I can prove it mathematically. I built a deterministic graph engine that offloads reasoning from the LLM. It reduces token usage by 89% and makes a tiny 0.8B model trace enterprise execution paths flawlessly. Here is the white paper and the reproducible benchmark.
Mar 2026 · github.com
- 11RA
I built a local-first UI that adds two reasoning architectures on top of small models like Qwen, Llama and Mistral: a sequential Thinking Pipeline (Plan → Execute → Critique) and a parallel Agent Council where multiple expert models debate in parallel and a Judge synthesizes the best answer. No API keys, zero .env setup — just pip install multimind. Benchmark on GSM8K shows measurable accuracy gains vs. single-model inference.
Mar 2026 · github.com
- 12SA
Hi HN, like many I've been interested in the direction software engineering is taking now that coding LLMs are becoming prevalent. It seems that we're not quite there for "natural language programming", but it seems new abstractions are already starting to form. In order to explore this further I've built semcheck (semantic checker). It's a simple cli tool that can be used in CI or pre-commit to check that your implementation matches your specification using LLMs. The inspiration came while I was working on another project where I needed a data structure for a GeoJSON object, I passed Claude…
2025 · github.com
- 13AD
Ever wish you could get the best arguments for both sides of a debate? I built an AI-powered debate platform that pits language models against each other on controversial topics. Each AI is randomly assigned a side (pro/con). You vote before and after to see if you were persuaded. Most content today presents lopsided arguments. They provide strong points for one side, weak ones for the other. This project aims to surface the strongest arguments from both sides, using LLMs to simulate a fair debate. With enough usage, I want to use it to benchmark LLMs. My hypothesis is that randomly…
2025 · bot-bicker.vercel.app
- 14BA
2025 · github.com
- 15IO
Hey folks, I’m the creator of WFGY — a semantic reasoning framework for LLMs. After open-sourcing it, I did a full technical and value audit — and realized this engine might be worth $8M–$17M based on AI module licensing norms. If embedded as part of a platform core, the valuation could exceed $30M. Too late to pull it back. So here it is — fully free, open-sourced under MIT. --- ### What does it solve? Current LLMs (even GPT-4+) lack *self-consistent reasoning*. They struggle with: - Fragmented logic across turns - No internal loopback or self-calibration - No modular thought units - Weak…
2025 · github.com
- 16
- 17AO
Hi, We are building an open-source framework for loading and structuring LLM context to create accurate and explainable LLM answers using knowledge graphs and vector stores. We built the tool with four main concepts in mind: 1. Loader -> uses dlt in the backend to load and structure the data 2. Cognify step -> creates a graph with summaries, labels and factoids that are interconnected across the documents and stored as a representation in the vector store 3. Optimizer -> Uses DSPy to optimize LLM queries, and we plan to extend it to most of the knobs we can turn, like chunking etc. 4. Search…
2024 · github.com
- 18
- 19CB
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com
- 20SD
2018 · silogizma.org
- 21IM
Got tired of my debates with my friend's ending in "I'm right bc I said so" so I made a platform where you can debate with your friend's(or a bot, recently added feature) about whatever you want, and after the debate is done a LLM judges who's more sound in logic. Gain points and climb the leaderboard! Feedback and criticism would be appreciated(there's a discord in there if you wanna talk more in depth) p.s.: I decided to use google acc sign in since it makes the whole thing easier to manage(live chatroom feature, etc.). I don't actually do anything with your emails
2025 · crs-prod-rankeddebate-l4dnggfaca-nn.a.run.app
- 22AT
We kept shipping “simple” LLM features that were fluent-but-wrong. After too many postmortems we wrote down the failure patterns and added a small reasoning layer in front of the model. It’s model-agnostic, sits beside your existing stack, and you can implement it from a single PDF (MIT). What’s inside the PDF A problem map of 16 failure modes we kept hitting in real systems (OCR/layout drift, table-to-question mismatches, embedding≠meaning, pre-deploy collapse, etc.). Four lightweight gates you can add today: Knowledge-boundary canaries (empty/adversarial/known-fact probes).…
2025 · github.com
- 23MD
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com
- 24SA
Writeup: https://www.deeptempo.ai/blogs/the-36-percent-false-positive...
Jul 2026 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →