Semcheck – AI Tool for checking implementation follows spec
Hi HN, like many I've been interested in the direction software engineering is taking now that coding LLMs are becoming prevalent. It seems that we're not quite there for "natural language programming", but it seems new abstractions are already starting to form. In order to explore this further I've built semcheck (semantic checker). It's a simple cli tool that can be used in CI or pre-commit to check that your implementation matches your specification using LLMs. The inspiration came while I was working on another project where I needed a data structure for a GeoJSON object, I passed Claude…
In plain words
Semcheck is a CLI tool that uses AI to verify whether code implementations match their specifications. It integrates into CI pipelines and pre-commit workflows to automatically check semantic alignment between code and spec documents using large language models. The tool addresses the challenge of keeping implementations consistent with specifications as they evolve during development, particularly useful when AI-generated code needs validation against original requirements or standards documentation.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hi HN, like many I've been interested in the direction software engineering is taking now that coding LLMs are becoming prevalent. It seems that we're not quite there for "natural language programming", but it seems new abstractions are already starting to form. In order to explore this further I've built semcheck (semantic checker). It's a simple cli tool that can be used in CI or pre-commit to check that your implementation matches your specification using LLMs. The inspiration came while I was working on another project where I needed a data structure for a GeoJSON object, I passed Claude the text of RFC-7946 and it gave me an implementation. It took some back and forth after that before I was happy with it, but this also meant the RFC went out of context for the LLM. That's why I asked Claude again to check the RFC to make sure we haven't strayed too far from the spec. It occurred to me that it would be good to have a formal way of defining these kinds of checks that can be run in a pre-commit or merge request flow. Creating this tool was itself an experiment to try "spec-driven-development" using Claude Code, a middle ground between completely vibe-coding and traditional programming. My workflow was as follows: ask AI to write a spec and implementation plan, edit these manually to my liking, then ask AI to execute one step at a time. Being careful that the AI doesn't drift too far from what I think is required. My very first commit [1] is the specification of the config file structure and an implementation plan. As soon as semcheck was in a state where it could check itself it started to find issues [2]. I found that this workflow improves not just your implementation but helps you refine your specification at the same time. Besides specification, I also started to include documentation in my rules, making sure that the configuration examples and CLI flags I have in my README.md file stay in line with implementation [3]. The best thing is that you can put found issues directly back into your AI editor for a quick iteration cycle. Some learnings: - LLMs are very good at finding discrepancies, as long as the number of files you pass to the comparison function isn't too large, in other words the true-positive results are quite good. - False-positives: the LLM is a know-it-all (literally) and often thinks it knows better. The LLM is eager to use its own world knowledge to find faults. This can both be nice and problematic. I've often had it complain that my Go version doesn't exist, but it was simply released after the knowledge cutoff of that model. I specifically prompt [4] the model to only find discrepancies, but it often "chooses" to use its knowledge anyway. - In an effort to reduce false-positives I ask the model to give me a confidence score (0-1), to indicate to me how sure it was that the issue it found is actually applicable in this scenario. The models are always super confident and output values > 0.7 almost exclusively. - One thing that did reduced false-positives significantly is asking the model to give its reasoning before assigning a severity level to an issue found. - In my (rudimentary) experiments I found that "thinking" models like O3 don't improve on performance much and are not worth the additional tokens/time. (likely because I already ask for the reasoning anyway) - The models that perform best are Claude 4 and GPT-4.1 Let me know if you could see this be useful in your workflow, and what feature you would need to make it functional. [1]: https://github.com/rejot-dev/semcheck/commit/ce0af27ca0077fe... [2]: https://github.com/rejot-dev/semcheck/commit/2f96fc428b551d9... [3]: https://github.com/rejot-dev/semcheck/blob/47f7aaf98811c54e2... [4]: https://github.com/rejot-dev/semcheck/blob/fec2df48304d9eff9...
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, July 2025
the whole month →
Trickle - Magic Canvas ▲1,064The 1st Agentic Canvas for building apps visually with AI
AI · 2025 · trickle.so
- TB
Life & fun · 2025 · trytender.app
- TY
Today marks ten years, 3653 consecutive days, of running at least one mile every day under the USRSA rules [1]. To celebrate, I built an interactive dashboard that turns a decade of GPX files into charts you can explore. Running has truly changed my life: I've made lifelong friends, explored beautiful places, and more importantly invested into my own health and fitness, which I'm starting to see the positive benefits as I get older. The stack is pretty simple: a NextJS app, with a Postgres database to keep all my running data, and all the stats are pre-computed and cached in Redis, so I…
Dev tools · 2025 · nodaysoff.run
- DA
Made this website as an exercise in vibe-coding and GCP. It was posted about a few times around the internet, on sites like Morning Brew (https://www.morningbrew.com/issues/business-buzzkill), MetaFilter (https://www.metafilter.com/209703/Draw-A-Fish), boingboing.net, etc. I think it's cute! I built a basic CNN trained against penises and swastikas, and then anything that doesn't hit the 63% confidence score gets sent to a mod queue, a vibe-coded fish-tinder…
Dev tools · 2025 · drawafish.com
