Alternatives
Products that do what ChainFactory – Run Structured LLM Inference with Easy Parallelism does
hi everyone. how does moving llm call prompts and output structure definitions away from code into configuration land sound? would you use something like this if it was stable and well documented enough? please don't hold back the criticism. i appreciate all feedback (constructive & otherwise).
- 1CM
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…
Mar 2026 · github.com
- 2A1
I've seen a lot of comments about how complex frameworks like LangChain can be. Over the holidays, I wanted to see how minimal an LLM framework could get if we stripped away everything non-essential. The result is an LLM framework in just 100 lines of code. These 100 lines capture what I see as the core abstraction of most LLM frameworks: a nested directed graph that breaks down tasks into multiple LLM steps, with branching and recursion to enable agent-like decision-making. From there, you can layer on more advanced features like agents, RAG, task decomposition, and more. I’ve intentionally…
2025 · github.com
- 3HP
Hi HN. I heard you like dev tools and AI, so we wanted to share our project that we’ve been working on. We’re working on Horizon [1] - a higher level abstraction for LLMs so that developers can spend less time trying to grapple with LLMs to make them work and more time with users. This is the starting feature set which takes an auto-ML approach to identify the optimal LLM model, hyperparameters, and prompt - instead of just giving you the tooling to figure it out yourself. You can read more about it in our documentations. Our view is that as LLMs become increasingly commoditized and prompts…
2023 · gethorizon.ai
- 4LB
For the past few months I've been building a lot of things with LLMs (GPT-3, Codex, etc.) as I've been trying to push them to their limits (especially towards applying them to the tabular data domain) When working on this, I've found there are some common patterns for solving problems (templating, chaining, functional-programming style operations, etc.) As I've iterated, I've come to believe that a functional style interface is likely going to power a new wave of systems I'm calling "prompt-machines"(systems where the core new unit of work is a "named" LLM prompt, extending the "function"…
2022 · github.com
- 5WB
Here is a production-first Keras-inspired LM framework, built with the advice of François Chollet (ex-Google, creator of Keras and ARC-AGI), our technical advisor. This system have already been deployed in production with our clients (which is why we have already every LLMOps practice implemented). It is also compatible with Jupyter and Marimo to integrate seamlessly in you Data Scientists workflows. You can try the code examples online on HF space and you can find more information in the documentation and FAQ. If you have any feedback for us don't hesitate to join our discord! More releases…
2025 · github.com
- 6AO
Hi, We are building an open-source framework for loading and structuring LLM context to create accurate and explainable LLM answers using knowledge graphs and vector stores. We built the tool with four main concepts in mind: 1. Loader -> uses dlt in the backend to load and structure the data 2. Cognify step -> creates a graph with summaries, labels and factoids that are interconnected across the documents and stored as a representation in the vector store 3. Optimizer -> Uses DSPy to optimize LLM queries, and we plan to extend it to most of the knobs we can turn, like chunking etc. 4. Search…
2024 · github.com
- 7LH
I work on inference scheduling — KV cache-aware routing, load balancing across GPU workers, that kind of thing. I wanted something like k9s but for my inference stack. Nothing existed, so I built it. llmtop is a real-time terminal dashboard for LLM inference workers. It scrapes the Prometheus /metrics endpoints that vLLM, SGLang, and LMCache already expose and shows everything in one view: KV cache usage, queue depth, TTFT/ITL latencies (P50/P99 from histogram buckets), token throughput, prefix cache hit rates. Color-coded — red means go fix it. ``` brew install…
Mar 2026 · github.com
- 8LF
I submitted an earlier version of this a few months ago (as llama2.f90). At that time it had a lot of steps to run and was just a toy, now it's easy to run and is a competitive option for llm inference. See the motivation section for discussion and the `Performance` issue for an ongoing discussion about performance.
2023 · github.com
- 9IL
LLM Application development is extremely iterative, more so than any other types of development. This is because in addition to all the activities involved in regular application development, we also need to make the LLM Application accurate and reduce hallucination. To improve performance, we need to trial and error various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt context with different RAG architecture, try different agent architecture, and more. There are thousands of permutations to try. We need to be able to easily experiment with these…
2024 · palico.ai
- 10NL
Built this because I was tired of every AI tool shipping my data to someone else server n0x runs the full stack LLM inference via WebGPU, autonomous ReAct agents, RAG over your own docs, sandboxed Python execution via Pyodide all inside a single browser tab. No account No keys No backend Models download once, cache in IndexedDB permanently. Biggest challenge was context window budgeting for the agent loop and making the WASM vector search non-blocking. Happy to talk architecture. GitHub: https://github.com/ixchio/n0x | Live demo: https://n0x-three.vercel.app
Mar 2026 · n0xth.vercel.app
- 11SB
*Motivation* Hi hackers, I'm Asif. I know we dislike premature standardization, but hear me out. LLM Application development is extremely iterative, more so than most other types of application development. We need a process that allows us to iterate faster. LLM Development is highly iterative due to the activities that come with regular software development, as well as the need to make the LLM Application accurate and reduce hallucination. To improve hallucination, we need to trial and error various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt…
2024 · github.com
- 12CA
Hi HN, I've been working with LLMs in production for a while both as a solo dev building apps for clients and working at an AI startup. The one thing that always was a pain was to pay OpenAI/Gemini/Anthropic a few dollars a month just for me to say "test" or have a CI runner validate some UI code. So I built this server called ChunkBack, that mocks the popular llm provider's functionality but allows you to type in a deterministic language: `SAY "cheese"` or `TOOLCALL "tool_name" {} "tool response"` I've had to work in some test environments and give good results for experimenting…
Nov 2025 · github.com
- 13KA
Hey HN! I've spent the past year full-time building Knowing, a tool for interacting with LLMs directly inside hierarchical structures instead of the usual prompt-response format. The idea started because I realized how much more intuitive it felt to build concept hierarchies continuously—no more endless copy-pasting or wondering how everything connects. The journey’s been a struggle. While I see huge potential in structuring AI interactions this way (writing books fast, planning projects, or organizing ideas), it’s been hard to pin down clear use cases in the market. I’m also working in near…
2024
- 14LS
Hi HN, I built llm.sql, an LLM inference framework that reimagines the LLM execution pipeline as a series of structured SQL queries atop SQLite. The motivation: Edge LLMs are getting better, but hardware remains a bottleneck, especially RAM (size and bandwidth). When available memory is less than the model size and KV cache, the OS incurs page faults and swaps pages using LRU-like strategies, resulting in throughput degradation that's hard to notice and even harder to debug. In fact, the memory access pattern during LLM inference is deterministic - we know exactly which weights are needed…
Apr 2026
- 15AT
We kept shipping “simple” LLM features that were fluent-but-wrong. After too many postmortems we wrote down the failure patterns and added a small reasoning layer in front of the model. It’s model-agnostic, sits beside your existing stack, and you can implement it from a single PDF (MIT). What’s inside the PDF A problem map of 16 failure modes we kept hitting in real systems (OCR/layout drift, table-to-question mismatches, embedding≠meaning, pre-deploy collapse, etc.). Four lightweight gates you can add today: Knowledge-boundary canaries (empty/adversarial/known-fact probes).…
2025 · github.com
- 16OS
Hi HN, we’re Dylan and Matthew, building sublingual (https://github.com/sublingual-ai/sublingual), an open-source LLM observability tool you can use with zero code changes. As developers focused on iterating and building features as fast as possible, we felt observability would’ve been a helpful tool to have, but we found existing solutions had too much overhead to set up. So we gave ourselves the challenge of building an observability tool that you can integrate without changing a single line of code in your project. How it works Run your python application as usual with…
2025 · github.com
- 17IO
Hey folks, I’m the creator of WFGY — a semantic reasoning framework for LLMs. After open-sourcing it, I did a full technical and value audit — and realized this engine might be worth $8M–$17M based on AI module licensing norms. If embedded as part of a platform core, the valuation could exceed $30M. Too late to pull it back. So here it is — fully free, open-sourced under MIT. --- ### What does it solve? Current LLMs (even GPT-4+) lack *self-consistent reasoning*. They struggle with: - Fragmented logic across turns - No internal loopback or self-calibration - No modular thought units - Weak…
2025 · github.com
- 18PA
Hey HN! We just launched PromptL: a templating language built to simplify writing complex prompts for LLMs like GPT-4 and Claude. Why PromptL? Creating dynamic prompts for LLMs can get tricky, even with standardized APIs that use lists of messages and settings. While these formats are consistent, building complex interactions with custom logic or branching paths can quickly become repetitive and hard to manage as prompts grow. PromptL steps in to make this simple. It allows you to define and manage LLM conversations in a readable, single-file format, with support for control flow and…
2024 · promptl.ai
- 19MA
Hi HN, A couple weeks ago I shared an early version of a side project I’ve been tinkering with called Persistent Mind Model. I built it at home on an i7-10700K / 32GB RAM / RTX 3080 because I was curious whether an AI could keep a stable “mind” over time, that could "think" about it's own identity as an LLM, instead of resetting every session. After a lot more tinkering, I think the architecture is finally in a solid place. Basically, it saves everything the AI does, thoughts, decisions, updates as a chain of events in a local SQLite database. Because the “identity” is stored in…
Nov 2025 · github.com
- 205L
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
2025 · github.com
- 21EC
Hi! I've found myself repeatedly writing little scripts to do bulk calls to LLMs for various tasks. For example, run some analysis on a large list of records. There are a few "gotchas" to doing this. For example, some service providers have rate limits, and some models will not reliably return JSON (if you're asking for it). So, I've written a command for this. What I've tried to do here is let the user break up prompts and configuration as they see fit. For example, you can have a prompt file which includes the API key, rate limit, settings, etc. all together, or break these up into…
2025 · github.com
- 22LG
Hi there, I've decided to jump on the AI train and put something together with low effort & high reward, to see if it can get any traction. What do you think? Is it a promising area? Do you guys have ideas for me? There is obviously going to be sea of LLM generated content out there and one project adding up to it might not necessarily be what world needs. In the same time there is something intriguing about the area. Well, please play with it and let me know what y'all think. Much appreciated.
2023 · canonica.ai
- 23IB
Hi HN, I'm pleased to share Promptspot, an open-source (Apache License 2.0) project that helps automate testing of large language model (LLM) prompts against an array of input data. Modern LLMs offer an enormous amount of leverage if you "teach the bot to fish" — i.e. simply prompt it with both a "system prompt" (which typically doesn't change often) and a dynamic input, which is often application state, search results, recent activity, user profile data, etc. Existing playgrounds and prompt management systems often lack the rigor and flexibility required for this dynamic approach — and as…
2023 · github.com
- 24UL
I was using LLM frameworks everywhere but had no idea what was happening inside them. One day I needed to optimize something and realized I couldn't. Hard truth: I didn't understand the fundamentals, just which framework function to call. So I stripped everything away. No abstractions. Just Python, HTTP requests, and the OpenAI/Anthropic APIs. What I found was anticlimactic in the best way: there's almost nothing there. - "AI agents" are just functions the model tells you to call - "Memory" is literally just a list you append to and send back - "RAG" is search, concatenate to prompt,…
Oct 2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →