Alternatives
Products that do what Trieve Vector Inference does
Deploy fast, unmetered embedding inference in your own VPC
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8DI
Hi HN! I’m so excited to show my another open-source project here. It is a PoC project. Distributed Inference is a project to demonstrate an approach to designing cross-language and distributed pipeline in deep learning/machine learning domain, using WebRTC and Redis Streams. This project consists of multiple services, which are written in Go, Python, and TypeScript, running on Docker. It allows setting up multiple inference services in multiple host machines, in a distributed manner. It does RPC-like calls and service discovery via my other open-source projects, go-inventa and…
2023 · github.com
- 9S1
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
Nov 2025 · github.com
- 10CA
We open-sourced catsu, a Python client for embedding APIs. The problem: every embedding provider has a different SDK with different bugs. OpenAI has undocumented token limits. VoyageAI's retry logic was broken until September. Cohere breaks downstream libraries every release. LiteLLM's embedding support is minimal. catsu provides: - One API for 11 providers (OpenAI, Voyage, Cohere, Jina, Mistral, Gemini, etc.) - Bundled database of 50+ models with pricing, dimensions, and benchmark scores - Built-in retry with exponential backoff - Automatic cost tracking per request - Full async support…
Dec 2025 · catsu.dev
- 11FC
Hi HN,I am Anubhav from RamanLabs.We have been developing dedicated modules based on deep-learning for purposes like face-detection,object-detection,pose-estimation etc. We hope to make it easy for developers,hobbyists to integrate such functionalities into their existing app/pipeline at the cost of a few milliseconds.All our modules run end to end in super-realtime even on consumer-grade CPUs[0]. For now we provide only Python based API. We provide Demo for each of the modules to allow testing for your desired data distribution.We also have a blog[1] where we hope to add more technical…
2022 · ramanlabs.in
- 125L
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
2025 · github.com
- 13
- 14MM
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional…
2024 · github.com
- 15BO
Read the full blogpost at https://rach.codes/blog/Introducing-Bhumi (click on reader to see the technical breakdown!) AI inference should be fast, but in practice it’s painfully slow. Inference bottlenecks slow down LLM-powered chatbots and AI workflows everywhere. I built Bhumi to fix that. Bhumi is a Python library designed for developers, yet its performance-critical core is implemented in Rust (via PyO3) for near-native speed. This hybrid approach delivers up to 2.5x faster response times across providers like OpenAI, Anthropic, and Gemini—without changing the…
2025 · bhumi.trilok.ai
- 16FM
As the title and description of the GitHub repo suggest, I’m working on a small project for purely educational purposes, with the goal of implementing generative model inference (small models capable of modeling 2D distributions) based on the Flow Matching paradigm in C. I’ve worked on generative AI models based on Flow Matching from a more “abstract” perspective, using frameworks like PyTorch, and I wanted to understand what goes on behind the scenes. The repository is still a work in progress and is also one of my first "serious" projects in C.
Jul 2026 · github.com
- 17SV
Hi HN, I'm Daniel from Superlinked! We have built an open-source framework that improves vector search relevance and usefulness by combining structured metadata with unstructured data in your embeddings. We included self-hostable API server that sits between your data sources and vector database. Docs: https://docs.superlinked.com/ We're launching our cloud offering soon where you can use Superlinked to orchestrate high-performance retrieval for RAG, Search & Recommendation apps in your own cloud. Looking for feedback and happy to answer questions!
2024 · github.com
- 18SF
Hey HN! We've just open-sourced Semble, a fast and accurate code search library built for agents. We're also releasing potion-code-16M, a small code-specialized static embedding model that powers it. Most embedding-based code search methods are either too slow to index on demand or need GPU infrastructure, while grep-style retrieval methods often cannot find the relevant content. Semble combines the speed and quality benefits of both, so agents waste less time and fewer tokens exploring. Main features: - Fast: indexes a full codebase in ~250 ms and answers queries in ~1.5 ms, all on CPU…
Apr 2026 · github.com
- 19SR
Hi HN! Sipp is an open-source AI inference library for running local models in browsers with up to 3x faster decode speeds than alternative libraries. My background is in HCI (human-computer interaction) and graphics programming. Me along with my co-founder have been experimenting and thinking a lot about what the next user experience will look like when tokens are commodified to the point of being essentially “free.” A motivation for us was to try to move beyond the chat app and information retrieval use cases that are dominant now, and figure out how AI could instead act as a continuous…
Jun 2026 · sipp.sh
- 20MM
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This reduction of course comes at a cost: distilled models are a lot worse than their parent models. Even so, they are actually a lot better than large sets of conventional static embeddings, such as GLoVe or word2vec-based models, which are many…
2024 · github.com
- 21IE
Quick note on how it works and how I've done my batch embedding engine IgniteMS. The whole thing runs as one process using Rust, reading input, tokenizing, packing batches, keeping the queue full. TensorRT handles inference. Python is only as a wrapper. I built it this way because when you use more than couple of GPUs, the GPUs stop being the problem. CPU cannot feed them fast enough. One A100 can go through batches faster than Python can tokenize and feed, so the GPU just sits there idle waiting for work. Most of my time went into optimizing this. At 8 GPUs that was basically the entire…
Jun 2026 · github.com
- 22MA
I've been working on training this small vision language model for the last month - excited to release the first prototype today! It is based on SigLIP (image encoder), Phi-1.5 (text model) and trained using the LLaVa-1.5 training dataset. It runs reasonably fast on CPU with ~8GB of RAM in full 32-bit precision. There's plenty of room to speed it up and reduce memory consumption by quantizing the model. I posted a video of it running on my M2 Macbook Air (on CPU not MPS, so performance should be comparable on other hardware) on Twitter to demonstrate inference speed:…
2023 · github.com
- 23BC
We are a small group of undergrads interested in building human in the loop coding agents. We dream of a world where building complex agent workflows feels as simple and creative as playing with legos. When we were building stuff we needed a tool that made it easy to try out different code embedding models so that we could see which ones worked best in different scenarios and understand their strengths and weaknesses. So to speed that process up we made PurpleSearch an 'instant' search engine for your local codebases. This tool lets you quickly deploy any open source embedding model on…
2025
- 24CM
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…
Mar 2026 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →