Alternatives
Products that do what Distributed Inference,Apply DL to WebRTC Video Frames via Redis Streams does
Hi HN! I’m so excited to show my another open-source project here. It is a PoC project. Distributed Inference is a project to demonstrate an approach to designing cross-language and distributed pipeline in deep learning/machine learning domain, using WebRTC and Redis Streams. This project consists of multiple services, which are written in Go, Python, and TypeScript, running on Docker. It allows setting up multiple inference services in multiple host machines, in a distributed manner. It does RPC-like calls and service discovery via my other open-source projects, go-inventa and…
- 1
- 2
- 3
- 4WI
2018 · github.com
- 5GOgRPC over WebRTC▲102
2020 · github.com
- 6RB
2018 · github.com
- 7TR
2018 · actcast.io
- 8
- 9GA
2021 · inferrd.com
- 10WE
2021 · github.com
- 11S1
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
Nov 2025 · github.com
- 12FS
Hi everyone! I've been loving building with AI, and over the past few years I've been leaning more and more into Typescript (and bun). My team at inference.net is constantly trying to get more leverage out of AI and find ways to setup our codebase to be able to increase the level of correctness that our AI is able to write code at. This starter repo is a very opinionated way to lay out a repo to lean into AI heavily. It leverages Cloudflare Workers as a deployment target for the API (my goal is to never have to deploy an API on a AWS/Azure/GCP server ever again unless I get to a…
2025 · abeahmed.com
- 13FM
As the title and description of the GitHub repo suggest, I’m working on a small project for purely educational purposes, with the goal of implementing generative model inference (small models capable of modeling 2D distributions) based on the Flow Matching paradigm in C. I’ve worked on generative AI models based on Flow Matching from a more “abstract” perspective, using frameworks like PyTorch, and I wanted to understand what goes on behind the scenes. The repository is still a work in progress and is also one of my first "serious" projects in C.
Jul 2026 · github.com
- 14WC
2019 · github.com
- 15LT
I wanted to share a project I've been working on for the past few weeks: llgtrt. It's a Rust implementation of a HTTP REST server for hosting Large Language Models using llguidance library for constrained output with NVIDIA TensorRT-LLM. The server is compatible with the OpenAI REST API and supports structured JSON schema enforcement as well as full context-free grammars (via Guidance). It's similar in spirit to the Python-based TensorRT-LLM OpenAI server example but written entirely in Rust and built with constraints in mind. No Triton Inference Server involved. This also serves as a demo…
2024 · github.com
- 16CM
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…
Mar 2026 · github.com
- 17FC
Hi HN,I am Anubhav from RamanLabs.We have been developing dedicated modules based on deep-learning for purposes like face-detection,object-detection,pose-estimation etc. We hope to make it easy for developers,hobbyists to integrate such functionalities into their existing app/pipeline at the cost of a few milliseconds.All our modules run end to end in super-realtime even on consumer-grade CPUs[0]. For now we provide only Python based API. We provide Demo for each of the modules to allow testing for your desired data distribution.We also have a blog[1] where we hope to add more technical…
2022 · ramanlabs.in
- 18SI
2019 · github.com
- 19IA
Hi HN, I'm one of the creators of Exosphere. Think of us like a reliability lab for agents. Today we are launching Exosphere Flex Inference APIs: Inference APIs should adapt to your constraints, not the other way around. Usually, when you need to run inference at scale, you are forced into rigid boxes: 1. "Real-time" APIs (Expensive, optimized for <1s latency, prone to 429s). 2. "Batch" APIs (Cheaper, but often force 24-hour windows and rigid file formats). 3. "Self-hosted" (Total control, but high ops overhead). We built a flexible inference engine that sits in the middle. You define the…
Jan 2026 · models.exosphere.host
- 205L
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
2025 · github.com
- 21
- 22IE
Quick note on how it works and how I've done my batch embedding engine IgniteMS. The whole thing runs as one process using Rust, reading input, tokenizing, packing batches, keeping the queue full. TensorRT handles inference. Python is only as a wrapper. I built it this way because when you use more than couple of GPUs, the GPUs stop being the problem. CPU cannot feed them fast enough. One A100 can go through batches faster than Python can tokenize and feed, so the GPU just sits there idle waiting for work. Most of my time went into optimizing this. At 8 GPUs that was basically the entire…
Jun 2026 · github.com
- 23IC
Built a community-driven database for inference hardware. All results are submitted by users and validated by volunteers.
Dec 2025 · inferbench.com
- 24CA
We open-sourced catsu, a Python client for embedding APIs. The problem: every embedding provider has a different SDK with different bugs. OpenAI has undocumented token limits. VoyageAI's retry logic was broken until September. Cohere breaks downstream libraries every release. LiteLLM's embedding support is minimal. catsu provides: - One API for 11 providers (OpenAI, Voyage, Cohere, Jina, Mistral, Gemini, etc.) - Bundled database of 50+ models with pricing, dimensions, and benchmark scores - Built-in retry with exponential backoff - Automatic cost tracking per request - Full async support…
Dec 2025 · catsu.dev
Ranked by how close each launch is in meaning, then by votes. Refine with a description →