Distributed Inference,Apply DL to WebRTC Video Frames via Redis Streams
Hi HN! I’m so excited to show my another open-source project here. It is a PoC project. Distributed Inference is a project to demonstrate an approach to designing cross-language and distributed pipeline in deep learning/machine learning domain, using WebRTC and Redis Streams. This project consists of multiple services, which are written in Go, Python, and TypeScript, running on Docker. It allows setting up multiple inference services in multiple host machines, in a distributed manner. It does RPC-like calls and service discovery via my other open-source projects, go-inventa and…
What it does
In the maker’s words, at launch
Hi HN! I’m so excited to show my another open-source project here. It is a PoC project. Distributed Inference is a project to demonstrate an approach to designing cross-language and distributed pipeline in deep learning/machine learning domain, using WebRTC and Redis Streams. This project consists of multiple services, which are written in Go, Python, and TypeScript, running on Docker. It allows setting up multiple inference services in multiple host machines, in a distributed manner. It does RPC-like calls and service discovery via my other open-source projects, go-inventa and py-inventa, you can find them in my profile too. Also includes a monitoring stack configuration using Grafana, InfluxDB, Telegraf, and Prometheus. The main idea is: - Webcam video will be streamed to the Media Bridge service via WebRTC, - Media Bridge service will capture frame images from the video as JPEG images, pushes them to Redis Streams, - One of available Inference services will pop a JPEG image data from Redis Streams stream, execute YOLOX inference model, push detected objects' name, box coordinates, prediction score, and resolution to other Redis Streams stream, - The Signaling service listens and consumes the Redis Streams stream (predictions), sends the results to relevant participant (by participantId in the JSON) via WebSockets. - Web client will draw boxes for each prediction, and writes results to the browser console. Please check it out and I’d love to read your thoughts! https://github.com/adalkiran/distributed-inference
Does the same job
all alternatives →
- GOgRPC over WebRTC2020 · github.com · ▲102
- RBRTCTunnel, build network tunnels over WebRTC2018 · github.com · ▲89
- TRTry running deep learning inference on Raspberry Pi2018 · actcast.io · ▲53
- WEWebRTC-Echoes: Interop for C#, C++, Python, TypeScript, Go and Servers2021 · github.com · ▲5
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
