liteLLM Proxy Server: 50+ LLM Models, Error Handling, Caching

In plain words

liteLLM Proxy Server is an open-source tool that provides a single API endpoint to access over 50 large language models from providers including OpenAI, Azure, Cohere, Anthropic, and Hugging Face. It standardizes input and output across different models using OpenAI's format, so developers can call any model with consistent syntax. The proxy handles logging, error tracking, caching, and streaming, and includes model fallback capabilities to automatically switch providers if one fails. It is designed for developers building applications that need to work with multiple LLM providers through a unified interface.

written from the facts on this page · September 2026

From the sources

In the maker’s words, at launch

Hello hacker news, I’m the maintainer of liteLLM() - package to simplify input/output to OpenAI, Azure, Cohere, Anthropic, Hugging face API Endpoints: https://github.com/BerriAI/litellm/ We’re open sourcing our implementation of liteLLM proxy…

• 
  liteLLM2023 · litellm.ai · ▲120

  One library to standardize all LLM APIs

• 
  LLM Gateway2025 · llmgateway.io · ▲296

  I built an OpenAI-compatible LLM gateway that routes requests to OpenAI, Anthropic, Ollama, vLLM, llama-server, SGLang... anything that speaks /v1/chat/completions. Single Go binary, one YAML config file, no infrastructure. It does the things you'd expect from this kind of gateway... semantic routing via a three-layer cascade (keyword heuristics, embedding similarity, LLM classifier) that picks the best model when clients omit the model field, weighted round-robin load balancing across local inference servers with health checks and failover. The part I think is most…

• 
  Litellm – Simple library to standardize OpenAI, Cohere, Azure LLM I/O2023 · github.com · ▲62

  I built this library because langchain was too bloated and I needed a simple abstraction to call multiple LLM APIs. litellm has two functions - completion(), embedding()

• 
  GoModel – an open-source AI gateway in GoApr 2026 · github.com · ▲217

  Hi, I’m Jakub, a solo founder based in Warsaw. I’ve been building GoModel since December with a couple of contributors. It's an open-source AI gateway that sits between your app and model providers like OpenAI, Anthropic or others. I built it for my startup to solve a few problems: - track AI usage and cost per client or team - switch models without changing app code - debug request flows more easily - reduce AI spendings with exact and semantic caching How is it different? - ~17MB docker image - LiteLLM's image is more than 44x bigger ("docker.litellm.ai/berriai/litellm:latest" ~…

• 
  Context GatewayMar 2026 · github.com · ▲220

  We built an open-source proxy that sits between coding agents (Claude Code, OpenClaw, etc.) and the LLM, compressing tool outputs before they enter the context window. Demo: https://www.youtube.com/watch?v=-vFZ6MPrwjw#t=9s. Motivation: Agents are terrible at managing context. A single file read or grep can dump thousands of tokens into the window, most of it noise. This isn't just expensive — it actively degrades quality. Long-context benchmarks consistently show steep accuracy drops as context grows (OpenAI's GPT-5.4 eval goes from 97.2% at 32k to 36.6% at 1M…

• 
  TokenwiseJun 2026 · tokenwisehq.com · ▲143

  A smart LLM proxy that shows where you're overpaying

• 
  Spira Maxima▲578

  Video model that turns script into viral social media videos

  AI · 17d ago · spira.ai

• 
  Rill Browser▲540

  The browser where Claude Code and Codex work beside you

  AI · 16d ago · rill.love

• 
  Germany's new sovereign AI model Kolibri▲424

  Kolibri is Aleph Alpha

  AI · 18d ago · tej.as

• 
  Giving Opus 5.5 a simulated paint canvas▲392

  Paintings by AI models: programs run through a simulation of oil paint.

  AI · 20d ago · stillwet.art

• 
  Onepin▲384

  AI voiceover with every name, price and date said right

  AI · 30d ago · onepin.ai

• 
  PixVerse R2▲381

  A real-time world model you can explore and change

  AI · 28d ago · world.pixverse.video

• 
  Resend▲1,325

  An entire collection of pre-built components with a much faster development experience.

  Dev tools · 2023 · resend.com

• 
  STORI AI▲1,316

  Your ideas become visually compelling branded social posts

  AI · 2023 · storiai.com

• 
  Kombai▲1,079

  A new AI model that you can prompt with UI designs

  AI · 2023 · kombai.com

• 
  User Persona Generator▲1,014

  Understand your audience without 50 interviews

  AI · 2023 · founderpal.ai

• 
  Lottielab▲869

  Create and ship lottie animations to sites and apps faster

  Dev tools · 2023 · lottielab.com

• 
  LLMs can generate valid JSON 100% of the time▲854

  Outlines is a Python library that focuses on text generation with large language models. Brandon and I are not LLM experts and started the project a few months ago because we wanted to understand better how the generation process works. Our original background is probabilistic, relational and symbolic programming. Recently we came up with a fast way to generate text that matches a regex (https://blog.normalcomputing.ai/posts/2023-07-27-regex-guide...). The basic idea is simple: regular expressions have an equivalent Deterministic-Finite Automaton (DFA) representation. We…

  AI · 2023 · github.com