💥 OpenAI Proxy Server

📄️ Quick Start

Quick start CLI, Config, Docker

Set model list, apibase, apikey, temperature & proxy server settings (master-key) on the config.yaml.

Features here are behind a commercial license in our /enterprise folder. See Code

Input, Output, Exceptions are mapped to the OpenAI format for all supported models

Track Spend, Set budgets and create virtual keys for the proxy

Requirements:

Route calls to different model groups based on the team-id

Create + delete keys through a UI

Alerts when a project will exceed it’s planned limit

2 items

Add new models + Get model info without restarting proxy.

Use this to health check all LLMs defined in your config.yaml

2 levels of debugging supported.

LiteLLM supports Microsoft Presidio for PII masking.

Cache LLM Responses

3 items

2 items

You can find the Dockerfile to build litellm proxy here

Cli arguments, --host, --port, --num_workers