Ollama-Compatible vLLM Proxy
Ollama's API in front of three vLLM servers
- Year
- 2026
- Kind
- Systems
- Status
- In use
A FastAPI proxy that speaks Ollama's API on port 11434 and routes each request to one of three vLLM servers on a single GPU, translating requests and token streams both ways.
An app built against Ollama's API keeps working unchanged while its models run on vLLM. A launch script starts Gemma 4 E4B, DeepSeek-OCR and nomic-embed-text as separate vLLM servers on one GPU, bound to localhost, then starts the proxy. The proxy maps Ollama's chat, generate and embedding endpoints to vLLM's OpenAI-style API, and converts its Server-Sent Events back into Ollama's NDJSON stream. A Cloudflare tunnel exposes the single proxy port. I built it to develop an air-gapped document assistant on a Colab GPU, since the models were too big for my own hardware.
The problem
An air-gapped document assistant I was building talked to Ollama's API on port 11434, and its models were too big for my own hardware. The goal was to serve them with vLLM on a Colab GPU during development, without changing the app. This proxy speaks Ollama's API on Ollama's port and forwards each request to the vLLM server that hosts the requested model, translating requests on the way in and token streams on the way out.
How it works
- 01
Start three models on one GPU
A launch script starts three vLLM servers, each bound to 127.0.0.1: Gemma 4 E4B on port 8001, DeepSeek-OCR on 8002, and nomic-embed-text v1.5 for embeddings on 8003. Each gets a fixed share of GPU memory, 35%, 35% and 15%, because vLLM reserves its share up front. The script polls each server's /v1/models every 5 seconds until all three answer, then starts the proxy on port 11434. A server that never answers stops the launch after 15 minutes (READY_TIMEOUT) instead of leaving it waiting, and each server runs under a small supervisor that restarts it if it crashes, waiting 5 seconds at first and doubling up to 5 minutes.
- 02
Route by model name
Ollama clients send names like gemma4:latest, so the proxy strips the tag and looks the base name up in a routing table. Each entry holds the backend URL, the Hugging Face model id and whether the model does chat or embeddings. Unknown names get a 404 with an Ollama-style error. GET /api/tags lists the three models as installed, and GET / answers "Ollama is running", the same string Ollama returns.
- 03
Translate the request
Ollama's /api/chat and /api/generate both become OpenAI-style chat completion calls to vLLM, and a generate prompt becomes a single user message, preceded by its system prompt if it has one. Base64 images become image_url parts with data URIs. Options are renamed where an equivalent exists, num_predict to max_tokens for example, and any that have no OpenAI equivalent are named in a warning in the log. Ollama's single and batch embedding endpoints both map to vLLM's /v1/embeddings.
- 04
Stream tokens back
vLLM streams Server-Sent Events: lines starting with data: that each carry a small token delta. The proxy reads them as they arrive and writes one line of NDJSON per delta in Ollama's shape, then turns the closing [DONE] into a final line with done set to true. Non-streaming calls get a single JSON reply, with token usage mapped to Ollama's eval_count and prompt_eval_count. Failures come back the way Ollama reports them: an {"error": ...} body, with upstream 5xx errors turned into 502, and a stream that fails midway still ends with an error line and a final done line, so the client never hangs.
Decisions
Keep the Ollama API
The app and its Cloudflare route stay as they were. In the proxy's own docstring, the app never knows vLLM is back there.
Expose one port
The vLLM servers listen only on 127.0.0.1. The proxy on 11434 is the only port the Cloudflare tunnel needs to reach, and it binds to 127.0.0.1 too, so only cloudflared on the same host can connect. Setting PROXY_API_KEY adds a bearer-token check on every request; a key that is set but blank stops the proxy from starting rather than quietly turning the check off.
Model quirks live in the proxy
DeepSeek-OCR needs extra sampling parameters to avoid degenerate output: an n-gram logits processor with an n-gram size of 30 and a window of 90. The routing table adds them to every request for that model, so clients send plain Ollama requests.