Documentation
Providers & self-hosted
Your own inference server, or somebody else's gateway.
Self-hosted / local LLM
Pass your upstream base URL with X-LLM-API-Base.
Billing for local models uses a flat $0.50 per 1M tokens saved
— see How billing works.
default_headers={
"X-LLM-API-Key": "YOUR_LOCAL_KEY_OR_PLACEHOLDER",
"X-LLM-API-Base": "http://your-vllm:8000/v1",
}
OpenRouter, Grok, Groq, Together and other gateways
You are not limited to calling OpenAI or Anthropic directly. Anything with
an OpenAI-compatible API works, and several are known by name so you do not
even need a base URL. Your key goes in
X-LLM-API-Key exactly as before — it is that
provider's key, and we still never store it.
| Provider | Model name to send | Base URL needed? |
|---|---|---|
| OpenRouter | openrouter/anthropic/claude-3.5-sonnet | No — the prefix is enough |
| Grok (x.ai) | xai/grok-4 | No |
| Groq | groq/llama-3.3-70b-versatile | No |
| Together, Fireworks, DeepSeek, Mistral… | together_ai/… · fireworks_ai/… · deepseek/… · mistral/… | No |
| Anything else OpenAI-compatible | openai/<their-model-id> | Yes — X-LLM-API-Base |
One trap worth knowing. When you supply your own X-LLM-API-Base, the model name decides which API dialect we speak. A name that begins with a known provider — anthropic/claude-3.5-sonnet — makes us talk Anthropic to your gateway, and an OpenAI-compatible gateway answers that with a 404. Prefix it: openai/anthropic/claude-3.5-sonnet keeps the dialect OpenAI and passes the full id through. Or skip the base URL entirely and use openrouter/…, which is simpler and was what we verified.
# OpenRouter — no base URL needed
client = OpenAI(
base_url="https://compreslm.com/v1",
api_key="YOUR_COMPRESLM_TOKEN",
default_headers={"X-LLM-API-Key": "YOUR_OPENROUTER_KEY"},
)
client.chat.completions.create(
model="openrouter/anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": question}],
extra_body={"context": long_document, "compression_profile": "auto"},
)
Compression is identical whichever provider you use — it happens before the request leaves us, on the message array, so it never depends on who answers.