Custom LLMs

Use Another LLM or Your Own Server

Vapi supports using any OpenAI-compatible endpoint as the LLM. This includes services like OpenRouter, AnyScale, Together AI, or your own server.

When to Use Custom LLMs

• For an open-source LLM, like Mixtral
• To update the context during the conversation
• To customize the messages before they’re sent to an LLM

Using an LLM provider

You’ll first want to POST your API key via the /credential endpoint:

{
  "provider": "openrouter",
  "apiKey": "<YOUR OPENROUTER KEY>"
}

Then, you can create an assistant with the model provider:

{
  "name": "My Assistant",
  "model": {
    "provider": "openrouter",
    "model": "cognitivecomputations/dolphin-mixtral-8x7b",
    "messages": [
      {
        "role": "system",
        "content": "You are an assistant."
      }
    ],
    "temperature": 0.7
  }
}

Using your server

To set up your server to act as the LLM, you’ll need to create an endpoint that is compatible with the OpenAI Client. For best results, your endpoint should also support streaming completions.

If your server is making calls to an OpenAI compatble API, you can pipe the requests directly back in your response to Vapi.

If you’d like your OpenAI-compatible endpoint to be authenticated, you can POST your server’s API key and URL via the /credential endpoint:

{
  "provider": "custom-llm",
  "apiKey": "<YOUR SERVER API KEY>"
}

If your server isn’t authenticated, you can skip this step.

Then, you can create an assistant with the custom-llm model provider:

{
  "name": "My Assistant",
  "model": {
    "provider": "custom-llm",
    "url": "<YOUR OPENAI COMPATIBLE ENDPOINT BASE URL>",
    "model": "my-cool-model",
    "messages": [
      {
        "role": "system",
        "content": "You are an assistant."
      }
    ],
    "temperature": 0.7
  }
}