🤖 Model Configuration

Supported Models

At a minimum, a Language Model must be configured in the AI Optimizer for basic functionality. For Retrieval-Augmented Generation (RAG), an Embedding Model will also need to be configured.

There is an extensive list of different model providers available to choose from.

Configuration

The models can either be configured using environment variables or through the AI Optimizer interface. To configure models through environment variables, please read the Additional Information about the specific model provider you would like to configure.

To configure a Language Models or Embedding Models, from the AI Optimizer, navigate to Configuration page and Models tab:

Add/Edit/Delete

Set the Provider, API Key, and Provider URL as required. For Language Models you can also set the Max Input Tokens (Context Length) and Max Output (Completion) Tokens; for Embedding Models, set the Max Chunk Size. These values can often be found on the model card—if they are not listed, the defaults are usually sufficient.

Most models ship pre-configured but disabled. When editing a model, tick the Enabled checkbox to activate it.

To remove a model, use the Delete button while editing it; any settings that referenced it are cleared automatically.

Provider

The AI Optimizer supports a number of model providers. When adding a model, choose the most appropriate provider. If unsure, or the specific provider is not listed, try a LiteLLM OpenAI-compatible provider, such as openai_like or custom_openai, before opening an issue requesting additional model provider support.

There are a number of local AI Model runners that use OpenAI-compatible APIs, including:

• LM Studio
• vLLM
• LocalAI

When using these local runners, select the appropriate LiteLLM provider. For example, use hosted_vllm for vLLM, or an OpenAI-compatible provider such as openai_like or custom_openai for other compatible endpoints.

Provider URL

The Provider URL for the model will either be the URL, including the IP or Hostname and Port, of a locally running model; or the remote URL for a Third-Party or Cloud model.

Examples:

• Third-Party: OpenAI - https://api.openai.com
• On-Premises: Ollama - http://localhost:11434
• On-Premises: LM Studio - http://localhost:1234/v1

API Keys

Third-Party cloud models, such as OpenAI and Perplexity AI, require API Keys. These keys are tied to registered, funded accounts on these platforms. For more information on creating an account, funding it, and generating API Keys for third-party cloud models, please visit their respective sites.

On-Premises models, such as those from Ollama or HuggingFace usually do not require API Keys. These values can be left blank.

CPU Optimization

When running models on CPU-only systems (without GPU acceleration), smaller models provide significantly better performance and responsiveness. The AI Optimizer includes built-in optimizations for CPU-friendly models.

Recommended CPU-Friendly Models

Small local models are often a better fit for CPU-only systems. When an Ollama server is configured with AIO_ON_PREM_OLLAMA_URL, the AI Optimizer discovers pulled Ollama models and enables them automatically.

Examples of CPU-friendly model choices include:

Vector Search Optimization

When a selected model name includes a parameter count below 7B, such as llama3.2:1b or gemma3:1b, the AI Optimizer automatically disables these Vector Search features:

• Store Discovery: Uses the Language Model to select the most appropriate Vector Store. When disabled, you must select the Vector Store manually if more than one is available.
• Document Grading: Evaluates retrieved documents for relevance before they are used.
• Prompt Rephrase: Rephrases queries using the conversation context to improve retrieval.

Disabling these features reduces the number of Language Model calls and can significantly improve response times. You can manually enable or disable each feature using the checkboxes in the Vector Search sidebar, regardless of model size.

Performance Tips

1. Model Selection: Choose the smallest model that meets your quality requirements
2. Reduce Top K: Lower the number of retrieved documents (e.g., Top K = 3-5)
3. Lower Max Tokens: Reduce maximum output tokens to speed up generation
4. Temperature 0: Use temperature 0 for deterministic, faster responses

Additional Information

genai_compartment_id=<OCI Compartment OCID of the OCI GenAI Service>
genai_region=<OCI GenAI Region>

sudo curl -fsSL https://ollama.com/install.sh | sh

ollama pull llama3.2:3b

ollama serve