Creating External Providers

Configuration

To enable external providers, you need to add module into your build yaml, allowing Llama Stack to install the required package corresponding to the external provider.

an example entry in your build.yaml should look like:

- provider_type: remote::ramalama
  module: ramalama_stack

Provider Types

Llama Stack supports two types of external providers:

1. Remote Providers: Providers that communicate with external services (e.g., cloud APIs)
2. Inline Providers: Providers that run locally within the Llama Stack process

Provider Specification (Common between inline and remote providers)

• provider_type: The type of the provider to be installed (remote or inline). eg. remote::ollama
• api: The API for this provider, eg. inference
• config_class: The full path to the configuration class
• module: The Python module containing the provider implementation
• optional_api_dependencies: List of optional Llama Stack APIs that this provider can use
• api_dependencies: List of Llama Stack APIs that this provider depends on
• provider_data_validator: Optional validator for provider data.
• pip_packages: List of Python packages required by the provider

Remote Provider Specification

Remote providers are used when you need to communicate with external services. Here's an example for a custom Ollama provider:

adapter_type: custom_ollama
provider_type: "remote::ollama"
pip_packages:
- ollama
- aiohttp
config_class: llama_stack_ollama_provider.config.OllamaImplConfig
module: llama_stack_ollama_provider
api_dependencies: []
optional_api_dependencies: []

Remote Provider Configuration

• adapter_type: A unique identifier for this adapter, eg. ollama

Inline Provider Specification

Inline providers run locally within the Llama Stack process. Here's an example for a custom vector store provider:

module: llama_stack_vector_provider
provider_type: inline::llama_stack_vector_provider
config_class: llama_stack_vector_provider.config.VectorStoreConfig
pip_packages:
  - faiss-cpu
  - numpy
api_dependencies:
  - inference
optional_api_dependencies:
  - vector_io
provider_data_validator: llama_stack_vector_provider.validator.VectorStoreValidator
container_image: custom-vector-store:latest  # optional

Inline Provider Fields

• container_image: Optional container image to use instead of pip packages

Required Fields

All Providers

All providers must contain a get_provider_spec function in their provider module. This is a standardized structure that Llama Stack expects and is necessary for getting things such as the config class. The get_provider_spec method returns a structure identical to the adapter. An example function may look like:

from llama_stack.providers.datatypes import (
    ProviderSpec,
    Api,
    RemoteProviderSpec,
)


def get_provider_spec() -> ProviderSpec:
    return RemoteProviderSpec(
        api=Api.inference,
        adapter_type="ramalama",
        pip_packages=["ramalama>=0.8.5", "pymilvus"],
        config_class="ramalama_stack.config.RamalamaImplConfig",
        module="ramalama_stack",
    )

Remote Providers

Remote providers must expose a get_adapter_impl() function in their module that takes two arguments:

1. config: An instance of the provider's config class
2. deps: A dictionary of API dependencies

This function must return an instance of the provider's adapter class that implements the required protocol for the API.

Example:

async def get_adapter_impl(
    config: OllamaImplConfig, deps: Dict[Api, Any]
) -> OllamaInferenceAdapter:
    return OllamaInferenceAdapter(config)

Inline Providers

Inline providers must expose a get_provider_impl() function in their module that takes two arguments:

1. config: An instance of the provider's config class
2. deps: A dictionary of API dependencies

Example:

async def get_provider_impl(
    config: VectorStoreConfig, deps: Dict[Api, Any]
) -> VectorStoreImpl:
    impl = VectorStoreImpl(config, deps[Api.inference])
    await impl.initialize()
    return impl

Dependencies

The provider package must be installed on the system. For example:

$ uv pip show llama-stack-ollama-provider
Name: llama-stack-ollama-provider
Version: 0.1.0
Location: /path/to/venv/lib/python3.10/site-packages

Best Practices

1. Package Naming: Use the prefix llama-stack-provider- for your provider packages to make them easily identifiable.

2. Version Management: Keep your provider package versioned and compatible with the Llama Stack version you're using.

3. Dependencies: Only include the minimum required dependencies in your provider package.

4. Documentation: Include clear documentation in your provider package about:

   • Installation requirements
   • Configuration options
   • Usage examples
   • Any limitations or known issues

5. Testing: Include tests in your provider package to ensure it works correctly with Llama Stack. You can refer to the integration tests guide for more information. Execute the test for the Provider type you are developing.

Troubleshooting

If your external provider isn't being loaded:

1. Check that module points to a published pip package with a top level provider module including get_provider_spec.
2. Verify that the YAML files are properly formatted.
3. Ensure all required Python packages are installed.
4. Check the Llama Stack server logs for any error messages - turn on debug logging to get more information using LLAMA_STACK_LOGGING=all=debug.

Examples

How to create an external provider module

If you are creating a new external provider called llama-stack-provider-ollama here is how you would set up the package properly:

1. First, create the provider package:

mkdir -p llama-stack-provider-ollama
cd llama-stack-provider-ollama
git init
uv init

2. Edit pyproject.toml:

[project]
name = "llama-stack-provider-ollama"
version = "0.1.0"
description = "Ollama provider for Llama Stack"
requires-python = ">=3.12"
dependencies = ["llama-stack", "pydantic", "ollama", "aiohttp"]

3. Install the provider:

uv pip install -e .

4. Edit provider.py

provider.py must be updated to contain get_provider_spec. This is used by llama stack to install the provider.

def get_provider_spec() -> ProviderSpec:
    return RemoteProviderSpec(
        api=Api.inference,
        adapter_type="llama-stack-provider-ollama",
        pip_packages=["ollama", "aiohttp"],
        config_class="llama_stack_provider_ollama.config.OllamaImplConfig",
        module="llama_stack_provider_ollama",
    )

5. Implement the provider as outlined above with get_provider_impl or get_adapter_impl, etc.

Example using module: ramalama-stack

ramalama-stack is a recognized external provider that supports installation via module.

To install Llama Stack with this external provider a user can provider the following build.yaml:

version: 2
distribution_spec:
  description: Use (an external) Ramalama server for running LLM inference
  container_image: null
  providers:
    inference:
    - provider_type: remote::ramalama
      module: ramalama_stack==0.3.0a0
image_type: venv
image_name: null
additional_pip_packages:
- aiosqlite
- sqlalchemy[asyncio]

No other steps are required beyond installing dependencies with llama stack list-deps <distro> | xargs -L1 uv pip install and then running llama stack run. The CLI will use module to install the provider dependencies, retrieve the spec, etc.

The provider will now be available in Llama Stack with the type remote::ramalama.