Skip to main content
Version: Next

Detailed Tutorial

Detailed Tutorial​

In this guide, we'll walk through how you can use the Llama Stack (server and client SDK) to test a simple agent. A Llama Stack agent is a simple integrated system that can perform tasks by combining a Llama model for reasoning with tools (e.g., RAG, web search, code execution, etc.) for taking actions. In Llama Stack, we provide a server exposing multiple APIs. These APIs are backed by implementations from different providers.

Llama Stack is a stateful service with REST APIs to support seamless transition of AI applications across different environments. The server can be run in a variety of ways, including as a standalone binary, Docker container, or hosted service. You can build and test using a local server first and deploy to a hosted endpoint for production.

In this guide, we'll walk through how to build a RAG agent locally using Llama Stack with Ollama as the inference provider for a Llama Model.

Step 1: Installation and Setup​

Install Ollama by following the instructions on the Ollama website, then download Llama 3.2 3B model, and then start the Ollama service.

ollama pull llama3.2:3b
ollama run llama3.2:3b --keepalive 60m

Install uv to setup your virtual environment

Use curl to download the script and execute it with sh:

curl -LsSf https://astral.sh/uv/install.sh | sh

Setup your virtual environment.

uv sync --python 3.12
source .venv/bin/activate

Step 2: Run Llama Stack​

Llama Stack is a server that exposes multiple APIs, you connect with it using the Llama Stack client SDK.

You can use Python to build and run the Llama Stack server, which is useful for testing and development.

Llama Stack uses a YAML configuration file to specify the stack setup, which defines the providers and their settings. The generated configuration serves as a starting point that you can customize for your specific needs. Now let's build and run the Llama Stack config for Ollama. We use starter as template. By default all providers are disabled, this requires enable ollama by passing environment variables.

llama stack build --distro starter --image-type venv --run

Step 3: Run Client CLI​

Open a new terminal and navigate to the same directory you started the server from. Then set up a new or activate your existing server virtual environment.

# The client is included in the llama-stack package so we just activate the server venv
source .venv/bin/activate

Now let's use the llama-stack-client CLI to check the connectivity to the server.

llama-stack-client configure --endpoint http://localhost:8321 --api-key none

You will see the below:

Done! You can now use the Llama Stack Client CLI with endpoint http://localhost:8321

List the models

llama-stack-client models list
Available Models

┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓
┃ model_type      ┃ identifier                          ┃ provider_resource_id                ┃ metadata                                  ┃ provider_id           ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩
│ embedding       │ ollama/all-minilm:l6-v2             │ all-minilm:l6-v2                    │ {'embedding_dimension': 384.0}            │ ollama                │
├─────────────────┼─────────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ ...             │ ...                                 │ ...                                 │                                           │ ...                   │
├─────────────────┼─────────────────────────────────────┼─────────────────────────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ llm             │ ollama/Llama-3.2:3b                 │ llama3.2:3b                         │                                           │ ollama                │
└─────────────────┴─────────────────────────────────────┴─────────────────────────────────────┴───────────────────────────────────────────┴───────────────────────┘

You can test basic Llama inference completion using the CLI.

llama-stack-client inference chat-completion --model-id "ollama/llama3.2:3b" --message "tell me a joke"

Sample output:

OpenAIChatCompletion(
    id="chatcmpl-08d7b2be-40f3-47ed-8f16-a6f29f2436af",
    choices=[
        OpenAIChatCompletionChoice(
            finish_reason="stop",
            index=0,
            message=OpenAIChatCompletionChoiceMessageOpenAIAssistantMessageParam(
                role="assistant",
                content="Why couldn't the bicycle stand up by itself?\n\nBecause it was two-tired.",
                name=None,
                tool_calls=None,
                refusal=None,
                annotations=None,
                audio=None,
                function_call=None,
            ),
            logprobs=None,
        )
    ],
    created=1751725254,
    model="llama3.2:3b",
    object="chat.completion",
    service_tier=None,
    system_fingerprint="fp_ollama",
    usage={
        "completion_tokens": 18,
        "prompt_tokens": 29,
        "total_tokens": 47,
        "completion_tokens_details": None,
        "prompt_tokens_details": None,
    },
)

Step 4: Run the Demos​

Note that these demos show the Python Client SDK. Other SDKs are also available, please refer to the Client SDK list for the complete options.

Now you can run inference using the Llama Stack client SDK.

i. Create the Script​

Create a file inference.py and add the following code:

from llama_stack_client import LlamaStackClient

client = LlamaStackClient(base_url="http://localhost:8321")

# List available models
models = client.models.list()

# Select the first LLM
llm = next(m for m in models if m.model_type == "llm" and m.provider_id == "ollama")
model_id = llm.identifier

print("Model:", model_id)

response = client.chat.completions.create(
    model=model_id,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about coding"},
    ],
)
print(response)

ii. Run the Script​

Let's run the script using uv

uv run python inference.py

Which will output:

Model: ollama/llama3.2:3b
OpenAIChatCompletion(id='chatcmpl-30cd0f28-a2ad-4b6d-934b-13707fc60ebf', choices=[OpenAIChatCompletionChoice(finish_reason='stop', index=0, message=OpenAIChatCompletionChoiceMessageOpenAIAssistantMessageParam(role='assistant', content="Lines of code unfold\nAlgorithms dance with ease\nLogic's gentle kiss", name=None, tool_calls=None, refusal=None, annotations=None, audio=None, function_call=None), logprobs=None)], created=1751732480, model='llama3.2:3b', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage={'completion_tokens': 16, 'prompt_tokens': 37, 'total_tokens': 53, 'completion_tokens_details': None, 'prompt_tokens_details': None})

You're Ready to Build Your Own Apps!

Congrats! 🥳 Now you're ready to build your own Llama Stack applications! 🚀