Skip to main content
Version: v0.2.23

Telemetry

The Llama Stack telemetry system provides comprehensive tracing, metrics, and logging capabilities. It supports multiple sink types including OpenTelemetry, SQLite, and Console output for complete observability of your AI applications.

Event Types​

The telemetry system supports three main types of events:

Free-form log messages with severity levels for general application logging:

unstructured_log_event = UnstructuredLogEvent(
    message="This is a log message",
    severity=LogSeverity.INFO
)

Spans and Traces​

  • Spans: Represent individual operations with timing information and hierarchical relationships
  • Traces: Collections of related spans that form a complete request flow across your application

This hierarchical structure allows you to understand the complete execution path of requests through your Llama Stack application.

Automatic Metrics Generation​

Llama Stack automatically generates metrics during inference operations. These metrics are aggregated at the inference request level and provide insights into token usage and model performance.

Available Metrics​

The following metrics are automatically generated for each inference request:

Metric NameTypeUnitDescriptionLabels
llama_stack_prompt_tokens_totalCountertokensNumber of tokens in the input promptmodel_id, provider_id
llama_stack_completion_tokens_totalCountertokensNumber of tokens in the generated responsemodel_id, provider_id
llama_stack_tokens_totalCountertokensTotal tokens used (prompt + completion)model_id, provider_id

Metric Generation Flow​

  1. Token Counting: During inference operations (chat completion, completion, etc.), the system counts tokens in both input prompts and generated responses
  2. Metric Construction: For each request, MetricEvent objects are created with the token counts
  3. Telemetry Logging: Metrics are sent to the configured telemetry sinks
  4. OpenTelemetry Export: When OpenTelemetry is enabled, metrics are exposed as standard OpenTelemetry counters

Metric Aggregation Level​

All metrics are generated and aggregated at the inference request level. This means:

  • Each individual inference request generates its own set of metrics
  • Metrics are not pre-aggregated across multiple requests
  • Aggregation (sums, averages, etc.) can be performed by your observability tools (Prometheus, Grafana, etc.)
  • Each metric includes labels for model_id and provider_id to enable filtering and grouping

Example Metric Event​

MetricEvent(
    trace_id="1234567890abcdef",
    span_id="abcdef1234567890",
    metric="total_tokens",
    value=150,
    timestamp=1703123456.789,
    unit="tokens",
    attributes={
        "model_id": "meta-llama/Llama-3.2-3B-Instruct",
        "provider_id": "tgi"
    },
)

Telemetry Sinks​

Choose from multiple sink types based on your observability needs:

Send events to an OpenTelemetry Collector for integration with observability platforms:

Use Cases:

  • Visualizing traces in tools like Jaeger
  • Collecting metrics for Prometheus
  • Integration with enterprise observability stacks

Features:

  • Standard OpenTelemetry format
  • Compatible with all OpenTelemetry collectors
  • Supports both traces and metrics

Configuration​

Meta-Reference Provider​

Currently, only the meta-reference provider is implemented. It can be configured to send events to multiple sink types:

telemetry:
  - provider_id: meta-reference
    provider_type: inline::meta-reference
    config:
      service_name: "llama-stack-service"
      sinks: ['console', 'sqlite', 'otel_trace', 'otel_metric']
      otel_exporter_otlp_endpoint: "http://localhost:4318"
      sqlite_db_path: "/path/to/telemetry.db"

Environment Variables​

Configure telemetry behavior using environment variables:

  • OTEL_EXPORTER_OTLP_ENDPOINT: OpenTelemetry Collector endpoint (default: http://localhost:4318)
  • OTEL_SERVICE_NAME: Service name for telemetry (default: empty string)
  • TELEMETRY_SINKS: Comma-separated list of sinks (default: console,sqlite)

Visualization with Jaeger​

The otel_trace sink works with any service compatible with the OpenTelemetry collector. Traces and metrics use separate endpoints but can share the same collector.

Starting Jaeger​

Start a Jaeger instance with OTLP HTTP endpoint at 4318 and the Jaeger UI at 16686:

docker run --pull always --rm --name jaeger \
  -p 16686:16686 -p 4318:4318 \
  jaegertracing/jaeger:2.1.0

Once running, you can visualize traces by navigating to http://localhost:16686/.

Querying Metrics​

When using the OpenTelemetry sink, metrics are exposed in standard format and can be queried through various tools:

Example Prometheus queries for analyzing token usage:

# Total tokens used across all models
sum(llama_stack_tokens_total)

# Tokens per model
sum by (model_id) (llama_stack_tokens_total)

# Average tokens per request over 5 minutes
rate(llama_stack_tokens_total[5m])

# Token usage by provider
sum by (provider_id) (llama_stack_tokens_total)

SQLite Querying​

The sqlite sink allows you to query traces without an external system. This is particularly useful for development and custom analytics.

Example Queries​

-- Query recent traces
SELECT * FROM traces WHERE timestamp > datetime('now', '-1 hour');

-- Analyze span durations
SELECT name, AVG(duration_ms) as avg_duration
FROM spans
GROUP BY name
ORDER BY avg_duration DESC;

-- Find slow operations
SELECT * FROM spans
WHERE duration_ms > 1000
ORDER BY duration_ms DESC;
Advanced Analytics

Refer to the Getting Started notebook for more examples on querying traces and spans programmatically.

Best Practices​

🔍 Monitoring Strategy​

  • Use OpenTelemetry for production environments
  • Combine multiple sinks for development (console + SQLite)
  • Set up alerts on key metrics like token usage and error rates

📊 Metrics Analysis​

  • Track token usage trends to optimize costs
  • Monitor response times across different models
  • Analyze usage patterns to improve resource allocation

🚨 Alerting & Debugging​

  • Set up alerts for unusual token consumption spikes
  • Use trace data to debug performance issues
  • Monitor error rates and failure patterns

🔧 Configuration Management​

  • Use environment variables for flexible deployment
  • Configure appropriate retention policies for SQLite
  • Ensure proper network access to OpenTelemetry collectors

Integration Examples​

Basic Telemetry Setup​

from llama_stack_client import LlamaStackClient

# Client with telemetry headers
client = LlamaStackClient(
    base_url="http://localhost:8000",
    extra_headers={
        "X-Telemetry-Service": "my-ai-app",
        "X-Telemetry-Version": "1.0.0"
    }
)

# All API calls will be automatically traced
response = client.inference.chat_completion(
    model="meta-llama/Llama-3.2-3B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

Custom Telemetry Context​

# Add custom span attributes for better tracking
with tracer.start_as_current_span("custom_operation") as span:
    span.set_attribute("user_id", "user123")
    span.set_attribute("operation_type", "chat_completion")

    response = client.inference.chat_completion(
        model="meta-llama/Llama-3.2-3B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}]
    )