Observability

The AI Optimizer Server can emit OpenTelemetry traces and logs to any OTLP-compatible backend (e.g. SigNoz, Jaeger, Grafana Tempo). Telemetry is opt-in — disabled by default and activated entirely via environment variables.

What Is Instrumented

Telemetry covers HTTP traffic, LangChain/LangGraph orchestration, LLM invocations, and application logs on the server:

A typical chat request produces a tree like:

POST /v1/chat                            SERVER
└── LangGraph invocation                 INTERNAL
    ├── retrieve node                    INTERNAL
    └── generate node                    INTERNAL
        └── ChatLiteLLM.invoke           INTERNAL  ← model, tokens, prompt/response
            └── HTTP POST <provider>     CLIENT    ← transport timing

The LangChain LLM span carries semantic LLM information (model, tokens, content); the child httpx span carries transport-level information (URL, status, duration). They are complementary, not duplicates.

Log records emitted during the lifetime of any span carry that span’s trace_id and span_id, so a backend can show the logs from a specific span when its trace is opened.

The Streamlit Client is not instrumented; it is a thin REST client whose work is reflected in the server-side traces it triggers.

Enabling

1. Install the otel extra

The OpenTelemetry packages live in an optional dependency group. They are not installed by default.

pip install -e ".[server,otel]"

For container builds, include otel in the extras passed to your install step.

2. Configure the exporter

Two paths, chosen with OTEL_TRACES_EXPORTER:

Backend export (production)

Point the server at any OTLP receiver:

# In .env.dev (or .env.prd, etc.)
OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4317

The default protocol is gRPC (port 4317). For HTTP/protobuf (port 4318):

OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Console export (debugging, no backend required)

Dumps each span as JSON to stdout. Useful for verifying instrumentation locally without standing up a backend.

OTEL_TRACES_EXPORTER=console

3. Start the server and verify

./src/entrypoint.py server
curl http://localhost:8000/v1/healthz

In console mode, span JSON is printed to stdout immediately. In backend mode, the server logs OTel telemetry initialized: service=ai-optimizer-server exporters=['otlp'] at startup; traces and logs then appear in the backend UI within a few seconds.

If you do not see this log line, telemetry did not initialize — check the Troubleshooting section below.

Environment Variable Reference

The AI Optimizer honors the standard OpenTelemetry SDK environment variables. The most relevant ones for operators:

Endpoint and protocol

Exporter selection

Resource attributes

Sampling

Batch span processor tuning

The OTEL_BSP_* family (OTEL_BSP_MAX_QUEUE_SIZE, OTEL_BSP_SCHEDULE_DELAY, etc.) is honored by the SDK without code changes; tune in production if you observe span drops or back-pressure. See the SDK configuration spec for the full list.

Resource Attributes Set by the Application

The server sets the following attributes by default. All can be overridden via OTEL_RESOURCE_ATTRIBUTES or the dedicated env var.

Operator-supplied values via OTEL_RESOURCE_ATTRIBUTES always take precedence over these defaults.

Deployment Patterns

Bare-metal / VM

Set the variables in .env.dev (or whichever .env.{AIO_ENV} file is loaded), or export them in the shell before running the entrypoint:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
./src/entrypoint.py server

Container (Docker / Podman)

Pass the variables at run time:

docker run --rm \
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4317 \
  -e OTEL_RESOURCE_ATTRIBUTES=deployment.environment=staging \
  ai-optimizer:latest server

Or via an env-file (--env-file).

If running SigNoz on the same Docker host (e.g. via the SigNoz docker-compose), point at the SigNoz collector container by service name on the shared network.

Kubernetes / Helm

The bundled Helm chart exposes OpenTelemetry settings under server.otel. Two paths:

1. Bring your own OTLP collector — point endpoint at a separately-deployed SigNoz, Jaeger, Tempo, or vendor agent:

# values overlay
server:
  otel:
    enabled: true
    endpoint: http://signoz-otel-collector.observability.svc.cluster.local:4317
    insecure: true   # plaintext gRPC inside the cluster
    resourceAttributes:
      service.namespace: ai-optimizer
    # logsEnabled: true   # opt-in; review backend retention first

2. Install SigNoz alongside the application — flip signoz.enabled=true and the chart deploys SigNoz as a subchart. The server’s OTLP endpoint is then auto-defaulted to the in-cluster collector service URL; you only configure the OTel-side switches:

signoz:
  enabled: true
server:
  otel:
    enabled: true
    insecure: true   # the in-chart collector serves plaintext gRPC

The published image already includes the [otel] extra, so enabled: true works against the default image. Within Kubernetes, service.instance.id is auto-populated from HOSTNAME (the pod name), giving stable per-pod identity in the backend. deployment.environment is set automatically from the chart’s global.env value.

If enabled: true is set without an endpoint (and without tracesExporter: console for local debugging or signoz.enabled=true to use the in-chart collector), helm template / helm install fails fast rather than silently producing zero telemetry.

Troubleshooting

Backend-Specific Quickstarts

For step-by-step instructions on standing up a specific OTLP backend and pointing the server at it, see:

• SigNoz Quickstart