Observability

DeepFellow Server and DeepFellow Infra expose three kinds of observability data: Prometheus metrics, OpenTelemetry traces and logs, and simple health checks. This page gathers the endpoints, environment variables, and metric names in one place. For how to view container logs from the CLI, see Logs and Status. For how to edit these settings after installation, see DeepFellow Server Web Panel and DeepFellow Infra Web Panel.

Quick Start

To go from a fresh installation to dashboards and traces, work through these steps in order:

1. Confirm the built-in health checks respond, so you know the endpoints are reachable before adding anything else.
2. Read the metrics auth credentials and confirm GET /metrics returns data.
3. Point a Prometheus server at both /metrics endpoints, then add it as a Grafana data source and build dashboards from the metric tables below.
4. Enable OpenTelemetry tracing and logging, and point otel_exporter_otlp_endpoint at a collector, either a local debug collector or Jaeger.
5. Send a request and confirm the trace and log records arrive at your collector before relying on them.

The sections below cover each step in detail, with example configuration and commands to verify each one works.

Health Checks

Both components expose a GET /health endpoint that requires no authentication and returns a static "ok" response. Use it for Docker health checks and load balancer probes.

curl http://localhost:8000/health
curl http://localhost:8086/health

Both components also expose a GET /info endpoint that requires authentication and returns version information:

• On DeepFellow Server, /info returns the server's version and the connectivity status and version of the configured DeepFellow Infra.
• On DeepFellow Infra, /info returns the running Infra's version.

DeepFellow Server additionally exposes GET /admin/infra/health, an admin-only endpoint that probes the configured DeepFellow Infra directly and reports one of ok, wrong_api_key, not_found, unreachable, or timeout.

Metrics (Prometheus)

Both components expose a GET /metrics endpoint formatted for Prometheus scraping. Metrics are always collected, independent of the OpenTelemetry tracing and logging settings described below.

The endpoint requires HTTP Basic authentication. Configure the credentials with the DF_METRICS_USERNAME and DF_METRICS_PASSWORD environment variables, or through the Metrics Auth section of the Server Configuration card and the equivalent Infra Configuration page.

Verify the endpoint with the credentials you configured:

curl -u "$DF_METRICS_USERNAME:$DF_METRICS_PASSWORD" http://localhost:8000/metrics
curl -u "$DF_METRICS_USERNAME:$DF_METRICS_PASSWORD" http://localhost:8086/metrics

Viewing Metrics in Grafana

Point a Prometheus server at /metrics on DeepFellow Server and DeepFellow Infra, with HTTP Basic authentication configured in its scrape config:

scrape_configs:
  - job_name: deepfellow-server
    basic_auth:
      username: <DF_METRICS_USERNAME>
      password: <DF_METRICS_PASSWORD>
    static_configs:
      - targets: ["deepfellow-server-host:8000"]

  - job_name: deepfellow-infra
    basic_auth:
      username: <DF_METRICS_USERNAME>
      password: <DF_METRICS_PASSWORD>
    static_configs:
      - targets: ["deepfellow-infra-host:8086"]

Then, in Grafana, go to Connections > Data sources > Add data source, choose Prometheus, and set its Connection URL to your Prometheus server's address, for example http://prometheus:9090. Click Save & test to confirm Grafana can reach it. From there, create a dashboard and add a panel for each metric you want to chart, using the metric names listed below as the query.

Server Metrics

Infra Metrics

Tracing and Logs (OpenTelemetry)

Both DeepFellow Server and DeepFellow Infra export traces and logs to an OpenTelemetry collector over OTLP gRPC. Both export under the service name llm-audit, so you will correlate spans and log records from both components in your observability backend. Tracing and log export are independent; enable each one separately.

On DeepFellow Server, these are dynamic settings, editable through the OpenTelemetry section of the Server Configuration card or the /admin/config API:

DeepFellow Infra exposes the same three settings (otel_exporter_otlp_endpoint, otel_tracing_enabled, otel_logging_enabled) as dynamic configuration, editable with deepfellow infra config set or from the Infra Web Panel's Configuration page.

For setup during installation, including running a local debug-only collector with --otel-local or connecting to an existing one with --otel-url, see Installing DeepFellow Server. The --otel-local collector only writes trace data to its own container logs; it has no browser UI. To browse traces, point otel_exporter_otlp_endpoint at a backend such as Jaeger instead, as described below.

What Gets Traced

When tracing is enabled, DeepFellow Server instruments:

• Every HTTP endpoint request, capturing the route, headers, authentication and project or organization IDs, request body fields, including full message lists for chat, messages, completions, and responses endpoints, with PII anonymization applied, and the response text and token counts.
• Vector store searches and other internal service calls.
• MCP tool calls, captured as mcp.call_tool spans with the tool name.

Viewing Traces in a Browser

DeepFellow exports plain OTLP, so any OTLP-compatible backend will work. Jaeger is a common choice for viewing these traces in a browser. Run it with its OTLP receiver enabled, reachable from DeepFellow Server and Infra:

docker run -d --name jaeger -p 4317:4317 -p 16686:16686 jaegertracing/all-in-one:latest

Set otel_exporter_otlp_endpoint to http://jaeger:4317 (or the host running Jaeger), enable otel_tracing_enabled, and send a request. Confirm the trace arrived before opening the UI:

curl 'http://localhost:16686/api/services'

The response lists llm-audit once a trace has arrived. Open http://localhost:16686 to search and inspect traces by service name.

Selecting a trace shows its spans and their tags, for example the request's model and messages on a /v1/chat/completions span:

Viewing Collector Logs

To tail the logs of a locally installed OpenTelemetry collector, use deepfellow otel logs. See OpenTelemetry Collector Logs.

Log Level

DeepFellow Server controls its own log verbosity with the DF_LOG_LEVEL environment variable, or the LOG_LEVEL dynamic setting in the Server Configuration card. It defaults to INFO. This is separate from otel_logging_enabled, which controls whether those log records also export to the OpenTelemetry collector. DeepFellow Server logs to standard output as plain text; it does not emit structured or JSON-formatted logs. DeepFellow Infra has no equivalent log level setting.