Observability

DeepFellow Server and DeepFellow Infra expose three kinds of observability data: Prometheus metrics, OpenTelemetry traces and logs, and simple health checks. This page gathers the endpoints, environment variables, and metric names in one place. For how to view container logs from the CLI, see Logs and Status. For how to edit these settings after installation, see DeepFellow Server Web Panel and DeepFellow Infra Web Panel.

Quick Start

To go from a fresh installation to dashboards and traces, work through these steps in order:

1. Confirm the built-in health checks respond, so you know the endpoints are reachable before adding anything else.
2. Read the metrics auth credentials and confirm GET /metrics returns data.
3. Point a Prometheus server at both /metrics endpoints, then add it as a Grafana data source and build dashboards from the metric tables below.
4. Enable OpenTelemetry tracing and logging, and point otel_exporter_otlp_endpoint at a collector, either a local debug collector or Jaeger.
5. Send a request and confirm the trace and log records arrive at your collector before relying on them.

The sections below cover each step in detail, with example configuration and commands to verify each one works.

Health Checks

Both components expose a GET /health endpoint that requires no authentication and returns a static "ok" response. Use it for Docker health checks and load balancer probes.

curl http://localhost:8000/health
curl http://localhost:8086/health

Both components also expose a GET /info endpoint that requires authentication and returns version information:

• On DeepFellow Server, /info returns the server's version and the connectivity status and version of the configured DeepFellow Infra.
• On DeepFellow Infra, /info returns the running Infra's version.

DeepFellow Server also exposes GET /ready, which requires no authentication and checks its dependencies. It returns 200 with {"status": "ready"} when MongoDB answers. Otherwise, it returns 503 with {"status": "not_ready"}. MongoDB is the only dependency the server cannot work without, so RabbitMQ, DeepFellow Infra, and the vector database do not affect /ready. The response does not say why the server is not ready. To check every dependency, read the dependency_up metric. Use /ready for readiness probes, and /health for liveness probes.

curl http://localhost:8000/ready

DeepFellow Server additionally exposes GET /admin/infra/health, an admin-only endpoint that probes the configured DeepFellow Infra directly and reports one of ok, wrong_api_key, not_found, unreachable, or timeout.

The Nightshift worker has no /health endpoint. To check it, run python -m server.workers.healthcheck in the worker's environment. See Nightshift Health Check.

Metrics (Prometheus)

Both components expose a GET /metrics endpoint formatted for Prometheus scraping. Metrics are always collected, independent of the OpenTelemetry tracing and logging settings described below.

The endpoint requires HTTP Basic authentication. Configure the credentials with the DF_METRICS_USERNAME and DF_METRICS_PASSWORD environment variables, or through the Metrics Auth section of the Server Configuration card and the equivalent Infra Configuration page.

Verify the endpoint with the credentials you configured:

curl -u "$DF_METRICS_USERNAME:$DF_METRICS_PASSWORD" http://localhost:8000/metrics
curl -u "$DF_METRICS_USERNAME:$DF_METRICS_PASSWORD" http://localhost:8086/metrics

Viewing Metrics in Grafana

Point a Prometheus server at /metrics on DeepFellow Server and DeepFellow Infra, with HTTP Basic authentication configured in its scrape config:

scrape_configs:
  - job_name: deepfellow-server
    basic_auth:
      username: <DF_METRICS_USERNAME>
      password: <DF_METRICS_PASSWORD>
    static_configs:
      - targets: ["deepfellow-server-host:8000"]

  - job_name: deepfellow-infra
    basic_auth:
      username: <DF_METRICS_USERNAME>
      password: <DF_METRICS_PASSWORD>
    static_configs:
      - targets: ["deepfellow-infra-host:8086"]

Then, in Grafana, go to Connections > Data sources > Add data source, choose Prometheus, and set its Connection URL to your Prometheus server's address, for example http://prometheus:9090. Click Save & test to confirm Grafana can reach it. From there, create a dashboard and add a panel for each metric you want to chart, using the metric names listed below as the query.

Server Metrics

Every label has a small, fixed set of values. No label holds a user, organization, project, file, or vector store ID, a URL, or a prompt. The http_route label holds the matched route template, for example /v1/files/{file_id}, or unmatched for a request that matched no route.

HTTP

GenAI and DeepFellow Infra Calls

For upstream_request_duration_seconds, service is embedding, sparse_embedding, reranker, lemmatizer, doc_chunker, entity_extractor, image_generation, llm_metadata, llm_summarization, or llm_df_abuse. outcome is success, timeout, quota (status 429), http_4xx, http_5xx, cancelled (the client left), or error.

Files and Vector Stores

For filetype, DeepFellow Server reports the lowercase file extension of a known type, for example .pdf. Any other extension is other, and a file without an extension is none.

Plugins

hook is before_request, after_request, or on_error. outcome is ok, blocked (the plugin stopped the request), cancelled (the client left), or error (the hook raised an error and DeepFellow Server skipped it).

For ti_check_duration_seconds, outcome is safe, unsafe, rejected, fail_open, unavailable_blocked, or error. A fail_open check means Threat Intelligence was unreachable and the plugin let the request through unchecked. To get notified, alert on increase(ti_check_duration_seconds_count{outcome="fail_open"}[10m]) > 0.

Datastores and Queues

DeepFellow Server polls the queue gauges every 15 seconds, so they can lag the broker slightly. While the broker is unreachable, they report nothing. When the check of a single queue fails, only that queue's gauges disappear, and rabbitmq_queue_poll_failures_total counts the failure.

Dependencies and Health

DeepFellow Server checks the dependencies every 15 seconds. The /ready endpoint does not use them, because it checks only MongoDB. DeepFellow Infra is 1 only if its /info endpoint answers with status 200, so a wrong API key reports 0.

Background Jobs, Webhooks, and MCP

For webhook_deliveries_total, result is delivered, http_error, timeout, network_error, dropped_endpoint_missing, or dropped_error. status_class is 2xx, 3xx, 4xx, 5xx, or none when there was no response. A webhook whose endpoint no longer exists counts as dropped_endpoint_missing, and an unexpected error while sending counts as dropped_error. A delivery that used up all its retries appears in nightshift_messages_total{queue="webhook_deliveries",outcome="dropped_exhausted"}.

The background_job label is not named job, because Prometheus reserves job for the scrape target and would rename it to exported_job. DeepFellow Server sets background_job_last_success_seconds when the job starts, so before the first run it holds the start time, not the time of a success. This way, a staleness alert also works after a restart.

Token Usage and Throughput Metrics

gen_ai_server_time_to_first_token_seconds and gen_ai_server_tokens_per_second cover /v1/chat/completions, /v1/completions, /v1/responses, /v1/messages, and the Ollama-compatible /api/chat. gen_ai_client_token_usage covers the same endpoints, plus /v1/embeddings and /v1/images/generations. All three metrics record only responses with status 200. gen_ai_server_time_to_first_token_seconds records only requests that set stream: true. The other two metrics record both streamed and non-streamed requests.

All three metrics carry the gen_ai_request_model, http_route, and http_request_method labels. gen_ai_server_tokens_per_second also carries stream. gen_ai_client_token_usage also carries gen_ai_token_type, and records a separate sample for each token type: input and output.

Throughput varies by endpoint. To compare gen_ai_server_tokens_per_second over time, filter it by http_route. For /v1/responses with server-side tool calls, the throughput time includes running the tools.

Token counts come from the usage object of the response. Anthropic-style cached prompt tokens count as input tokens. If a response reports no usage, for example a streamed /v1/completions request without stream_options.include_usage, DeepFellow Server records neither a token usage sample nor a throughput sample for it. If a stream fails or the client leaves it early, the output so far is partial. In that case, DeepFellow Server records only the input tokens and no throughput sample.

The _sum series of gen_ai_client_token_usage is the total token count. To chart the token rate per model and token type, use this query:

sum by (gen_ai_request_model, gen_ai_token_type) (rate(gen_ai_client_token_usage_sum[5m]))

Infra Metrics

Nightshift Metrics

The Nightshift worker serves its own /metrics endpoint on port DF_NIGHTSHIFT_METRICS_PORT (default 9464). On every scrape of its own /metrics, DeepFellow Server fetches the worker's endpoint from DF_NIGHTSHIFT_METRICS_URL and serves the worker's metrics too. The DeepFellow Server scrape config above covers both processes. You don't need a separate scrape job for the worker. For the settings, see Nightshift Monitoring.

Every sample from the worker carries the label process="nightshift". The label keeps the worker's samples apart from the server's samples of the same metric, such as target_info or upstream_request_duration_seconds.

nightshift_metrics_up reports whether the worker's metrics were collected. It is 1 when DeepFellow Server fetched them for the current scrape, and 0 when the worker did not answer, or did not answer within 2 seconds. A worker that fails or answers slowly never fails the DeepFellow Server scrape.

If you scrape the worker's endpoint directly, set DF_NIGHTSHIFT_METRICS_URL to an empty value. Otherwise, Prometheus collects every worker series twice.

Every Nightshift metric carries a queue label with the name of the RabbitMQ queue: metadata_expansion, document_summarization, webhook_deliveries, vector_store_file_processing, ontology_discovery, or graph_document_ingestion.

The outcome label takes one of the following values:

A queue with waiting messages (rabbitmq_queue_messages{kind="main"} > 0) whose nightshift_last_message_seconds is old has a stuck consumer. The worker also reports the server metrics that apply to it, such as upstream_request_duration_seconds from GenAI and DeepFellow Infra Calls and webhook_deliveries_total from Background Jobs, Webhooks, and MCP, labelled process="nightshift". For a webhook delivery, failed means the worker dropped it because of an unexpected error while sending (result="dropped_error"). A webhook whose endpoint no longer exists ends as success, with result="dropped_endpoint_missing".

A nightshift_consumer_up series appears for a queue only once its consumer has started. A worker that is still connecting to RabbitMQ reports no nightshift_consumer_up series at all.

Alerts and Dashboard

The monitoring/ directory of the DeepFellow Server repository holds a ready-made setup:

• prometheus.yml: scrapes the DeepFellow Server (with basic authentication), which also serves the Nightshift worker's metrics.
• rules/df-server.rules.yml: alert rules for these conditions:
  • API: the server is down, a 5xx rate above 5% (DfServerHigh5xxRate), requests near max_concurrent_requests, a spike in authentication failures (DfServerAuthFailureSpike), failing streams, and a slow time to first token.
  • Dependencies: a dependency that is down, failing calls to DeepFellow Infra services (DfServerInfraCallsFailing), vector database errors, MongoDB command failures, MongoDB pool checkout failures (DfServerMongoPoolCheckoutFailures), a saturated worker pool, and Threat Intelligence failing open.
  • Background jobs: a stale job (DfServerBackgroundJobStale) and a job that fails to handle some items (DfServerBackgroundJobItemsFailing).
  • Nightshift and RabbitMQ: the worker's metrics are unavailable while a queue has waiting messages (NightshiftDown), a stopped or stuck consumer, a queue backlog, a queue without consumers (NightshiftQueueWithoutConsumers), dropped messages (NightshiftMessagesDropped), queue depth checks that keep failing (NightshiftQueueMonitoringBlind), and a lost RabbitMQ connection (RabbitMqDisconnected).
  • Webhooks: failing deliveries and deliveries that were permanently lost.
• grafana/df-server-dashboard.json: a Grafana dashboard that charts the metrics above. To use it, go to Dashboards > New > Import and upload the file.

The rules are a starting point. Adjust the thresholds to your traffic, and check the rule file with promtool check rules.

Tracing and Logs (OpenTelemetry)

Both DeepFellow Server and DeepFellow Infra export traces and logs to an OpenTelemetry collector over OTLP gRPC. Both export under the service name llm-audit, so you will correlate spans and log records from both components in your observability backend. The Nightshift worker exports under the service name nightshift. Tracing and log export are independent; enable each one separately.

On DeepFellow Server, these are dynamic settings, editable through the OpenTelemetry section of the Server Configuration card or the /admin/config API:

DeepFellow Infra exposes the same three settings (otel_exporter_otlp_endpoint, otel_tracing_enabled, otel_logging_enabled) as dynamic configuration, editable with deepfellow infra config set or from the Infra Web Panel's Configuration page.

For setup during installation, including running a local debug-only collector with --otel-local or connecting to an existing one with --otel-url, see Installing DeepFellow Server. The --otel-local collector only writes trace data to its own container logs; it has no browser UI. To browse traces, point otel_exporter_otlp_endpoint at a backend such as Jaeger instead, as described below.

What Gets Traced

When tracing is enabled, DeepFellow Server instruments:

• Every HTTP endpoint request, capturing the route, headers, authentication and project or organization IDs, request body fields, including full message lists for chat, messages, completions, and responses endpoints, with PII anonymization applied, and the response text.
• Token usage of AI endpoint requests, as described in Token Usage in Traces.
• Vector store searches and other internal service calls.
• MCP tool calls, captured as mcp.call_tool spans with the tool name.

Token Usage in Traces

For both streamed and non-streamed requests to /v1/chat/completions, /v1/completions, /v1/responses, /v1/messages, /api/chat, /v1/embeddings, and /v1/images/generations, the endpoint span carries the following OpenTelemetry GenAI attributes:

For /v1/responses with server-side tool calls, the token counts cover every model call that DeepFellow Server makes while running the tools. If a response reports no usage, its span gets no gen_ai.usage.* attributes. The span still gets gen_ai.response.finish_reasons if the response reports a finish reason. If a stream fails or the client leaves it early, its span gets only gen_ai.usage.input_tokens, because the output so far is partial.

To keep memory use bounded, DeepFellow Server reads only the usage object at the end of a response body or stream event larger than 256 KiB, such as a large embeddings batch or a base64-encoded image. For such a response, the span gets no gen_ai.response.finish_reasons. Ollama reports token counts outside usage, so for such an /api/chat response, neither the span nor the token usage metrics get token counts.

Nightshift Traces

The Nightshift worker reads the same otel_* settings from config.json as DeepFellow Server, and applies a change on its next configuration reload. With otel_tracing_enabled, the worker handles each message in a nightshift <queue> span, for example nightshift metadata_expansion. The span continues the trace of the API request that queued the message, so one trace shows both the request and the background work it started. Every redelivery of a failed message stays in the same trace.

Each span records the attempt number in nightshift.attempt, the attempt limit in nightshift.max_attempts, and how the message ended in nightshift.outcome, with the same values as the outcome metric label. A span whose attempt raised an error has an error status and records the exception.

Messages queued before an upgrade to a DeepFellow Server version with Nightshift tracing carry no trace context. The worker processes them as before, each in a new trace.

Viewing Traces in a Browser

DeepFellow exports plain OTLP, so any OTLP-compatible backend will work. Jaeger is a common choice for viewing these traces in a browser. Run it with its OTLP receiver enabled, reachable from DeepFellow Server and Infra:

docker run -d --name jaeger -p 4317:4317 -p 16686:16686 jaegertracing/all-in-one:latest

Set otel_exporter_otlp_endpoint to http://jaeger:4317 (or the host running Jaeger), enable otel_tracing_enabled, and send a request. Confirm the trace arrived before opening the UI:

curl 'http://localhost:16686/api/services'

The response lists llm-audit once a trace has arrived. Open http://localhost:16686 to search and inspect traces by service name.

Selecting a trace shows its spans and their tags, for example the request's model and messages on a /v1/chat/completions span:

Viewing Collector Logs

To tail the logs of a locally installed OpenTelemetry collector, use deepfellow otel logs. See OpenTelemetry Collector Logs.

Log Level

DeepFellow Server controls its own log verbosity with the DF_LOG_LEVEL environment variable, or the LOG_LEVEL dynamic setting in the Server Configuration card. It defaults to INFO. The Nightshift worker applies the same setting, and logs one line per message with its queue, outcome, duration, and attempt number. This is separate from otel_logging_enabled, which controls whether those log records also export to the OpenTelemetry collector. DeepFellow Server logs to standard output as plain text; it does not emit structured or JSON-formatted logs. DeepFellow Infra has no equivalent log level setting.