Nightshift Background Jobs

Nightshift is the background worker behind vector store file processing, Metadata Expansion, Document Summarization, Knowledge Graph file ingestion and ontology auto-discovery, webhook delivery, vector store migration, and fine-tuning pipelines. It runs as a process separate from DeepFellow Server, so an API restart or crash does not kill an in-flight job.

DeepFellow Server queues all of these jobs in RabbitMQ, and only Nightshift processes them. Every deployment needs RabbitMQ and a running Nightshift worker. Without the worker, the jobs stay queued and never finish.

Run Nightshift

If you install DeepFellow Server with deepfellow server install or deepfellow suite install, the CLI sets up RabbitMQ and the Nightshift worker container. To manage the worker after installation, or to run it only during set hours, use deepfellow server nightshift.

In shift mode, user-level systemd timers start and stop the worker at set hours. Only one install per user can use shift mode. See Nightshift Operating Modes.

To run Nightshift on a server you did not install with the CLI, start the worker next to the server with:

• The same Docker image as the server, with the command ./.venv/bin/python -m server.workers.nightshift
• The same environment variables as the server
• The same storage directory as the server (/app/storage in the container)
• Access to the same RabbitMQ and MongoDB as the server
• A stop timeout above DF_NIGHTSHIFT_RELOAD_DRAIN_TIMEOUT_SECONDS, so a stop lets jobs in progress finish (see Stop the Worker)

Give the server and the worker exactly the same environment. Both processes create config.json from their environment on first boot, so whichever starts first decides what config.json contains.

Nightshift reads uploaded files, the Doc Chunker conversion cache, and config.json from the server's storage directory, so it runs on the same host as the server. To run the worker on another host, share the storage directory through a shared file system, the same way as for multiple server replicas.

Run with Docker Compose

The compose.yaml file in the DeepFellow Server repository runs the server, Nightshift, and their backing services together. The server and nightshift services inherit one shared definition, x-df-server, so they always get the same build, .env file, environment overrides, storage mount, and network.

To start the stack:

docker network create deepfellow-infra-net

docker compose up -d --build

To use MongoDB, RabbitMQ, or a vector database that runs outside the Compose project, start only the server and Nightshift. The --no-deps flag skips the bundled backing services:

docker compose up -d --no-deps server nightshift

Then point both services at your own services:

• MongoDB: change DF_MONGO_URL in the x-df-server block of compose.yaml. The server reads it on every boot.
• Vector database and RabbitMQ: change them with the Server Configuration card or PUT /admin/config, because they are dynamic settings. Before the first boot, edit DF_VECTOR_DATABASE__PROVIDER__URL and DF_RABBITMQ_URL in the x-df-server block instead. DF_RABBITMQ_URL also takes effect after the first boot, as long as the rabbitmq section of config.json is not configured yet.

Verify the Setup

To confirm that Nightshift processes jobs:

RabbitMQ Requirements

RabbitMQ must be reachable from both the server and the Nightshift worker. Its consumer_timeout must be above nightshift_jobs_timeout_seconds, or the broker redelivers jobs that are still running. See Job Timeouts. The local RabbitMQ that the CLI installs runs in a container with the fixed hostname rabbitmq and a 4-hour consumer_timeout. Its username and password cannot contain whitespace or any of the characters @ / : ? # [ ] %.

DF_RABBITMQ_URL and DF_RABBITMQ_CONNECT_TIMEOUT_SECONDS only seed the rabbitmq section of config.json, on any boot where that section is not configured yet. After the section is saved, the server ignores them. See RabbitMQ Connection.

Retry Behavior

A document that fails while processing is redelivered with exponential backoff: 5 seconds, then 10, then 20, doubling up to a cap of 5 minutes between attempts. A document that keeps failing is marked failed once it reaches the job type's configured attempt limit (metadata_expansion_max_attempts or summarization_max_attempts, both dynamic settings; see below). If Doc Chunker rejects the file with 413, 415, or 422, or the error is another one that a retry will not fix, the document is marked failed on the first attempt. A malformed message is dropped immediately, since redelivering it can never make it valid.

Knowledge Graph jobs use one message per job instead of one per document:

• A Knowledge Graph ingestion message covers a whole file. A single document that fails to ingest is marked failed without a redelivery, and the job moves on to the next document. The message is redelivered only when a step shared by the whole file fails, such as reading or chunking it, until it reaches graph.document_ingestion_max_attempts deliveries. A file that Doc Chunker rejects with 413, 415, or 422 during chunking fails the job without a redelivery. See Retries and Failures.
• An ontology discovery message is redelivered on any failure, until it reaches graph.ontology_discovery_max_attempts deliveries. See Discovery Failures.

Configuration Reload

The Nightshift worker checks config.json every DF_NIGHTSHIFT_CONFIG_POLL_INTERVAL_SECONDS seconds. When a dynamic setting changes, for example through deepfellow server config set or PUT /admin/config, the worker stops taking new messages, waits up to DF_NIGHTSHIFT_RELOAD_DRAIN_TIMEOUT_SECONDS seconds for the messages in progress to finish, and then restarts itself with the new configuration. RabbitMQ redelivers any message still in progress after the timeout, without counting it as a failed attempt.

The reload also covers graph.enabled: the worker starts consuming the Knowledge Graph queues when you enable Knowledge Graph, and stops when you disable it.

Both are bootstrap settings, read from .env at startup.

Stop the Worker

The worker stops when it receives SIGTERM or SIGINT. docker stop, docker compose stop, a redeploy, deepfellow server stop, deepfellow server nightshift stop, the nightshift-stop timer in shift mode, and Ctrl+C in a terminal all send one of these signals.

On a stop signal, the worker stops taking new messages, waits up to DF_NIGHTSHIFT_RELOAD_DRAIN_TIMEOUT_SECONDS seconds for the messages in progress to finish, and then exits with code 0. RabbitMQ redelivers any message still in progress after the timeout, without counting it as a failed attempt. Its job starts over from the beginning on the next run, except for a Doc Chunker conversion, which resumes its Doc Chunker job.

• If no message is in progress, the worker exits at once.
• If the worker is still connecting to RabbitMQ, it exits at once.
• If the signal arrives during a configuration reload, the worker finishes the wait and then exits instead of restarting.
• If a second stop signal arrives during the wait, the worker skips the rest of the wait, interrupts the messages in progress, and exits.

Docker waits for a container to exit only for its stop timeout, 10 seconds by default, and then kills it. To give the worker time to finish its messages, set the stop timeout above DF_NIGHTSHIFT_RELOAD_DRAIN_TIMEOUT_SECONDS. The nightshift service in the DeepFellow Server compose.yaml and the one the CLI installs both use:

services:
  nightshift:
    stop_grace_period: 330s
    init: true

With init: true, Docker runs an init process as the container's main process, and that process passes the stop signal on to the worker.

Monitoring

The Nightshift worker serves its own Prometheus /metrics endpoint. DeepFellow Server fetches it and serves the worker's metrics on the DeepFellow Server /metrics endpoint, so one scrape target covers both. The worker reports how many messages each queue handled and how each one ended, how long each message took, how many messages are in progress, whether each queue's consumer is running, and when it last finished a message. For the metric names and labels, see Nightshift Metrics.

All three are bootstrap settings, read from .env at startup. The default DF_NIGHTSHIFT_METRICS_URL matches the nightshift container that the installer and the DeepFellow Server compose.yaml create.

The default 127.0.0.1 keeps the endpoint off the host's network interfaces when you run the worker directly on a host. If the worker runs in a Docker container, set DF_NIGHTSHIFT_METRICS_HOST=0.0.0.0 so DeepFellow Server reaches it from its own container, and do not publish the port. The endpoint is then reachable only on the Docker network. The nightshift service in the DeepFellow Server compose.yaml uses this setup.

If the worker cannot listen on the metrics port, for example because another process already uses it, the worker logs the error and keeps processing jobs. The health check below then reports the worker as unhealthy.

Health Check

To check the worker's health, run the following command in the worker's environment:

python -m server.workers.healthcheck

The command reads the worker's /metrics endpoint and exits with 0 when the worker is healthy, or 1 when it is not. It reports the worker as unhealthy when:

• The /metrics endpoint does not answer.
• No queue consumer has started yet, for example while the worker is still connecting to RabbitMQ.
• Any queue's consumer has stopped while the others keep running.

The check ignores the http_proxy and HTTP_PROXY environment variables, so a proxy configured for the container does not intercept it.

Use the command as a Docker health check:

services:
  nightshift:
    healthcheck:
      test: ["CMD", "./.venv/bin/python", "-m", "server.workers.healthcheck"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

With this health check, docker compose ps shows the nightshift container as unhealthy when a queue's consumer has stopped.

Tracing and Logs

The worker uses the same log_level, otel_tracing_enabled, otel_logging_enabled, and otel_exporter_otlp_endpoint settings as DeepFellow Server, and applies a change on its next configuration reload. With tracing enabled, each message runs in a span that continues the trace of the API request that queued it. See Nightshift Traces.

Inspect and Purge Queues

An admin can check the backlog and consumer count on every Nightshift queue, including its retry tiers, and clear a stuck queue:

curl -X 'GET' \
  'https://deepfellow-server-host/admin/queues' \
  -H "Authorization: Bearer DEEPFELLOW_ADMIN_API_KEY"

Response:

{
    "connected": true,
    "queues": [
        {
            "name": "metadata_expansion",
            "main": { "name": "metadata_expansion", "message_count": 0, "consumer_count": 1 },
            "retries": [null, null, null, null, null, null, null]
        }
    ]
}

To show the same formatted data from the command line, run deepfellow server nightshift queue. The CLI has no command to purge a queue.

retries has one entry per backoff tier described in Retry Behavior (5s, 10s, 20s, 40s, 80s, 160s, then the 5-minute cap), in order. connected is false (with an empty queues list) when the server has no live RabbitMQ connection. A queue that has never been created shows as null instead of a stats object.

Purge a specific queue by name:

curl -X 'POST' \
  'https://deepfellow-server-host/admin/queues/metadata_expansion/purge' \
  -H "Authorization: Bearer DEEPFELLOW_ADMIN_API_KEY"

queue_name must be one of the queues GET /admin/queues lists (a base queue or one of its retry tiers); any other name returns 404 Not Found, as does a known queue name that RabbitMQ has not created yet (for example, a retry tier that has never had a failed message). The same page is available from the admin Dashboard under Queues.

RabbitMQ Connection

The RabbitMQ connection is a dynamic setting. It lives in the rabbitmq section of config.json. Change it with the RabbitMQ (Nightshift broker) section of the Server Configuration card or with PUT /admin/config, without a restart.

For example, to move the server to another broker:

curl -X PUT "$DF_SERVER_URL/admin/config" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"rabbitmq": {"host": "rabbitmq.example.com", "username": "deepfellow", "password": "BROKER_PASSWORD"}}'

When you change the connection:

• The server connects to the new broker first and drops the old connection only after that succeeds, so job dispatch continues during the change.
• If the new broker does not answer within connect_timeout_seconds, PUT /admin/config returns 502 Bad Gateway. The old settings and the old connection stay in place.
• If the old broker is unreachable while the server moves back to it after a failed update, requests that need RabbitMQ return 503 Service Unavailable until the connection returns.
• A change to connect_timeout_seconds alone does not reconnect the server. It applies to the next connection attempt.
• The Nightshift worker reloads itself like it does for any config.json change. It also reloads while it is still trying to reach the broker, so correcting the settings of an unreachable broker reaches the worker without a manual restart.

The server seeds the rabbitmq section from DF_RABBITMQ_URL and DF_RABBITMQ_CONNECT_TIMEOUT_SECONDS on any boot where DF_RABBITMQ_URL is set and the section is not configured yet. The section counts as not configured when config.json has no rabbitmq section, or when the section holds exactly the built-in defaults (guest@localhost). Until DF_RABBITMQ_URL is set, the server and the worker use those defaults without writing them to config.json, so an install updated before it had the variable picks it up on the next boot after you add it. Give the server and the worker the same DF_RABBITMQ_URL.

After a non-default section is saved, edits to those variables in .env or with deepfellow server env set have no effect. When DF_RABBITMQ_URL differs from the saved section, the server and the worker log a warning at boot. While DF_RABBITMQ_URL is set to a broker other than the built-in defaults, PUT /admin/config rejects setting a configured section back to the defaults with 400 Bad Request, because the next boot would seed it from DF_RABBITMQ_URL again. Without the variable, the server accepts that update and leaves the section out of config.json. The seed does not carry over URL query parameters such as heartbeat, and the server logs a warning when it drops one.

Broker-side settings such as consumer_timeout stay on RabbitMQ. See Job Timeouts.

Configuration

The following are dynamic settings: change them with the Server Configuration card or the /admin/config API, without a restart. See Bootstrap Settings vs. Dynamic Settings.

Job Timeouts

A Nightshift job step includes the Doc Chunker conversion of the file, so nightshift_jobs_timeout_seconds must stay above doc_chunker_convert_timeout_seconds. Its default is 3 hours.

If your server still uses the old default of 1800 after an upgrade, the server moves it to 10800 automatically. A value you set yourself stays as it is.

Nightshift acknowledges a message only after its job finishes. If a job runs longer than the RabbitMQ consumer_timeout, the broker redelivers the message while the first run is still going. The bundled compose.yaml sets consumer_timeout to 4 hours. A stock RabbitMQ broker defaults to 30 minutes. If you run your own RabbitMQ, set consumer_timeout above nightshift_jobs_timeout_seconds:

environment:
  RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: "-rabbit consumer_timeout 14400000"