Nightshift Background Jobs

Nightshift is the background worker behind Metadata Expansion and Document Summarization. It runs as a process separate from DeepFellow Server, so an API restart or crash does not kill an in-flight job.

Enable Nightshift

Nightshift is disabled by default. With it disabled, Metadata Expansion and Document Summarization still work: each job dispatches its documents in-process instead, one at a time, with no retry across API restarts. See Dispatch without Nightshift for what changes when Nightshift is off.

To enable Nightshift:

1. Run a RabbitMQ instance reachable from your server, and set DF_RABBITMQ_URL to its connection string.
2. Set DF_NIGHTSHIFT_ENABLED=true.
3. Run the Nightshift worker process alongside your server.

DF_NIGHTSHIFT_ENABLED, DF_RABBITMQ_URL, and DF_RABBITMQ_CONNECT_TIMEOUT_SECONDS are bootstrap settings: the server reads them from .env at startup only, and changing one requires a restart. See Bootstrap Settings vs. Dynamic Settings.

Dispatch without Nightshift

With DF_NIGHTSHIFT_ENABLED set to false, POST /v1/metadata/expand and POST /v1/summarization/summarize still create a job and process its documents, but through a different path:

• Each document runs as an in-process background task instead of a RabbitMQ message.
• A document gets a single attempt instead of the broker-driven retry with backoff described in Retry Behavior. If a document fails, mark it for another attempt by calling the job's retry endpoint.
• An API restart while a job is running leaves any pending documents in that job stuck. Call the job's retry endpoint after the restart to resume them.

Pausing, resuming, and retrying a job work the same way regardless of whether Nightshift is enabled. See Metadata Expansion and Document Summarization for their job endpoints.

Retry Behavior

With Nightshift enabled, a document that fails while processing is redelivered with exponential backoff: 5 seconds, then 10, then 20, doubling up to a cap of 5 minutes between attempts. A document that keeps failing is marked failed once it reaches the job type's configured attempt limit (metadata_expansion_max_attempts or summarization_max_attempts, both dynamic settings; see below). A malformed message is dropped immediately, since redelivering it can never make it valid.

Configuration

The following are dynamic settings: change them with the Server Configuration card or the /admin/config API, without a restart. See Bootstrap Settings vs. Dynamic Settings.