Fine-Tuning Pipelines

A fine-tuning pipeline keeps a model trained on your project's own documents. On a schedule, or when you start it, a pipeline turns the documents in your vector stores into training data, fine-tunes a new version of a base model, scores that version, and serves it under a stable model name only if it beats the version already in use.

Your applications call the pipeline's model name, its alias, like any other model. Each promoted version replaces the previous one behind the same alias, so the applications never change.

Pipelines run in the background via Nightshift.

Requirements

• Nightshift, with its RabbitMQ connection. See Nightshift Background Jobs.
• A fine-tuning worker of type infra: the DF-Finetune service installed on your DeepFellow Infra and registered as a fine-tuning worker. The worker needs eval support (eval_file). Infra serves adapters only from the DF-Finetune service it installed itself, so a worker of type url doesn't work for pipelines.
• The vLLM service on DeepFellow Infra, which serves the promoted versions.
• Infra's admin key in DeepFellow Server's infra.admin_api_key setting. Pipelines return 503 Service Unavailable while it is empty. See Settings.
• A vector store with your documents, in the same project. See Using Vector Stores.
• Doc Chunker on DeepFellow Infra, which converts the files of your vector stores and splits them into chunks. See Doc Chunker. Without it, runs end as dataset_generation_failed.
• The generation and judge models on DeepFellow Infra: the models set in finetune_pipelines.generation_model and finetune_pipelines.judge_model, which DeepFellow Server calls on Infra. Without the generation model, runs end as dataset_generation_failed. See Settings.

How a Run Works

On a pipeline's first run, a set of question-and-answer pairs is held out as the eval set, up to finetune_pipelines.eval_size pairs. Every later version is scored on the same eval set, so the scores of two versions are comparable. The chunks the eval set came from never enter training.

A run whose source files and training settings are the same as those of the version in use ends as no_new_data, without generating data or training. The training settings are hyperparameters, quantization, and finetune_pipelines.generation_model. Changing one of them makes the next run train a new version even when the files are the same. Uploaded files never change, so a changed document always arrives as a new file.

Create a Pipeline

curl -X 'POST' \
  'https://deepfellow-server-host/v1/fine_tuning/pipelines' \
  -H "Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "Support Bot",
  "base_model": "Qwen/Qwen3-8B",
  "source": {"vector_store_ids": ["VECTOR_STORE_ID"]},
  "schedule": {"interval_hours": 24},
  "promotion": {"min_delta": 0.2}
}'

import requests

response = requests.post(
    "https://deepfellow-server-host/v1/fine_tuning/pipelines",
    headers={"Authorization": "Bearer DEEPFELLOW_PROJECT_API_KEY"},
    json={
        "name": "Support Bot",
        "base_model": "Qwen/Qwen3-8B",
        "source": {"vector_store_ids": ["VECTOR_STORE_ID"]},
        "schedule": {"interval_hours": 24},
        "promotion": {"min_delta": 0.2},
    },
)

print(response.json())

Response (201 Created):

{
    "id": "68da445c5186deb8bca2bde9",
    "object": "fine_tuning.pipeline",
    "project_id": "68da445c5186deb8bca2bdea",
    "name": "Support Bot",
    "alias": "ft-68da445c5186deb8bca2bdea-support-bot",
    "base_model": "Qwen/Qwen3-8B",
    "source": {"vector_store_ids": ["VECTOR_STORE_ID"]},
    "schedule": {"interval_hours": 24},
    "hyperparameters": {"n_epochs": "auto", "batch_size": "auto", "learning_rate_multiplier": "auto"},
    "quantization": null,
    "worker_id": "68da445c5186deb8bca2bdeb",
    "serving_strategy": "blue_green",
    "promotion": {"min_delta": 0.2},
    "enabled": true,
    "active_run_id": null,
    "next_run_at": 1735776000,
    "eval_set": null,
    "champion": null,
    "created_at": 1735689600
}

alias, base_model, and worker_id don't change after creation. Creating a pipeline returns:

• 400 Bad Request if a field is invalid, or no usable infra worker exists.
• 404 Not Found if a vector store doesn't belong to your project.
• 409 Conflict if your project already has a pipeline with the same alias.
• 503 Service Unavailable if infra.admin_api_key is empty.

Use the Promoted Model

Once a run is promoted, call the pipeline's alias as the model name:

curl -X 'POST' \
  'https://deepfellow-server-host/v1/chat/completions' \
  -H "Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "ft-68da445c5186deb8bca2bdea-support-bot",
  "messages": [{"role": "user", "content": "How many vacation days do I get?"}]
}'

The alias belongs to the project that created the pipeline, since the model learned from that project's documents. Other projects get 403 Forbidden when they call it, and don't see it in GET /v1/models, even when their model access is set to all models. When the owning project's model access is a list, promotion adds the alias to that list.

Manage a Pipeline

Both list endpoints, GET /v1/fine_tuning/pipelines and GET /v1/fine_tuning/pipelines/{pipeline_id}/runs, are paginated. They take limit, from 1 to 100 with a default of 20, and after, the ID of the last item on the previous page. The response's has_more is true while more items follow.

A pipeline has at most one run in progress. Starting a run, or deleting the pipeline, while one is in progress returns 409 Conflict. Starting a run also returns 409 Conflict while finetune_pipelines.max_active_runs runs are in progress across all projects; try again once one ends. Starting a run while the server isn't connected to RabbitMQ returns 503 Service Unavailable.

You also manage pipelines on the Fine-tuning pipelines page of a project in the server panel.

Runs

GET /v1/fine_tuning/pipelines/{pipeline_id}/runs lists a pipeline's runs, newest first, and GET .../runs/{run_id} returns one run:

{
    "id": "68da4a015186deb8bca2bdf0",
    "object": "fine_tuning.pipeline.run",
    "pipeline_id": "68da445c5186deb8bca2bde9",
    "version": 3,
    "trigger": "schedule",
    "status": "promoted",
    "dataset": {"training_pairs": 412, "chunks": 160, "failed_chunks": 2},
    "adapter_id": "ft-support-bot-v3-7d6c5b4a",
    "eval": {"score": 3.82, "n": 50, "champion_score": 3.41},
    "decision": {"promoted": true, "reason": "Score 3.82 against v2's 3.41 plus the required improvement of 0.20."},
    "error": null,
    "created_at": 1735776000,
    "finished_at": 1735790400
}

A run's status is one of the following:

Each run creates a training file and an eval predictions file in your project, and a regular fine-tuning job. The first run of a pipeline also creates its eval file. They appear in GET /v1/files and GET /v1/fine_tuning/jobs like any other file or job.

A pipeline's first version has no version to beat, so it needs a score of at least finetune_pipelines.first_version_min_score.

Run Errors

The version in use keeps serving after any of these, with one exception described under Serving Strategies. The next scheduled run tries again.

Serving Strategies

• blue_green: DeepFellow Infra installs the new version next to the version in use, and removes the old version once the new one serves. Requests to the alias are served throughout. The GPU needs room for two copies of the base model while both run.
• replace: DeepFellow Infra removes the version in use first, then installs the new version. The alias doesn't answer until the new version serves, and no extra GPU memory is needed.

Settings

Pipelines use the finetune_pipelines section of DeepFellow Server's dynamic settings, and the infra.admin_api_key setting. Change them with deepfellow server config set or with PUT /admin/config. The Server Configuration card doesn't show them yet. The finetune_pipelines section has no .env equivalent.

To enable pipelines, set infra's admin key, the DF_INFRA_ADMIN_API_KEY of your Infra installation, on the server:

deepfellow server config set infra='{"admin_api_key": "INFRA_ADMIN_API_KEY"}'

The server merges the object into the existing infra section, so infra.url and infra.api_key keep their values. GET /admin/config masks the key, like the other credentials.

To change a pipeline setting, pass the finetune_pipelines fields to change:

deepfellow server config set finetune_pipelines='{"max_active_runs": 2, "judge_model": "qwen3:8b"}'