Vector Store Embedding Migration

A project's vector stores are created with a specific embedding model and vector dimension, chosen once at creation time. Vector Store Embedding Migration lets you move every vector store in a project to a different model, dimension, or sparse/dense setting. Search keeps working throughout, and every vector store keeps its original ID.

Migration operates on the whole project at once. Vector stores in one project always migrate together. Migrating a single vector store on its own is not supported.

Requirements

• Nightshift must be running and connected to RabbitMQ. DeepFellow re-embeds files in the background with Nightshift. If it is not available, POST /v1/migrate and POST /v1/migrate/retry return 503 Service Unavailable and change nothing. See Nightshift Jobs.
• Nightshift must have a vector database and an embedding service configured.
• To start, cancel, or retry a migration, authenticate as an owner of the project's organization, or as a server admin. Use a user token (dfuser_) or an organization API key (dforg_) whose owner is an organization owner. Pass the OpenAI-Project header with the project ID, and the OpenAI-Organization header when the organization differs from the user's default. See Authorization.
• To check the status, use any credential with access to the project, including a project API key.

Requests from a regular organization member return 403 Forbidden. Project API keys always return 403 Forbidden for start, cancel, and retry. Every migration endpoint applies to the project named in OpenAI-Project, so no request body takes a project ID.

Start a Migration

To start a migration, send the target model, dimension, and sparse setting to POST /v1/migrate.

curl -X 'POST' \
  'https://DEEPFELLOW_SERVER_HOST/v1/migrate' \
  -H "Authorization: Bearer DEEPFELLOW_USER_TOKEN" \
  -H "OpenAI-Organization: ORGANIZATION_ID" \
  -H "OpenAI-Project: PROJECT_ID" \
  -H 'Content-Type: application/json' \
  -d '{
  "model": "granite-embedding:278m",
  "size": 768,
  "sparse": false
}'

The response describes the migration, starting in migrating status:

{
  "id": "68a1c2f0e4b0f1a2b3c4d5e6",
  "project_id": "68a1c2f0e4b0f1a2b3c4d5e6",
  "status": "migrating",
  "target": {"model": "granite-embedding:278m", "size": 768, "sparse": false, "max_context_size": 512},
  "attempt_id": "...",
  "vector_store_ids": ["..."],
  "target_generation": 1,
  "error": null,
  "created_at": "...",
  "updated_at": "...",
  "finished_at": null
}

DeepFellow reads max_context_size from the model's max_context_window property. If the model does not report a valid value, DeepFellow uses 512 and writes a warning to the log. You cannot set max_context_size in the request.

If the project has no completed files, the migration finishes immediately and the response has done status.

Validation

Before it locks the project, DeepFellow embeds a probe text with the target model and checks the result. The migration does not start in the following cases, and each returns 409 Conflict:

If the inference service rejects the API key when DeepFellow looks up the model, the request returns 401 Unauthorized.

What Happens During a Migration

DeepFellow builds a new, empty collection for every vector store in the project and re-embeds each already-ingested file into it in the background. Only files with completed status are migrated. Nightshift processes one file at a time per worker.

Whether a file's cached chunk text is reused, or the file is split into chunks again, depends only on how the file was originally chunked, not on the target model:

• Files chunked in skip or doc_chunker mode reuse their cached chunk text. If the cached chunks.json uses an outdated schema, DeepFellow recomputes it.
• Files chunked in token_chunker mode are always split again against the new model's context size.
• Files ingested before the chunking_mode field existed are treated as doc_chunker files. If that fails, the file fails.

While a migration is in the migrating, failed, cutting_over, or cancelling status:

• Search keeps working against the existing data. Reads are never blocked.
• The following requests return 409 Conflict with the detail Project embedding migration in progress; vector store writes are blocked until it completes.
  • POST /v1/vector_stores
  • DELETE /v1/vector_stores/VECTOR_STORE_ID
  • POST /v1/vector_stores/VECTOR_STORE_ID/files
  • DELETE /v1/vector_stores/VECTOR_STORE_ID/files/FILE_ID
  • The request that updates the attributes of a vector store file
  • POST /v1/vector_stores/VECTOR_STORE_ID/file_batches
  • DELETE /v1/files/FILE_ID
• Uploading a file with POST /v1/files, listing and reading vector stores, and updating a vector store's own metadata keep working.
• Automatic deletion of expired vector stores (expires_after) skips the project's vector stores until the migration ends.
• Vector store IDs never change.

:::caution

A migration in failed status keeps blocking writes. Retry the migration or cancel it to unblock the project.

:::

Automatic Retries

If a file fails to re-embed, DeepFellow retries it with exponential backoff: 5, 10, 20, and 40 seconds between attempts. Each file gets 5 attempts by default. To change the limit, set the DF_VECTOR_STORE_MIGRATION_MAX_ATTEMPTS environment variable and restart Nightshift. The setting is not available in /admin/config or in the web UI. Every error is retried until the attempts run out, and then the file fails permanently.

The migration jobs run in the vector_store_migration_file_reembedding queue, which you can see in GET /admin/queues.

DeepFellow sets no timeout for re-embedding a single file. Only the RabbitMQ consumer_timeout applies. When the broker exceeds it, the broker closes the channel and returns the job to the queue.

Cutover

Once every file finishes re-embedding, DeepFellow starts the cutover automatically. For each vector store in turn, DeepFellow points it at the new collection, updates its embedding_config and sparse setting, and removes the old collection. Search then answers from the new data. When the last vector store switches, the status changes to done.

The switch is atomic for each vector store, not for the whole project. If the cutover is interrupted, the status changes to failed with a Cutover failed error, and a retry resumes where it stopped. Vector stores that already switched are skipped.

If DeepFellow cannot remove an old collection, the migration still finishes as done. DeepFellow writes a warning to the log, and you have to remove the leftover collection manually.

Check Migration Status

Poll GET /v1/migrate/status for the project's current status and per-file progress. Without a migration, or after a cancellation, the endpoint returns 200 with the idle status and zero counts.

{
  "project_id": "68a1c2f0e4b0f1a2b3c4d5e6",
  "status": "failed",
  "error": null,
  "file_counts": {"pending": 1, "in_progress": 0, "completed": 3, "failed": 1, "total": 5},
  "failed_files": [
    {"vector_store_id": "...", "file_id": "...", "error": "Re-embedding failed after 5 attempt(s): ValueError: Embedding dimension mismatch: 768 != 1024"}
  ],
  "stalled_files": [
    {"vector_store_id": "...", "file_id": "..."}
  ]
}

failed_files lists every file that permanently failed, with the reason. stalled_files lists files that never got a chance to be attempted in the current run, because another file's failure ended the migration first. It is filled only in failed status. A retry re-attempts stalled files too.

After a migration reaches done, the counts keep describing that migration until you start the next one or cancel.

Retry a Failed Migration

POST /v1/migrate/retry resumes a migration in the failed status, and also one stuck in cutting_over. It re-attempts only the files that have not yet succeeded. Files that already finished are left untouched. There is no limit on how many times you can retry a migration.

The response contains the new status only, without counts. The status is usually migrating, but it is done if only the cutover remained and succeeded, or failed if the cutover fails again.

If the migration is in any other status, the request returns 409 Conflict. Retry needs Nightshift, like starting a migration.

Cancel a Migration

POST /v1/migrate/cancel abandons a migration and returns the project to its state from before the migration started. Cancelling is available from migrating, failed, and cancelling (to resume an interrupted cancellation). The response is 200 with the idle status and empty counts.

DeepFellow removes the new collections and the per-file progress records. The existing collections are never touched, so search works the whole time. If a new collection cannot be removed, DeepFellow logs a warning and finishes the cancellation anyway. The next migration overwrites the leftover collection.

Cancelling from idle, done, or cutting_over, or without a migration, returns 409 Conflict. Once the cutover has started, you cannot cancel. Cancelling does not need Nightshift.

Web UI

The web UI has a Vector Store Migration page for each project at /dashboard/projects/PROJECT_ID/migration. To open it, click Vector Store Migration in the row of buttons on the project's page. The sidebar has no entry for it. The page shows:

• The model and dimension pairs the project's vector stores use now
• A form with the target model, size, and sparse setting, and a Start button with a confirmation dialog
• The migration status, file counts, errors, and a list of failed files

The page has a Retry button for failed migrations, and a Cancel button for migrating and failed migrations. It refreshes the status automatically while the migration is migrating, cutting_over, or cancelling.

After a Migration

Every vector store response includes an embedding_config object with the model, size, and max_context_size of the vector store. A migration updates it during the cutover.

A vector store created after a migration uses the server's default embedding model, not the project's migrated model. Files' chunking_mode, attributes, and chunking_strategy stay as they were.

Limitations

• Migrating a single vector store, independently of the rest of its project, is not supported.
• Files are not written to both the old and new collection during migration. Writes are blocked until the migration finishes.
• The old collection is removed as soon as the vector store switches over. There is no grace period for rolling back to it.
• The knowledge graph is not affected by a vector store migration.