Uploading Large Files

Upload a file too big for a single request by sending it to DeepFellow in parts.

The Uploads API handles files that exceed the size limit of a single POST /v1/files request. You split the file into parts and send each part separately. When you complete the upload, DeepFellow joins the parts into a single file. You use this file the same way as any other uploaded file, for example, in a vector store.

How It Works

An upload goes through the following statuses:

pending: the upload accepts parts.
completing: DeepFellow joins the parts into a file. The upload no longer accepts parts.
cancelling: DeepFellow is removing the parts of a cancelled upload. An upload gets this status when the cancel request fails with status 500, and DeepFellow finishes the removal in the background.
completed: the file exists, and the response contains it in file.
cancelled: you cancelled the upload, and DeepFellow removed its parts.
expired: the upload had no activity for 24 hours, and DeepFellow removed its parts.

DeepFellow accepts parts in parallel and in any order. The order of part_ids in the complete request decides the order of the final file.

The Upload Object

The create, complete, and cancel requests respond with the Upload object. It has these fields:

Field	Description
`status`	Status of the upload. See the list in How It Works.
`file`	The created file. DeepFellow sets it only for the `completed` status.
`part_count`	Number of parts accepted so far, from 0 to 10,000.
`received_bytes`	Total size of the parts accepted so far. It never exceeds the `bytes` you declared.
`expires_at`	Moment the upload expires, as a Unix timestamp in seconds. The countdown restarts after every accepted part.

Upload a File

Create the upload. Declare the total size of the file in bytes.

curl -X POST "https://deepfellow-server-host/v1/uploads" \
  -H "Authorization: Bearer DEEPFELLOW-PROJECT-API-KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "filename": "recordings.zip",
    "purpose": "assistants",
    "bytes": 3221225472,
    "mime_type": "application/zip"
  }'

Save the id from the response as UPLOAD_ID.

Send each part. Repeat this request for every part.

curl -X POST "https://deepfellow-server-host/v1/uploads/UPLOAD_ID/parts" \
  -H "Authorization: Bearer DEEPFELLOW-PROJECT-API-KEY" \
  -F "file=@part-001"

The response is an upload.part object with the fields id, upload_id, object, and created_at. Save its id as PART_ID.

Complete the upload. List the part IDs in the order of the final file.

curl -X POST "https://deepfellow-server-host/v1/uploads/UPLOAD_ID/complete" \
  -H "Authorization: Bearer DEEPFELLOW-PROJECT-API-KEY" \
  -H "Content-Type: application/json" \
  -d '{"part_ids": ["PART_ID_1", "PART_ID_2", "PART_ID_3"]}'

The response has "status": "completed" and the new file in file.

Limits

The parts must add up to exactly the bytes you declared on create. Otherwise, the complete request fails.
An upload accepts up to 10,000 parts. The part_count field shows how many parts DeepFellow has accepted so far.
DeepFellow rejects a part with status 413 if the upload already has 10,000 parts, or if the parts would total more than bytes. Use fewer, larger parts to stay within the part limit.
DeepFellow rejects an empty part with status 400.
One complete request lists up to 10,000 parts, without duplicates.

File Content Type

The mime_type field is required when you create an upload. It must have 1 to 255 characters. DeepFellow sets the content type of the final file as follows:

If mime_type is a valid type/subtype value other than application/octet-stream, the file gets that type. DeepFellow converts the value to lowercase and removes parameters, so Text/Plain; charset=utf-8 becomes text/plain.
If mime_type is application/octet-stream or is not a valid type/subtype value, DeepFellow guesses the type from the extension of filename.
If DeepFellow finds no type for the extension of filename, the file gets application/octet-stream.

Cancel an Upload

To discard an upload, send POST /v1/uploads/UPLOAD_ID/cancel. DeepFellow removes all parts from the disk and the database.

curl -X POST "https://deepfellow-server-host/v1/uploads/UPLOAD_ID/cancel" \
  -H "Authorization: Bearer DEEPFELLOW-PROJECT-API-KEY"

The response is the upload with "status": "cancelled". DeepFellow cancels only an upload with the pending status. If DeepFellow fails to remove the parts, it responds with status 500 and finishes the removal in the background. For any other status, DeepFellow responds with status 409.

Expiration

An upload with no activity for 24 hours expires, and DeepFellow removes its parts. Every new part restarts the 24 hours.

To change how often DeepFellow looks for expired uploads, set upload_expiry_sweep_interval_seconds (environment variable DF_UPLOAD_EXPIRY_SWEEP_INTERVAL_SECONDS).