Doc Chunker

Doc Chunker is an optional DeepFellow component that turns diverse knowledge sources into clean, searchable text. It converts documents, presentations, spreadsheets, images, audio, and video into text and splits that text into chunks ready for embedding.

What You Can Convert

Doc Chunker handles a wide range of formats.

• Documents and Markup: pdf, docx, pptx, xlsx, html, md. OCR is enabled by default and extracts text from scanned PDFs and images embedded in documents.
• Legacy Office formats: doc, ppt, xls, odt, odp, ods, rtf. Doc Chunker converts these with LibreOffice before parsing.
• Images: png, jpg, jpeg, tiff, bmp. OCR reads text from the image. Optional image description adds a natural-language summary of the picture.
• Audio: mp3, wav, m4a, ogg, flac, opus, aac, wma. Doc Chunker transcribes speech to text through a speech-to-text model.
• Video: mp4, mkv, mov, avi, webm. Doc Chunker extracts the audio track and transcribes it.

Doc Chunker handles any other file type in two steps. First it tries to read the file as plain text. If that succeeds, the text becomes a document whose paragraphs are split on blank lines. If it fails, Doc Chunker falls back to Apache Tika, which is what happens with a format such as epub.

Install Doc Chunker

Install Doc Chunker from the Infra Web Panel as a model of the custom service.

Doc Chunker Settings

Installing the model exposes the Doc Chunker endpoint. DeepFellow Server reads the following settings to route vector store files through it.

Doc Chunker itself holds no model configuration, and every request names the model it needs. These settings are what DeepFellow Server puts into that request.

• doc_chunker_prefix: Endpoint prefix the server uses to reach Doc Chunker. It must match the prefix you set when you installed the model. Default doc_chunker.
• doc_chunker_picture_description_model: Vision-language model the server sends as the vision_model parameter when it asks Doc Chunker to describe pictures. Default gemma4:e4b.
• doc_chunker_stt_model: Speech-to-text model the server sends as the stt_model parameter with every conversion. Doc Chunker uses it only for audio and video. Default Systran/faster-whisper-large-v3.
• doc_chunker_convert_timeout_seconds: Seconds the server waits for one conversion to finish, including the time the file waits in Doc Chunker's queue. It applies to every conversion: vector store ingestion, Document Summarization, Knowledge Graph ingestion, and file text retrieval. When the time runs out, the request fails with 504. Default 7200.
• doc_chunker_poll_interval_seconds: Seconds between two reads of a conversion job's status. Default 5.
• doc_chunker_request_timeout_seconds: Seconds one file upload or one result download takes at most. It does not limit the conversion itself. Default 300.

All six are dynamic settings, so a change applies immediately and never needs a restart:

deepfellow server config set doc_chunker_prefix=doc_chunker

The Doc Chunker section of the Server Web Panel holds all six: doc_chunker_prefix as Prefix, doc_chunker_picture_description_model as Picture Description Model, doc_chunker_stt_model as Speech-to-Text Model, doc_chunker_convert_timeout_seconds as Conversion Timeout (seconds), doc_chunker_poll_interval_seconds as Conversion Poll Interval (seconds), and doc_chunker_request_timeout_seconds as Request Timeout (seconds).

Conversion Timeouts

A conversion of a large file, such as a long recording or a PDF with many pictures, takes up to about an hour. Five timeouts apply to it, and each one must be at least as long as the one it wraps. Otherwise an outer layer cuts the conversion short or runs it a second time:

To convert files that take longer than the defaults allow, raise them together and keep them in this order. For more on the last two, see Nightshift Background Jobs.

Update Doc Chunker

Updating DeepFellow Infra does not update an installed Doc Chunker. To move it to a newer image, click Update in the ⋮ menu of the doc_chunker card in the Infra Web Panel, or run deepfellow infra model update custom doc_chunker in the CLI. deepfellow infra model versions custom doc_chunker lists the available versions. See Updates.

The update recreates the container, so register your tokenizers again afterward. See Data Storage.

Container Configuration

Set these environment variables on the Doc Chunker model in the Infra Web Panel.

Configuration Options

The Doc Chunker installation form exposes the following fields.

Processing

The defaults work for every format listed in What You Can Convert. Doc Chunker holds further timeouts for its Apache Tika and Docling backends, and the form leaves them at their defaults.

Worker Slots

/convert, /chunk, and conversion jobs share the MAX_CONCURRENT worker slots. Each slot has a role, which decides what it takes first when it frees up.

Waiting work of one kind runs in arrival order.

With the default of 2 slots, one slot prefers conversion jobs and the other prefers /convert and /chunk. While jobs wait in the queue, /convert and /chunk effectively have one slot.

Doc Chunker refuses to start unless ASYNC_PRIORITY_WORKERS plus SYNC_ONLY_WORKERS is at most MAX_CONCURRENT, and each of the two is below MAX_CONCURRENT. Then at least one slot takes /convert and /chunk first, and at least one slot takes conversion jobs.

The Infra install form exposes neither ASYNC_PRIORITY_WORKERS nor SYNC_ONLY_WORKERS, so an install from the Infra Web Panel runs with their defaults.

Chunk Size

A chunk holds at most as many tokens as the chunking request asks for in max_tokens. A request that omits the parameter gets the largest chunk recorded for the tokenizer that measures it, which is 512 tokens for the built-in default. See Chunking Tokenizers.

Doc Chunker fills 70 percent of that budget with text unique to the chunk, and spends the remaining 30 percent on overlap. The overlap runs forward: Doc Chunker takes whole sentences from the start of a chunk and appends them to the chunk before it. The earlier chunk grows, and the later one stays as it was. Each overlap also stays within about 30 percent of the length of the chunk it extends, so a short chunk grows by far less than the 30 percent ceiling of the budget.

Doc Chunker overlaps only consecutive chunks of running text, so a table never bleeds into its neighbors.

Chunk boundaries are measured with the tokenizer of the embedding model the chunks are destined for, so the token counts match the ones that model counts. Name that model in the embedding_model parameter of a chunking request, and register its tokenizer first. See Chunking Tokenizers.

Data Storage

Doc Chunker keeps everything it stores under DATA_DIR, in two subdirectories it creates there: tokenizers/ holds the registered tokenizers, and jobs/ holds the conversion jobs with their results. Chunking never writes to DATA_DIR.

Doc Chunker refuses to start when it can't write to both subdirectories. It doesn't check that DATA_DIR is a mounted volume. Without one, Doc Chunker writes to the container's own file system, and every registered tokenizer and every job result is lost when the container is recreated, for example on a reinstall or an upgrade.

Registering a tokenizer past MAX_TOKENIZERS deletes the least recently used one from memory and from the volume. That model then answers like one that was never registered, and registering its tokenizer again brings it back. Recency lives in memory, so right after a restart, before anything has been used, Doc Chunker deletes the tokenizer that was registered longest ago.

Inference Gateway

Image description and audio transcription both call an OpenAI-compatible inference gateway, so they share one address and one key. Point INFRA_API_URL at the gateway root, not at a specific endpoint. Doc Chunker appends /v1/chat/completions for image description and /v1/audio/transcriptions for transcription.

An install through Infra fills in INFRA_API_URL with http://infra:8086 when you leave it empty. In a Doc Chunker container run outside Infra, an empty INFRA_API_URL makes both features unavailable, and Doc Chunker rejects a request that needs one rather than returning a document without it.

Image Descriptions

Image description is opt-in per request. Pass describe_images=true and name the vision-language model with vision_model. Both are required together: a request that asks for descriptions without vision_model is rejected with 422, and so is a request naming a model the gateway doesn't offer.

Doc Chunker describes every picture the document holds. There is no size threshold, so icons and logos are described too.

PICTURE_DESCRIPTION_CONCURRENCY bounds a single document, and Doc Chunker converts MAX_CONCURRENT documents at a time. The gateway therefore sees up to the product of the two, which is 8 requests with both at their defaults. Raise it for a gateway that takes the load.

Audio Transcription

For audio and video, transcription is the conversion: the transcript becomes the document's text. Name the speech-to-text model per request with stt_model. It is required, so uploading audio or video without it is rejected with 422.

Doc Chunker splits the transcript on silence, and every piece keeps the time span it was spoken in. A silence longer than AUDIO_SILENCE_THRESHOLD always ends a chunk, so no chunk ever spans a pause.

Converted Documents

When Doc Chunker is active, adding a file to a vector store with POST /v1/vector_stores/{vector_store_id}/files converts the file through Doc Chunker first, and the vector store stores the chunks that come back.

DeepFellow keeps the converted document next to the file, under the path set by DF_FILESTORAGE_META_PATH, which defaults to storage/. Adding the same file again, or asking for its text later, reuses that document instead of converting the file a second time. A conversion that stopped halfway leaves nothing reusable behind, so the next request converts the file again.

POST /v1/metadata/expand, which extracts metadata fields from a document with an LLM, uses the same cache. When a requested field is not present in a document's embedded metadata, DeepFellow uses Doc Chunker to extract the document's text, converting the file if no conversion is cached yet. To have a vision model describe images during that conversion, set image_processing_mode to description in the request body. The default value, ignore, skips image descriptions. The setting applies only when the conversion runs.

Chunking Tokenizers

Every embedding model counts tokens with its own tokenizer, so a chunk that fits one model overflows another. Doc Chunker measures each chunking request with the tokenizer of the embedding model the chunks are destined for. Two query parameters select it, and both POST /custom/doc_chunker/chunk and POST /custom/doc_chunker/convert?chunk=true accept them.

One tokenizer ships inside the image, for the mxbai-embed-large model, with a maximum of 512 tokens. A request that names no embedding_model uses it, and a request that names mxbai-embed-large resolves to the same tokenizer. It never counts against MAX_TOKENIZERS, and Doc Chunker never downloads it, so a container with no outbound network and an empty volume still chunks.

To chunk for any other embedding model, register that model's tokenizer first.

Register a Tokenizer

POST /custom/doc_chunker/tokenizers accepts multipart/form-data with a ZIP archive in the file field, and registers the tokenizer it holds for one embedding model.

Spell model exactly as your inference gateway spells it, tags and namespaces included, for example mxbai-embed-large:latest or mixedbread-ai/mxbai-embed-large-v1. Doc Chunker takes the name as a key rather than a path and returns it verbatim, so a-model:v1 and a-model:v2 are two separate registrations. A name that is blank or longer than 200 characters is rejected with 400.

curl -X 'POST' \
  'https://deepfellow-server-host/custom/doc_chunker/tokenizers?model=nomic-embed-text&max_tokens=8192' \
  -H 'Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY' \
  -F 'file=@nomic-embed-text-tokenizer.zip;type=application/zip'

The response repeats what is now registered.

{
    "model": "nomic-embed-text",
    "max_tokens": 8192
}

Registering a model that already has a tokenizer replaces it. Doc Chunker loads the archive before it stores anything, so a rejected upload leaves the previous tokenizer in place and usable.

What the archive holds

Pack a Hugging Face tokenizer directory with the files at the root of the archive, not inside a folder. It is what AutoTokenizer.from_pretrained(model).save_pretrained(directory) writes:

Everything else in a model repository is the model rather than the tokenizer, so config.json, model.safetensors, onnx/, and 1_Pooling/ stay out of the archive. Doc Chunker accepts only a tokenizer backed by a tokenizer.json, and rejects a sentencepiece-only tokenizer or one that ships its own Python class with 400. It loads tokenizers with no remote code and no network access, so an upload never executes code and never pulls files from outside the archive.

Read a Registered Tokenizer

GET /custom/doc_chunker/tokenizers/{model} returns the model name and its recorded max_tokens, in the shape POST /custom/doc_chunker/tokenizers returns. It answers 404 when nothing is registered for that model. The name is taken as a path, so a namespaced name needs no escaping.

curl -X 'GET' \
  'https://deepfellow-server-host/custom/doc_chunker/tokenizers/nomic-embed-text' \
  -H 'Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY'

Call Doc Chunker Directly

For advanced use cases, call Doc Chunker through the /custom/doc_chunker endpoint without a vector store. The two operations are separate endpoints, and conversion never happens implicitly: POST /convert turns a file into a Docling document, and POST /chunk turns that document into chunks.

Address these calls to DeepFellow Server with your project API key. The server proxies them to Doc Chunker and supplies the Infra credentials itself, so your client never holds the Infra API key. See Custom Endpoints.

Convert a File

POST /custom/doc_chunker/convert accepts multipart/form-data with the file in the file field, and returns a ZIP archive named after the uploaded file.

The archive holds document.json always, assets/ when the document has pictures, and chunks.json when you pass chunk=true.

Doc Chunker checks embedding_model and max_tokens before it starts converting, so an unregistered model or an impossible token limit costs one rejected request rather than a conversion whose result never reaches you.

curl -X 'POST' \
  'https://deepfellow-server-host/custom/doc_chunker/convert?chunk=true' \
  -H 'Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY' \
  -F 'file=@report.pdf;type=application/pdf' \
  -o report.zip

import io
import json
import zipfile

import requests

with open("report.pdf", "rb") as f:
    response = requests.post(
        "https://deepfellow-server-host/custom/doc_chunker/convert",
        params={"chunk": "true"},
        files={"file": ("report.pdf", f, "application/pdf")},
        headers={"Authorization": "Bearer DEEPFELLOW_PROJECT_API_KEY"},
    )

with zipfile.ZipFile(io.BytesIO(response.content)) as archive:
    chunks = json.loads(archive.read("chunks.json"))

print(chunks)

import * as fs from 'fs';

const buffer = await fs.promises.readFile('report.pdf');
const formData = new FormData();
formData.append('file', new Blob([buffer], { type: 'application/pdf' }), 'report.pdf');

const response = await fetch('https://deepfellow-server-host/custom/doc_chunker/convert?chunk=true', {
    method: 'POST',
    headers: {
        Authorization: 'Bearer DEEPFELLOW_PROJECT_API_KEY'
    },
    body: formData
});

const archive = Buffer.from(await response.arrayBuffer());
await fs.promises.writeFile('report.zip', archive);

Convert a File in the Background

POST /custom/doc_chunker/convert answers only once the conversion is done, which takes up to an hour for a large file. A conversion job lets you submit the file, disconnect, and come back for the result. DeepFellow Server itself always converts through POST /convert, so conversion jobs are for direct calls only.

The status and result endpoints answer 404 for an id that Doc Chunker doesn't hold, either because it never existed or because the job has expired.

Submit the file:

curl -X 'POST' \
  'https://deepfellow-server-host/custom/doc_chunker/convert/jobs?chunk=true' \
  -H 'Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY' \
  -F 'file=@report.pdf;type=application/pdf'

The response, and every status request after it, returns the job's status:

{
    "id": "3f9a0c1e5b7d4e8a9c217d4e5f6a8b90",
    "status": "failed",
    "filename": "report.pdf",
    "created_at": "2026-09-29T12:00:00Z",
    "started_at": "2026-09-29T12:00:04Z",
    "finished_at": "2026-09-29T12:41:10Z",
    "error": { "status_code": 422, "detail": "..." }
}

status is queued, running, succeeded, or failed. A failed job's error holds the status code and the detail POST /convert would have answered with for the same failure. Treat the id as an opaque string.

Once status is succeeded, download the result and delete the job:

curl -X 'GET' \
  'https://deepfellow-server-host/custom/doc_chunker/convert/jobs/JOB_ID/result' \
  -H 'Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY' \
  -o report.zip

curl -X 'DELETE' \
  'https://deepfellow-server-host/custom/doc_chunker/convert/jobs/JOB_ID' \
  -H 'Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY'

Job Lifecycle

• Retention: Doc Chunker removes a finished job and its result JOB_RESULT_TTL seconds after the job finished. Delete a job once you have downloaded its result, so it doesn't occupy the volume until then.
• Restarts: Finished jobs survive a restart of Doc Chunker. A job that was queued or running when Doc Chunker stopped becomes failed, with the detail interrupted by restart. Doc Chunker doesn't run it again, so submit the file again if you still need the result.
• Full queue: When MAX_QUEUED_JOBS jobs already wait for a slot, a submission answers 503 with the detail queue_full_retry_later and no Retry-After header, because Doc Chunker has no estimate of when a slot frees up.
• Full volume: When the volume has no room for the uploaded file, a submission answers 507.

Conversion jobs run on the same worker slots as /convert and /chunk, as described in Worker Slots. The Infra install form doesn't expose JOB_RESULT_TTL or MAX_QUEUED_JOBS, so an install from the Infra Web Panel runs with their defaults. Job results live under DATA_DIR, as described in Data Storage.

Chunk a Document

POST /custom/doc_chunker/chunk accepts multipart/form-data with the document.json from a POST /convert archive in the file field, and returns JSON. It converts nothing and contacts no inference gateway, so it needs no conversion model and leaves the document untouched. The assets aren't needed, because chunk content comes from the document's own text and metadata.

curl -X 'POST' \
  'https://deepfellow-server-host/custom/doc_chunker/chunk' \
  -H 'Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY' \
  -F 'file=@document.json;type=application/json'

The embedding_model and max_tokens parameters decide the chunk boundaries. This call measures with the built-in default tokenizer, at its own maximum. To measure for another embedding model, add both parameters, as described in Chunking Tokenizers.

curl -X 'POST' \
  'https://deepfellow-server-host/custom/doc_chunker/chunk?embedding_model=nomic-embed-text&max_tokens=2048' \
  -H 'Authorization: Bearer DEEPFELLOW_PROJECT_API_KEY' \
  -F 'file=@document.json;type=application/json'

The response holds the chunks and a total count.

{
    "chunks": [
        {
            "text": "Quarterly revenue grew by 12 percent compared to the previous period.",
            "meta": {
                "headings": ["Financial Summary"],
                "pages": [3],
                "bboxes": [{ "page": 3, "l": 72.0, "t": 120.5, "r": 540.0, "b": 200.3 }],
                "self_refs": ["#/texts/12"],
                "labels": ["text"]
            },
            "start_index": 0,
            "end_index": 69
        }
    ],
    "count": 1
}

Every key in meta is conditional, and a chunk carries only the ones that apply to it.

start_index and end_index are character offsets into the concatenation of the chunk texts, in the order Doc Chunker emitted them. They are not positions in the original file. Because the overlap extends the earlier chunk, one chunk's end_index reaches past the next chunk's start_index, and the difference is the length of the duplicated text, which is what you drop when you join chunks back together. To locate a chunk within the original document, use self_refs, bboxes, or pages.

Errors

Doc Chunker enforces no size limit of its own on the files it converts, so it never answers 413. The limit on the files you add to DeepFellow comes from DeepFellow Server. A tokenizer upload is the one exception: Doc Chunker rejects an archive over TOKENIZER_MAX_ARCHIVE_BYTES with 400, from the Content-Length header, before it reads the archive into memory.

Doc Chunker also exposes GET /custom/doc_chunker/health, which returns {"status": "ok"} when the service is ready. It returns 503 when one of the conversion backends has exited or a conversion has been running past its soft timeout. GET /custom/doc_chunker/version returns the version of the running container.