Documents
Uploading, indexing, chunking configuration, processing statuses, and sharing one document across knowledge bases.
The life of a document
The knowledge base keeps the document's metadata: its original filename, content type,
size, where the file is stored, its processing status and how many chunks it produced. The
file itself stays in object storage, so it can be downloaded again later, and the chunks
live in your tenant's knowledge schema.
Processing status
| Status | Meaning |
|---|---|
pending | Uploaded, not yet picked up. |
processing | Being extracted, split and embedded. |
completed | Indexed. Its chunk count is on the row. |
failed | Could not be indexed, with the reason recorded alongside. |
A failed document stays in the list rather than disappearing, because the file is still
there and the usual fix is to look at it. Nothing is retried automatically.
Check the chunk count, not only the status. A long document that produced a handful of chunks
extracted badly: a scanned PDF with no text layer, or a file whose text is in images. The status
says completed because the pipeline ran, and retrieval will quietly find almost nothing.
One document, several knowledge bases
Documents and knowledge bases are many-to-many. The same file can be indexed into more than one knowledge base, and its chunks are tracked per pairing rather than per document.
Two consequences worth knowing:
- Chunks are per knowledge base. A document in two bases is chunked and embedded for each, so the same text occupies the index twice. That is what lets two bases chunk the same source differently.
- Removing a document from one base leaves the other alone. The chunks that belong to the pairing go; the file and the other base's chunks stay.
Chunking
How a document is split decides what retrieval can find, and it is the setting worth understanding before any other.
| Setting | Range | What it does |
|---|---|---|
chunkSize | 50 to 2000 | Tokens per chunk. |
chunkOverlap | 0 to 500 | Tokens each chunk repeats from the previous one. |
textSplitStrategy | recursive, semantic, sentence | Where it prefers to cut. |
embeddingModel | text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002 | What turns a chunk into a vector. |
Size is a trade. Small chunks match a question precisely and return little context, so an agent gets a sentence that answers the question with nothing around it. Large chunks carry their context but dilute the match, because a vector averaged over 2000 tokens is close to everything and specific to nothing.
Overlap exists for the sentence that straddles a boundary. With no overlap, a split in the middle of the one paragraph that answers the question leaves neither half matching it well. A few dozen tokens is usually enough; large overlaps mostly duplicate the index.
Strategy decides where the cut lands. recursive backs off through paragraph, line and
word boundaries to fit the size. sentence keeps sentences whole. semantic tries to cut
where the subject changes.
Per content type
Chunking can be overridden per content type, so PDFs and Markdown in one base can be split differently. Useful when a base mixes long-form prose with structured reference material, which want different sizes.
Changing the chunking configuration does not re-index anything by itself. It applies to documents indexed afterwards, so a base whose settings changed halfway contains chunks of both shapes. Use Rebuild index to apply the new settings to everything already there.
Rebuilding the index
Rebuilding re-processes every document in the knowledge base from the files already in storage, replacing their chunks. Nothing is re-uploaded, so it costs embedding calls and no bandwidth.
Reach for it when:
- You changed the chunking or embedding settings and want them applied to existing documents.
- The index is incomplete: counts that disagree, or chunks without embeddings. The health check below names this case.
Rebuilding takes an advisory lock per knowledge base and commits its index writes
together, so two rebuilds of one base cannot interleave. A second attempt while one is
running is refused with a 409 rather than queued.
Checking the index
A health check per knowledge base compares what the platform thinks it has against what is actually indexed:
| Reported | Meaning |
|---|---|
expected vs indexed documents and chunks | Whether the counts agree. |
embeddingsComplete | Whether every indexed chunk actually has a vector. |
pgvectorEnabled | Whether the vector extension is present at all. |
tables | Which of the three knowledge tables exist. |
recommendations | What to do about anything above that is wrong. |
A chunk without an embedding is the quiet failure worth watching for: it exists, it is counted, and it can never be retrieved, because retrieval only ever compares vectors.
Downloading and removing
The original file can be fetched back, which is how you check what was actually indexed rather than what you think you uploaded.
Removing a document from a knowledge base deletes that pairing's chunks. The totals on the base are recomputed, so the document and chunk counts stay honest.
Across tenants
Knowledge bases are syncable, like tools and agents, so one can be promoted from a development tenant to staging or production. See Syncing for what a sync carries.