The collection format

What a knowledge collection is on disk — the contract between whatever builds an index and whatever answers from it.

*The contract between the thing that builds an index and the thing that answers from it — DSL 0.31, rewritten 2026-10-22.*

Two programs have to agree about this and they are written at different times by different people: an indexing agent produces it, a search server reads it. Two implementations that disagree quietly is the failure this whole design has been avoiding since 0.30, which is why the format is specified here rather than left to whoever writes the first one.

This page described a directory until 2026-10-22 — collection.json, passages.jsonl and a bare vectors.bin — and the owner chose SQLite instead. That was the better choice and the change is recorded rather than tidied away, because three of the four refusals this page used to demand existed only to survive a raw float file. See What SQLite removed.

The shape#

One SQLite file, opened with node:sqlite, which is built into Node and already used by the board — so this adds no dependency.

collections.db
  store_meta         what schema this is
  agents_chunks      one row per chunk, for every agent here
  collection_built   what built each collection, and when

One table for every agent, with agent_id on every row, so one installation of the indexer serves many agents and a row always says whose it is.

agents_chunks#

agent_id and collection are both here because they answer different questions. The owner's spec asked for agent_id; the collection name is what a flow and the board already speak.

collection_built#

One row per collection: model, dimensions, method, built_from, passages, built_at, agent_id, refused.

• model is a name, not a description, and it is the field the record exists for: a query embedded by a different model than the index returns noise, silently. The board compares it against the previous build and records a change as something that went wrong, with both names in it.
• built_from is for a person, not a program. The HR share, *the policy site* — never a path, for the same reason a flow never carries one.
• refused is what the last build could not read, by name, as a JSON array - and it has three answers, which a reader must keep apart:

Collapsing null into [] turns nobody recorded this into *nothing was refused*, which is the one reading this record must never produce. The names only, never the reason: two of the three reasons a document is skipped are safe, and the third is the underlying read error - which for a missing file is an ENOENT carrying the whole path. This record is read by a board, and the rule that keeps a folder out of the DSL keeps it out of here.

It is not part of STORE_SCHEMA. The column is added when a store is opened and is nullable, so an older store gets it with every row null and an older build reading a newer store ignores a column it does not know. A schema bump would have refused every store in existence until somebody rebuilt - a hard break bought for one field that is readable as not said by construction.

• A collection with no row here has never been built, which is a third answer and not a synonym for empty.

An empty answer, and which kind of empty it is#

A search has five nearest paragraphs to hand back whether or not any of them answers the question, and an agent handed five paragraphs writes an answer out of them. So search carries refused, and it has three values which a reader must keep apart:

null must never be read as false. A collection that cannot refuse and one whose judge approved are different facts, and treating them alike reports a guard that is not there - which is worse than having no guard, because somebody then stops checking.

Why it is asked rather than computed#

Two arithmetic answers were measured on 2026-10-27, on two corpora, in every mode a deployment can be set to, and both failed:

• a floor on the score. Seven of the eight corpus-and-mode pairs overlap: an answer scores as low as a control. The one pair that separates is a column built from hand-written wordings and is refuted by the other corpus.
• the margin - the best score against the second, which is scale-free and so survives a fused score being a cosine in one mode and a matched-wording fraction in another. It does not separate either, and in five of eight pairs the controls have the wider margin. The reason is legible once seen: a question a corpus cannot answer often returns one passage matching on something incidental with nothing else near it, while a real question's answer comes back with three or four genuinely related paragraphs. *Confidence looks like isolation, and on these corpora isolation is what a wrong answer has.*

So does this passage answer this question is not a property of the score, in any arithmetic over the scores retrieval already produces. Asking a model took refusal from 1 of 8 and 0 of 8 controls to 7-8 of 8 and 8 of 8 over three rounds on each corpus, costing nothing on prose and one answer in eleven on a legal document in two rounds of three.

It costs one model call per question, at query time - the first call in this platform that exists to prevent an answer rather than to produce one. When it refuses, the answering call never happens.

What SQLite removed, and what it did not#

Three of the four things the directory format made a reader refuse simply cannot happen now:

A build replaces, in one transaction. That is the decisive reason for SQLite rather than a convenience: a rebuild has a window between *deleted the old* and wrote the new, and in that window the collection answers nothing — so an agent asking it would be told, truthfully and uselessly, that nothing there answers the question. Inside a transaction the window does not exist: a reader sees the old collection until the moment it sees the whole new one, and a build that fails half way leaves it exactly as it was, still answering.

Nothing here answers this needs a floor#

This page used to claim that no paragraphs means nothing here answers this, and with a real embedding model that state was unreachable. Cosine over normalised vectors is almost always positive, so score > 0 kept every paragraph: a question the corpus could not answer came back with unrelated paragraphs at a low score, and the llm node below was handed them as though they bore on it.

Measured on 1,355 paragraphs of this project's own docs with text-embedding-3-small:

So a search keeps only paragraphs at or above MIN_SCORE, 0.25 by default and 0 to turn it off. With the floor the four unanswerable questions return nothing at all, which is the sentence this page always said they should.

• The number is the deployment's. A usable threshold depends on which model is in use, and the model is already the deployment's — a flow naming one would be a flow that knew what indexed it.
• The floor travels back with every answer. Without it, *nothing here answers this* and the floor is set too high for this model are indistinguishable from the asking side, and the second is a configuration mistake somebody can fix in a second if they are told.
• It only showed up with a real model, which is why it is written down here rather than quietly fixed: every test before the acceptance used hand-made vectors on three axes, where an unrelated question really does score zero. The suite reached the nothing found path and no deployment could.

What a score does and does not tell#

It tells you whether the corpus is about the subject. It does not tell you whether the right paragraph came back. Measured on 565 paragraphs of this project's own reference docs, eight questions in three kinds:

The middle band sits inside the first and above part of it. So a floor separates absent from present, which is the job it was put here to do, and nothing about a score separates found from not found.

Do not build a thin-answer warning on it. One would have fired on nothing useful and stayed silent on both questions it was meant to catch. What is worth reporting is the fork that is knowable - nothing above the floor, or paragraphs that came back and did not answer - and src/knowledge.js reports it to the trace, never into the channel: *a question goes in words and paragraphs come back in words* is a rule of the language, and a score in the channel is a score in the prompt.

A hit may arrive with the paragraphs around it#

A floor cannot be reached by depth, and that is not obvious until it costs an answer. Measured on the acceptance corpus — a 23-page Hebrew articles of association, fixed at 350, the multilingual model — one right runs about 2,000 characters and became six consecutive paragraphs. Asked the question that clause answers:

So the best paragraph in the collection, for a question asking whether the right exists, was the sentence saying where it does not — and asking for fifty paragraphs still returned only four of the six, because two of them scored under the floor and no depth reaches what the floor has already removed.

AROUND is how many paragraphs either side of a hit come back with it, 1 by default — measured rather than chosen, and AROUND=0 is the behaviour before it existed. Nothing said and zero are different answers: the first is this installation's, the second is somebody deciding that the matching paragraph alone is what they want. The pieces of a clause are adjacent, and ordinal is already in the table, so this is one range over rows already read — not a second search and not a different index:

• It is not a substitute for depth, nor depth for it. most reaches further across the document, which a question answered in two unrelated places needs. AROUND reaches further around what already matched, which a question answered by one long clause needs.
• Adjacent paragraphs come back stitched into one, in document order, carrying the best score of their pieces and a passages count saying how many they are. The alternative — returning them separately — puts the same sentence in the prompt more than once whenever two hits are near each other, and a model handed a sentence twice reads it as emphasis. Ranking a stitched paragraph by its mean rather than its best would push a long clause below a single sharp unrelated hit, which is the opposite of the intent.
• They are rejoined with a single space. fixedChunks trims each piece and steps over the whitespace it broke on, so the boundary's own character is gone; joining with nothing glues the last word of one paragraph to the first of the next, which is the same run-together word this corpus already suffers from in extraction.
• around travels back with the answer, as the floor does and for the same reason: with a window on, one paragraph in the answer is several of the index's, and a reader comparing what came back against what was indexed is otherwise comparing two different units.
• This is the cheap half of parent-child, which has been in CHUNK_STRATEGIES since 0.31 and is still null — *index the small chunk, return the section around it*. It is a retrieval setting rather than a chunking strategy: it does not change how a document is split, only what a hit brings back. The real parent-child returns the enclosing section, which needs the document's sections and so needs structure first; adjacency approximates it with what the store already records.

Two things a window must never do, both of them silent:

1. Reach a paragraph the reader may not read. Widening is done over the rows mayRead has already allowed, never over the table — widening by ordinal against the store hands a reader the paragraph beside one they are allowed to see, which is exactly the disclosure the reader filter exists to prevent. Where a restricted paragraph sits in the middle of a clause the stitched run breaks around it rather than closing over the gap.
2. Cross from one document into another. Ordinals run across the whole collection, so the paragraph after a document's last is the next document's first. A stitch over that seam invents a sentence spanning two unrelated documents and attributes it to one of them, and nothing about the result looks wrong.

What is left to refuse, because each still produces a worse answer rather than an error:

1. A schema it does not know. Refused, never guessed: a store one version ahead looks almost right, and almost right is the worst thing an index can be. The refusal closes the file before throwing, so whoever catches it can still move the store aside.
2. A question embedded by a model of a different width than the rows. Checked per row, because the model can change underneath.
3. A build that would replace a collection with nothing. Refused in two places — src/building.js and the store — because it is reachable from both, and because the result is a collection that answers from an empty index rather than erroring.

Who may read a paragraph#

A collection whose paragraphs depend on the reader puts labels on them, and a reader is accepted when it holds at least one of a paragraph's labels.

What this file specifies is the shape, not the policy. How a reader's labels are decided — a directory group, a role on the board, a claim in a token — is the installation's, and the search server is where that is resolved. The platform's half is narrower and is the half it can keep: a run with no reader is not allowed to ask a collection that declares one (src/knowledge.js), so the question never leaves.

A paragraph with no labels is readable by anybody. That is a decision, and it is the dangerous direction: it means a collection marked per-reader leaks any paragraph somebody forgot to label. It is this way because the alternative — no labels means nobody — makes a half-built index look empty rather than wrong, and an index that looks empty gets debugged as a bug in the data while the real answer is that nothing has been labelled yet.

Whoever builds a per-reader collection should treat an unlabelled paragraph as a defect, and a build that produces one should say so.

Who is asking, which is not who the answer is for#

A search call carries onBehalfOf beside reader, and the two had to stop being one field.

A search server that answers from an index with no model behind it can ignore onBehalfOf entirely. One that uses a model — to rewrite the question, to judge what came back, to rerank — spends on behalf of the asking agent, and this is the field that says which one. The reference implementation charges each call to that agent and reports it over the board's own log frame; a cost with nobody to attribute it to is a cost no budget can enforce, and both the daily budget and the runaway rules are per agent. Fourteen agents may ask one server.

**Absent is nobody said, and nothing is charged.** Not charged to the server, not charged to a bucket of its own, not charged to a placeholder named after the missing field. A search with no onBehalfOf works exactly as it did and costs its model call to nobody — which is the right refusal: a figure filed under nothing is a figure no screen can show and no threshold can read.

And it never reaches a paragraph, a trace or a label. What travels back to a board is which call it was and for which collection — judge for handbook — and never the question or the passage. The journal rule, for the journal's reason: a board is read by other people.

How a document becomes paragraphs#

The strategy is named by the flow (index node → method) and never described to it. src/constants.js → CHUNK_STRATEGIES holds the whole vocabulary including what is not built yet, each mapped to the version that introduced it or to null, so a flow naming a planned strategy is a validation error at design time rather than a failure when the trigger fires.

Two of the eight are built, and that is what a board offers and what a measurement compares — *if it is in the list it is in the compare, and if it is not in the list it is not in the compare.* Both come from CHUNK_STRATEGIES_AVAILABLE, which is this table's first column with the null rows dropped, so none of the three can drift from the others. The version in this column is the one in CHUNK_STRATEGIES, and a test fails when they disagree — structure shipped in 0.32 and this table went on calling it unbuilt until 2026-10-25, which is the reason that test exists.

fixed is deliberately crude: it is the only one whose behaviour a person can work out from its name, which is what makes it the right first one — when retrieval is poor, the chunking is not the thing to suspect. It breaks at whitespace within the last tenth of a chunk so a word is never cut in half, and that is the only judgement it makes.

Its weakness is real and is why phase 2 exists. A fact spanning a boundary is split across two chunks — Carry-over is capped at five in one and days in the next — and neither chunk then answers the question on its own.

Chunking is written, not generated from a description. It is the exact shape of the failure docs/decisions.md records under *a model writing a parser from prose*: a chunker that is subtly wrong produces a worse index rather than an error, and nothing downstream can tell a bad answer caused by a chunk that stopped mid-sentence from one caused by the model.

Where the documents come from#

KNOWLEDGE_<COLLECTION>_FOLDER, from the deployment's .env. Never from a flow and never from a call, because a path is a fact about one machine: a flow that named a directory would stop working the first time it was installed anywhere else, which is the rule requires.mcp has kept since 0.23.

It is also what keeps a webhook from choosing what gets indexed. The engine sends documents or sends nothing; there is nowhere in the DSL to put an address.

Extraction is extractDocument in src/read.js — the same PDF and Word reading a read node uses, shared rather than copied. It was split out of readDocument for this, and the reason is worth keeping: readDocument takes an address, a local path is not one, and a bare string falls through to the text is the document — so the first version of the folder reader produced paragraphs whose text was the file's own path, silently, straight past the rule that a deployment's paths never travel. The address rules stay in readDocument: a read node still fetches http and https only and still refuses file://.

A document that cannot be read is skipped and counted, not thrown — the behaviour the DSL cannot express, since onError names a node and so a fan-out of read nodes cannot skip one copy. One scanned PDF in five hundred must not index none.

What a build answers with#

A build is reported, and the record outlives the run — so what it answers with is part of this contract rather than a convenience of one implementation.

**The word for one of these is now paragraph; the key is still passages. Everything a person reads — a label on a screen, a hint, a sentence in these pages — says paragraph. The key on the wire, the column in the board's database and the field a collection server answers with did not move, because they are a protocol and a schema: renaming them is a protocol bump, a migration, and every collection server deployed anywhere needing to be rebuilt in the same moment, and an older agent's passages arriving at a newer board would be read as nobody said** rather than as a count. So the two layers are deliberately out of step, and a table of wire names below is the place where the old word is correct.

The last four are a report and are coerced rather than refused: a server that does not send them is an older one, not a broken one, and null is *did not say* — the third answer this page keeps everywhere. They reach a board through knowledge-built on the agent's own socket, which LGStudioMonitor/docs/protocol.md defines, and are kept per run so that was it always 129 paragraphs and did somebody change the model under it are answerable at all.

What reads it#

collection/collection-server.mjs is the reference implementation of both halves: the search tool a knowledge node asks, and the build tool an index node calls. One server, because the thing that answers search already holds the embedder — it has to embed the query — and the store.

And a third tool, measure, which no node calls. It answers a question about a collection rather than out of one: given questions whose answers are known, it indexes the same documents several ways and says which way found them and at what rank. It is here because the choices this page specifies — how a document is split, how big a paragraph is, the floor under which nothing answers — are not choosable from first principles. Measured on two real corpora with one model, structure found 6 of 8 known answers where fixed found 3; on prose the two swapped places. No default is right for both, so the format ships a way to find out rather than a recommendation.

It is taken unfloored, unless the caller asks for a floor. A floor is applied before anything comes back, so a measurement that inherits one cannot see either of the two bands a floor is set from: an answer scoring below it is absent and reads as not found, and a control scoring below it reports no score at all. Every row carries its own score instead, and the floor is read off the result — the lowest an answer scored against the highest a control reached, with any number between them separating the two.

Every variant reports what it was asked, and two measurements are comparable only when that matches as well as the corpus and the model. Measured: the same folder, digest, model and chunking gave fixed/350 4 of 11 answers at around 0 and 7 of 11 at around 1, because stitching a hit to its neighbours puts the marker inside the returned text. That is a larger swing than the difference between the two chunking strategies, so a reader comparing two tables has to be told both numbers.

It writes nothing into the collection. Each design goes into a scratch store that is thrown away, so comparing four ways of indexing does not leave the collection built the fourth way — which is what the obvious implementation does, and it would destroy what every agent is answering from. A server implementing this format may leave measure out; a board that asks for it and gets no answer says the server may be too old to have one, and nothing else changes.

collection/collection-store.mjs is the schema and the queries; collection/chunk.mjs is the strategies; collection/collection-folder.mjs reads the folder. test/collection-server.test.js holds every refusal on this page, and holds the unlabelled-paragraph decision too.

The embedder is EMBED_URL: point it at a model on this machine and nothing leaves the network, point it at a hosted one and the documents do. One line, which is where § 4.3 put that choice — and the same endpoint embeds the documents and the questions, so they cannot disagree.

What this does not specify#

• How the index is searched. Brute-force cosine over every row is what the reference does, and node:sqlite has no vector index, so it is also the only option there. It is fine well past the size at which anything else here matters. A server wanting an approximate index may build one beside these tables; nothing else reads it.
• Where the database lives. The deployment's, via .env. It is deliberately not the board's: indexing-agent-scope.md § 5 records that the board is the single point and holds no documents, and the measured sizing says why — about 4.2 KB a row with a 384-dimension local model and 8.3 KB with a 1536-dimension hosted one, which puts fifty agents at 1–2 GB against the board's ~373 MB in total, scanned on every question because node:sqlite has no vector index.
• Reclaiming space. Nothing here vacuums. A collection rebuilt smaller keeps its file at the largest it has ever been — measured: 11.2 MB holding 5.7 MB after a rebuild from 1536 dimensions to 384. That is SQLite's ordinary behaviour and is left alone deliberately: a vacuum rewrites the whole file and would block the collection being asked, which is a worse thing to do to a running agent than holding some free pages.