Skip to main content

internal/ingest

github.com/ensera-ai/taisce/internal/ingest · 1 files · 276 lines · 6 tests · source

Package ingest turns a document on disk into the observations the write contract already accepts, and nothing else. It decides three things: which files are documents and how one is refused, where a long text is cut so that every piece fits one message, and how a piece names itself so that sending it twice stores it once. It does not talk to the network, the database or a model; the command in cmd/taisce does the sending, and the API applies every rule about roles, subjects and sizes exactly as it does for a conversation. A document is not a new kind of memory. It is a sequence of turns whose speaker is a third party.

Why segments are separate observations rather than one long message​

Formation extracts one message per model call, so a message is the unit of extraction cost and of failure: a message that exceeds the turn budget is parked whole, and its neighbours with it if they share a turn. A document cut into segments that are each their own observation is formed piece by piece, retried piece by piece, and cited piece by piece. Evidence is a byte span against the chunk it indexes, so a segment is also the unit of citation: the span is exact within the segment, and the segment's own offset within the document, recorded in the manifest, maps it back to the file. Splitting inside a code point or without recording the offset would make every citation into a document point at the wrong bytes, which is the corruption #91 names.

Why the cut prefers a paragraph, then a sentence, then a space​

The extractor locates a quote inside the message it was given; a claim whose sentence is cut in half is a quote it cannot locate, and the claim is refused as unlocatable. Cutting on a paragraph keeps claims with their context; on a sentence keeps each quote whole; on whitespace keeps words whole; and the last resort, a code-point boundary, keeps the text valid UTF-8 and nothing more. The ceiling is measured, not guessed: on a news corpus the articles that overran a two-minute model budget under a shared server averaged 10.7 KB, and the rest, averaging 4.5 KB, did not (docs/31), so the default ceiling sits at 8 KiB and an operator who has measured their own model moves it.

What is refused, and why before anything is sent​

A file over the size ceiling, one that is not valid UTF-8, one carrying a NUL byte, one with no text once trimmed, an unsupported extension, malformed JSON or JSON with a field this contract does not define: each is refused with no segment sent, because a document half stored is a document whose citations point at bytes the operator cannot reproduce. Nothing here decompresses anything, so an archive is an unsupported format rather than a bomb.

Where it sits​

Imports: internal/domain

Imported by: cmd/taisce

Files​

FileLinesDeclarationsWhat it is for
ingest.go27611Carries the package documentation.

6 tests hold this package's behaviour; the list is named for what each one proves.