internal/extract/extract.go
internal/extract · 336 lines · 11 declarations · source
This file carries the package documentation, rendered on the package page.
Declarations
type Proposal
type Proposal struct {
Subject string
Predicate string
Object string
Statement string
Confidence float32
SubjectType string
ObjectType string
// Quote is what the model says it read. It is treated as a SEARCH TERM, never as evidence:
// what is stored is the text found in the message, which is the only string a byte span can
// honestly index.
Quote string
// Polarity is what the message does with the relation: states it, denies it, hedges it, supposes
// it, or reports somebody else saying it. Only the first becomes a fact.
//
// An empty value is not "asserted". It means the model did not answer, and a claim nobody has
// decided the polarity of is refused — see the reasoning at the refusal site.
Polarity domain.Polarity
// Tense is when the message puts the relation: now, over, or not yet. Only the first becomes a
// fact, and an empty value is refused for the same reason an empty polarity is.
//
// Separate from polarity because they are independent and both have to be right: "I used to live
// in Amman" is asserted, positively, about a period that is over.
Tense domain.Tense
}
Proposal is what a model returns, before anything has been checked.
It is deliberately not a domain.Claim. A Claim carries a byte span, and a span this package has not verified is the defect this package exists to prevent — so the model is not given a type it could put one in.
type Model
type Model interface {
Propose(ctx context.Context, message domain.Message, vocabulary domain.Ontology) ([]Proposal, error)
}
Model proposes claims for one message.
The vocabulary is passed in rather than held by the implementation, because it is a per-tenant table and an implementation holding its own copy is a second definition of a closed set.
type Result
type Result struct {
Claims []domain.Claim
// Rejected is not an error. Most messages contain no assertable claim at all, and a message
// whose every proposal was rejected is the ordinary case rather than a failure.
Rejected []domain.RejectedClaim
}
Result is everything one message produced.
type Extractor
type Extractor struct {
model Model
vocabulary domain.Ontology
// unresolvable are terms that cannot be a subject, from the table that holds them. Empty is
// valid and means the check does nothing — a deployment that has not seeded the table gets the
// behaviour it had before, rather than a refusal it cannot explain.
unresolvable map[string]struct{}
}
Extractor applies the vocabulary and the span rule to a model's proposals.
func New
func New(model Model, vocabulary domain.Ontology) *Extractor
New builds an extractor over a model and a tenant's vocabulary.
func NewWith
func NewWith(model Model, v domain.Vocabulary) *Extractor
NewWith builds an extractor over both closed sets: the relations it may admit, and the terms that cannot be a subject.
method Extractor.Extract
func (e *Extractor) Extract(ctx context.Context, message domain.Message) (Result, error)
Extract returns the claims one message supports.
The span is the whole point
Every returned claim satisfies `message.Content[ByteStart:ByteEnd] == Quote`, and it satisfies it by construction rather than by the model's cooperation: the quote is located IN THE MESSAGE and the stored quote is the text that was found there. A model that paraphrased has produced a claim with no locatable quote, and that claim is not a weaker claim — it is an uncitable one, in a product whose argument is citation.
An empty result is not an error
Most messages assert nothing. "Thanks, that worked" produces no claims, and a caller that has to distinguish that from a failure will end up treating failures as ordinary.
func reject
func reject(p Proposal, reason string, ordinal int) domain.RejectedClaim
func Locate
func Locate(content, quote string) (start, end int, found bool)
Locate finds a quote in a message and returns its byte span, or reports that it is not there.
Exact first, then whitespace-tolerant, and nothing else
A model copying text out of a message routinely changes its whitespace: a line break inside a sentence comes back as a space, and indentation disappears. That difference carries no meaning — the words are the same words — so refusing it would discard correct citations for a formatting artefact.
Nothing else is tolerated, and case in particular is not. A model that changed the case has retyped rather than copied, and once retyping is admitted there is no line left to draw: a tolerance wide enough to match "moved to dublin" against "moved to Dublin" is wide enough to match a sentence the model composed.
The span is mapped back to the ORIGINAL bytes
Matching happens against a whitespace-normalised copy, but the span returned indexes the message as it was stored. A span into a normalised string that exists nowhere is precisely the failure this package exists to prevent, arrived at from a different direction.
The first occurrence wins
A quote appearing twice in one message is the same words in both places, so a citation resolves to the same text either way; only the instance differs, and nothing reads the instance. Refusing repeats instead would discard a claim for being quoted about something mentioned twice.
type normalised
type normalised struct {
text string
start []int
end []int
}
normalised is a whitespace-collapsed copy of a string together with, for each of its bytes, where that byte came from in the original.
Two parallel slices rather than one, because a match needs the START of its first byte and the END of its last, and for a multi-byte rune those are not one number apart.
func normalise
func normalise(s string) normalised
normalise collapses every run of whitespace to a single space and drops it at both ends.
Runs collapse rather than disappear, because removing whitespace entirely would match "works at" against "worksat" and, worse, would join two words across a line break into one that appears in neither.