internal/infra/pg/rebuildstaleness.go
internal/infra/pg · 165 lines · 5 declarations · source
Declarations
type Stale
type Stale struct {
// Entities is how many entities no current fact refers to. A rebuild that changed its mind
// about a claim leaves the entity it used to point at, and nothing removes it: the alias cache
// then answers with names the current extractor never produced.
Entities int `json:"unreferenced_entities"`
// NameReceipts is how many alias receipts belong to those entities.
NameReceipts int `json:"unreferenced_name_receipts"`
// Communities is how many subjects are waiting for a report. The invalidation trigger deletes a
// report when its facts change; the subject pass writes them back a few per tick, so this is the
// size of the window in which a recall that would have returned a theme returns nothing.
Communities int `json:"communities_without_a_report"`
// MessageEmbeddings, EntityEmbeddings and ReportEmbeddings are whether the active generation of
// each still covers what is stored. False means a search is reading an index built for content
// that has since changed — which `Activate` would refuse today, but a generation activated
// before the rebuild is not asked again.
MessageEmbeddings EmbeddingCoverage `json:"message_embeddings"`
EntityEmbeddings EmbeddingCoverage `json:"entity_embeddings"`
ReportEmbeddings EmbeddingCoverage `json:"report_embeddings"`
}
Stale is what a rebuild left behind.
Why an operation reports what it broke
A rebuild publishes a new interpretation of a project's turns and finishes. From the operator's side that looks complete, and it is not: the entities the old interpretation made are still there, the embedding generations were built over content that has changed, and the reports were deleted by their invalidation trigger and come back only when the worker's subject pass reaches them. Each has a different remedy and none of them is visible.
This does not repair any of it — re-deriving those projections is larger work, and doing it silently inside a rebuild would be the same mistake in the other direction. It reports, so an operator knows what to do next and a monitoring system can see that something is owed.
Every number here is a count against a table that already exists. Nothing is stored; a staleness report is true at the moment it is read and would be a lie the moment after.
type EmbeddingCoverage
type EmbeddingCoverage struct {
// Active is false when no generation has been activated for this surface, which is not
// staleness: it is a deployment that has never built one, and saying "stale" would send an
// operator looking for a problem they do not have.
Active bool `json:"active"`
// Covers is what a message generation was built up to and Current is where the log is now;
// Behind is the difference. Absent for the other two surfaces, which have no offset.
Covers int64 `json:"covers_through_offset,omitempty"`
Current int64 `json:"current_offset,omitempty"`
Behind int64 `json:"behind,omitempty"`
// Stale is the database's own verdict on an entity or report generation whose input set has
// changed since it was built.
Stale bool `json:"stale,omitempty"`
}
EmbeddingCoverage is one semantic surface's active generation, and whether it still covers.
Why the three surfaces answer differently
A message generation records the log offset it was built through, so its staleness is arithmetic: the log has moved on by this much. An entity or report generation records how many targets existed instead, and the database marks its build `stale` when that set changes — which is a stronger signal than a count, because it fires on the change itself. Reporting an offset for a surface that does not have one would be inventing a number, so each is read by the mechanism it actually has.
method EmbeddingCoverage.Behindhand
func (c EmbeddingCoverage) Behindhand() bool
Behindhand reports whether this surface needs a new generation, by whichever measure it has.
method Stale.Anything
func (s Stale) Anything() bool
Anything reports whether a rebuild left anything worth telling an operator about.
method RebuildJobStore.Staleness
func (s *RebuildJobStore) Staleness(ctx context.Context, schema Schema, scope string) (Stale, error)
Staleness reads what a rebuild of this project left behind.
Read rather than remembered: these are consequences of the state the database is in now, and a number recorded at the end of a rebuild would be wrong by the time anybody looked — the subject pass writes reports back continuously, so the honest answer changes every few seconds.