Storage failures and restart | Omnigraph

Storage failures and restart

A served write sends many storage requests: data files, a transaction file, and last the conditional create of the graph's next manifest version, which publishes the commit. This page describes what a client and an operator see when one of those requests fails or gets no answer. A request's outcome belongs to the write that sent it: one failed or unanswered request never stops the server. Process shutdown and admission are described under Deployment.

Publication outcomes

A graph commit becomes visible only when its manifest version is created, put-if-absent. A failure before that request is definite: nothing was published, the write answers its error, and the server keeps serving once the write's own storage requests have settled. That includes a failed put of the transaction file Lance writes just before the manifest, for example one the store throttled with SlowDown, and a manifest request the store refused outright.

A manifest request whose answer was lost, timed out or throttled is re-issued unchanged, up to four times with about three seconds of pauses in total, plus the time spent on each storage request. A re-issue that creates the version publishes the commit once. A version that already exists is read back: it holds either this commit, and the write succeeds, or another commit, and the write meets an ordinary conflict. Re-issuing the same conditional request never publishes a duplicate. When a re-issue meets a version already held and the read-back then finds it missing, the outcome stays unknown rather than failed: on S3 that conflict can answer the first request, still landing.

Unknown publications

The outcome stays unknown if no re-issue settles the request or the committed version cannot be read back reliably. A local filesystem error can also occur after a write has taken effect, so an error alone does not prove that nothing was published. When the outcome remains unknown, the write answers 500 outcome_unknown on HTTP and MCP alike, and the server keeps serving. Each manifest version is created only if absent, so the request lands before the branch's next publication or never: the branch's next write either finds it and builds on it, or takes that version first, after which the late request can no longer land. A read that does not show the write is therefore not proof that it failed until another commit follows its base; read the branch's commits before repeating it.

The same rule applies when creating a proposal or an unattached snapshot: if its final authority check fails, deleting its tag does not prove that an earlier create request cannot still complete. That outcome remains unknown unless the candidate is proven expired.

Retries

The server's storage clients retry reads, and writes of fresh uniquely named files, within Lance's own bounds: three client retries and three throttle retries. A conditional request, such as a manifest create, and a write to a reused key make one attempt each, and their owner resolves an ambiguous answer itself. OBJECT_STORE_CLIENT_MAX_RETRIES, if set, must be 0.

On S3 and Azure, a conditional write of a control object (the cluster lock and state, a deployment resource, a lease heartbeat or claim) whose answer was lost, timed out or throttled is re-issued unchanged, as a manifest request is: up to four times with about three seconds of pauses in total. A request the store refused before applying it, such as a missing bucket or container, fails at once, and so does a permission or authentication error on the server. When an earlier copy of the request may be what the object holds, the object is read back, and holding this request's bytes makes it this write's. An object still absent, or still at the expected version, once no re-issue settles the request leaves the write failed with its outcome unknown.

A command that writes storage directly, without a server, uses a client that retries a request itself before any re-issue: up to its max_retries (10 by default), and on S3 an if-match that meets a conflicting write in flight too. Those attempts are not logged. Because one of them may have applied, that client's permission or authentication error is re-issued like a lost answer, and its first conflict is read back.

A write to a reused key whose outcome is unknown refuses later writes to that key only, until a clean shutdown; a clean shutdown with such a key retains admission. Its owner's re-issue of a conditional write is still admitted. The key settles when that re-issue applies, or when a read of the key shows its condition no longer holds. A request the store refused before applying it leaves its key settled.

When the server stops

A panic, a lost write owner, a publication adopted in storage but not in memory (RecoveryRequired), an opaque execution failure, or an uncertain startup or deployment activation still closes admission for every graph and exits 2. Their effects are not scoped to one write's storage keys.

Restart after a failure

On S3 roots a server's cluster lock carries a lease, so an orchestrator can restart a failed server with no operator step. Set its restart policy to restart on failure (Railway ON_FAILURE, Kubernetes restartPolicy: Always).

A lock with no lease still needs the exact-ID unlock described under writer topology. Local and Azure locks have no lease, and neither does a CLI writer's lock or a lock an older binary wrote. A local root has no conditional update, so a successor could not replace the lock without racing a release the old owner already sent. Older binaries refuse the leased (version 3) lock record; do not delete it to bypass the version check.

Logs

No storage-failure log carries an error's message, which can name storage paths; each records the failure's class.

Rust error API

Rust embedders updating from 0.14.0 must pass the publication's graph commit ID before the diagnostic message to OmniError::manifest_publish_in_doubt(graph_commit_id, message). The ID must name the attempted publication; do not substitute a storage path or invent an ID when the outcome is unknown.

ManifestError.publication_in_doubt changes from bool to Option<Box<String>>: Some carries that commit ID, and None means this is not an in-doubt publication. Prefer the constructor over struct literals, and use is_manifest_publish_in_doubt() or publication_id() to inspect an OmniError. HTTP error responses and CLI syntax are unchanged.