troubleshooting.md

Troubleshooting

CLI failures include a human-readable message. Add --json where supported to receive structured fields suitable for automation. Every HTTP error the server answers, an unknown route, a method the route does not serve and a malformed body, query string or path parameter included, uses a JSON body with error, code, disposition and optional details for conflicts, limits, Blob ranges, external sources, or recovery. A malformed request is 400, a body without a JSON content type 415. MCP tool calls report the same body beside the HTTP status (MCP).

Do not parse human-readable error text when a structured field is present.

Server discovery

Remote graph commands first make an anonymous HEAD /healthz request to the configured service root. Discovery sends neither the data bearer token nor URL Basic-auth credentials; reverse proxies must allow that public probe. A timeout or connection failure is reported with a credential-safe cause category and means the data request was not sent. HTTP discovery refusals retain their status.

HTTP errors

Each code is sent under the statuses ErrorCode in openapi.json names beside its meaning: one, except branch_start_refusal, which is 409, or 410 for not_retained.

Status Meaning Usual action
400 Invalid request, query, schema, configuration, or external-Blob policy, including an update that must carry a stored external Blob reference the policy does not admit Correct the request; retrying unchanged will fail again. For a stored reference, assign that property in the same update (a new value or null), or admit its base in the policy
401 Missing or invalid bearer token. WWW-Authenticate says which: error="invalid_token" when the server refused the token sent, and resource_metadata when the server accepts OAuth sign-in Supply a token configured by the server. The CLI sends a token only to the server it is keyed to; see server credentials
403 The resolved actor is not authorized (forbidden), or a valid OIDC token's subject is not in the server's admission file (not_admitted) For forbidden, change policy or use an authorized identity. For not_admitted, signing in again cannot help: the server's operator must admit the identity
404 Graph, query, branch, entity, or route is unavailable Check the name and applied cluster revision; stored-query denials may also appear as 404
405 The path is served, but not with this method (method_not_allowed) Use a method the Allow header lists
408 The request body did not arrive before its deadline (request_timeout) Send the request again; nothing was admitted
409 Concurrent change, duplicate ID, merge conflict, existing resource, incompatible full-text index, a branch whose writer epoch refuses the writing process (writer_fenced), or a commit no branch can start at (branch_start_refusal) Inspect structured details; not every conflict is retryable
410 Required change-feed history was reclaimed, or the commit a branch start names (branch_start_refusal, not_retained) Capture and durably install a new baseline, then resume from its terminal cursor; start the branch at a retained commit
412 Blob entity-tag or graph-commit precondition failed (precondition_failed), or Omnigraph-If-Actor named another actor than the credential (actor_mismatch) For precondition_failed, refresh the Blob ETag (a Blob write's blob_precondition_failure names the current one), or re-read the branch and retry the mutation with its current graph commit. For actor_mismatch nothing ran: use the named actor's credential or correct the header; retrying unchanged fails again
413 Request or operation exceeded a bounded resource limit (payload_too_large) Split or reduce the operation using the reported limit. Blob limits are listed in Blob limits
415 The body's content type is not one the route accepts (unsupported_media_type) Send JSON with Content-Type: application/json, or the type the route names
416 Blob byte range is outside the value (range_not_satisfiable) Use the returned length to choose a valid range
424 An allowed external Blob source could not be read Restore source availability or correct its URI/credentials
429 Server or per-actor admission limit reached Use the whole-command disposition below before retrying; preserve Retry-After
500 Server or stored-data integrity failure Check server logs; do not assume partial success. A Blob delivery, change-route or streamed-response failure logs its error class (error_variant, and storage_kind for a storage failure), never its storage path; see below
503 Admission is closed, the server's OIDC identity snapshot has expired, or the graph is not served (service_unavailable, graph_unavailable, both with Retry-After), or a published schema change requires completion (recovery_required) Inspect the structured error; generic 503 is not permission to repeat a write

A GET or HEAD /blob 500 logs error_kind="blob_pre_header_internal" with a stage. target is a failure resolving a snapshot target for a policy-gated request. cell is any failure of the engine's Blob read, including the branch or snapshot resolution the engine does itself. transport is the server's own refusal before headers, logged with error_variant="unclassified". A managed Blob body that fails after the headers logs the byte range under one of blob_payload_read (with the error class), blob_payload_short_read (with the returned and expected byte counts) or blob_payload_permit_closed.

A change route (GET /changes, POST /changes/baseline, GET /commits/{commit_id}/changes) answers an internal failure with a fixed 500 and logs error_kind="change_route_internal" with error_variant, storage_kind and manifest_kind. A change route that requires recovery logs the operation_id the 503 reports. An export or change-baseline response that fails after its 200 headers ends with an incomplete body and logs error_kind="served_stream_failed" with stream (export or change_baseline) and the same error class. None of these logs carries the error's message, which can hold object URIs or credentials.

A graph-head 412 includes precondition_failure with expected and, when available, actual. A change-feed 410 includes change_feed_gap; retrying the same cursor cannot recover its missing history.

A 400 that refuses a query includes diagnostic: the stable code (Q… parse, T… typecheck), position (line, column, byte) for a parse refusal or stage (and expression when known) for a later one, expected, and fix when one exists. Act on the fix; a retry of the same source fails the same way. See Diagnostics.

Failed data-write commands

Every server error body, and every CLI data-write and read failure, carries a disposition: effects is none, partial or unknown, and action is fix, refresh, retry, recover or reconcile. The server decides it for the request it answered by the first row that matches:

Error effects action
Any error before the request submitted a write none the refusal's own
A write whose completion the server could not settle; code becomes outcome_unknown unless the action is recover unknown reconcile, or recover
A refusal before the write started (401, 403, 412, 429, api_contract_mismatch, actor_mismatch), a typed refusal of its one publication (a merge, key, read-set or dataset-version conflict, a writer-epoch refusal, a resource limit, an external Blob source, a query diagnostic), or a failure the engine proved published nothing, such as a storage failure before publication; code is kept none, or partial when an earlier step of the write published, such as a load that forked its missing branch the refusal's own
Any other error after a write, which may follow its publication; code is kept unknown reconcile, or recover

A refusal's own action follows its code: recover for a recovery intent or a full-text index to build, retry for too_many_requests, service_unavailable, graph_unavailable and request_timeout, fix for unauthorized, bad_request, api_contract_mismatch, actor_mismatch, not_found, method_not_allowed, unattached_cannot_publish, branch_start_refusal, payload_too_large, unsupported_media_type and range_not_satisfiable, and refresh for the rest: forbidden, precondition_failed, conflict, writer_fenced, external_blob_source, change_feed_gap and internal. A write the server could not settle answers 500.

The CLI reports the server's disposition when that request was the command's only effectful one, and otherwise its own account of the whole command, so a lost response is unknown with reconcile. A read command changes nothing, so its failure always has effects: none. JSON mode writes the failure to stdout, or to stderr when part of the result, such as a streamed page, already reached stdout. Stdout then keeps the bytes already written, which can end inside an unfinished object, as for diff, branch diff, commit changes and changes poll with --json when they follow every page, and the failure document is on stderr. Human mode prints the disposition line on stderr. A target refusal on these commands exits 1 with action: fix. HTTP failures also retain http_status and, when present, the original retry_after string.

This includes commit publish and commit drop, as well as their mutate statement forms. If a response or result output is lost, inspect the commit's current state before another attempt; an output failure does not undo the write. The same applies to load, blob put, blob clear and schema apply: a closed output pipe or failed result flush can follow a successful write. The CLI exits with a failure and does not repeat the operation. If stdout cannot carry a JSON failure, stderr reports the undelivered diagnostic and its disposition, and the exit still follows it.

For the data commands the CLI classifies (queries, reads, load, mutate, branch, commit, schema and Blob commands, and aliases), the exit follows from the disposition:

CLI exit What to do
0 The operation succeeded or was a no-op. A merge can separately report a source-deletion failure without losing its successful result.
1 effects: none: the command changed nothing. Inspect the action and details, correct the cause and run it again.
2 effects: unknown or partial: the command may have had effects, or an earlier step of it did. Read the target, and reconcile or recover as action says, before another attempt; a conflict, resource limit, incomplete response or lost connection does not by itself permit repeating the write.
4 A condition you set did not hold, with no earlier command effects: a verified write precondition or an --as assertion. Re-read the branch and choose a new precondition deliberately.
75 effects: none with retry: the server refused before starting, as when the actor is busy. Honor Retry-After and bound any caller retry.
64 The argument parser refused the command line before anything ran (EX_USAGE); an addressing refusal inside a command, such as two scope flags, exits 1.

Maintenance commands (init, upgrade, optimize, index build, index rebuild, repair, cleanup, schema upgrade-system-columns), cluster, policy, target, credentials, mcp, studio and the managed commands keep their own exits for now: their exit 1 does not by itself say that nothing changed, so read their report; mcp answers each relayed request with its own disposition.

A served write refused with 401 unauthorized, 403 forbidden or 400 api_contract_mismatch exits 1 with effects: none, and action: refresh for the policy denial and fix for the others. The server answers these before it admits the write: bearer authentication, then the API contract header, then policy. Fix the credential, the policy grant or the client before running the command again. If a server target with auth: {provider: anonymous} receives a 401, the CLI adds missing_credential with target, auth: "anonymous" and a setup hint. A missing credential for a token or exec provider refuses locally before any HTTP request. The server's error text is unchanged. The status counts only with its matching code in a response that carries the server's API contract header: a bare 403 from a proxy stays unknown with reconcile, and a refusal that also reports recovery work stays recover.

fix means correct the credential, input, path or client; sent unchanged, the command fails the same way. refresh means read the current state and decide again; it does not grant automatic replay. partial effects mean an earlier step stands although the failing one wrote nothing; read the target before deciding. recover means finish the reported recovery obligation. reconcile means establish the original operation's outcome before another attempt. Only retry with exit 75 grants the bounded retry described above; the CLI performs no automatic retry.

The disposition covers the entire command. A later refused subrequest cannot undo an earlier branch creation, load or publication. An embedded write's own refusal says what the engine did, projected as a server would project it: a policy refusal precedes every publication, a load --from fork included, so it has effects: none on both planes. The engine marks an error that follows a fork which did publish: a typed refusal then has effects: partial on both planes, and an error the engine cannot prove wrote nothing has unknown. A load --from onto a branch that already exists forks nothing. Any later failure, such as rendering a committed receipt, stays unknown. Disconnecting an admitted HTTP write does not cancel it: the server keeps running the original operation, and a lost response can leave the caller's outcome unknown. There is no durable request-result lookup in this release.

Conflicts

A 409 is not one universal retry signal:

Writer epoch

A branch at storage format v17 carries a writer epoch, a number that starts at 0. A claim raises it by one and makes the claiming process the branch's owner; a claim by the process that already holds the epoch writes nothing. From then on, every publication to the branch, which is how a mutation, a load, a merge into the branch and a schema apply land, is refused unless the writing process holds the branch's current epoch. A process whose own claim another process has since replaced is refused like one that never claimed. The owner serves reads of a held branch its handle is bound to from its own last publication, without checking the branch first. So a replaced owner can keep answering reads from before the claim until a write of its own is refused. The refusal is HTTP 409, code: "writer_fenced" and writer_fenced: { "held": …, "read": …, "current": … }: current is the branch's epoch, held the refused process's own and absent when it holds none, read the epoch a refused claim read. Nothing was written. Deleting the branch is not refused: an owner whose branch another process deleted keeps answering reads from the deleted branch until its own next write there.

No CLI command or HTTP route claims a branch in this release. Only a program that embeds the engine does, through Omnigraph::claim_branch_writer, under the branch_claim action, and only on a branch at storage format v17: below it the claim is refused and names omnigraph upgrade. A branch nobody claimed stays at epoch 0 and accepts every writer as before. A process holds its claim until it exits or another process claims the branch, so an owner that restarts claims again before it writes. The epoch of a claimed branch never goes back: after the owner exits, every writer is refused until a process claims the branch again. The graph records the epoch and not its holder, so no command names the process that holds a branch. So omnigraph-server and direct CLI commands cannot write a claimed branch in this release; write to it through the embedding program that holds it. A branch deleted and created again under its name starts at epoch 0.

Writes are atomic at the graph-commit boundary. A normal validation, conflict, or limit error does not mean that a subset became visible. When recovery_required identifies an already published schema change, its table pins and accepted schema contract are durable together; only the process view may still need refresh. Reopen the graph read-write or restart its server rather than replaying the change. Legacy recovery sidecars require the original release, as described under maintenance failures.

Storage-format mismatch

If a graph was written by a different storage-format generation, the binary refuses to open it and names the required release line. Follow the qualified explicit routes in Upgrading to preserve graph identity and history; normal open does not migrate or reset the graph.

Do not edit internal metadata or copy files from individual backing datasets between graph roots.

Cluster failures

See Operating a cluster.

Maintenance failures

If optimize, index build, index rebuild, repair, cleanup or upgrade cannot write or flush its output, the CLI returns a failure without repeating the operation. An output failure does not undo completed work. Inspect the graph's state before another attempt; for an interrupted upgrade, follow the upgrade recovery procedure.

Foreign drift

omnigraph repair classifies a graph table as foreign_drift when its Lance linear history carries commits above the table's recorded last linear version (omnigraph.last_linear_version on the registration). Every OmniGraph write is a detached commit that a graph commit pins, so such a commit came from something else writing the table directory directly. The condition is per table.

There is no command to run before writing: the write path has no precondition on a table's linear history. To discard the foreign commits, export the graph and load it into a new one.

See Maintenance and Deployment.

Useful diagnostics

omnigraph version
omnigraph snapshot --target ./graph.omni --json
omnigraph commit list --target ./graph.omni --json
omnigraph cluster status --target ./company-brain --json
omnigraph repair --target ./graph.omni --json

For server requests, retain the HTTP status, response body, request path, timestamp, and server logs. Redact bearer tokens and storage credentials before sharing diagnostics.