blobs.md

Blobs

Source: https://www.omnigraph.dev/docs/schema/blobs

Use Blob properties for bytes that should not be projected through .gq. Blob values can contain graph-managed bytes, an external URI reference, or null. A valid empty managed Blob is distinct from null.

node Document {
  slug: String @key
  content: Blob?
}

Writing Blob values

Load and mutation input use one String representation:

{"type":"Document","data":{"slug":"manual","content":"base64:SGVsbG8="}}

New external references are denied by default. A cluster-served graph must list allowed URI bases in its graph configuration. Direct --target CLI access has no external-source allowlist, so it accepts managed base64: input but rejects new external references. Credentials must not appear in stored URIs. An allowed base must lie outside the cluster's storage root and every graph root; see External Blob references. The same bases admit the source of a load csv statement (CSV loading), which reads the object once and stores no reference to it.

Write mode determines ownership:

An update never reads the old value of a Blob it assigns. When it assigns no Blob and is the only statement of its mutation that writes the type, it can write just the assigned properties into the rows' existing storage: the row's Blob and vector values are neither read nor rewritten, and a stored external reference stays as it is. It does so whenever a matched row holds a stored external reference, and otherwise when it writes fewer bytes than rewriting the matched rows whole. That is the usual case when the rows hold large Blobs or vectors. A one-row update in a large table of small rows rewrites the row. Such an update is never refused for the Blobs it does not assign: when rewriting the rows would exceed write_max_bytes, or when the mutation also writes another type, it writes just the assigned properties. Whether it is admitted never depends on an index, and only the matched rows count toward write_max_bytes, not the other rows stored beside them.

An update that rewrites whole rows carries every Blob cell of a matched row it does not assign: it reads the cell and rewrites it as managed bytes, and those bytes count toward write_max_bytes. Carrying a stored external reference therefore needs the graph's external Blob policy to admit the reference's source. Otherwise the update fails with a 400 that names the type, id, and property. Assign that property in the same update, to a new value or to null, to replace or clear the reference without reading it. An update rewrites whole rows when it assigns a Blob, or when another statement of the same mutation writes the type.

Only a graph written outside OmniGraph can hold a stored reference to a byte range of an object. Export writes an external reference as a bare URI, which reloads as the whole object, so export refuses a ranged reference instead of widening it. Change-feed images, the change-feed baseline and entity reads by id describe it exactly, as {\"uri\": …, \"offset\": …, \"length\": …} with a positive length, without reading the object. The feed passes the commit that holds it like any other, and a baseline taken while the row exists succeeds; that baseline does not reload with load. A stored descriptor with an offset but no length is refused as a Blob integrity error.

OmniGraph never deletes the object named by an external reference.

Replacing one Blob value

One Blob cell of an existing node or edge, addressed by exact id like a Blob read, can be replaced or cleared without a load or a mutation. Over HTTP, PUT stores the raw request body as a managed value and DELETE clears the cell:

PUT /graphs/knowledge/blob?entity=node&type=Document&id=manual&property=content&branch=main
Omnigraph-Http-Api: 0.14
Content-Type: application/octet-stream
If-Match: \"<current ETag>\"

<raw bytes>

An embedded Session does the same with put_blob_at_as and clear_blob_at_as. A put returns the stored length, its ETag and the commit. Clearing a cell that is already null publishes nothing and returns no commit.

Each write that changes the cell is one graph commit and needs the change action on the branch. Neither verb inserts a row: a missing entity is NotFound (404). Clearing a property that is not nullable is refused (400). The write never reads the cell's old value. Lance replaces whole rows, so the row's other cells are carried as an update carries them: a stored external reference among them needs the external Blob policy to admit its source, and their managed payloads count toward the operation's Blob payload allowance, write_max_bytes, together with the new value.

A precondition makes the write conditional on the cell's current value: over HTTP an If-Match field, embedded a BlobPrecondition. A list of entity tags (BlobPrecondition::Tags) holds when the current ETag is one of them; weak tags never match. * (BlobPrecondition::AnyExisting) holds when the cell is not null. When it fails, the write changes nothing and returns BlobWritePreconditionFailed, over HTTP a 412 precondition_failed whose blob_precondition_failure.current_etag and ETag header carry the cell's current ETag when it holds a managed value. A malformed If-Match is a 400. A commit that lands while the write is in flight makes it start again from the new head and check again, so a stale ETag fails instead of overwriting the newer value; a write without a precondition replaces whatever is current. A schema apply or a delete and recreate of the branch landing in that window refuses the write instead, over HTTP a 409 read_set_conflict.

The returned ETag is the one a read at the returned commit reports. When a call fails after its commit became durable, for example because its acknowledgement was lost, the value is published once; read the cell for its current ETag before retrying with a precondition.

Over HTTP:

From the CLI, blob put stores the bytes of --file or stdin and blob clear nulls the cell. Both take --branch, --if-match and --json, and print the receipt the server returns:

omnigraph blob put node Document manual content --file manual.pdf \
  --if-match '\"<current ETag>\"' --target ./graph.omni
omnigraph blob clear node Document manual content --target prod --graph knowledge

The CLI refuses input over 32 MiB and a malformed --if-match before it addresses the graph. The value must also fit write_max_bytes: the process's OMNIGRAPH_WRITE_MAX_BYTES for a --target write, the server's for a served one. --as names the actor of a --target write; a served write takes its actor from the bearer token, and --as there only asserts it. A Blob precondition failure exits 4, embedded or served, as a graph-commit precondition does. Each write is sent once: after an unknown outcome, read the cell before writing again.

Query behavior

Blob properties are not ordinary .gq read values. They cannot be projected, filtered, ordered, or aggregated. Write them through load or mutation assignment, then read an individual Blob value through the dedicated CLI or HTTP surface.

To replace or clear one value, use the CLI, HTTP or embedded writes in Replacing one Blob value; each is an atomic graph commit.

CLI reads

The selector is ENTITY TYPE ID PROPERTY, where ENTITY is node or edge. Blob commands address the graph with --target NAME|ADDRESS; they do not take a positional graph URI.

# Stream managed bytes to a file.
omnigraph blob get node Document manual content \
  --out manual.bin --target ./graph.omni

# Inspect the value without reading payload bytes.
omnigraph blob stat node Document manual content \
  --json --target ./graph.omni

Reads default to main. --branch <name> and --snapshot <commit-id> are mutually exclusive.

blob get supports --offset, --length, and --out. With no range flags it streams the complete value. --offset N reads from N to the end; --length M reads the first M bytes; together they read N..N+M. A length of zero is rejected. An end beyond the value is clamped to EOF, while a start at or beyond the end of a non-empty value is unsatisfiable.

A failure before transfer begins leaves an existing --out file untouched. A successful empty value creates or truncates the file. If transfer fails after output begins, stdout or --out may contain the prefix already written. For atomic file replacement, write to a temporary path, check the exit status, then rename it.

blob stat --json reports the selector, value kind, and exact resolved snapshot. A managed value also reports size and etag; a whole-object external value reports its stored uri. Fields that do not apply are omitted rather than set to null.

The CLI never follows an external URI. blob get refuses a whole-object reference and points to blob stat, which returns it without opening the target. A persisted ranged external descriptor is rejected by both commands; OmniGraph never widens it to the complete target object.

HTTP reads

Servers expose the same logical selector with GET and HEAD:

GET /graphs/knowledge/blob?entity=node&type=Document&id=manual&property=content&branch=main
Omnigraph-Http-Api: 0.14

Both methods follow the HTTP contract, inclusive checking the response header before consuming bytes. Use snapshot=<commit-id> instead of branch for an immutable historical read.

For managed values:

Treat ETags as opaque validators for the selected graph representation, not as content hashes. An unrelated change to the same entity type can produce a new ETag even when this cell's bytes are unchanged.

For a whole-object external value, GET and HEAD return 302 with the stored URI in Location. The server does not fetch, sign, authorize, or proxy that object and does not claim its size or ETag. A persisted ranged external descriptor fails loudly instead of redirecting to a wider value.

Limits

Blob limits bound the memory one operation needs. An operation over a limit fails before it changes the graph. Over HTTP it returns 413 with a resource_limit detail naming the resource, its limit and the actual value observed. For a write, the CLI reports the same three fields; the omnigraph blob commands report an over-limit range read as Blob read range exceeds the limit. Split the work into smaller operations and retry.

Limit Applies to Reported resource
write_max_bytes decoded bytes Each node or edge type in one load, in every mode, including overwrite decoded blob input bytes for <table>
write_max_bytes decoded bytes One base64: value in an insert or update mutation; a statement's values add up across its Blob properties and an update's matched rows. A load reports its decoded bytes under the for <table> and per operation names decoded blob input bytes per operation
write_max_bytes row bytes per type and across all touched types Incremental writes: append and merge loads, inserts and updates. Counts ordinary columns, Blob descriptors and Arrow bookkeeping; excludes logical Blob payload buffers keyed write bytes for <table>, keyed entity bytes for <table>, retained keyed batch bytes per operation, keyed parsed entity bytes for <table>, keyed parsed entity bytes per operation
write_max_bytes logical Blob payload bytes One incremental write across all types: inline payloads, copied external payloads and Blob values carried by updates or by a Blob put count together materialized blob payload bytes, decoded blob input bytes per operation
write_max_bytes bytes, inclusive The bytes of one Blob put, embedded or the body of an HTTP PUT, checked before the write opens a table Blob write payload bytes
write_max_bytes Blob payload bytes One branch merge that writes rows, across all types, managed and external bytes together materialized blob payload bytes
8,192 external references One write operation or merge external Blob reference cells
32 MiB of retained URI metadata One write operation or merge. Every copy of a URI the operation keeps counts, plus 24 bytes per copy: admission keeps each reference's text twice and each distinct object's normalized URI twice, so distinct URIs reach the limit at about 8 MiB of text external Blob URI metadata bytes
64 KiB One external URI external Blob URI bytes
4 MiB One embedded managed range read Blob read range bytes

Superseded versions of a key written in one operation count as input (decoded blob input bytes per operation) until the last-write-wins fold. An update of a row inserted earlier in the same mutation is such a version: it carries the pending row's Blob, so that payload counts twice, the carried copy as materialized blob payload bytes.

write_max_bytes defaults to 32 MiB and accepts any integer of at least 1; a Blob put stays bounded at 32 MiB by the put limit whatever the setting. Payload and row allowances are independent: a payload exactly at the limit fits when its row and descriptors also fit. Repeated external references each count their payload length even when one fetch serves them. External-only admission can also report materialized external blob payload bytes.

<table> names the type as node:<Type> or edge:<Type>, for example keyed entity bytes for node:Document.

The HTTP load request body is also capped at 32 MiB. That cap counts the encoded request, so one request carries about 24 MiB of decoded base64: data. A Blob PUT carries raw bytes, not base64: text, so its 32 MiB body holds a full 32 MiB value. Every HTTP request other than a load (/load and /load/ndjson) or a Blob PUT is bounded by the default 1 MiB request body limit, so a base64: literal in an HTTP mutation hits that limit first.

Values larger than these limits stay readable. The CLI and the HTTP server read managed values in 4 MiB ranges, so a large value streams without a whole-value buffer, and one HTTP response holds at most two ranges at a time. omnigraph optimize bounds the Blob payload of each compaction batch separately; see Optimize.

Lifecycle

Blob readers stay pinned to the snapshot selected when they were opened. They never switch to newer bytes when a branch advances. Branch deletion and destructive cleanup can remove storage needed by a long-running reader; quiesce those readers first when they must finish reliably.

Historical reads are fail-closed. Type renames remain addressable through the current type name, but a historical property rename, drop/re-add, or branch incarnation may be refused when the selected snapshot does not carry enough identity information to prove it is the same logical Blob property. OmniGraph returns an error rather than guessing from a reused name or physical field position.