Skip to content

Compartments

orqest.compartments stores information that should stay masked: only assistants holding a grant can read it, and it leaves toward the owner only through a confirmed, single-use release. It is separate from memory: compartment contents never enter recall, memory search or summaries.

The keeper is the only way in. It is code, not a model, so no language model is ever the guard. Assistants that hold a grant may be language models; the keeper decides what they can reach.

import asyncio
from pathlib import Path
from tempfile import TemporaryDirectory

from cryptography.fernet import Fernet  # pip install cryptography; any authenticated Cipher works

from orqest.compartments import Keeper

KEYS = {"private-notes": Fernet(Fernet.generate_key())}  # one key per compartment, from your secret store


async def is_owner_turn(turn_id: str) -> bool:
    # In your application: true only for the owner's message whose turn is in progress.
    # A check that accepts any past owner message also accepts a stale turn.
    return turn_id == "message-in-progress"


async def main() -> None:
    with TemporaryDirectory() as tmp:
        path = Path(tmp) / "compartments.db"
        async with Keeper(path, cipher_for=KEYS.__getitem__, is_owner_turn=is_owner_turn) as keeper:
            await keeper.grant("specialist", "private-notes", "read_write", approved_by="owner")
            pointer = await keeper.write(
                "specialist", "private-notes", "door note", {"code": "4417-XK", "where": "side door"}
            )

            # Working access: the specialist uses what it holds for its own work.
            records = await keeper.read("specialist", "private-notes")
            print(records[0].value["code"])

            # Anyone can learn that a compartment holds something, not what.
            print([p.label for p in await keeper.pointers("private-notes")])

            # An outgoing message is checked against what the specialist can read.
            print(await keeper.leak_check("specialist", "The code is 4417-xk, by the way"))  # a hit
            print(await keeper.leak_check("specialist", "It starts with 4417"))  # passes: a fragment

            # Release: prepared in the owner's turn, redeemed after the owner confirms.
            ticket = await keeper.prepare_release(
                "specialist",
                "private-notes",
                pointer.record_id,
                owner_turn_id="message-in-progress",
            )
            record = await keeper.redeem(ticket.ticket_id, confirmed_by="owner-verified-by-the-app")
            print(record.value["code"])  # your own endpoint returns it to the owner, and only that

            # "Forget that": the label and the ciphertext leave the disk.
            await keeper.delete("specialist", pointer.record_id)

            for event in await keeper.audit(compartment="private-notes"):
                print(event.action, event.actor, event.allowed, event.reason)


asyncio.run(main())

What the keeper takes on trust

The keeper enforces grants against the names it is given, and takes every actor name on trust: it cannot tell who is really calling. It does not verify approved_by or confirmed_by, and both reach the audit trail and the logs as given, so pass an identity there and never a credential. The ticket id is a bearer credential: whoever holds it can redeem it. So the application must verify the owner first and only then call redeem, grant and revoke; those three and audit must never be model tools, and actor names must come from the runtime, never from a model's arguments.

Encryption

cipher_for(compartment) returns a Cipher (encrypt(bytes) -> bytes, decrypt(bytes) -> bytes). It is required: there is no plaintext mode, and the battery brings no cryptography dependency of its own. The cipher must be authenticated: decrypt must raise on a wrong key or altered bytes. cryptography.fernet.Fernet fits as it is. Each compartment should have its own key, so that one key opens one compartment. A cipher that fails, or returns its input unchanged, fails the write with RuntimeError.

What is encrypted is the JSON of the value together with the record's id, compartment, label, author and time, so altering any of those columns, or moving a row or ciphertext elsewhere, raises CompartmentCorrupt, even when compartments share a key. Only a read through the cipher (read, redeem, leak_check) can detect this. pointers() never decrypts, so it shows an altered label as stored.

What this does not cover: deleting a whole row, restoring an older copy of the file, and the grants, tickets and audit tables, none of which are encrypted or authenticated. Someone who can write to the file can give themselves a grant; only the values stay out of reach without the key.

Labels are plaintext by design, and so are grants, actors and the audit trail. pointers() shows labels without a grant, and the leak check returns them. A label holding a string value from its own record is refused (dict keys don't count, so {"reading": 120} can be labelled "reading log"), but a label can still give away a secret in other words: write "door note", not "the side door code". The file is created with mode 0600, and SQLite gives its log the same mode.

Names

Assistant and compartment names must fit the identifier grammar: start with a letter or digit, then letters, digits, _ or -, at most 64 characters. Grants match them exactly. A record id must have the shape the keeper issues (32 lowercase hex characters).

A malformed name may be something a caller typed by mistake, a secret included, so it is refused with CompartmentDenied without being repeated: the error says only "malformed name", and the audit event has that reason with the malformed fields NULL. grants_for, pointers and leak_check refuse one too, so an empty result always means a real name with nothing to show. A well-formed compartment or record id that does not exist gets the same treatment in the audit and logs. An actor name is recorded as given when it is well formed, since recording who tried is the point of the audit.

Grants and working access

A grant names one assistant, one compartment and an access level (read, write or read_write), and records who approved it. A second grant replaces the first. revoke applies to the very next call.

read(assistant, compartment, record_id=None, limit=20) returns records newest first. It is for an assistant's own work, including on a schedule, with no owner present.

write takes a dict of strict JSON with string keys. Anything that would come back changed, such as an integer key or a tuple, is refused rather than coerced.

Record leaves value out of repr and str, so logging a record does not reveal it. model_dump() does return it.

Deleting

delete(actor, record_id) needs a write grant on the record's compartment. It spends any open ticket on the record, overwrites the freed space (secure_delete) and truncates the write-ahead log, so the label and ciphertext are gone from the file and the log. The audit trail keeps the record's id, which says nothing about its content.

Another connection reading the file can keep the log from being truncated. delete still returns, because the row is gone, but its bytes stay in the log until the next truncating checkpoint: a later delete (one that finds no such record included), the next write, the next keeper opened on the file, or the last connection closing. An ordinary checkpoint does not remove them. A warning names the record, and a second audit event says the purge is pending; no event marks it finished. Backups and earlier copies of the file are out of the keeper's reach.

Release

Anything leaving a compartment toward the owner takes two steps, so that a manipulated assistant can at worst show the owner their own data:

  1. prepare_release(requester, compartment, record_id, owner_turn_id=...) needs a read grant, and is_owner_turn(owner_turn_id) must return exactly True. An exception from that check counts as a refusal. No value is touched. The caller must take the turn id from the message that triggered the work, not from a model.
  2. redeem(ticket_id, confirmed_by=...) is called by the application's own verified channel after the owner confirms there, and its result goes to the screen. Never expose it as a tool, and never pass the result to a model. A model may see a ticket id; what stops it redeeming one is that the application offers it no way to, not the keeper.

A redeemed value goes back in one HTTP response, and nowhere else

The application's own authenticated endpoint returns the value in its response to the owner. It must never go into a view, an artifact, state or a message, because all of them persist. A code handler that shows it with ctx.show_artifact or ctx.publish_view leaves it in the views row and, once dismissed, in artifacts.snapshot, where render_view serves it with no ticket and no passkey. Orqest has no release event, so nothing else carries a value to the screen.

The keeper does not check that a turn is fresh. It stores owner_turn_id and asks your callback, so the callback must refuse anything but the turn in progress. A callback that accepts any message ever sent by the owner lets an old turn id prepare a release.

A ticket works once, for ticket_ttl_s seconds (default 300). redeem refuses with TicketInvalid if the ticket is unknown, spent or expired, if confirmed_by is blank, or if the requester's grant is gone. revoke, a grant without read access and delete spend open tickets, so granting access again does not revive them. Otherwise only a release spends a ticket: a refusal for a blank confirmation or a corrupt record leaves it usable until it expires, and a crash right after the commit loses that one release, so prepare another. The database holds only a hash of each ticket id, so tickets stay single-use and time-limited across restarts, and a copy of the file holds no ticket that can be redeemed.

Leak check

leak_check(assistant, text, *, match_keys=True) returns the labels whose stored strings appear in text, looking only in compartments the assistant can read. It checks string values and dict keys, nested ones included; match_keys=False leaves the keys out. Both sides are normalized first (NFKC, casefold, runs of whitespace collapsed), so case, line wrapping, spacing and Unicode form do not hide a match. Strings under six characters after normalizing are skipped. A hit is audited against its record, with allowed=False.

It compares whole stored strings, and that has two costs:

  • A fragment of a longer value passes. If you store {"text": "the code is 4417-XK"}, the text "it is 4417-XK" gets through. So does a paraphrase, a number, an encoding or a translation, and this battery does not claim otherwise.
  • A short common value causes false refusals. A stored "active" refuses "the plan is inactive", and a key such as "reading" refuses any text with that word in it.

So structured short fields work better than prose: {"code": "4417-XK"} catches the code wherever it appears, as long as the fields are not ordinary words. Short key names (under six characters) stay out of the way.

A corrupt record stalls the check. leak_check raises CompartmentCorrupt rather than letting a text pass unchecked, so one bad record blocks everything that depends on the check for that assistant until the record is dealt with. Each call also decrypts every record the assistant can read while holding the keeper's lock, so other calls on that keeper wait for it.

In the assistant runtime

AssistantRuntime(keeper=..., owner_senders=...) connects a keeper the application has opened to its LLM assistants. The keeper's is_owner_turn should be AssistantRuntime.is_owner_turn, which needs a runtime that needs the keeper, so bind it late:

import asyncio
from pathlib import Path
from tempfile import TemporaryDirectory

from cryptography.fernet import Fernet  # any authenticated Cipher works
from pydantic_ai.messages import ModelResponse, ToolCallPart
from pydantic_ai.models.function import FunctionModel

from orqest.assistants import AssistantRuntime, AssistantSpec
from orqest.autonomy.factory import AgentFactory
from orqest.autonomy.spec import AgentSpec
from orqest.compartments import Keeper

KEYS = {"private-notes": Fernet(Fernet.generate_key())}
OWNERS = frozenset({"owner"})  # senders whose messages count as the owner's

SPEC = AssistantSpec(
    agent=AgentSpec(
        name="specialist",
        system_prompt="You keep private notes.",
        output_schema={"type": "object", "properties": {"summary": {"type": "string"}}},
    ),
    state_schema={"type": "object"},
)


def model_turn(messages, info):
    """Stands in for a model: reads the compartment, then finishes."""
    if len(messages) == 1:
        call = ToolCallPart(tool_name="compartment_read", args={"compartment": "private-notes"})
        return ModelResponse(parts=[call])
    print(messages[-1].parts[0].content)  # what the tool returned to the model
    output = ToolCallPart(tool_name=info.output_tools[0].name, args={"summary": "reviewed"})
    return ModelResponse(parts=[output])


async def main() -> None:
    runtimes: list[AssistantRuntime] = []

    async def is_owner_turn(turn_id: str) -> bool:
        return await runtimes[0].is_owner_turn(turn_id)

    with TemporaryDirectory() as tmp:
        async with Keeper(
            Path(tmp) / "compartments.db", cipher_for=KEYS.__getitem__, is_owner_turn=is_owner_turn
        ) as keeper:
            runtime = AssistantRuntime(
                Path(tmp) / "assistants.db",
                factory=AgentFactory(),
                model=FunctionModel(model_turn),
                keeper=keeper,
                owner_senders=OWNERS,
            )
            runtimes.append(runtime)
            async with runtime:
                await runtime.create("specialist", SPEC)
                await keeper.grant("specialist", "private-notes", "read_write", approved_by="owner")
                await keeper.write("specialist", "private-notes", "door note", {"code": "4417-XK"})
                await runtime.send("owner", "specialist", "review", {})
                await runtime.tick()


asyncio.run(main())

Tools follow grants. Each wake reads keeper.grants_for(address), and an assistant gets only what those grants allow. Without a grant it has no compartment tool at all, and its prompt and sends are as they were. A grant also adds a ## Your compartments section to its wake prompt.

Tool Needs Returns
compartment_read(compartment, record_id=None, limit=20) read grant JSON: record_id, label, created_at, created_by, value, newest first
compartment_prepare_release(compartment, record_id) read grant, a message from the owner in this wake only the ticket id
compartment_write(compartment, label, value) write grant stored: <record id>
compartment_delete(record_id) write grant deleted: <record id>

Grants are read once, as the wake begins. A revoke during a run takes the tool's power away at once, since the keeper refuses and audits the next call, and the tool disappears at the next wake. If the grants can't be read, the wake fails and is retried.

The runtime supplies every actor name (record.address), so no tool takes one. grant, revoke, redeem and audit are never tools. Refusals, malformed names, unknown records and corrupt records come back as ERROR: strings the model can read. A database failure fails the run, which is retried.

These tools act on the keeper at once, unlike the runtime's other tools, whose effects wait for the wake's commit. The two files share no transaction, so a wake that fails afterwards keeps what they did: a write stays and its retry can write a second record, a delete stands, and a prepared ticket stays open until its TTL.

A wrongly typed tool argument goes back to the model through pydantic-ai's retries. If they run out the run fails, without the argument appearing in the message's last_error or in events.

Release. The model never gives a turn id: the runtime uses the most recent message in the wake's batch whose sender is in owner_senders, and refuses when there is none, as on a scheduled wake. That refusal shows in the audit trail with the reason no owner message, through Keeper.refuse_release(requester, compartment, record_id), which records the attempt and does nothing else. Malformed or unknown names are blanked, as in any refusal. AssistantRuntime.is_owner_turn(message_id, owner_senders=None) is true only for an existing message from one of those senders (by default the constructor's) that is being processed right now, so a message already handled, or one still waiting, doesn't count. rt.send's sender is taken on trust, so only your verified channel should send as an owner sender. create refuses an owner sender's address with ValueError, since that assistant's messages would count as the owner's. register_code does not, because a handler is your own code. A message being retried counts while it is claimed again, so a retry after a provider failure can still release.

What is checked. For an assistant with a read grant, these tools run leak_check first, and a hit means nothing changes:

  • reply_to_message and contract tools, on the payload and a reply's message type, against stored values and record keys, because a send crosses to another assistant;
  • compartment_write, on the label, the same way, because pointers() shows a label without any grant;
  • update_state, on the patch's keys and values, against stored values only. A key set to null is skipped, because a removal must always work, and so is any string or key the state already holds exactly, because that says nothing new and refusing it would block every later change to it;
  • publish_view, against stored values only, on the view's name, version and the values typed into its nodes: ids, data values and the hint group. Block types, hint levels, dict keys and binding paths aren't text: a binding path names a state key, which a model can't rephrase and which was checked when it was written. There is no exception for what state holds, because a value in a view reaches the owner's screen with no fresh confirmation, which is what a release exists to require.
  • show_artifact, the same way as publish_view for its tree (including the view name art-<artifact_id>), plus the title, intent, accessibility label, source refs and allowed actions. Its search_text is built from the title and intent, so it is covered.

The check sees every decoded string and str() of every number and bool, in document order, joined into one text. This is not the JSON, which escapes non-ASCII, quotes and newlines, so that real values would miss. On a hit the tool returns ERROR: this would reveal stored private information (<labels>); say what changed, not the private detail. This was not accepted: do not retry it unchanged. Leave the value out or refer to the record by its label; if the match may be an ordinary word, rephrase. If the check can't run because a record can't be decrypted, the tool refuses too, and any other keeper failure fails the run, so nothing is kept. remember_run applies the send's check to the run's output and message types, and skips storing them with a warning that names the assistant only.

What a record costs its assistant. Every key and string value of six characters or more in a record the assistant can read refuses any send holding that word, and remember_run skips memory for any output or message type holding it. A record {"account": "active"} therefore stops every send that says "account" or "inactive". Short keys are better, and values that aren't ordinary words.

What the state and view rules leave. A stored key such as {"codename": true} isn't matched in state or views, though it is in sends and labels. A value used as a dict key inside a view's data isn't matched. A string or key state already holds can be repeated or moved within state. A binding path isn't checked, and any word-character text can be one, not just an old key, so a stored value made of word characters can sit in a path: it reaches the views row, the unresolved-binding warning and ResolvedView.problems, but not the tree, which shows a placeholder. And a short common stored value can still refuse new text: a stored active refuses a new inactive, and a new state key equal to a stored value is refused. The model can rephrase that, unless a state_schema requires that exact value or key, and then it can't be newly set until the record changes.

What this guarantees is the literal stored value, as a string of six characters or more. Numbers and shorter strings are not matched, so a stored {"account": 4417889012} passes in a sentence: store it as a string to have it matched. Everything in Leak check still applies: a paraphrase, a fragment of a longer value, an encoding or a translation gets through. In addition, the check does not cover:

  • a write-only assistant, one revoked after reading, or a value read and then deleted mid-run. leak_check looks only at what an assistant can read at that moment and what is stored then, so the first is never checked, and the other two are no longer checked for what was read;
  • anything before storage. The guarantee starts at storage. A single response can send a value and store it, and the send isn't caught. The same goes for a send queued earlier in a run when compartment_write stores that value later in it;
  • wake reasons. schedule_wake is not checked, because a reason returns only to the same assistant. The wake prompt tells the model to keep values out of them, but that is an instruction, not a guard. A reason sits in plaintext in the runtime's file, and compartment_delete doesn't remove a copy a model already made there;
  • a record's value. A value read from compartment A and written to B isn't checked, so it becomes readable by whoever holds B;
  • state and views from before the check. Only a patch or view being written is checked, so what state already holds stays, including a value that was stored in a compartment afterwards. A value already in state can be bound into a view, or shown by the default_views fallback, and is copied into a frozen artifact's snapshot, all without a passkey;
  • code assistants. They get no tools and no check: a handler is your own code, and it calls the keeper directly if it needs to.

Failures can carry text. A provider's error text is stored as given in the message's last_error and in assistant.message.dead. It holds a value only if a provider echoes one back.

Messages and retirement. The runtime's messages table keeps inbound text in plaintext, so what a sender typed stays there after the same value is stored or deleted here. retire neither revokes an assistant's grants nor spends its open tickets: do that yourself if it should lose access.

Instrumentation sees everything. Logfire, Agent.instrument_all and the like capture tool arguments and results, values included, so don't enable them on a keeper's assistants.

Failures and the audit trail

Every failure raises. Unlike memory, which is best-effort, nothing here is swallowed:

Error When
CompartmentDenied (a PermissionError) No grant of the needed kind, not an owner turn, a malformed name, or a label holding a string value from its record
TicketInvalid (a PermissionError) A ticket that cannot be redeemed
CompartmentCorrupt (a RuntimeError) A record cannot be decrypted intact (wrong key, altered or truncated data), or a stored timestamp cannot be read. Names the record. One corrupt record fails the whole read, and leak_check raises too
KeyError The record is not there, or not in this compartment
ValueError / TypeError A blank label or approver, a bad limit, an unknown access level, a value that is not a strict-JSON dict with string keys
RuntimeError A cipher that fails or does not encrypt, a closed keeper, or a file from a newer schema version, which is left as it is
sqlite3.OperationalError The database itself failed, for example another process held it past the busy timeout (5 s). No value is returned without its audit row

audit(compartment=None, limit=100) returns attempts newest first: who, what, when, whether it was allowed, and why. Each refusal is committed in the transaction that decides it, before the error reaches the caller. An event with no known compartment, such as a malformed name or an unknown ticket, shows only in the unfiltered list. A successful pointers() or grants_for() is not audited: neither has an actor to name or touches a value. Spent tickets and audit rows are never pruned, so both tables grow for the life of the file.

Logs name records and actors. Values, labels and ticket ids stay out of logs and of error text, and a cipher failure logs only the exception's type.

Some of this is yours to set up, because a tool that captures variables sees your own code's too:

  • Set loguru's diagnose=False on your sinks in production. It is on by default and prints the variables on each traceback line, and Orqest can't control your sinks. With a keeper, Orqest's own crash logs in the assistant runtime carry only frames (file, line, function) and exception type names for the whole chain, with message text only for sqlite3.Error and OSError, because a message can hold a value: pydantic's input_value=..., or a provider's echo. Without a keeper they attach the exception as any loguru call does (logger.opt(exception=exc)), because there are no stored values to protect. With a keeper, an error leaving tick() or run_forever() has lost its chain and the frames that name a run's arguments, state and batch, and the log has its location, but anything your own handler holds is still yours to keep out of a log. Without a keeper, an error leaves with its frames and chain, and no extra log.
  • Turn off local-variable capture in Sentry (include_local_variables=False) and do not use traceback's capture_locals=True, since either can record a value passed to write or a record you are holding.
  • Do not log a validation error's errors() or json(). The models keep input out of the error's text only; those two still contain it.

What this does not protect against

Compartments protect against accidental disclosure and manipulated assistants. They do not protect against:

  • an assistant with a grant paraphrasing something private, or passing on a fragment of it;
  • the provider of a model that reads a value: whatever a granted model reads goes to that provider;
  • anything said or typed before it was stored, which already sits in transcripts and provider logs;
  • code in your own process that makes up an actor name or redeems a ticket: the keeper takes both on trust;
  • a compromised server with its keys exposed.