Skip to content

Generative UI

Orqest lets agents emit typed component specifications that a frontend resolves dynamically. Instead of returning text for the UI to render, the agent emits a UIComponentSpec[T] (chart, table, form, plan, takeover dialog, Vega visualization, Mermaid diagram, sandboxed HTML, etc.). The frontend hot-loads a renderer keyed on component_type. Subsequent state changes flow as UIDeltaEvent records on the same SSE stream.

What problem does this solve?

Agents that return prose and let the UI render it leave a lot of capability on the table. If the agent is analyzing data, you want a chart — not a paragraph describing one. If it needs your approval, you want a dialog — not "should I proceed? (yes/no)" in chat. The agent already knows the shape of the answer; let it pick the surface. Generative UI flips the protocol: the agent describes what to render, the frontend resolves how. Same backend produces a Plan, a chart, and a PDF report based on what the work needs.

UIComponentSpec

Generic Pydantic with a component_type Literal discriminator. Each first-party component is a subclass that fixes component_type and types the data: T payload.

Field Type Description
component_id str Stable identifier (the frontend uses this to route deltas to the right instance)
component_type Literal["plan" \| "chart" \| ...] Discriminator the frontend resolves to a renderer
data T (typed payload) Component-specific payload — strongly typed per subclass
metadata dict[str, Any] Free-form (used for layout hints, accessibility, etc.)
created_at datetime UTC timestamp

UIDeltaEvent — partial updates

Once a component is initialized, you patch it with delta events. Avoids re-shipping the full payload on every change.

Field Type Description
op Literal["replace", "merge", "append", "remove"] The mutation kind
path str Dot-path into the data payload (e.g., "series.0.points")
value Any New value (or to-merge object, or to-append item)

Ops: - replace — overwrite the value at path - merge — shallow-merge value (object) into the dict at path - append — append value to the list at path - remove — delete the key/element at path

Three layers of first-party components

Components ship in three layers — pick the lowest layer that does the job.

Layer 1 — Compositional primitives

Hand-typed components for common UI patterns:

  • PlanComponent — task list (carries PlanTask instances; ties into ExecutionPlan)
  • ChartComponent — line / bar / scatter / pie / heatmap with typed ChartSeries
  • TableComponent — typed TableColumn + rows
  • FormComponent — typed FormField + submit event handle
  • TakeoverDialogComponent — confirm / input / choice (the agent asks the user a question)
  • LayoutComponent, TextComponent, MarkdownComponent, ImageComponent, BadgeComponent, ButtonComponent, InputComponent
  • StatTileComponent — one headline number with label, unit and optional trend
  • ChecklistComponent — items with a checked flag
  • TimelineComponent — dated events in order
  • MapComponent — markers, an optional center and a zoom level
  • MediaComponent — image, video or audio by URL (https:// only)
  • PathwayComponent — a titled route of steps (todo / current / done) with overall progress in [0, 1] and an optional action name
  • LessonStepComponent — one guided step: objective, at most three cues, sub-steps with status, an optional safety note, optional https:// media_url and a voice_hint for renderers that speak

Layer 2 — Declarative grammars

When you need richer visualization than Layer 1 covers — pass a spec the frontend already knows how to render:

  • VegaChartComponent — Vega-Lite spec passed through
  • MermaidComponent — Mermaid diagram source
  • LatexComponent — KaTeX/MathJax-renderable LaTeX
  • JsonViewerComponent — collapsible JSON tree

Layer 3 — Sandboxed escape hatch

For one-offs the framework can't anticipate:

  • SandboxedHTMLComponent — HTML/CSS/JS in an iframe sandbox

UIEmitter — convenience facade

The minimal API. Wraps an EventBus so init/delta/remove are one line each.

import asyncio
from orqest.observability import EventBus
from orqest.ui import (
    ChartComponent,
    ChartComponentData,
    ChartSeries,
    UIEmitter,
)


async def main():
    bus = EventBus()
    emitter = UIEmitter(bus)

    # Initialize a chart
    chart = ChartComponent(
        component_id="latency-chart",
        data=ChartComponentData(
            kind="line",
            title="Request latency",
            x_label="time",
            y_label="ms",
            series=[
                ChartSeries(
                    name="p50",
                    points=[{"x": 0, "y": 12}, {"x": 1, "y": 14}],
                ),
            ],
        ),
    )
    emitter.init(chart)

    # Append a new point
    emitter.delta(
        component_id="latency-chart",
        op="append",
        path="series.0.points",
        value={"x": 2, "y": 18},
    )

    # Remove the chart entirely
    emitter.remove(component_id="latency-chart")


asyncio.run(main())

View specs and bindings

A ViewSpec is a serializable screen: a named, versioned tree of ViewNodes whose leaves name a registered component_type. An assistant (or any code) publishes the spec once; the data flows in separately through bindings, so the screen updates without the spec being rewritten.

  • Nodes. id (unique within the view, same grammar as addresses), component_type, literal data, bindings, hints and, for layout nodes only, children. The root must be a layout. A layout's data must not contain children, and a layout must not bind children. ViewNode, Hints and ViewSpec reject unknown keys, so a typo such as props for data is an error rather than silently dropped. A view holds at most 200 nodes (MAX_VIEW_NODES) nested at most 16 deep (MAX_VIEW_DEPTH, root = 1).
  • Bindings. bindings={"value": "state.done"} fills the top-level data field value from the context passed to resolve_view. The key must be a field of the block's data model (a misspelt key is a structural problem). A path is dot-separated: dict keys, and integers to index lists (state.plan.today.drills.0.label). resolve_path raises BindingError on a miss.
  • Hints. Hints(importance="primary" | "secondary" | "glanceable", size="compact" | "wide", group=None). They are copied to each component's metadata["hints"] (the owning view's name goes in metadata["view"]). Assistants do not set colours, fonts or spacing; the frontend decides how to present hints.
  • Placeholders. resolve_view never raises. A node whose type is unknown, whose binding does not resolve, or whose data is invalid or not strict JSON (a bound NaN or infinity) renders as a text component whose content is "Unavailable", and the reason is appended to ResolvedView.problems. Resolved trees and blocks are therefore always strict JSON. A degraded non-root layout becomes a placeholder too (its subtree is not rendered). A degraded root stays a layout holding one placeholder child with the reserved id root_unavailable, so ResolvedView.tree is always a layout.
  • Validation. validate_view reports only structural problems (unknown component_type, a binding key that is not a data field, invalid literal data, invalid data after bindings that did resolve). A binding that does not resolve is not a problem there, because it may resolve later; it still shows up in ResolvedView.problems, renders as a placeholder, and is listed by unresolved_bindings(spec, registry, context) (useful as a publish-time warning). Non-strict-JSON data is not structural either: it comes from the context, not the spec.
  • Result. ResolvedView carries tree (nested, ready to render), blocks (a flat map of node id to component_type, data without children, and metadata) and problems. fallback_view(name) builds a minimal view that shows state in a JSON viewer.

Diffing two states

diff_view(old, new) returns the UIDeltaEvents that turn one resolution into the next:

  • one replace (path=<key>) per changed or added top-level data key, and one remove per dropped key, ordered by block id then key. Values inside a field are not diffed: a changed list is sent whole.
  • None when a full re-init is required: the view name or version changed, the set of block ids changed, or any block changed component_type (for example a placeholder that became a real block). Send the whole tree again in that case.
  • [] when nothing changed. Metadata (hints) is not diffed.

apply_deltas(blocks, deltas) is the reference applier for exactly these ops; it returns a new block map. It raises ValueError for an unknown op, an unknown component_id or a component_type that differs from the block's. diff_view assumes both resolutions come from the same spec: it ignores child order and hints, so re-init whenever the spec changes.

from orqest.ui import (
    ViewNode,
    ViewSpec,
    apply_deltas,
    default_registry,
    diff_view,
    resolve_view,
)

registry = default_registry()
view = ViewSpec(
    name="today",
    root=ViewNode(
        id="root",
        component_type="layout",
        children=[
            ViewNode(
                id="done",
                component_type="stat_tile",
                data={"label": "Drills done"},
                bindings={"value": "state.done"},
            ),
            ViewNode(
                id="missing",
                component_type="stat_tile",
                data={"label": "Streak"},
                bindings={"value": "state.streak"},
            ),
        ],
    ),
)

before = resolve_view(view, registry, {"state": {"done": 2}})
after = resolve_view(view, registry, {"state": {"done": 3}})

assert before.blocks["done"]["data"]["value"] == 2
assert before.blocks["missing"]["data"]["content"] == "Unavailable"
assert before.problems == ["binding 'state.streak' did not resolve at node 'missing'"]

deltas = diff_view(before, after)
assert [(d.component_id, d.op, d.path, d.value) for d in deltas] == [
    ("done", "replace", "value", 3)
]
assert apply_deltas(before.blocks, deltas) == after.blocks

Views are events too: ui.view.init carries the full tree, ui.view.delta carries the deltas, and ui.view.escape_hatch flags a sandboxed_html node. When the views belong to a long-lived assistant, the runtime emits them for you; see Long-Lived Assistants.

Renderer contract

What a frontend can rely on when it renders views from the event stream (as emitted by orqest.assistants):

  • Event shapes. ui.view.init: {"address", "view", "version", "seq", "tree"}, where tree is ResolvedView.tree (a layout component whose data.children nests the rest). ui.view.delta: {"address", "view", "version", "seq", "deltas"}, where deltas is a list of UIDeltaEvent dicts with op "replace" (path = a top-level data key, value = its new value) or "remove" (path = the key to drop). ui.view.remove: {"address", "view", "seq"}, sent when a view no longer exists (for example after a rollback): drop that view and its last seq. ui.view.escape_hatch: {"address", "view", "node_id"}.
  • Ordering with seq. seq counts events per (address, view), grows by one per event and never goes back within a process. Seq restarts with the server process — reset your last seq on reconnect. An init starts a new base. Keep the last seq applied per view: drop an event whose seq is not larger (you already have it), let an init replace the tree, and apply a delta only when its seq is exactly last + 1. Any other delta (a gap) means you missed something: refetch the snapshot.
  • Reconnect: reconcile the view set. On every (re)connect, call AssistantRuntime.list_views(address) and drop any view you hold that it does not list, then snapshot the listed ones as below. ui.view.remove is only sent for views the current server process has emitted, so a view that disappeared before a restart, or while you were disconnected, sends no remove event.
  • Retire. On assistant.lifecycle with data["action"] == "retire" (the address is the event's agent_name), drop all of that assistant's views: a retired assistant gets no ui.view.remove events.
  • Subscribe, then snapshot. Subscribe to the stream first, then fetch the snapshot (AssistantRuntime.render_view(address, name) returns a ViewSnapshot with resolved and seq), then process buffered and new events with the rule above. A snapshot with seq 0 was never emitted; the next event for that view is an init. A snapshot may predate an event already delivered to you — keep that event and re-apply it by the seq rule.
  • Applying deltas. A delta targets the tree node whose component_id matches and replaces or removes one key of its data. It never touches a layout's children: structure changes always arrive as an init. A delta whose component_id is unknown, or whose component_type differs from that node's, means you are out of sync: refetch the snapshot (apply_deltas raising is the reference form of this signal).
  • Layout. Honour a layout's direction and grid_columns as intent. Ignore gap and align: the renderer owns spacing. Hints (metadata["hints"]) are presentation intent too.
  • version. Informational (the spec's version); ordering comes from seq.
  • JSON. Trees and deltas are strict JSON (no NaN or infinity), so any standard parser reads them.

Artifacts: show, update, pin, dismiss, expire

An artifact is a view with a lifecycle, for a screen an assistant generates on demand. While it is live it is the ordinary view art-<artifact_id> (so render_view, deltas and the renderer contract keep working) plus a metadata row, committed in the same transaction as the wake. A failed wake shows nothing.

from orqest.assistants import AssistantRuntime
from orqest.ui.view import ViewNode

class Briefer:
    async def handle(self, message, ctx):
        ctx.show_artifact(
            "morning-brief",
            intent="summarise today",
            title="Morning brief",
            a11y_label="Today's summary",
            root=ViewNode(id="root", component_type="layout"),
            allowed_actions=["done"],
            expires_in_s=3600,
        )

# async with AssistantRuntime(path) as rt:
#     await rt.register_code("briefer", Briefer())
#     await rt.send("app", "briefer", "go", {})
#     await rt.tick()
#     await rt.list_artifacts(status={"active"})
#     await rt.pin_artifact("briefer", "morning-brief", True)   # never expires while pinned
#     await rt.artifact_action("briefer", "morning-brief", "done", {}, sender="owner-app")
#     await rt.delete_artifact("briefer", "morning-brief")      # forget it for real
  • Ids belong to their address. The key is (address, artifact_id), as for views, so two assistants can use the same id and one can never squat on or probe another's. pin_artifact, artifact_action and delete_artifact take the address first. Ids are identifiers of at most 60 characters. Events carry the address in agent_name and {artifact_id, version, status} in data.
  • Statuses. active, pinned, dismissed, expired. Showing the same id again updates it (version +1); showing a dismissed or expired one makes it live and active again; a pinned one stays pinned. Two shows of an id in one wake keep the last and bump once. Dismissing after showing a new id in the same wake shows nothing. show_artifact and dismiss_artifact are sequential tools, so a model response holding either runs its calls in order.
  • Frozen trees. When an artifact is dismissed or expires, or its assistant retires, its tree is resolved once against the state of that moment and kept as a snapshot, and its view is removed (ui.view.remove). A frozen artifact is not emitted live, is not in list_views or in update_state's view problems, and adds nothing to later commits. render_view(address, "art-<id>") returns the snapshot (seq 0), also for a retired address. A show that reactivates it clears the snapshot and goes live again. A name with the art- prefix never falls back to a view of the raw state: an artifact that is unknown, deleted or undecodable raises UnknownArtifact, and one frozen with no usable view keeps an empty layout.
  • Deadlines. expires_in_s counts from the commit (on the runtime's clock), not from the start of the wake. Showing again without expires_in_s keeps the deadline a live artifact has; a dismissed or expired one starts without one. To clear a deadline, pin it. Expiry runs in tick() and run_forever; each artifact emits expire once, also across restarts (a crash between the commit and the emit loses the event rather than repeating it).
  • Pin. The owner can pin an artifact in any status, a dismissed or expired one keeping its snapshot, and no assistant needs to be alive. A pinned artifact never expires. Unpinning one that isn't pinned does nothing. Unpinning a pinned one returns a frozen one to dismissed and a live one to active with the deadline it had, so one whose time has passed expires at the next poll. An assistant cannot undo a pin: dismiss_artifact on a pinned artifact is refused (the commit ignores it too). A show that updates a pinned live artifact is allowed and it stays pinned. A pinned artifact that is also frozen is kept as the owner saw it: show_artifact on it is refused by the tool and ignored (with a log line) at commit.
  • Retire. Retiring an assistant dismisses its active artifacts (with ui.artifact.dismiss) and freezes all of them; a pinned one stays pinned, frozen. They stay searchable and renderable.
  • Rollback. rollback() and redesign() leave artifacts alone: version snapshots skip art-* views, and a restore does not touch them.
  • Delete. delete_artifact(address, artifact_id) removes the row, search text, snapshot and view for real (the runtime keeps secure_delete on, so freed pages are zeroed, and the write-ahead log is truncated) and emits ui.artifact.delete, without emitting a retired assistant's screens again. If another connection holds the log, the truncation waits only 0.1 s: the call still succeeds because the row is gone, a warning names the artifact, and the truncation is retried on every poll until it empties the log, and once at the next open (the text stays in the log until then). It is for the owner, not a model tool, so "forget that" can remove a fact from a screen that was shown.
  • Events. ui.artifact.show, update, dismiss, expire, pin and delete, emitted after the commit. Emission is best-effort, and one batch of events is bounded as a whole by one run_timeout_s (events not sent in time are dropped and counted in a warning); a subscriber that hangs or raises CancelledError costs one logged warning, not the loop.
  • Reserved names. publish_view refuses names starting with art-. An id that would land on an existing non-artifact view named art-<id> is refused (early by the LLM tool, authoritatively at commit).
  • Search. search_artifacts("review") matches every word in the title and intent, for any status, after NFKC and case folding (één finds Één). At most 20 words and 1000 characters of the query count, limit is clamped to 1..100, and a query that isn't valid text finds nothing. %, _ and \ match literally.
  • Actions. artifact_action refuses an action not in allowed_actions and any action on a frozen artifact (dismissed, expired, or retired with it, even pinned), since a stale tap must not reach an assistant. value must be a dict (TypeError). Otherwise it sends the assistant a ui.action message {artifact_id, action, value}; an address with registered contracts must list ui.action (ContractViolation otherwise). The sender is trusted as given: pass a verified owner sender (one in owner_senders) for a voice or touch action to count as the owner's turn.
  • Model prompt. An assistant with live artifacts sees a ## Your artifacts section (up to 20, id and title, newest first) with a note on show_artifact and dismiss_artifact; kept (pinned and frozen) screens are listed as read-only. update_state reports view problems for the artifacts live when the wake was claimed and those shown in it, so one that expires during the wake may still be reported.
  • Private information. With a read grant, show_artifact is leak-checked like publish_view; see compartments.

Event-type conventions

The emitter maps to SSE event types via dedicated helpers. The frontend resolves a renderer by parsing the event type.

Event type When Helper
ui.<component_type>.init Component first appears ui_init_event_type(component_type)
ui.<component_type>.delta State patch ui_delta_event_type(component_type)
ui.<component_type>.remove Component dismissed ui_remove_event_type(component_type)

For ChartComponent: ui.chart.init, ui.chart.delta, ui.chart.remove.

ComponentRegistry — per-Workbench, no module singleton

Each Workbench carries its own registry. This avoids the rigidity of module-level state and lets two consumers in the same process register different component sets.

from orqest.ui import ComponentRegistry, default_registry, UIComponentSpec
from pydantic import BaseModel
from typing import Literal


# Define a custom component for your domain
class MoleculeViewerData(BaseModel):
    smiles: str
    color_by: Literal["element", "charge"] = "element"


class MoleculeViewerComponent(UIComponentSpec[MoleculeViewerData]):
    component_type: Literal["molecule_viewer"] = "molecule_viewer"
    data: MoleculeViewerData


# Build a registry that includes both first-party and your custom one
registry = default_registry()  # PlanComponent, ChartComponent, etc.
registry.register(MoleculeViewerComponent)

Workbench(ui_registry=registry, auto_register_first_party_ui=True) injects it.

ExecutionPlan dual emission

ExecutionPlan opt-in flag-gates dual emission of legacy plan.* events alongside typed ui.plan.* events. Default off — existing emission-count assertions stay byte-identical.

from orqest.plan import ExecutionPlan

plan = ExecutionPlan(...)
plan.enable_ui_events(component_id="plan")  # opt-in dual emission

SSE sidecar — flushing to the frontend

sse_sidecar(bus, ...) is an async iterator yielding SSE-formatted strings. Plug into FastAPI / Starlette / aiohttp / etc. The frontend subscribes and resolves renderers by event type.

from fastapi import FastAPI
from sse_starlette import EventSourceResponse
from orqest.observability import sse_sidecar

app = FastAPI()


@app.get("/sessions/{session_id}/events")
async def events(session_id: str):
    bus = get_bus_for(session_id)
    return EventSourceResponse(sse_sidecar(bus, replay=(), heartbeat_s=15.0))

Custom component example — end to end

import asyncio
from typing import Literal
from pydantic import BaseModel
from orqest.observability import EventBus
from orqest.ui import (
    ComponentRegistry,
    UIComponentSpec,
    UIEmitter,
    default_registry,
)


class RiskHeatmapData(BaseModel):
    rows: list[str]
    cols: list[str]
    cells: list[list[float]]  # 0 (low) → 1 (high)


class RiskHeatmapComponent(UIComponentSpec[RiskHeatmapData]):
    component_type: Literal["risk_heatmap"] = "risk_heatmap"
    data: RiskHeatmapData


async def main():
    registry = default_registry()
    registry.register(RiskHeatmapComponent)

    bus = EventBus()
    emitter = UIEmitter(bus)

    spec = RiskHeatmapComponent(
        component_id="q3-risk",
        data=RiskHeatmapData(
            rows=["EU", "US", "APAC"],
            cols=["liquidity", "credit", "operational"],
            cells=[[0.2, 0.4, 0.1], [0.3, 0.5, 0.2], [0.7, 0.6, 0.3]],
        ),
    )
    emitter.init(spec)
    # Emits an "ui.risk_heatmap.init" event on the bus.
    # The frontend (which previously registered a renderer for "risk_heatmap")
    # resolves and mounts it.


asyncio.run(main())

Best practices

  • Start at the highest layer that fits. PlanComponent over a custom component over SandboxedHTMLComponent. The frontend already knows how to render Layer 1 — you pay design surface for every layer below.
  • component_id is stable. It's how deltas route. Using random UUIDs per-emit defeats the delta contract.
  • Deltas, not full re-emits. A 50KB chart that gains one point should emit a delta(append, "series.0.points", {x,y}), not a fresh init.
  • Keep frontend renderers thin. The component is a typed contract; the frontend just maps fields to DOM. Behavior belongs in the agent, not the renderer.
  • Don't reach into Layer 3 (SandboxedHTMLComponent) by default. It's for one-offs that don't deserve a typed component. Frequent use is a smell — promote it to a typed component instead.

Pitfalls

  • Don't share ComponentRegistry across consumers that need different schemas. The per-Workbench design is intentional.
  • Don't emit init for the same component_id twice in a session — that's a re-mount; the frontend may flicker. Use a delta(replace, "", new_data) instead, or remove then init.
  • Don't trust the frontend to validate. The registry's validate_payload runs at the backend boundary; the frontend just renders.
  • UIEmitter never raises on bus failure — it logs at DEBUG. If you need delivery guarantees, add a separate observability layer.
  • Polymath uses every layer of generative UI because it's the substrate's flagship demo. Most consumer apps pick 2-3 components and stay there.

What's happening under the hood

  1. Agent computes a typed component (or accumulates one across tool calls)
  2. UIEmitter.init(spec) emits AgentEvent(event_type="ui.<type>.init", data=spec.model_dump()) on the EventBus
  3. sse_sidecar(bus, ...) translates events to SSE frames; ring-buffered against slow consumers
  4. Frontend's resolver parses the event type, finds the renderer, mounts the component
  5. Subsequent delta events patch the component's data in place via dot-path navigation
  6. remove event tears the component down

Runnable demo

notebooks/03_generative_ui.ipynb — agents emit typed components (PlanComponent, TableComponent, ChartComponent) onto the EventBus; sse_sidecar streams to a frontend.