Generative UI¶
Orqest lets agents emit typed component specifications that a frontend resolves dynamically. Instead of returning text for the UI to render, the agent emits a UIComponentSpec[T] (chart, table, form, plan, takeover dialog, Vega visualization, Mermaid diagram, sandboxed HTML, etc.). The frontend hot-loads a renderer keyed on component_type. Subsequent state changes flow as UIDeltaEvent records on the same SSE stream.
What problem does this solve?¶
Agents that return prose and let the UI render it leave a lot of capability on the table. If the agent is analyzing data, you want a chart — not a paragraph describing one. If it needs your approval, you want a dialog — not "should I proceed? (yes/no)" in chat. The agent already knows the shape of the answer; let it pick the surface. Generative UI flips the protocol: the agent describes what to render, the frontend resolves how. Same backend produces a Plan, a chart, and a PDF report based on what the work needs.
UIComponentSpec¶
Generic Pydantic with a component_type Literal discriminator. Each first-party component is a subclass that fixes component_type and types the data: T payload.
| Field | Type | Description |
|---|---|---|
component_id |
str |
Stable identifier (the frontend uses this to route deltas to the right instance) |
component_type |
Literal["plan" \| "chart" \| ...] |
Discriminator the frontend resolves to a renderer |
data |
T (typed payload) |
Component-specific payload — strongly typed per subclass |
metadata |
dict[str, Any] |
Free-form (used for layout hints, accessibility, etc.) |
created_at |
datetime |
UTC timestamp |
UIDeltaEvent — partial updates¶
Once a component is initialized, you patch it with delta events. Avoids re-shipping the full payload on every change.
| Field | Type | Description |
|---|---|---|
op |
Literal["replace", "merge", "append", "remove"] |
The mutation kind |
path |
str |
Dot-path into the data payload (e.g., "series.0.points") |
value |
Any |
New value (or to-merge object, or to-append item) |
Ops:
- replace — overwrite the value at path
- merge — shallow-merge value (object) into the dict at path
- append — append value to the list at path
- remove — delete the key/element at path
Three layers of first-party components¶
Components ship in three layers — pick the lowest layer that does the job.
Layer 1 — Compositional primitives¶
Hand-typed components for common UI patterns:
PlanComponent— task list (carriesPlanTaskinstances; ties intoExecutionPlan)ChartComponent— line / bar / scatter / pie / heatmap with typedChartSeriesTableComponent— typedTableColumn+ rowsFormComponent— typedFormField+ submit event handleTakeoverDialogComponent— confirm / input / choice (the agent asks the user a question)LayoutComponent,TextComponent,MarkdownComponent,ImageComponent,BadgeComponent,ButtonComponent,InputComponentStatTileComponent— one headline number with label, unit and optional trendChecklistComponent— items with a checked flagTimelineComponent— dated events in orderMapComponent— markers, an optional center and a zoom levelMediaComponent— image, video or audio by URL (https://only)PathwayComponent— a titled route of steps (todo/current/done) with overall progress in[0, 1]and an optional action nameLessonStepComponent— one guided step: objective, at most three cues, sub-steps with status, an optional safety note, optionalhttps://media_urland avoice_hintfor renderers that speak
Layer 2 — Declarative grammars¶
When you need richer visualization than Layer 1 covers — pass a spec the frontend already knows how to render:
VegaChartComponent— Vega-Lite spec passed throughMermaidComponent— Mermaid diagram sourceLatexComponent— KaTeX/MathJax-renderable LaTeXJsonViewerComponent— collapsible JSON tree
Layer 3 — Sandboxed escape hatch¶
For one-offs the framework can't anticipate:
SandboxedHTMLComponent— HTML/CSS/JS in an iframe sandbox
UIEmitter — convenience facade¶
The minimal API. Wraps an EventBus so init/delta/remove are one line each.
import asyncio
from orqest.observability import EventBus
from orqest.ui import (
ChartComponent,
ChartComponentData,
ChartSeries,
UIEmitter,
)
async def main():
bus = EventBus()
emitter = UIEmitter(bus)
# Initialize a chart
chart = ChartComponent(
component_id="latency-chart",
data=ChartComponentData(
kind="line",
title="Request latency",
x_label="time",
y_label="ms",
series=[
ChartSeries(
name="p50",
points=[{"x": 0, "y": 12}, {"x": 1, "y": 14}],
),
],
),
)
emitter.init(chart)
# Append a new point
emitter.delta(
component_id="latency-chart",
op="append",
path="series.0.points",
value={"x": 2, "y": 18},
)
# Remove the chart entirely
emitter.remove(component_id="latency-chart")
asyncio.run(main())
View specs and bindings¶
A ViewSpec is a serializable screen: a named, versioned tree of ViewNodes whose leaves name a registered component_type. An assistant (or any code) publishes the spec once; the data flows in separately through bindings, so the screen updates without the spec being rewritten.
- Nodes.
id(unique within the view, same grammar as addresses),component_type, literaldata,bindings,hintsand, forlayoutnodes only,children. The root must be alayout. A layout'sdatamust not containchildren, and a layout must not bindchildren.ViewNode,HintsandViewSpecreject unknown keys, so a typo such aspropsfordatais an error rather than silently dropped. A view holds at most 200 nodes (MAX_VIEW_NODES) nested at most 16 deep (MAX_VIEW_DEPTH, root = 1). - Bindings.
bindings={"value": "state.done"}fills the top-level data fieldvaluefrom the context passed toresolve_view. The key must be a field of the block's data model (a misspelt key is a structural problem). A path is dot-separated: dict keys, and integers to index lists (state.plan.today.drills.0.label).resolve_pathraisesBindingErroron a miss. - Hints.
Hints(importance="primary" | "secondary" | "glanceable", size="compact" | "wide", group=None). They are copied to each component'smetadata["hints"](the owning view's name goes inmetadata["view"]). Assistants do not set colours, fonts or spacing; the frontend decides how to present hints. - Placeholders.
resolve_viewnever raises. A node whose type is unknown, whose binding does not resolve, or whose data is invalid or not strict JSON (a boundNaNor infinity) renders as atextcomponent whosecontentis"Unavailable", and the reason is appended toResolvedView.problems. Resolved trees and blocks are therefore always strict JSON. A degraded non-root layout becomes a placeholder too (its subtree is not rendered). A degraded root stays a layout holding one placeholder child with the reserved idroot_unavailable, soResolvedView.treeis always a layout. - Validation.
validate_viewreports only structural problems (unknowncomponent_type, a binding key that is not a data field, invalid literal data, invalid data after bindings that did resolve). A binding that does not resolve is not a problem there, because it may resolve later; it still shows up inResolvedView.problems, renders as a placeholder, and is listed byunresolved_bindings(spec, registry, context)(useful as a publish-time warning). Non-strict-JSON data is not structural either: it comes from the context, not the spec. - Result.
ResolvedViewcarriestree(nested, ready to render),blocks(a flat map of node id tocomponent_type,datawithoutchildren, andmetadata) andproblems.fallback_view(name)builds a minimal view that showsstatein a JSON viewer.
Diffing two states¶
diff_view(old, new) returns the UIDeltaEvents that turn one resolution into the next:
- one
replace(path=<key>) per changed or added top-leveldatakey, and oneremoveper dropped key, ordered by block id then key. Values inside a field are not diffed: a changed list is sent whole. Nonewhen a full re-init is required: the view name or version changed, the set of block ids changed, or any block changedcomponent_type(for example a placeholder that became a real block). Send the whole tree again in that case.[]when nothing changed. Metadata (hints) is not diffed.
apply_deltas(blocks, deltas) is the reference applier for exactly these ops; it returns a new block map. It raises ValueError for an unknown op, an unknown component_id or a component_type that differs from the block's. diff_view assumes both resolutions come from the same spec: it ignores child order and hints, so re-init whenever the spec changes.
from orqest.ui import (
ViewNode,
ViewSpec,
apply_deltas,
default_registry,
diff_view,
resolve_view,
)
registry = default_registry()
view = ViewSpec(
name="today",
root=ViewNode(
id="root",
component_type="layout",
children=[
ViewNode(
id="done",
component_type="stat_tile",
data={"label": "Drills done"},
bindings={"value": "state.done"},
),
ViewNode(
id="missing",
component_type="stat_tile",
data={"label": "Streak"},
bindings={"value": "state.streak"},
),
],
),
)
before = resolve_view(view, registry, {"state": {"done": 2}})
after = resolve_view(view, registry, {"state": {"done": 3}})
assert before.blocks["done"]["data"]["value"] == 2
assert before.blocks["missing"]["data"]["content"] == "Unavailable"
assert before.problems == ["binding 'state.streak' did not resolve at node 'missing'"]
deltas = diff_view(before, after)
assert [(d.component_id, d.op, d.path, d.value) for d in deltas] == [
("done", "replace", "value", 3)
]
assert apply_deltas(before.blocks, deltas) == after.blocks
Views are events too: ui.view.init carries the full tree, ui.view.delta carries the deltas, and ui.view.escape_hatch flags a sandboxed_html node. When the views belong to a long-lived assistant, the runtime emits them for you; see Long-Lived Assistants.
Renderer contract¶
What a frontend can rely on when it renders views from the event stream (as emitted by orqest.assistants):
- Event shapes.
ui.view.init:{"address", "view", "version", "seq", "tree"}, wheretreeisResolvedView.tree(alayoutcomponent whosedata.childrennests the rest).ui.view.delta:{"address", "view", "version", "seq", "deltas"}, wheredeltasis a list ofUIDeltaEventdicts withop"replace"(path= a top-level data key,value= its new value) or"remove"(path= the key to drop).ui.view.remove:{"address", "view", "seq"}, sent when a view no longer exists (for example after a rollback): drop that view and its lastseq.ui.view.escape_hatch:{"address", "view", "node_id"}. - Ordering with
seq.seqcounts events per(address, view), grows by one per event and never goes back within a process. Seq restarts with the server process — reset your last seq on reconnect. An init starts a new base. Keep the lastseqapplied per view: drop an event whoseseqis not larger (you already have it), let an init replace the tree, and apply a delta only when itsseqis exactly last + 1. Any other delta (a gap) means you missed something: refetch the snapshot. - Reconnect: reconcile the view set. On every (re)connect, call
AssistantRuntime.list_views(address)and drop any view you hold that it does not list, then snapshot the listed ones as below.ui.view.removeis only sent for views the current server process has emitted, so a view that disappeared before a restart, or while you were disconnected, sends no remove event. - Retire. On
assistant.lifecyclewithdata["action"] == "retire"(the address is the event'sagent_name), drop all of that assistant's views: a retired assistant gets noui.view.removeevents. - Subscribe, then snapshot. Subscribe to the stream first, then fetch the snapshot (
AssistantRuntime.render_view(address, name)returns aViewSnapshotwithresolvedandseq), then process buffered and new events with the rule above. A snapshot withseq0 was never emitted; the next event for that view is an init. A snapshot may predate an event already delivered to you — keep that event and re-apply it by the seq rule. - Applying deltas. A delta targets the tree node whose
component_idmatches and replaces or removes one key of itsdata. It never touches a layout'schildren: structure changes always arrive as an init. A delta whosecomponent_idis unknown, or whosecomponent_typediffers from that node's, means you are out of sync: refetch the snapshot (apply_deltasraising is the reference form of this signal). - Layout. Honour a layout's
directionandgrid_columnsas intent. Ignoregapandalign: the renderer owns spacing. Hints (metadata["hints"]) are presentation intent too. version. Informational (the spec's version); ordering comes fromseq.- JSON. Trees and deltas are strict JSON (no
NaNor infinity), so any standard parser reads them.
Artifacts: show, update, pin, dismiss, expire¶
An artifact is a view with a lifecycle, for a screen an assistant generates on demand. While it is live it is the ordinary view art-<artifact_id> (so render_view, deltas and the renderer contract keep working) plus a metadata row, committed in the same transaction as the wake. A failed wake shows nothing.
from orqest.assistants import AssistantRuntime
from orqest.ui.view import ViewNode
class Briefer:
async def handle(self, message, ctx):
ctx.show_artifact(
"morning-brief",
intent="summarise today",
title="Morning brief",
a11y_label="Today's summary",
root=ViewNode(id="root", component_type="layout"),
allowed_actions=["done"],
expires_in_s=3600,
)
# async with AssistantRuntime(path) as rt:
# await rt.register_code("briefer", Briefer())
# await rt.send("app", "briefer", "go", {})
# await rt.tick()
# await rt.list_artifacts(status={"active"})
# await rt.pin_artifact("briefer", "morning-brief", True) # never expires while pinned
# await rt.artifact_action("briefer", "morning-brief", "done", {}, sender="owner-app")
# await rt.delete_artifact("briefer", "morning-brief") # forget it for real
- Ids belong to their address. The key is
(address, artifact_id), as for views, so two assistants can use the same id and one can never squat on or probe another's.pin_artifact,artifact_actionanddelete_artifacttake the address first. Ids are identifiers of at most 60 characters. Events carry the address inagent_nameand{artifact_id, version, status}indata. - Statuses.
active,pinned,dismissed,expired. Showing the same id again updates it (version +1); showing a dismissed or expired one makes it live and active again; a pinned one stays pinned. Two shows of an id in one wake keep the last and bump once. Dismissing after showing a new id in the same wake shows nothing.show_artifactanddismiss_artifactare sequential tools, so a model response holding either runs its calls in order. - Frozen trees. When an artifact is dismissed or expires, or its assistant retires, its tree is resolved once against the state of that moment and kept as a snapshot, and its view is removed (
ui.view.remove). A frozen artifact is not emitted live, is not inlist_viewsor inupdate_state's view problems, and adds nothing to later commits.render_view(address, "art-<id>")returns the snapshot (seq0), also for a retired address. A show that reactivates it clears the snapshot and goes live again. A name with theart-prefix never falls back to a view of the raw state: an artifact that is unknown, deleted or undecodable raisesUnknownArtifact, and one frozen with no usable view keeps an empty layout. - Deadlines.
expires_in_scounts from the commit (on the runtime's clock), not from the start of the wake. Showing again withoutexpires_in_skeeps the deadline a live artifact has; a dismissed or expired one starts without one. To clear a deadline, pin it. Expiry runs intick()andrun_forever; each artifact emitsexpireonce, also across restarts (a crash between the commit and the emit loses the event rather than repeating it). - Pin. The owner can pin an artifact in any status, a dismissed or expired one keeping its snapshot, and no assistant needs to be alive. A pinned artifact never expires. Unpinning one that isn't pinned does nothing. Unpinning a pinned one returns a frozen one to
dismissedand a live one toactivewith the deadline it had, so one whose time has passed expires at the next poll. An assistant cannot undo a pin:dismiss_artifacton a pinned artifact is refused (the commit ignores it too). A show that updates a pinned live artifact is allowed and it stays pinned. A pinned artifact that is also frozen is kept as the owner saw it:show_artifacton it is refused by the tool and ignored (with a log line) at commit. - Retire. Retiring an assistant dismisses its active artifacts (with
ui.artifact.dismiss) and freezes all of them; a pinned one stays pinned, frozen. They stay searchable and renderable. - Rollback.
rollback()andredesign()leave artifacts alone: version snapshots skipart-*views, and a restore does not touch them. - Delete.
delete_artifact(address, artifact_id)removes the row, search text, snapshot and view for real (the runtime keepssecure_deleteon, so freed pages are zeroed, and the write-ahead log is truncated) and emitsui.artifact.delete, without emitting a retired assistant's screens again. If another connection holds the log, the truncation waits only 0.1 s: the call still succeeds because the row is gone, a warning names the artifact, and the truncation is retried on every poll until it empties the log, and once at the next open (the text stays in the log until then). It is for the owner, not a model tool, so "forget that" can remove a fact from a screen that was shown. - Events.
ui.artifact.show,update,dismiss,expire,pinanddelete, emitted after the commit. Emission is best-effort, and one batch of events is bounded as a whole by onerun_timeout_s(events not sent in time are dropped and counted in a warning); a subscriber that hangs or raisesCancelledErrorcosts one logged warning, not the loop. - Reserved names.
publish_viewrefuses names starting withart-. An id that would land on an existing non-artifact view namedart-<id>is refused (early by the LLM tool, authoritatively at commit). - Search.
search_artifacts("review")matches every word in the title and intent, for any status, after NFKC and case folding (éénfindsÉén). At most 20 words and 1000 characters of the query count,limitis clamped to 1..100, and a query that isn't valid text finds nothing.%,_and\match literally. - Actions.
artifact_actionrefuses an action not inallowed_actionsand any action on a frozen artifact (dismissed, expired, or retired with it, even pinned), since a stale tap must not reach an assistant.valuemust be a dict (TypeError). Otherwise it sends the assistant aui.actionmessage{artifact_id, action, value}; an address with registered contracts must listui.action(ContractViolationotherwise). Thesenderis trusted as given: pass a verified owner sender (one inowner_senders) for a voice or touch action to count as the owner's turn. - Model prompt. An assistant with live artifacts sees a
## Your artifactssection (up to 20, id and title, newest first) with a note onshow_artifactanddismiss_artifact; kept (pinned and frozen) screens are listed as read-only.update_statereports view problems for the artifacts live when the wake was claimed and those shown in it, so one that expires during the wake may still be reported. - Private information. With a read grant,
show_artifactis leak-checked likepublish_view; see compartments.
Event-type conventions¶
The emitter maps to SSE event types via dedicated helpers. The frontend resolves a renderer by parsing the event type.
| Event type | When | Helper |
|---|---|---|
ui.<component_type>.init |
Component first appears | ui_init_event_type(component_type) |
ui.<component_type>.delta |
State patch | ui_delta_event_type(component_type) |
ui.<component_type>.remove |
Component dismissed | ui_remove_event_type(component_type) |
For ChartComponent: ui.chart.init, ui.chart.delta, ui.chart.remove.
ComponentRegistry — per-Workbench, no module singleton¶
Each Workbench carries its own registry. This avoids the rigidity of module-level state and lets two consumers in the same process register different component sets.
from orqest.ui import ComponentRegistry, default_registry, UIComponentSpec
from pydantic import BaseModel
from typing import Literal
# Define a custom component for your domain
class MoleculeViewerData(BaseModel):
smiles: str
color_by: Literal["element", "charge"] = "element"
class MoleculeViewerComponent(UIComponentSpec[MoleculeViewerData]):
component_type: Literal["molecule_viewer"] = "molecule_viewer"
data: MoleculeViewerData
# Build a registry that includes both first-party and your custom one
registry = default_registry() # PlanComponent, ChartComponent, etc.
registry.register(MoleculeViewerComponent)
Workbench(ui_registry=registry, auto_register_first_party_ui=True) injects it.
ExecutionPlan dual emission¶
ExecutionPlan opt-in flag-gates dual emission of legacy plan.* events alongside typed ui.plan.* events. Default off — existing emission-count assertions stay byte-identical.
from orqest.plan import ExecutionPlan
plan = ExecutionPlan(...)
plan.enable_ui_events(component_id="plan") # opt-in dual emission
SSE sidecar — flushing to the frontend¶
sse_sidecar(bus, ...) is an async iterator yielding SSE-formatted strings. Plug into FastAPI / Starlette / aiohttp / etc. The frontend subscribes and resolves renderers by event type.
from fastapi import FastAPI
from sse_starlette import EventSourceResponse
from orqest.observability import sse_sidecar
app = FastAPI()
@app.get("/sessions/{session_id}/events")
async def events(session_id: str):
bus = get_bus_for(session_id)
return EventSourceResponse(sse_sidecar(bus, replay=(), heartbeat_s=15.0))
Custom component example — end to end¶
import asyncio
from typing import Literal
from pydantic import BaseModel
from orqest.observability import EventBus
from orqest.ui import (
ComponentRegistry,
UIComponentSpec,
UIEmitter,
default_registry,
)
class RiskHeatmapData(BaseModel):
rows: list[str]
cols: list[str]
cells: list[list[float]] # 0 (low) → 1 (high)
class RiskHeatmapComponent(UIComponentSpec[RiskHeatmapData]):
component_type: Literal["risk_heatmap"] = "risk_heatmap"
data: RiskHeatmapData
async def main():
registry = default_registry()
registry.register(RiskHeatmapComponent)
bus = EventBus()
emitter = UIEmitter(bus)
spec = RiskHeatmapComponent(
component_id="q3-risk",
data=RiskHeatmapData(
rows=["EU", "US", "APAC"],
cols=["liquidity", "credit", "operational"],
cells=[[0.2, 0.4, 0.1], [0.3, 0.5, 0.2], [0.7, 0.6, 0.3]],
),
)
emitter.init(spec)
# Emits an "ui.risk_heatmap.init" event on the bus.
# The frontend (which previously registered a renderer for "risk_heatmap")
# resolves and mounts it.
asyncio.run(main())
Best practices¶
- Start at the highest layer that fits.
PlanComponentover a custom component overSandboxedHTMLComponent. The frontend already knows how to render Layer 1 — you pay design surface for every layer below. component_idis stable. It's how deltas route. Using random UUIDs per-emit defeats the delta contract.- Deltas, not full re-emits. A 50KB chart that gains one point should emit a
delta(append, "series.0.points", {x,y}), not a freshinit. - Keep frontend renderers thin. The component is a typed contract; the frontend just maps fields to DOM. Behavior belongs in the agent, not the renderer.
- Don't reach into Layer 3 (
SandboxedHTMLComponent) by default. It's for one-offs that don't deserve a typed component. Frequent use is a smell — promote it to a typed component instead.
Pitfalls¶
- Don't share
ComponentRegistryacross consumers that need different schemas. The per-Workbench design is intentional. - Don't emit
initfor the samecomponent_idtwice in a session — that's a re-mount; the frontend may flicker. Use adelta(replace, "", new_data)instead, orremovetheninit. - Don't trust the frontend to validate. The registry's
validate_payloadruns at the backend boundary; the frontend just renders. UIEmitternever raises on bus failure — it logs at DEBUG. If you need delivery guarantees, add a separate observability layer.- Polymath uses every layer of generative UI because it's the substrate's flagship demo. Most consumer apps pick 2-3 components and stay there.
What's happening under the hood¶
- Agent computes a typed component (or accumulates one across tool calls)
UIEmitter.init(spec)emitsAgentEvent(event_type="ui.<type>.init", data=spec.model_dump())on theEventBussse_sidecar(bus, ...)translates events to SSE frames; ring-buffered against slow consumers- Frontend's resolver parses the event type, finds the renderer, mounts the component
- Subsequent
deltaevents patch the component'sdatain place via dot-path navigation removeevent tears the component down
Related Concepts¶
- Workbench — bundles
ComponentRegistry+EventBus - Observability — the
EventBusunderneath everything - SSE Sidecar — flushing events to the frontend
- Execution Plan —
PlanComponentintegration - Event Bus Hook — emits
tool.*events alongsideui.*
Runnable demo¶
notebooks/03_generative_ui.ipynb — agents emit typed components (PlanComponent, TableComponent, ChartComponent) onto the EventBus; sse_sidecar streams to a frontend.