---
date: 2026-10-07
title: "A2UI & a 3D Marathon Simulator"
status: published
youtubeId: "m6EyN2Z7HJk"
tags: "AI", "demos", "games"
---

# Why We Put A2UI Report Cards in a 3D Marathon Simulator

When we started building **Race Condition** for the Google Cloud Next Developer Keynote, one of the technologies we wanted to show off in a real application was **A2UI** (Agent-to-User Interface) – the declarative protocol where an AI agent streams structured UI components back to the client instead of plain text or markdown.

It was a great fit for the project. Not because we wanted the agent to draw the whole application – it doesn't – but because Race Condition is built around a freeform agentic chat panel where you can ask open-ended questions (_"Plan a scenic marathon in Las Vegas for 10,000 runners"_, _"Simulate it"_) and get back rich, interactive report cards about how a course scored or how the race actually unfolded.

Here is how we drew the line between traditional frontend code and A2UI, what the report cards actually look like on the wire, and where we'd take it next.

## What is (and isn't) A2UI

Let's clear up a common misconception first: **not all of Race Condition's UI is built with A2UI.**

If you look at `web/frontend/src/app/components/`, the heavy visual machinery of the app is standard, hand-crafted **Angular 21 web app with Three.js**:

- The **3D Las Vegas viewport** (road splines, building meshes, camera rigs, weather particles, and the 1,000 moving runner sprites) is pure Three.js.
- The **top-level HUD**, camera switcher, countdown overlay, and live runner telemetry bars are standard Angular components.

Where A2UI lives is specifically inside the **agent chat panel** (`ChatNavPanel` → `A2uiControllerComponent`).

```mermaid
flowchart LR
    Agent["Planner Agent<br/>(Google ADK + Gemini 3 Flash)"]

    subgraph Browser["Angular 21 + Three.js Frontend"]
        direction TB
        Chat["Agent Chat Panel<br/>(Freeform Prompting)"]
        Renderer["A2uiControllerComponent<br/>(18-Primitive A2UI Renderer)"]
        Viewport["3D Vegas Viewport & HUD<br/>(Standard Three.js / Angular)"]
        Chat --> Renderer
    end

    Agent -- "validate_and_emit_a2ui (surfaceUpdate + beginRendering)" --> Renderer
    Renderer -- "A2uiActionsService (run_simulation / show_route)" --> Viewport
    Renderer -- "Broadcast Action" --> Agent
```

Why carve out that specific slice for A2UI?

1. **Freeform prompts produce different shapes of answers.** Depending on whether you ask the Planner to design a new route, compare three stored courses from AlloyDB, or summarise a finished 200-tick race, the response needs a different layout.
2. **Chat walls fail the 5-foot test.** When a race finishes or a route is evaluated across seven safety and logistics pillars, dumping thirty lines of bulleted Markdown into a narrow sidebar is unreadable on stage. You want a scannable scorecard with a big headline number, clean dividers, and a button to kick off the next step.
3. **Buttons need to talk back to both the 3D map and the agent.** An A2UI card isn't just a static image; its buttons carry structured actions (`run_simulation`, `show_route`) that bridge the chat conversation directly to the 3D viewport and simulator pipeline.

## Anatomy of an A2UI Report Card

In Race Condition, agents opt into A2UI through a shared skill (`agents/skills/a2ui-rendering/`). The protocol gives the agent a fixed catalogue of **18 capitalized primitives** (`Card`, `Column`, `Row`, `List`, `Tabs`, `Modal`, `Divider`, `Text`, `Image`, `Icon`, `Video`, `AudioPlayer`, `Button`, `TextField`, `MultipleChoice`, `CheckBox`, `Slider`, `DateTimeInput`).

Instead of deeply nested JSON trees – which LLMs love to mangle with mismatched closing braces – A2UI v0.8.0 uses a **flat component array**. Every component has a unique `id`, wraps its literal values in typed objects (`literalString`, `literalNumber`, `literalBoolean`), and references child IDs via `explicitList`.

To render a card, the agent calls the `validate_and_emit_a2ui` tool twice: first with a `surfaceUpdate` payload (defining the flat list of components), and second with `beginRendering` (declaring which component ID is the root).

### 1. The Post-Race & Plan Report Card (`sim_results` / `dashboard`)

When a simulation finishes (or when `planner_with_eval` / `planner_with_memory` scores a course), the agent composes a report card (`surfaceId: "sim_results"`). Look at `web/frontend/src/app/demo-config.ts` and `agents/planner_with_eval/prompts.py` to see what's actually inside:

- **The Header Row**: Split into two columns. The left column stacks a status badge ("PLAN", "SIMULATED", or "STORED") alongside the short simulation/route ID ("#881f"), right above the route title ("Las Vegas Neon Night Marathon"). The right column renders the overall evaluation score as a giant headline number ("91") over a "Score" caption.
- **Core Telemetry Rows**: Key-value `Row` pairs for **Total distance** ("26.2 miles") and **Participants (target / simulated)** ("10,000 / 1,000" — showing the 10,000-runner target event capacity alongside the 1,000-agent simulation cohort), followed by a horizontal `Divider`.
- **The Evaluation Breakdown**: In the compact simulation report card, three 0–100 integer rows summarise Safety Score (80), Runner Experience (80), and City Disruption (100). In the full planning evaluation card (`surfaceId: "dashboard"`), this expands into a 2-column grid covering all **7 evaluation pillars** (Safety Compliance, Logistics Completeness, Participant Experience, Distance Compliance, Community Impact, Financial Viability, and Intent Alignment) plus a **Findings** section flagging MAJOR or MODERATE issues (for example: _"Insufficient emergency access points — Route segments 3-5 lack ambulance-width clearance within 200m"_).
- **The Action Footer**: A primary `Button` ("Run Simulation" or "Re-run Simulation") wired to `"action": {"name": "run_simulation"}`.

Here is a trimmed slice of the actual `sim_results` payload from `agents/planner_with_eval/prompts.py`:

```json
{
  "surfaceUpdate": {
    "surfaceId": "sim_results",
    "components": [
      {
        "id": "tag",
        "component": {
          "Text": {
            "text": { "literalString": "SIMULATED" },
            "usageHint": "label"
          }
        }
      },
      {
        "id": "sim-meta",
        "component": {
          "Text": {
            "text": { "literalString": "#1234" },
            "usageHint": "caption"
          }
        }
      },
      {
        "id": "tag-row",
        "component": {
          "Row": { "children": { "explicitList": ["tag", "sim-meta"] } }
        }
      },
      {
        "id": "title",
        "component": {
          "Text": {
            "text": { "literalString": "Neon & neighbourhoods" },
            "usageHint": "h2"
          }
        }
      },
      {
        "id": "left-col",
        "component": {
          "Column": { "children": { "explicitList": ["tag-row", "title"] } }
        }
      },
      {
        "id": "score-num",
        "component": {
          "Text": { "text": { "literalString": "75" }, "usageHint": "h1" }
        }
      },
      {
        "id": "score-lbl",
        "component": {
          "Text": {
            "text": { "literalString": "Score" },
            "usageHint": "caption"
          }
        }
      },
      {
        "id": "score-col",
        "component": {
          "Column": {
            "children": { "explicitList": ["score-num", "score-lbl"] }
          }
        }
      },
      {
        "id": "header",
        "component": {
          "Row": { "children": { "explicitList": ["left-col", "score-col"] } }
        }
      },
      {
        "id": "dist-l",
        "component": {
          "Text": {
            "text": { "literalString": "Total distance" },
            "usageHint": "body"
          }
        }
      },
      {
        "id": "dist-v",
        "component": {
          "Text": {
            "text": { "literalString": "26.2 miles" },
            "usageHint": "body"
          }
        }
      },
      {
        "id": "dist-r",
        "component": {
          "Row": { "children": { "explicitList": ["dist-l", "dist-v"] } }
        }
      },
      {
        "id": "part-l",
        "component": {
          "Text": {
            "text": { "literalString": "Participants (target / simulated)" },
            "usageHint": "body"
          }
        }
      },
      {
        "id": "part-v",
        "component": {
          "Text": {
            "text": { "literalString": "10,000 / 1,000" },
            "usageHint": "body"
          }
        }
      },
      {
        "id": "part-r",
        "component": {
          "Row": { "children": { "explicitList": ["part-l", "part-v"] } }
        }
      },
      { "id": "d1", "component": { "Divider": {} } },
      {
        "id": "safe-l",
        "component": {
          "Text": {
            "text": { "literalString": "Safety Score" },
            "usageHint": "body"
          }
        }
      },
      {
        "id": "safe-v",
        "component": {
          "Text": { "text": { "literalString": "80" }, "usageHint": "body" }
        }
      },
      {
        "id": "safe-r",
        "component": {
          "Row": { "children": { "explicitList": ["safe-l", "safe-v"] } }
        }
      },
      { "id": "d2", "component": { "Divider": {} } },
      {
        "id": "rerun-txt",
        "component": {
          "Text": { "text": { "literalString": "Re-run Simulation" } }
        }
      },
      {
        "id": "rerun-btn",
        "component": {
          "Button": {
            "child": "rerun-txt",
            "action": { "name": "run_simulation" },
            "primary": { "literalBoolean": true }
          }
        }
      },
      {
        "id": "content",
        "component": {
          "Column": {
            "children": {
              "explicitList": [
                "header",
                "dist-r",
                "part-r",
                "d1",
                "safe-r",
                "d2",
                "rerun-btn"
              ]
            }
          }
        }
      },
      { "id": "card", "component": { "Card": { "child": "content" } } }
    ]
  }
}
```

### 2. Multi-Card Route Lists (`route_list`) and Two-Way Actions

When you ask `planner_with_memory` to _"list the top 3 best routes for the organizer UI"_, it queries PostgreSQL/AlloyDB (`get_planned_routes_data(limit=3)`) and emits a `List` of three `STORED` route cards inside a single `surfaceUpdate` (`surfaceId: "route_list"`), suffixing component IDs with `-1`, `-2`, and `-3`.

Instead of a `"Run Simulation"` button, each card in the list renders two secondary actions:

1. **`"Open Report"`** (`action: {"name": "organizer_show_scorecard"}`) – toggles the expandable summary drawer right inside `a2-ui-controller.component.html` (`unfold_more` / `unfold_less`) so all three cards fit cleanly in the chat column without scrolling off-screen.
2. **`"Show Route"`** (`action: {"name": "show_route", "payload": {"seed": "<route_id>"}}`) – handled by `A2uiActionsService` (`web/frontend/src/app/components/a2ui/a2ui-actions.service.ts`), which either swaps the cached GeoJSON spline on the 3D map immediately or sends a message back to the Planner agent (`Get the route for the seed <id>`) so the agent calls `report_marathon_route` and redraws the 3D Las Vegas course live.

### 3. Teaching the LLM Not to "Freestyle" the Schema

Two practical lessons jumped out while getting Gemini 3 Flash to emit these cards reliably:

First, **validate at the tool boundary**. `validate_and_emit_a2ui` (`agents/skills/a2ui-rendering/tools.py`) checks every component ID, wrapper type, and child reference _before_ the tool finishes. If the model forgets a `literalString` wrapper or references a missing ID, the tool returns a structured list of violations so the model fixes its own JSON inside the same turn.

Second, **lock down schema drift — and let deterministic guardrails emit A2UI directly**. In `prompts.py`, we had to add an explicit `Hard Constraints` block banning rows like `Spectators (expected/attendance)` or float scores like `7.2`, because the model occasionally got creative and added extra marathon metrics that didn't match the card design. And when enforcing hard policy boundaries — such as rejecting an unauthorized budget hike with a styled red `a2ui-Card--financial-refusal` card — don't disable your Python `before_model_callback` guardrail just to let the LLM author the refusal UI. Keep the deterministic callback in front of the model and have the guardrail emit the pre-validated A2UI refusal payload directly, so out-of-policy prompts are blocked before reaching the LLM while the user still gets a rich, structured refusal card.

## Where A2UI could go next in Race Condition

Right now, A2UI only utilises seven of the 18 primitives in the `standard_catalog_0_8_0` spec (`Card`, `Column`, `Row`, `List`, `Divider`, `Text`, `Button`), and only at the bookends of a run (planning and post-race summary).

Because the Angular `A2uiControllerComponent` (`web/frontend/src/app/components/a2ui/a2-ui-controller.component.html`) _already_ implements the remaining primitives – including `Tabs`, `Modal`, `Slider`, `MultipleChoice`, `CheckBox`, and `TextField` – there are some obvious places in Race Condition where A2UI could take over next:

### 1. Live Runner Bio & Medical Triage Cards

During a race, you can watch 1,000 runners move along the Strip and see their status emojis degrade from `🏃` to `🥵` to `🧟`. Imagine clicking a struggling runner in Three.js or asking the chat _"How is Runner #42 holding up?"_ and having the agent emit a live A2UI runner card:

- Their target pace vs. current velocity (`mph`) and hydration percentage (`water%`).
- A scrolling `List` of their recent 5-word `inner_thought` monologues from the tick stream.
- Interactive intervention `Button` controls (`"Trigger Crowd Cheer Boost"` or `"Dispatch Medical Cart"`) that send a targeted event to that single runner's session ID.

### 2. Mid-Race Chaos & Weather Controls (`Slider` + `MultipleChoice`)

The Simulator tracks weather, traffic, and crowd density tick by tick, and we even have a `simulator_with_failure` agent variant. Instead of pre-baking failure scenarios in config, a user could type _"Let me mess with the weather at Mile 18"_ and receive an interactive A2UI chaos card built with `Slider` (Temperature: 65°F–110°F, Wind Speed) and `MultipleChoice` (Road Closure at Bellagio / Hydration Station Shortage) that mutates the Simulator's live session state on the next tick.

### 3. Interactive Course Remediation (`CheckBox` + `Slider`)

When `planner_with_eval` flags a `MAJOR` finding on the Planning Dashboard – say, _"Route segments 3-5 lack ambulance-width clearance within 200m"_ or _"Estimated traffic-control cost exceeds allocated budget by 18%"_ – the findings list today is read-only text. Using `CheckBox` and `Slider` primitives, the Planner could attach a remediation form directly below the findings (_"Add 2 mobile medical tents (+$14k)"_, _"Reroute segment 4 off Las Vegas Blvd"_), letting the organizer apply fixes and re-score the plan with one click.

### 4. Head-to-Head Run Comparisons (`Tabs`)

Once `planner_with_memory` has stored three or four simulations in AlloyDB (`compile_results` records `finished_count`, `dnf_count`, `vitals_trend`, and `notable_events`), asking _"Compare my last two runs on Strip Classic"_ could emit a tabbed A2UI card (`Tabs`) flipping between **Overview Scores**, **DNF & Hydration Curves**, and **Notable Race Events** without needing a single line of new Angular template code.

## The takeaway

- **Don't force A2UI to be your 3D engine.** Let Three.js and standard Angular own the high-frequency viewport and HUD; use A2UI where user intent is open-ended and the response shape depends on what the agent just discovered.
- **Validate A2UI inside the tool loop.** Having `validate_and_emit_a2ui` reject unwrapped strings or dangling component IDs in Python means your frontend renderer never has to guess what the LLM meant.
- **Wire A2UI actions back into your app state.** A report card is twice as useful when clicking `"Show Route"` or `"Run Simulation"` inside the chat bubble drives the 3D map right next to it.
