Why We Put A2UI Report Cards in a 3D Marathon Simulator

When we started building Race Condition for the Google Cloud Next Developer Keynote, one of the technologies we wanted to show off in a real application was A2UI (Agent-to-User Interface) – the declarative protocol where an AI agent streams structured UI components back to the client instead of plain text or markdown.

It was a great fit for the project. Not because we wanted the agent to draw the whole application – it doesn't – but because Race Condition is built around a freeform agentic chat panel where you can ask open-ended questions ("Plan a scenic marathon in Las Vegas for 10,000 runners", "Simulate it") and get back rich, interactive report cards about how a course scored or how the race actually unfolded.

Here is how we drew the line between traditional frontend code and A2UI, what the report cards actually look like on the wire, and where we'd take it next.

What is (and isn't) A2UI

Let's clear up a common misconception first: not all of Race Condition's UI is built with A2UI.

If you look at web/frontend/src/app/components/, the heavy visual machinery of the app is standard, hand-crafted Angular 21 web app with Three.js:

• The 3D Las Vegas viewport (road splines, building meshes, camera rigs, weather particles, and the 1,000 moving runner sprites) is pure Three.js.
• The top-level HUD, camera switcher, countdown overlay, and live runner telemetry bars are standard Angular components.

Where A2UI lives is specifically inside the agent chat panel (ChatNavPanel → A2uiControllerComponent).

flowchart TD
    subgraph Browser["Angular 21 + Three.js Frontend"]
        Viewport["3D Vegas Viewport & HUD<br/>(Standard Three.js / Angular)"]
        Chat["Agent Chat Panel<br/>(Freeform Prompting)"]
        Renderer["A2uiControllerComponent<br/>(18-Primitive A2UI Renderer)"]
        Chat --> Renderer
    end

    Agent["Planner Agent<br/>(Google ADK + Gemini 3 Flash)"] -->|"validate_and_emit_a2ui<br/>(surfaceUpdate + beginRendering)"| Renderer
    Renderer -->|"A2uiActionsService<br/>(run_simulation / show_route)"| Viewport
    Renderer -->|"Broadcast Action"| Agent

Why carve out that specific slice for A2UI?

1. Freeform prompts produce different shapes of answers. Depending on whether you ask the Planner to design a new route, compare three stored courses from AlloyDB, or summarise a finished 200-tick race, the response needs a different layout.
2. Chat walls fail the 5-foot test. When a race finishes or a route is evaluated across seven safety and logistics pillars, dumping thirty lines of bulleted Markdown into a narrow sidebar is unreadable on stage. You want a scannable scorecard with a big headline number, clean dividers, and a button to kick off the next step.
3. Buttons need to talk back to both the 3D map and the agent. An A2UI card isn't just a static image; its buttons carry structured actions (run_simulation, show_route) that bridge the chat conversation directly to the 3D viewport and simulator pipeline.

Anatomy of an A2UI Report Card

In Race Condition, agents opt into A2UI through a shared skill (agents/skills/a2ui-rendering/). The protocol gives the agent a fixed catalogue of 18 capitalized primitives (Card, Column, Row, List, Tabs, Modal, Divider, Text, Image, Icon, Video, AudioPlayer, Button, TextField, MultipleChoice, CheckBox, Slider, DateTimeInput).

Instead of deeply nested JSON trees – which LLMs love to mangle with mismatched closing braces – A2UI v0.8.0 uses a flat component array. Every component has a unique id, wraps its literal values in typed objects (literalString, literalNumber, literalBoolean), and references child IDs via explicitList.

To render a card, the agent calls the validate_and_emit_a2ui tool twice: first with a surfaceUpdate payload (defining the flat list of components), and second with beginRendering (declaring which component ID is the root).

1. The Post-Race & Plan Report Card (sim_results / dashboard)

When a simulation finishes (or when planner_with_eval / planner_with_memory scores a course), the agent composes a report card (surfaceId: "sim_results"). Look at web/frontend/src/app/demo-config.ts and agents/planner_with_eval/prompts.py to see what's actually inside:

• The Header Row: Split into two columns. The left column stacks a status badge ("PLAN", "SIMULATED", or "STORED") alongside the short simulation/route ID ("#881f"), right above the route title ("Las Vegas Neon Night Marathon"). The right column renders the overall evaluation score as a giant headline number ("91") over a "Score" caption.
• Core Telemetry Rows: Key-value Row pairs for Total distance ("26.2 miles") and Participants ("10,000/1,000"), followed by a horizontal Divider.
• The Evaluation Breakdown: In the compact simulation report card, three 0–100 integer rows summarise Safety Score (80), Runner Experience (80), and City Disruption (100). In the full planning evaluation card (surfaceId: "dashboard"), this expands into a 2-column grid covering all 7 evaluation pillars (Safety Compliance, Logistics Completeness, Participant Experience, Distance Compliance, Community Impact, Financial Viability, and Intent Alignment) plus a Findings section flagging MAJOR or MODERATE issues (for example: "Insufficient emergency access points — Route segments 3-5 lack ambulance-width clearance within 200m").
• The Action Footer: A primary Button ("Run Simulation" or "Re-run Simulation") wired to "action": {"name": "run_simulation"}.

Here is a trimmed slice of the actual sim_results payload from agents/planner_with_eval/prompts.py:

{
  "surfaceUpdate": {
    "surfaceId": "sim_results",
    "components": [
      {
        "id": "tag",
        "component": {
          "Text": {
            "text": { "literalString": "SIMULATED" },
            "usageHint": "label"
          }
        }
      },
      {
        "id": "sim-meta",
        "component": {
          "Text": {
            "text": { "literalString": "#1234" },
            "usageHint": "caption"
          }
        }
      },
      {
        "id": "tag-row",
        "component": {
          "Row": { "children": { "explicitList": ["tag", "sim-meta"] } }
        }
      },
      {
        "id": "title",
        "component": {
          "Text": {
            "text": { "literalString": "Neon & neighbourhoods" },
            "usageHint": "h2"
          }
        }
      },
      {
        "id": "left-col",
        "component": {
          "Column": { "children": { "explicitList": ["tag-row", "title"] } }
        }
      },
      {
        "id": "score-num",
        "component": {
          "Text": { "text": { "literalString": "75" }, "usageHint": "h1" }
        }
      },
      {
        "id": "score-lbl",
        "component": {
          "Text": {
            "text": { "literalString": "Score" },
            "usageHint": "caption"
          }
        }
      },
      {
        "id": "score-col",
        "component": {
          "Column": {
            "children": { "explicitList": ["score-num", "score-lbl"] }
          }
        }
      },
      {
        "id": "header",
        "component": {
          "Row": { "children": { "explicitList": ["left-col", "score-col"] } }
        }
      },
      {
        "id": "dist-l",
        "component": {
          "Text": {
            "text": { "literalString": "Total distance" },
            "usageHint": "body"
          }
        }
      },
      {
        "id": "dist-v",
        "component": {
          "Text": {
            "text": { "literalString": "26.2 miles" },
            "usageHint": "body"
          }
        }
      },
      {
        "id": "dist-r",
        "component": {
          "Row": { "children": { "explicitList": ["dist-l", "dist-v"] } }
        }
      },
      { "id": "d1", "component": { "Divider": {} } },
      {
        "id": "safe-l",
        "component": {
          "Text": {
            "text": { "literalString": "Safety Score" },
            "usageHint": "body"
          }
        }
      },
      {
        "id": "safe-v",
        "component": {
          "Text": { "text": { "literalString": "80" }, "usageHint": "body" }
        }
      },
      {
        "id": "safe-r",
        "component": {
          "Row": { "children": { "explicitList": ["safe-l", "safe-v"] } }
        }
      },
      { "id": "d2", "component": { "Divider": {} } },
      {
        "id": "rerun-txt",
        "component": {
          "Text": { "text": { "literalString": "Re-run Simulation" } }
        }
      },
      {
        "id": "rerun-btn",
        "component": {
          "Button": {
            "child": "rerun-txt",
            "action": { "name": "run_simulation" },
            "primary": { "literalBoolean": true }
          }
        }
      },
      {
        "id": "content",
        "component": {
          "Column": {
            "children": {
              "explicitList": [
                "header",
                "dist-r",
                "d1",
                "safe-r",
                "d2",
                "rerun-btn"
              ]
            }
          }
        }
      },
      { "id": "card", "component": { "Card": { "child": "content" } } }
    ]
  }
}

2. Multi-Card Route Lists (route_list) and Two-Way Actions

When you ask planner_with_memory to "list the top 3 best routes for the organizer UI", it queries PostgreSQL/AlloyDB (get_planned_routes_data(limit=3)) and emits a List of three STORED route cards inside a single surfaceUpdate (surfaceId: "route_list"), suffixing component IDs with -1, -2, and -3.

Instead of a "Run Simulation" button, each card in the list renders two secondary actions:

1. "Open Report" (action: {"name": "organizer_show_scorecard"}) – toggles the expandable summary drawer right inside a2-ui-controller.component.html (unfold_more / unfold_less) so all three cards fit cleanly in the chat column without scrolling off-screen.
2. "Show Route" (action: {"name": "show_route", "payload": {"seed": "<route_id>"}}) – handled by A2uiActionsService (web/frontend/src/app/components/a2ui/a2ui-actions.service.ts), which either swaps the cached GeoJSON spline on the 3D map immediately or sends a message back to the Planner agent (Get the route for the seed <id>) so the agent calls report_marathon_route and redraws the 3D Las Vegas course live.

3. Teaching the LLM Not to "Freestyle" the Schema

Two practical lessons jumped out while getting Gemini 3 Flash to emit these cards reliably:

First, validate at the tool boundary. validate_and_emit_a2ui (agents/skills/a2ui-rendering/tools.py) checks every component ID, wrapper type, and child reference before the tool finishes. If the model forgets a literalString wrapper or references a missing ID, the tool returns a structured list of violations so the model fixes its own JSON inside the same turn.

Second, watch out for schema drift and code-level short-circuits. In prompts.py, we had to add an explicit Hard Constraints block banning rows like Spectators (expected/attendance) or float scores like 7.2, because the model occasionally got creative and added extra marathon metrics that didn't match the card design. And in planner_with_memory/agent.py, we deliberately removed the Python before_model_callback financial guardrail so that when a user asks for an unauthorized budget hike, the LLM isn't short-circuited before it can emit a styled red A2UI refusal card (a2ui-Card--financial-refusal).

Where A2UI could go next in Race Condition

Right now, A2UI only utilises about six of the 18 primitives in the standard_catalog_0_8_0 spec (Card, Column, Row, List, Divider, Text, Button), and only at the bookends of a run (planning and post-race summary).

Because the Angular A2uiControllerComponent (web/frontend/src/app/components/a2ui/a2-ui-controller.component.html) already implements the remaining primitives – including Tabs, Modal, Slider, MultipleChoice, CheckBox, and TextField – there are some obvious places in Race Condition where A2UI could take over next:

1. Live Runner Bio & Medical Triage Cards

During a race, you can watch 1,000 runners move along the Strip and see their status emojis degrade from 🏃 to 🥵 to 🧟. Imagine clicking a struggling runner in Three.js or asking the chat "How is Runner #42 holding up?" and having the agent emit a live A2UI runner card:

• Their target pace vs. current velocity (mph) and hydration percentage (water%).
• A scrolling List of their recent 5-word inner_thought monologues from the tick stream.
• Interactive intervention Button controls ("Trigger Crowd Cheer Boost" or "Dispatch Medical Cart") that send a targeted event to that single runner's session ID.

2. Mid-Race Chaos & Weather Controls (Slider + MultipleChoice)

The Simulator tracks weather, traffic, and crowd density tick by tick, and we even have a simulator_with_failure agent variant. Instead of pre-baking failure scenarios in config, a user could type "Let me mess with the weather at Mile 18" and receive an interactive A2UI chaos card built with Slider (Temperature: 65°F–110°F, Wind Speed) and MultipleChoice (Road Closure at Bellagio / Hydration Station Shortage) that mutates the Simulator's live session state on the next tick.

3. Interactive Course Remediation (CheckBox + Slider)

When planner_with_eval flags a MAJOR finding on the Planning Dashboard – say, "Route segments 3-5 lack ambulance-width clearance within 200m" or "Estimated traffic-control cost exceeds allocated budget by 18%" – the findings list today is read-only text. Using CheckBox and Slider primitives, the Planner could attach a remediation form directly below the findings ("Add 2 mobile medical tents (+$14k)", "Reroute segment 4 off Las Vegas Blvd"), letting the organizer apply fixes and re-score the plan with one click.

4. Head-to-Head Run Comparisons (Tabs)

Once planner_with_memory has stored three or four simulations in AlloyDB (compile_results records finished_count, dnf_count, vitals_trend, and notable_events), asking "Compare my last two runs on Strip Classic" could emit a tabbed A2UI card (Tabs) flipping between Overview Scores, DNF & Hydration Curves, and Notable Race Events without needing a single line of new Angular template code.

The takeaway

• Don't force A2UI to be your 3D engine. Let Three.js and standard Angular own the high-frequency viewport and HUD; use A2UI where user intent is open-ended and the response shape depends on what the agent just discovered.
• Validate A2UI inside the tool loop. Having validate_and_emit_a2ui reject unwrapped strings or dangling component IDs in Python means your frontend renderer never has to guess what the LLM meant.
• Wire A2UI actions back into your app state. A report card is twice as useful when clicking "Show Route" or "Run Simulation" inside the chat bubble drives the 3D map right next to it.

Article published on: 7 Oct, 2026