Adventure & A11Y: The Invisible Layer Under the Game

Most video games don't do much for blind and low-vision players. That's not a criticism, just the reality of the medium. Games are visual, real-time and spatial, and the usual accessibility toolkit (semantic HTML, alt text, a sensible tab order) assumes a document, not a world you walk around in.

I want to highlight one approach we took to that problem, because I still think it's one of the more interesting design spaces in games: what does a genuinely screen-reader-friendly video game look like?

The game was Adventure, the browser-based, multiplayer pixel-art world we built for mulitple Google I/O events and its sister events for the Indie Games Accelerator and Indie Games Festival. I designed the accessibility feature together with the team at Set Snail, the agency that I collaborated with to build the project.

Here's the problem in one line: Adventure renders its game world into a single <canvas> with PixiJS. To a screen reader, a canvas is a big rectangle of nothing. Every NPC, every gift box, every teleporter is just pixels.

Our answer: build a second, invisible game out of plain HTML, and keep it in sync with the visible one.

The problem: a conference you can't attend

A virtual conference venue isn't really a game. It's a building with doors. Behind those doors are talks, sandboxes, office hours and swag. If you can't navigate the building, you can't attend the conference. And "sorry, the venue is a canvas" isn't a great accessibility statement for a developer event whose content is about building for everyone.

So the bar wasn't "add some aria-labels." It was: a keyboard or screen-reader user should be able to walk around, talk to people, open gifts and teleport across the map. The same verbs everyone else gets.

The trick: a parallel DOM, built by game components

The engine mounts one container on top of the canvas and then deliberately makes it invisible, but not hidden:

// Hide container from view, this is only for the screen-readers.
this._outputContainer.style.pointerEvents = "none";
// Here we use opacity instead of visibility:hidden and display:none;
// We still want the dom elements to be visible og interact-able by screen-readers
this._outputContainer.style.opacity = "0";

That one comment is the whole design in miniature. display: none and visibility: hidden remove things from the accessibility tree. opacity: 0 doesn't. Real buttons, real focus, real tab order. Just not painted.

What goes into that container is decided by the game itself. Any entity component can opt in by implementing a tiny interface:

export interface IA11ySettings {
  group?: string; // which <ul> this lands in
  alwaysRender?: boolean;
  priority?: number; // components on the same entity "fight" for the slot
  ignoreAutoFocus?: boolean;
  autoFocusRadiusInTiles?: number; // defaults to 8
}

export interface IAccessible extends Component {
  readonly a11ySettings: IA11ySettings;
  a11y(module: A11yHTMLItem): void;
}

Here's an NPC opting in. That's it. That's the whole integration:

public readonly a11ySettings: IA11ySettings = { group: 'dialog' };

public a11y(module: A11yHTMLItem): void {
  module.addInteractiveButton(
    'talk with ' + this.getPlugin(DialogPopupPlugin).getTranslatedName(this.state.npcId),
  );
}

addInteractiveButton creates a <li><button> inside a group container: a <div> with an <h2> and a <ul aria-labelledby> pointing at it. So a screen reader user hears something like "dialog, list, 3 items – talk with Ana, button". Headings for groups, lists for things, buttons for verbs. Boring HTML. Exactly what you want.

And the button's default click handler doesn't call some special accessibility path. It fires the same player interaction action a mouse click on the sprite would:

private interact = () => {
  const localPlayer = this._component.getPlugin(LocalPlayerPlugin);
  const action = PlayerInteractionBehaviour.actions.interact(this._component.gameObject.state.id);
  localPlayer.applyAction(action);
};

One code path for everyone. No parallel "accessible mode" that quietly rots after launch.

flowchart LR
  subgraph Canvas["PixiJS canvas (what you see)"]
    NPC["NPC sprite"]
    Gift["Gift box sprite"]
    WP["Waypoint sprite"]
  end
  subgraph DOM["opacity:0 DOM (what AT sees)"]
    G1["h2 dialog + ul"] --> B1["button: talk with Ana"]
    G2["h2 gifts + ul"] --> B2["button: Open gift"]
    G3["h2 waypoints + ul"] --> B3["button: Go north: Sandbox"]
  end
  NPC -- "a11y(module)" --> B1
  Gift -- "a11y(module)" --> B2
  WP -- "a11y(module)" --> B3
  B1 -- "focus" --> NPC
  B1 -- "click = same interact action" --> Server[("Game server")]

The bit I like most: focus flows back into the world

Here's the catch with invisible buttons. A sighted keyboard user tabs, and… nothing visibly happens. The focus ring is on an element with opacity: 0.

So we pushed focus the other direction. Every focus/blur on a DOM button raises an event on that entity's AccessibilityComponent, and the game draws the focus indicator:

• Sprites get a two-tone outline: a dark 0x202124 outer stroke and a white inner stroke via OutlineFilter, so it reads on both light and dark tiles.
• That outline wipes in bottom-to-top over 0.3s using a tiny custom fragment shader. The tween runs to 1.05 rather than 1.0 "to allow the black outline to be 0.05 ahead". Yes, we cared about that.
• NPC avatars play a highlight animation on a separate Spine track.
• Waypoints grow a little arrow that rotates to point at the destination of whichever button is focused. Tab between "Go north: Sandbox" and "Go east: Stage" and the arrow swings around with a power3.out ease, taking the short way round, too.
• Sound effects helped orientate players based on the type of floor they stepped on (e.g. footstep sounds were different based on grass, concrete, sand, etc.)

That waypoint arrow is the moment the system stops being a compliance feature and becomes a game mechanic. It's a focus ring that tells you where you're going. It passes the 5-foot test: someone looking over your shoulder on a train gets it immediately.

Focus that follows your feet

Tab order in a static page is easy. Tab order in a world where you're walking around is not. If you're standing next to a gift box, the gift box should be the first thing you reach, not whatever entity happened to load first.

So the a11y plugin listens to the player's position and, on every move, auto-focuses the closest interactive thing within range:

if (distance < item.getAutoFocusRadiusInTiles() && distance < closestDistance) {
  closestDistance = distance;
  closestItem = item;
}

The tuning knobs are where most of the design work went:

• Per-entity radius. Default is 8 tiles. Waypoints use 0.5. You have to basically stand on a teleporter before it grabs focus, otherwise walking past one would hijack your keyboard.
• ignoreAutoFocus. Other players get an "Inspect the lanyard of {nickname}" button, but they never steal focus. In a busy multiplayer room, people walking past you shouldn't yank your screen reader around.
• Wait for the camera. Focus doesn't move until the player is roughly centred in view. A later fix accounted for the camera's out-of-bounds offset near map edges. Otherwise the outline would appear on something still off-screen.
• Graceful fallback. If the focused entity despawns, focus drops back to the canvas instead of vanishing into <body>.

Making the canvas itself a good citizen

The canvas isn't ignored either. It's focusable and described, and the label updates on every zone change:

canvas.setAttribute("tabindex", "0");
canvas.setAttribute("aria-label", "Game view - " + zone);
canvas.setAttribute("role", "img");
canvas.setAttribute(
  "aria-description",
  "Move around using arrow keys or W A S D.",
);

And because keyboard movement only works while the canvas has focus, the game watches document.activeElement every frame. If focus wanders outside the game (say, into chat), it locks movement and shows a role="status" hint: focus the character to move around. Small thing. Saves a lot of "why won't my character move?" confusion, for everyone, not just keyboard users.

Overlays round it out: dialogs use role="dialog" + aria-modal, a priority-stacked focus trap (so a modal over an overlay over the game hands focus back down the stack correctly), a visually hidden <h1> for the page, and liberal aria-hidden on purely decorative stuff like the minimap and tooltips.

AI-assisted A11Y?

When we made Adventure it was just before the modern boom of Generative AI and last year I started to wonder how I would reimagine Adventure today with some of the AI technologies we now have. That's what led to AIventure which we demoed at Google I/O this year and included in the Gemma 4 demos.

I've published another article about AIventure here. It has the beginning of some ideas borrowed from Adventure's A11Y, such as being able to "chat" via the keyboard with Gemini or Gemma to interact with the world. This provides an interesting new approach to designing A11Y for games in my opinion, as we can now provide chat-driven tool calling for interacting with a game's UI and receiving a more textual description of the game world as we play it.

I feel this is a space where AI can greatly assist the user experience.

Conclusion

1. Don't make the canvas accessible. Make a DOM twin of it. Let each game entity describe itself as real HTML, and hide it with opacity: 0, never display: none.
2. Route focus both ways. DOM focus should drive in-world visuals (outlines, animations, a rotating arrow), so sighted keyboard users aren't tabbing blind. And interaction should reuse the exact same action path as a mouse click.
3. In a spatial world, tab order is a gameplay problem. Proximity-based auto-focus with per-entity radii and opt-outs (other players, teleporters) is what makes it feel playable rather than technically compliant.

Article published on: 1 Oct, 2026