How to use Claude to play native macOS chess
Claude Code and Codex played native macOS chess through accessibility labels and MCP. A deterministic referee rejected illegal moves before real UI actions.
medium.com
Suggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Build a local MCP bridge for macOS Chess using accessibility square labels and AX press actions. Validate every proposed move with python-chess before touching the UI, return illegal-move errors to the agent and verify complete matches without relying on screenshots. Keep host-control permissions explicit.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- Janek Lopez
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The author reports complete legal native-app chess matches with structured accessibility input and deterministic move checks. The bridge was a macOS-only quick demo.
- Human involvement
- Lopez inspected accessibility controls, tested the bridge and imposed deterministic legal-move validation before allowing UI actions.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Lopez inspected the Chess app's accessibility labels and used them to reconstruct board state.
- Claude Code built an MCP bridge that pressed source and destination squares.
- python-chess rejected illegal proposals before UI actions, and narration paced the demonstrated matches.
Janek Lopez used Claude Code to build an MCP bridge that let Claude Code and Codex play the native macOS Chess application. Screenshot-and-coordinate experiments had been slow and fragile, so he inspected the operating system's accessibility interface instead.
Each board square exposed a text label describing the piece and coordinate. The bridge read those labels to reconstruct the board, then triggered accessibility press actions on source and destination squares. This removed the need to recognize the board from pixels on every turn.
Lopez placed python-chess between the model and the real interface. An illegal proposed move returned an error to the agent before the application received any click. That strict referee reduced desynchronization rather than trusting a fluent explanation of the move. A local voice narrator read each agent's commentary, and waiting for speech also paced the moves.
The author reports that two models could complete legal games through the native app and published the bridge. They did not play at grandmaster strength. The result is a specific structured-interface experiment with a deterministic action gate. It was tested only on macOS and requires an application that actually supplies useful accessibility labels; arbitrary canvas interfaces do not meet that condition.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Medium · Janek Lopez
Frequently asked questions
Did the agents use screenshots?
The final bridge read structured macOS accessibility labels.
What prevented illegal UI moves?
python-chess validated proposals before the bridge pressed board squares.
Does this work with every app?
No. It depends on useful accessibility labels and was tested only on macOS.
More details
About this account
Lopez inspected the Chess app's accessibility labels and used them to reconstruct board state. Claude Code built an MCP bridge that pressed source and destination squares.
Original account opened. The author reports full legal games, not strong chess performance. Tested on macOS only and dependent on meaningful accessibility labels. Suggested prompt written by Agents In Use; no original task prompt was published in the verified account.
Credit: Janek Lopez — Medium
Source published
Added to Agents In Use
Source checked
Tools and topics
Claude Code · Codex · macOS Accessibility API · python-chess · Kokoro TTS
Native app control · Legal-move gate
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.