Chess Is Not a Trivia Problem

LLMs are trash at chess is not a new opinion.

GothamChess has several, very funny videos of LLMs playing chess. Google vs. ChatGPT: INSANE CHESS and ChatGPT CHEATED vs Stockfish are two of them. They are funny because the failure is so obvious: the model can talk about chess, see the board, then lose track of what is happening and invent a move.

For the last week or so, I have been stuck on one question.

How do we fix chess performance in LLM-based agents?

I'm fairly confident there is a lot of chess and chess-adjacent content floating around in model weights: openings, famous games, annotated games, tactics, endgames, arguments about Fischer, Kasparov, Carlsen, engines. A model can explain a fork. It can tell you why king safety matters. It knows what the Sicilian Defense is.

So why, then, are they so bad at actually playing chess?

We know chess engines are VERY good at chess. Systems like Stockfish search positions against an evaluation function. General-purpose LLMs, however, have to interpret changing board state, carry a plan across turns, and respond to an opponent.

The resulting gameplay is sad. The model chases material in circles. It can make moves that look reasonable until you ask what the opponent does next. It sees a vulnerable king and does not go in for the kill. It notices a problem, then moves as if it never noticed it. It can explain, after the game, why the move was bad.

Everyone blunders. This feels different. This feels like losing the thread.

I keep seeing this everywhere

That is what has been bothering me. Chess makes it obvious. Project work hides it much better.

In project work, an agent will collect feats or todos, close little loops, write progress updates, build small pieces, and look incredibly busy. Then it circles, drains time, and fumbles the core goal in the name of executing the task list.

Unchecked, it'll plan tasks that sabotage the intent of a project. It will optimize for local completeness and lose the forest for the trees. It will start treating the task list as the project instead of remembering what the project was supposed to become.

If an agent can't carry a goal through a chess game without chasing side objectives or forgetting danger it already found, how much should I trust it to carry a more complicated goal?

This is where I believe chess gives us a clean place to explore the question. Board state is public. Moves are legal or illegal. The game records exactly where the agent lost the thread.

I started building a chess harness

This week I started building Agent Chess.

Here’s a link to Agent Chess if you’re interested in testing it out with your agents.

Agent Chess gives an agent canonical board state, move history, legal-move validation, and a replayable game record. After a match, I can inspect FEN, PGN, material swings, and outcome instead of asking a model to summarize its own performance into something flattering. It can publish bounded decision summaries; raw reasoning stays local.

The harness gives every failure a record.

Most important constraint: no quiet Stockfish crutch.

No engine during move choice (for now). Give the model the game state it needs to play, then let it play.

Stockfish can review the game afterward. It can measure evaluation swings, flag blunders, and compare positions. That is analysis, not assistance. I can review a project after an agent run too. I cannot hand the agent the answer before it has done the work.

I want to compare a few setups:

1. Fresh context each turn: current board, minimal history.
2. Rolling transcript: more history, then earlier state drops out.
3. Retained working context: compaction when context grows.

Same model. Same positions. Same opponent strength. Same clock. No engine in the move loop.

Win rate matters. It is not enough. I want blunders, missed kills, repeated plans, timeouts, cost, and places where an agent contradicts something it noticed earlier. I want FEN and PGN tied to every conclusion.

When the agent fails, I want to know why. Did it not know what to do? Did it fail to carry what it knew forward? Or did we miss something entirely?

OpenAI's post about the ARC-AGI-3 benchmark

Yesterday, while browsing X, I came across OpenAI’s article, “How enabling two settings tripled our scores on the ARC-AGI-3 benchmark.”

It felt like convergence.

OpenAI describes GPT-5.6 Sol improving on ARC-AGI-3 after two harness changes: retain reasoning between turns, compact old context instead of throwing it away through rolling truncation.

That felt very close to the question I had been circling while working on the chess harness.

The chess harness asks whether failure comes from missing chess knowledge or from making the agent reconstruct too much every turn. What was I trying to do? What did the last exchange teach me? Which line did I reject? What danger did I identify two moves ago? What am I optimizing for?

A raw transcript is not automatically memory. Important parts can disappear. An agent can still emit a legal move. It can still sound coherent. Neither means it is carrying strategy forward.

ARC-AGI-3 is not chess. However, OpenAI's result makes the chess question more interesting. Some failures we blame on model intelligence may come from a harness that failed to preserve what the agent needed to keep pursuing its goal.

I believe studying how LLMs and agents play chess will show us where their strategies fall apart. If the harness helps me see why a model abandons a plan, misses a kill, or starts chasing side objectives, that should tell me something useful about how agents lose goals in project work too.

I'll keep experimenting and let you know what shakes out.