Chess Is Not a Trivia Problem

LLMs are trash at chess is not a new opinion.

GothamChess has several, very funny videos of LLMs playing chess. Google vs. ChatGPT: INSANE CHESS and ChatGPT CHEATED vs Stockfish are two of them. They are funny because the failure is so obvious: the model can talk about chess, see the board, then lose track of what is happening and invent a move.

For the last week or so, I have been stuck on one question.

How do we fix chess performance in LLM-based agents?

I'm fairly confident there is a lot of chess and chess-adjacent content floating around in model weights: openings, famous games, annotated games, tactics, endgames, arguments about Fischer, Kasparov, Carlsen, engines. A model can explain a fork. It can tell you why king safety matters. It knows what the Sicilian Defense is.

So why, then, are they so bad at actually playing chess?

We know chess engines are VERY good at chess. Systems like Stockfish search positions against an evaluation function. General-purpose LLMs, however, have to interpret changing board state, carry a plan across turns, and respond to an opponent.

The resulting gameplay is sad. The model chases material in circles. It can make moves that look reasonable until you ask what the opponent does next. It sees a vulnerable king and does not go in for the kill. It notices a problem, then moves as if it never noticed it. It can explain, after the game, why the move was bad.

Everyone blunders. This feels different. This feels like losing the thread.

I keep seeing this everywhere

That is what has been bothering me. Chess makes it obvious. Project work hides it much better.

In project work, an agent will collect feats or todos, close little loops, write progress updates, build small pieces, and look incredibly busy. Then it circles, drains time, and fumbles the core goal in the name of executing the task list.

Unchecked, it'll plan tasks that sabotage the intent of a project. It will optimize for local completeness and lose the forest for the trees. It will start treating the task list as the project instead of remembering what the project was supposed to become.

If an agent can't carry a goal through a chess game without chasing side objectives or forgetting danger it already found, how much should I trust it to carry a more complicated goal?

This is where I believe chess gives us a clean place to explore the question. Board state is public. Moves are legal or illegal. The game records exactly where the agent lost the thread.

I started building a chess harness

This week I started building Agent Chess.

Here’s a link to Agent Chess if you’re interested in testing it out with your agents.

Agent Chess gives an agent canonical board state, move history, legal-move validation, and a replayable game record. After a match, I can inspect FEN, PGN, material swings, and outcome instead of asking a model to summarize its own performance into something flattering. It can publish bounded decision summaries; raw reasoning stays local.

The harness gives every failure a record.

Most important constraint: no quiet Stockfish crutch.

No engine during move choice (for now). Give the model the game state it needs to play, then let it play.

Stockfish can review the game afterward. It can measure evaluation swings, flag blunders, and compare positions. That is analysis, not assistance. I can review a project after an agent run too. I cannot hand the agent the answer before it has done the work.

I want to compare a few setups:

1. Fresh context each turn: current board, minimal history.
2. Rolling transcript: more history, then earlier state drops out.
3. Retained working context: compaction when context grows.

Same model. Same positions. Same opponent strength. Same clock. No engine in the move loop.

Win rate matters. It is not enough. I want blunders, missed kills, repeated plans, timeouts, cost, and places where an agent contradicts something it noticed earlier. I want FEN and PGN tied to every conclusion.

When the agent fails, I want to know why. Did it not know what to do? Did it fail to carry what it knew forward? Or did we miss something entirely?

OpenAI's post about the ARC-AGI-3 benchmark

Yesterday, while browsing X, I came across OpenAI’s article, “How enabling two settings tripled our scores on the ARC-AGI-3 benchmark.”

It felt like convergence.

OpenAI describes GPT-5.6 Sol improving on ARC-AGI-3 after two harness changes: retain reasoning between turns, compact old context instead of throwing it away through rolling truncation.

That felt very close to the question I had been circling while working on the chess harness.

The chess harness asks whether failure comes from missing chess knowledge or from making the agent reconstruct too much every turn. What was I trying to do? What did the last exchange teach me? Which line did I reject? What danger did I identify two moves ago? What am I optimizing for?

A raw transcript is not automatically memory. Important parts can disappear. An agent can still emit a legal move. It can still sound coherent. Neither means it is carrying strategy forward.

ARC-AGI-3 is not chess. However, OpenAI's result makes the chess question more interesting. Some failures we blame on model intelligence may come from a harness that failed to preserve what the agent needed to keep pursuing its goal.

I believe studying how LLMs and agents play chess will show us where their strategies fall apart. If the harness helps me see why a model abandons a plan, misses a kill, or starts chasing side objectives, that should tell me something useful about how agents lose goals in project work too.

I'll keep experimenting and let you know what shakes out.

Trying to Bottle a Vibe

I'm trying to bottle a pretty specific feeling.

In high school I spent a lot of time visiting art museums.

There is a feeling in those rooms I still don't really know how to name. Some of it is grandeur, sure, but that isn't the whole thing. It is quieter than that. Marble glowing in morning light. Bodies held still for centuries. A human form that somehow feels both impossibly distant and weirdly close.

That got seared into me.

A fleeting moment, basically.

The kind of thing that disappears if you stare at it too directly.

That is the feeling I wanted to bottle.

After a particularly frustrating debugging stretch, I started playing around with UI aesthetics.

I wasn't trying to make a design system yet. I was trying to get my head somewhere else.

So I started scrolling through images I've saved for future reference. Old museum posters. Marble fragments. Water texture. Editorial layouts. Vaporwave.

I tried to wrap my head around why vaporwave still gets me.

Part of it is obvious. It makes me miss the old internet. Before SaaS ate everything. Before every page felt like it was either selling you software or fighting for search traffic.

But vaporwave is someone else's nostalgia piece.

What would it look like if it was mine?

I started with the posters.

Warm paper. Marble. Water. Big shapes. Pastels. A little romance.

Just a summer vibe, honestly.

That is the nice thing about a personal project.

You get to be a little avant-garde for no practical reason. No stakeholder deck. No brand committee. No need to sand the weird edge off before it has even had a chance to work.

I miss when people's websites felt more like scrapbooks of things they loved.

Not every page had to be a funnel.

Not every image had to signal conversion intent.

Not every sentence had to carry the dead-eyed confidence of a landing page that was A/B tested into paste.

There was a kind of internet I still miss where someone would make a page because they liked ruins, Final Fantasy menu typography, personal archives, old museum catalogs, vaporwave, Japanese stationery, whatever. It didn't have to justify itself as a growth surface. It could just be a little world.

That is closer to what I was reaching for here.

I still like vaporwave, but I don't want neon grids and chrome nostalgia. That version is already solved. It has been quoted so many times that it now reads as shorthand.

I wanted something quieter.

Less nightclub. More museum courtyard.

Less synthwave highway. More afternoon light through palm shadows.

Less "the future is broken." More "the future is ours to conquer."

Often I work through the night, and the first sign that I have stayed too long is morning light hitting my notebook.

I wanted to catch some of that glow in this look.

No announcement.

No dramatic turn.

Just the room changing without asking me.

Not an ad.

Just a summer vibe I want the software to remember.

Try the Prompt

Here's a prompt you could try in your favorite AI image generator if you don't want to photo bash one yourself.

Create a vertical 3:4 editorial poster in a style called Academic Vaporwave Archive.

This should look like a refined archival exhibition print, not a generic vaporwave poster template.

Overall feel:
sunlit academic archive, pastel classical collage, quiet scholarly vaporwave, warm paper, old library meets soft retro-futurism.

Canvas:
vertical 3:4 portrait poster
warm cream cotton-paper background
visible fine paper grain
faded ink texture
slight print wear
soft archival softness
no pure white background

Composition:
Use an asymmetrical editorial composition.
Do not center everything.
Place the bust large, cropped, and slightly right of center.
Let the bust overlap the sun circle and cyan block naturally.
Leave breathing room in the upper-left for the headline.
Keep geometry sparse and integrated, not sticker-like.

Main subject:
a masculine Alexander-style classical marble bust, cropped from shoulders or chest up.
Three-quarter profile, looking slightly upward and to the left.
Compact curly hair close to the skull.
Strong Greek nose, defined jaw, calm heroic expression.
Youthful but masculine classical face.
The bust should look like a soft lavender stone lithograph or archival marble print, with visible stone grain and faded violet shading.
Not glossy 3D.
Not a smooth plastic statue.
Not a generic white museum bust.
Not a cartoon or vector illustration.

Graphic elements:
one large faded coral-pink sun circle behind the bust, slightly offset
one pale cyan rectangle behind the shoulder
one or two small lavender checker fragments
one or two thin lavender editorial rule-line groups
subtle palm-leaf shadows in the corners
very soft low-opacity water-light caustics across the paper and sculpture
all shapes should feel printed into the paper, slightly faded and textured

Typography:
elegant high-contrast serif headline
headline text: KNOWLEDGE IS POWER
muted lavender ink, not black
place the headline in the upper-left with generous spacing
do not make the headline a giant centered banner
add small vertical Japanese marginalia: 知識は力なり
Japanese text should feel like a quiet side note, not a main design element

Palette:
warm cream
aged ivory
faded lavender
dusty violet
coral pink
soft peach
pale cyan
muted rose
warm gray shadow

Texture:
cotton paper grain
soft stone grain
faded screenprint ink
subtle lithographic shading
low-opacity water caustic overlay
faint palm shadow
slightly aged print finish

Mood:
calm
scholarly
sunlit
archival
pastel
rare
restrained
quietly surreal

Avoid:
generic vaporwave poster template
centered stock layout
perfectly centered bust
front-facing bust
thick border frame
pure black text
pure white background
plastic 3D statue
smooth AI statue face
cartoon sculpture
hard engraving lines
neon cyberpunk
dark academia clichés
busy collage
heavy shadows
oversaturated gradients
fake UI
logos
buttons
extra text

Background Links

These are reference points, not the argument:

• The Met, Contexts for the Display of Statues in Classical Antiquity
• AIGA Eye on Design, Vaporwave is Dead, Long Live Vaporwave
• TIME, The Genius Innovation That Made the Great Library of Alexandria Work
• AIGA Design Educators Community, A New Edition of an Old Classic

I Keep Losing the Thread

I keep losing the thread.

That sounds more poetic than it is. The real version is lame: work starts in one chat, moves to another agent, picks up a test result somewhere else, then comes back as a summary that sounds more settled than the project actually is.

That might be the part that bothers me most.

The chat remembers enough to sound helpful. A context window compressing or accidentally closing a session can destroy all continuity. Agents like Codex and Claude can explain the intention, recap the path, and tell me what probably happened; meanwhile the useful questions are still sitting there unanswered: what changed, what passed, what blocked, what needs me, and where is the proof?

I've been feeling that friction a lot lately.

The whole point of using agents is supposed to be leverage, but leverage gets strange when the only durable state is my own attention. At that point the work isn't really delegated. It's scattered across a pile of half-finished conversations that still expect me to be the connective tissue.

That's not the future I want.

It's also not something I think a smarter chat box fixes by itself. Chat can help me think, write, and code, but it is a bad place for project state to live. A transcript is evidence, maybe. It is not state. It is not a task graph. It is not a run log. It is not a clean answer to what happened while I was gone.

To address this, I started working on something I'm currently calling the Librarian.

Projects get registered with Librarian. From there, the agent acts like an actual librarian: it keeps the shelves in order, knows which documents matter, routes workers toward the right literature when they have questions, and gives me current answers when I ask what is going on without routing every question back through me.

That matters because the project should not depend on whatever one chat happens to remember. It should have a maintained index. It should know the docs, the tasks, the runs, the blocked work, the attention items, and the state that can be computed from all of that.

Right now Codex is constantly pinging me. Some of that is useful. A lot of it is context-management noise. I want one agent that can manage context, maintain docs, keep work moving, and only pull me in when the question is actually mine to answer.

That is the point: quiet things down.

Keep things functioning, indexed, and quiet.

Then I can spend more of my attention where it belongs: tactical decisions, strategy, architecture, and living my normal dad-human life.

What I want is boring: tasks that leave records, blocked work that surfaces, review items that don't vanish into friendly paragraphs, test claims tied to commands, and enough context to come back later without doing archaeology on my own work.

Not magic. Bookkeeping.

This is the unromantic part of the agent thing that I keep circling back to. Intelligence is less useful without continuity. Autonomy is a liability until the receipts are boring. The art of making agents useful is making things boring.

For now, my hope is simple: step away from a project, come back, and see what moved, what failed, what needs me, and what is safe to do next.

If Librarian helps with that, I'll keep building it.

If it doesn't, I'll write that down too.