LLM benchmark

BAA-chmark: the same prompt, different games.

BAA-chmark compares how different models handled the assignment to create a short 3D arcade game about herding sheep.

Play the top result · Claude Sonnet 5.5

Updated October 20, 2026Reading time 7 min

Results: 66
Playable results: 39
Demos to try: 65
Method: Same prompt

Claude Sonnet 5.5

Claude Opus 5.5

DeepSeek V4.1 Flash 1017

Kimi K3

GLM 5.3 Flash

GPT-6 Astra

Union Alpha

Qwen 3.8 Flash Q2

Gemini 4 Argon

In brief

Every model received the same assignment to create a short 3D sheep-herding game.
The comparison covers screenshots, playability, bugs, and demos that can be launched directly.
Claude Sonnet 5.5 currently holds first place.
Original model outputs remain preserved; the manually completed game is listed separately.

Full ranking

Results and playable demos

The default ranking reflects the practical result. Filters keep the ranking intact and only hide runs that do not match.

Filters

Model

Benchmark ranking Subjective ranking by practical result. Playability and correct controls are evaluated strictly, so visually stronger games with broken controls rank lower.
Model	Type	Parameter count / requirements**	Result	Bugs	Note	Demo
1Claude Sonnet 5.5	Closed	—	Matched the assignment Playable	Few issues	Session report: 9 h 47 min from the task to the report snapshot, including 8 h 06 min with the lead agent or subagents active; this is not the time to the first output alone. 37 agent runs, 182.14M price-equivalent tokens; USD 364.28 estimated at API list prices, not a confirmed invoice.Show moreShow less Session report: 9 h 47 min from the task to the report snapshot, including 8 h 06 min with the lead agent or subagents active; this is not the time to the first output alone. 37 agent runs, 182.14M price-equivalent tokens; USD 364.28 estimated at API list prices, not a confirmed invoice.	Demo
2Claude Opus 5.5	Closed	—	Matched the assignment Playable	No visible issues	About 2 h 35 min generation after the first prompt; 2 optional follow-up prompts adjusted difficulty, late-game pen entry, and audio volume.Show moreShow less About 2 h 35 min generation after the first prompt; 2 optional follow-up prompts adjusted difficulty, late-game pen entry, and audio volume.	Demo
3Claude Opus 5	Closed	—	Matched the assignment Playable	No issues found	-	Demo
4Claude Fable 5.1	Closed	—	Matched the assignment Fully playable	Minimal minor issues	About 45 min generation after the first prompt; USD 12.56; 200K context.	Demo
5Claude Fable 5	Closed	—	Matched the assignment Fully playable	Minimal, a few minor issues	Reasoning effort 4/6, not maximum. Small Mac viewport fix.	Demo
6DeepSeek V4.1 Flash 1017	Open-weight	—	Matched the assignment Playable	Many minor issues	Generation after the first prompt took about 1 h 45 min including two resumptions after stopping; OpenCode via OpenRouter, USD 1.56, over 450K context with compaction.Show moreShow less Generation after the first prompt took about 1 h 45 min including two resumptions after stopping; OpenCode via OpenRouter, USD 1.56, over 450K context with compaction.	Demo
7Kimi K3	Open-weight	2.8T 1.56 TB ≈147.9 GB / 100K	Matched the assignment Fully playable	Minimal, minor issues	Generation cost: USD 7.31.	Demo
8GLM 5.3 Flash	Open-weight	320B ≈196.6 GB	Matched the assignment Playable	Few minor issues	Generation stopped twice and required short prompts to continue. The first-prompt generation took about 75 min.Show moreShow less Generation stopped twice and required short prompts to continue. The first-prompt generation took about 75 min.	Demo
9GPT-6 Astra	Closed	—	Matched the assignment Playable	Minor issues	Generation after the first prompt took 23 min.	Demo
10Union Alpha	Closed	—	Matched the assignment Playable	No major issues	Time after the first prompt: ≈27 min · ≈USD 3 · 1 continuation prompt. Later revealed as Pareto 26.9 (Circuit & Chisel).Show moreShow less Time after the first prompt: ≈27 min · ≈USD 3 · 1 continuation prompt. Later revealed as Pareto 26.9 (Circuit & Chisel).	Demo
11Qwen 3.8 Flash Q2	Open-weight	180B ≈110.1 GB	Matched the assignment Playable	Few issues	Time after the first prompt: 4 h 38 min · Q2 · max. 262K context with compaction · 2 corrective prompts after stopping. Exact Q2 artifact/revision not supplied.Show moreShow less Time after the first prompt: 4 h 38 min · Q2 · max. 262K context with compaction · 2 corrective prompts after stopping. Exact Q2 artifact/revision not supplied.	Demo
12DeepSeek V4.1 Flash	Open-weight	763.2B ≈466.9 GB	Matched the assignment Playable	Many issues	Generation after the first prompt took 1 h 55 min; cost USD 0.30.	Demo
13Gemini 4 Argon	Closed	—	Matched the assignment Playable	Many issues	Generation after the first prompt took 20 min 52 s; Google Antigravity, high effort, 95K context, about USD 1.45.Show moreShow less Generation after the first prompt took 20 min 52 s; Google Antigravity, high effort, 95K context, about USD 1.45.	Demo
14Qwen 3.8 27B ThinkingCap	Open-weight	27B 19 GB ≈6.71 GB / 100K	Matched the assignment Playable	Few issues	About 8 interruptions during generation due to file-writing problems; resumption required smaller blocks. Context reached 120K tokens. Exact generation time and checkpoint revision are unavailable.Show moreShow less About 8 interruptions during generation due to file-writing problems; resumption required smaller blocks. Context reached 120K tokens. Exact generation time and checkpoint revision are unavailable.	Demo
15GPT-6.0 Sol	Closed	—	Matched the assignment Playable	Many bugs	-	Demo
16GPT-6.1 Sol	Closed	—	Matched the assignment Playable	Many bugs	-	Demo
17GPT-5.6 Sol	Closed	—	Matched the assignment Playable	Few issues	-	Demo
18Claude Opus 4.8	Closed	—	Matched the assignment Playable	Few issues	-	Demo
19Muse Spark 1.3	Closed	—	Matched the assignment Playable	Many issues	Generation after the first prompt took 3 h 55 min; cost USD 0.33.	Demo
20Grok 4.7	Closed	—	Matched the assignment Playable	Very many issues	Generation after the first prompt took about 54 min; 109K context, about USD 6.	Demo
21Grok 4.6	Closed	—	Partially matched the assignment Playable	Many serious issues	Generation after the first prompt took 19 min; cost USD 1.54.	Demo
22Grok 4.5	Closed	—	Matched the assignment Playable	No obvious serious issue	-	Demo
23Gemini 3.8 Flash	Closed	—	Matched the assignment Playable	Many minor issues	About 7.5 min generation after the first prompt; USD 0.58.	Demo
24Gemini 3.5 Flash	Closed	—	Matched the assignment Playable	Few issues	Required 1 corrective prompt.	Demo
25GPT-6 Luna	Closed	—	Matched the assignment Playable	Very many bugs	1 corrective prompt to display the game.	Demo
26Qwen 3.8 27B	Open-weight	27B 19 GB ≈6.71 GB / 100K	Matched the assignment Playable	Few minor issues	Generation after the first prompt took 4 h and stopped at 187K context tokens. Early on, it needed two short prompts to continue.Show moreShow less Generation after the first prompt took 4 h and stopped at 187K context tokens. Early on, it needed two short prompts to continue.	Demo
27Gemini 3.6 Flash	Closed	—	Matched the assignment Playable	No serious bugs	-	Demo
28Qwen 3.6 27B ThinkingCap	Open-weight, GGUF	27B 16.8 GB ≈6.71 GB / 100K	Partially matched the assignment Playable	No serious bugs	ThinkingCap modification by Bottlecap. Generation after the first prompt took 17 minutes; no corrective prompt.Show moreShow less ThinkingCap modification by Bottlecap. Generation after the first prompt took 17 minutes; no corrective prompt.	Demo
29Muse Spark 1.2	Closed	—	Partially matched the assignment Playable	Many bugs; freezes after a while	Time after the first prompt: 4 min 19 sec.	Demo
30GPT-5.6 Terra	Closed	—	Matched the assignment Playable	Many bugs	Time after the first prompt: 20 min.	Demo
31Qwen 3.5 397B A17B	Open-weight	397B 235.7 GB ≈3.27 GB / 100K	Partially matched the assignment Playable	Many serious bugs	-	Demo
32MiniMax M2.7	Open-weight	229.9B ≈139.9 GB ≈25.4 GB / 100K	Partially matched the assignment Playable	Many serious bugs	Characters barely move.	Demo
33Qwen 3.8 Max Preview	Closed	2.4T	Fulfilled most of the assignment Playable	Unusual controls, few other issues	Generation stopped unexpectedly before completion.	Demo
34DeepSeek V4 Pro 0831	Open-weight	1.6T 864.7 GB ≈1.01 GB / 100K	Fulfilled most of the assignment Playable	Many issues; unusual controls	Generation after the first prompt took more than 2 h and cost USD 6.11. After stopping once, it needed a short prompt to continue.Show moreShow less Generation after the first prompt took more than 2 h and cost USD 6.11. After stopping once, it needed a short prompt to continue.	Demo
35DeepSeek V4 Flash 0731 DS4	Open-weight	284B 164.6 GB ≈1.22 GB / 100K	Partially matched the assignment Playable	Unusual camera and many bugs	A more memory-efficient Q2 quantization was used, fitting within 96 GB of RAM. Time after the first prompt: 1 h 49 min.Show moreShow less A more memory-efficient Q2 quantization was used, fitting within 96 GB of RAM. Time after the first prompt: 1 h 49 min.	Demo
36GLM 5.3 Flash + Claude Code	Open-weight through Claude Code	320B ≈196.6 GB	Partially matched Playable with problems	Many bugs; reversed controls	GLM 5.3 Flash through Claude Code, max effort. Generation after the first prompt took 2 h 49 min; launch succeeded only after several attempts.Show moreShow less GLM 5.3 Flash through Claude Code, max effort. Generation after the first prompt took 2 h 49 min; launch succeeded only after several attempts.	Demo
37Fable 5 UltraCode	Closed	—	Matched the assignment Playable	Distant camera	UltraCode mode.	Demo
38Gemini 3.7 Flash	Closed	—	Partially matched the assignment Not playable	Many issues, including reversed forward and backward movement	Required 1 short prompt to continue generating the second level. Generation after the first prompt took about 7 min.Show moreShow less Required 1 short prompt to continue generating the second level. Generation after the first prompt took about 7 min.	Demo
39DeepSeek V4 Flash Vision Exp	Closed	—	Partially matched the assignment Unplayable	Many serious issues	Generation after the first prompt took 11 min; cost approximately 0.64 cents. Two corrective prompts were required; the second restored the fences, but they remained broken.Show moreShow less Generation after the first prompt took 11 min; cost approximately 0.64 cents. Two corrective prompts were required; the second restored the fences, but they remained broken.	Demo
40GPT-5.5	Closed	—	Partially matched the assignment Not playable	Many bugs	-	Demo
41MiniMax M3	Open-weight	428B ≈261.3 GB	Partially matched the assignment Not playable	Many bugs	-	Demo
42Claude Sonnet 4.6	Closed	—	Partially matched the assignment Not playable	Several issues	Generation after the first prompt took unusually long, over an hour.	Demo
43GLM 5.2	Open-weight (OpenRouter)	744B 464.8 GB ≈9.52 GB / 100K	Partially matched the assignment Not playable	Many bugs	-	Demo
44MiMo v2.6 Pro	Open-weight	—	Partially matched the assignment Unplayable	Very many serious bugs	Generation cost USD 0.12.	Demo
45MiMo v2.6 Flash	Open-weight	—	Partially matched the assignment Unplayable	Very many serious bugs	Generation cost USD 0.03.	Demo
46Mistral Large 4	Open-weight	—	Partially matched the assignment Unplayable	Very many critical issues	Generation after the first prompt took about 1 h 55 min including 4–5 resumptions after stopping; Mistral Vibe via OpenRouter, high effort (not max), 49.4K context, USD 1.22.Show moreShow less Generation after the first prompt took about 1 h 55 min including 4–5 resumptions after stopping; Mistral Vibe via OpenRouter, high effort (not max), 49.4K context, USD 1.22.	Demo
47DeepSeek V4 Flash	Open-weight (OpenRouter)	284B 159.6 GB ≈0.71 GB / 100K	Partially matched the assignment Not playable	Many bugs	-	Demo
48Qwen 3.7 Max	Closed	—	Partially matched the assignment Not playable	Very many bugs	Required 1 corrective prompt.	Demo
49DeepSeek V4 Pro	Open-weight (OpenRouter)	1.6T 864.7 GB ≈1.01 GB / 100K	Partially matched the assignment Not playable	Many bugs	Required 1 corrective prompt.	Demo
50Big Pickle	Unknown	—	Partially matched the assignment Not playable	Many bugs	-	Demo
51Kimi 2.7 Code	Open-weight (OpenRouter)	1T 595.2 GB	Partially matched the assignment Not playable	Many bugs	Required 3 corrective prompts.	Demo
52Qwen 3.6 27B MLX	Open-weight	27B 19.1 GB ≈6.71 GB / 100K	Partially matched the assignment Unplayable	Many bugs	MLX run; 1 corrective prompt.	Demo
53Qwen 3.6 27B	Open-weight	27B 19.1 GB ≈6.71 GB / 100K	Partially matched the assignment Not playable	Many bugs	-	Demo
54Qwen 3.5 122B A10B	Open-weight	122B 78.9 GB ≈2.61 GB / 100K	Partially matched the assignment Not playable	Very many bugs	Both the sheep and dog fly around.	Demo
55GLM 4.7 Flash	Open-weight (OpenRouter)	30B ≈19.1 GB ≈5.41 GB / 100K	Partially matched the assignment Not playable	Many bugs	Required 1 corrective prompt.	Demo
56Qwen 3.6 35B A3B	Open-weight	35B ≈21.4 GB ≈2.11 GB / 100K	Partially matched the assignment Not playable	Many bugs	Required 1 corrective prompt.	Demo
57Muse Glimmer 30B	Open-weight	29.6B 16.8 GB ≈1.41 GB / 100K	Partially matched the assignment Playable	Very many serious issues; static camera	Generation after the first prompt took 6 minutes.	Demo
58Gemma 4 31B	Open-weight	30.7B 17.7 GB ≈9.03 GB / 100K	Matched the assignment only very partially Not playable	Many bugs	Required 1 corrective prompt.	Demo
59Laguna S 2.1	Open-weight	118B 71.9 GB ≈4.99 GB / 100K	Did not match the assignment Unplayable	Very many serious bugs; crashes	Required 5 corrective prompts; generation after the first prompt took 1 h 31 min.	Demo
60GPT-5.6 Luna	Closed	—	Did not match the assignment Playable	A few bugs	Time after the first prompt: 13 min 26 sec.	Demo
61MiMo v5	Open-weight	—	Partially matched the assignment Not playable	Many bugs	Required 1 corrective prompt.	Demo
62Mistral Small 4	Open-weight	119B 70.8 GB ≈2.3 GB / 100K	Did not match the assignment Completely unplayable	Very many serious bugs	Required 3 corrective prompts.	Demo
63Ornith 1.0 35B	Open-weight	35B 21.2 GB ≈2.11 GB / 100K	Partially matched the assignment Completely unplayable	Many bugs	Required 2 corrective prompts.	Demo
64Qwen 3.5 A3B	Open-weight	—	Partially matched the assignment Completely unplayable	Many bugs	-	Demo
65Qwen 3.6 27B GGUF	Open-weight	27B 19.1 GB ≈6.71 GB / 100K	Partially matched the assignment Unplayable	Crash when starting a level	GGUF run.	Demo
66Nemotron 3 Ultra	Open-weight	550B 352.3 GB	Did not match the assignment Produced nothing	—	-	—

**Shown when available: parameter count, Q4 weight size at 0K, and cache/state at 100K. We use a concrete Q4 checkpoint or a labeled Q4_K_M estimate; 100K assumes one sequence in the documented runtime. Weight quantization does not change the cache while the runtime keeps the same cache layout and data types. Hybrid models can differ substantially in another runtime. Unverifiable values are omitted, and a lone dash means no value is known. Runtime buffers, vision components, and offload are excluded; this is not a universal minimum RAM/VRAM requirement.

Notes refer to corrective prompts needed to launch the game (for example a JavaScript error) and to any run settings such as reasoning effort. No corrective prompt was used to tune the game itself.

On mobile or tablet, Demo opens an arrow and A/B control panel; a game with built-in touch controls opens the original. The original benchmark code remains unchanged.

Download benchmark results as JSON

Method

Assignment used for the comparison

The complete prompt is documented below; where a translation is shown, the original wording remains available. Evaluation is based on actually launching the output, playability, bugs, and constraints of each run.

Prompt used for the comparison

The models received the original Slovak prompt. A faithful English translation is shown below.

I would like to create a simple arcade game that feels a little like a Nintendo game, but in 3D, from a third-person perspective with a somewhat distant camera. The game should be set in mountain meadows, on hilly, rolling terrain where sheep graze. Most of the area should consist of meadows, with mountains, forests, and a few houses in the background. The player should control a sheepdog or another type of dog used for herding sheep, with a choice of several breeds that differ in some of their traits. The player should be able to move with the arrow keys and bark using the space bar or another suitable keyboard key. Barking should also affect the sheep, if that realistically helps with herding, and should scare away other animals—for example other dogs, wolves, and a bear—as well as people, except for the shepherd, who should also be present in the meadow. The sheep should react to the dog: they should move as a flock, while also moving in response to the way the dog herds them. They should bleat (“baa”), and the game should have an arcade feel. The main objective should be to guide the sheep into a pen, from point A to point B, while facing hazards such as occasional attacks by other dogs and, more rarely, wolves or a bear. Wolves and the bear should act as bosses, or simply as larger obstacles in later levels. Come up with smaller, easier obstacles for the early levels. There should be a reasonable number of levels; keep in mind that this is meant to be a short game for fun, lasting a few minutes, at most a few dozen minutes, or perhaps one hour if played all the way through the final level. Other animals could also appear on the map; include, for example, cows, foxes, and the occasional bird—in short, nature. In addition to the meadows, include a dirt road and occasional obstacles such as fences, cliffs, and bushes that the sheep cannot pass through. The graphics should be attractive, colorful, and arcade-like, somewhat reminiscent of a Nintendo game, but do not copy any specific game or its design. Write the game in HTML/JavaScript.

Show full English translation Hide English translation

Show original Slovak prompt Hide original Slovak prompt

Chcel by som vytvorit jednoduchu arkadovu hru, trocha akoby to bola nintendo hra ale v 3D s pohladu z tretej osoby ale trocha vzdialena kamera. Hra bude v prostredi luk v horach na kopcovitom vlnenom terene kde sa pasu ovce, Vacsina plochy budu luky, v pozadi hory a lesy a nejake domy. Hrac bude hrat za ovciarskeho alebo ineho psa ktory sa pouziva pri zahanani oviec, nech si vie vybrat z par plemien ktore sa budu lisit niektorymi vlastnostami. Hrac bude mat moznost sa pohybovat sipkami, medzerou alebo nejakym vhodnym key na klavesnici bude moct brechat co bude moct pouzit aj na ovce teda ak to realne pomaha pri zahanani, ale aj na odplasenie inych zvierat, napr. inych psov, vlkov, medveda, ludi okrem pastiera ktory sa bude tiez nachadzat na luke. Ovce nech reaguju na psa cize budu sa pohybovat v stade ale budu sa pohybovat tak ako ich pes zazenie, nech beekaju (baa), ale nech to je arkadove. Podstata hry ma byt zaviest ovce do kosiara cize, z bodu A do bodu B a celit nastraham ako su obcasne vypady inych psov, vynimocne vlkov, alebo medveda. Vlci a medved nech su nieco ako bosovia alebo proste vacsie prekazky v dalsich levelov. Vymysli si mensie lahsie prekazky v prvych leveloch. Levelov nech je nejaky rozumny pocet, mysli na to ze toto ma byt kratka hra na pobavenie na par minut max par desiatok minut mozno 1 hodina ak by sa to hralo do konca do posledneho levela. Na mape by sa mohli objavovat aj ine zvierata, navhrni napr. kravy, lisky obcas niekde nejake vtaky atd. proste priroda. Nech su okrem luk aj nejaka ta polna cesta, obcas nejake prekazdky ci uz v podobe ohrad, utesov, krikov cez ktore ovce neprejdu. Grafika nech je pekna, farebna, arkadova, trocha nech pripomina akoby to bolo nintendo hra ale nekopiruj ziadnu konkretnu hru respektive jej dizajn. Hru napis v html/javascript/

Standalone demo

Final BAA game

The manually completed playable version is separate from the model ranking and serves as a standalone demo of the original idea.

Playable

BAA - final game

A short 3D arcade game about a sheepdog herding a flock across meadows into a pen. This is the playable version; the benchmark above shows how each model handled the same assignment.

Play the final game

FAQ

BAA-chmark FAQ

What is BAA-chmark?

It compares AI models on the same assignment: create a short 3D arcade game about a sheepdog herding sheep.

Can the games be played?

Yes. Every result that can be launched includes a direct playable demo.

How is the ranking decided?

The ranking is based on practical evaluation of assignment completion, playability, controls, visual quality, and bugs.

Are the original model outputs modified?

The original benchmark projects remain preserved. The site may provide a separate mobile control wrapper that does not change game rules or logic.

Does the final BAA game count in the ranking?

No. The manually completed version is a separate demo of the original idea and is excluded from the model ranking.