- Playable
BAA - final game
A short 3D arcade game about a sheepdog herding a flock across meadows into a pen. This is the playable version; the benchmark above shows how each model handled the same assignment.
LLM benchmark
BAA-chmark: the same prompt, different games.
BAA-chmark compares how different models handled the assignment to create a short 3D arcade game about herding sheep.
- Results
- 45
- Playable results
- 24
- Demos to try
- 44
- Method
- Same prompt
In brief
- Every model received the same assignment to create a short 3D sheep-herding game.
- The comparison covers screenshots, playability, bugs, and demos that can be launched directly.
- Claude Opus 5 currently holds first place.
- Original model outputs remain preserved; the manually completed game is listed separately.
Full ranking
Results and playable demos
The default ranking reflects the practical result. Filters keep the ranking intact and only hide runs that do not match.
Filters
Type
Parameters
Authors
No model matches the selected filters.
| Preview | Model | Type | Parameter count / requirements** | Result | Bugs | Note | Demo |
|---|---|---|---|---|---|---|---|
|
1Claude Opus 5 | Closed | — | No issues found | - | Demo | |
|
2Claude Fable 5.1 | Closed | — | Minimal minor issues | About 45 min generation; USD 12.56; 200K context. | Demo | |
|
3Claude Fable 5 | Closed | — | Minimal, a few minor issues | Reasoning effort 4/6, not maximum. Small Mac viewport fix. |
Demo | |
|
4Kimi K3 | Open-weight | 2.8T 1.56 TB ≈147.9 GB / 100K | Minimal, minor issues | Generation cost: USD 7.31. | Demo | |
|
5GLM 5.3 Flash | Open-weight | 320B ≈196.6 GB | Few minor issues | Generation stopped twice and required short prompts to continue. It took about 75 min. | Demo | |
|
6GPT-6 Astra | Closed | — | Minor issues | Generated in 23 min. | Demo | |
|
7GPT-5.6 Sol | Closed | — | Few issues | - | Demo | |
|
8Claude Opus 4.8 | Closed | — | Few issues | - | Demo | |
|
9Grok 4.5 | Closed | — | No obvious serious issue | - | Demo | |
|
10Gemini 3.8 Flash | Closed | — | Many minor issues | About 7.5 min generation; USD 0.58. | Demo | |
|
11Gemini 3.5 Flash | Closed | — | Few issues | Required 1 corrective prompt. | Demo | |
|
12Qwen 3.8 27B | Open-weight | 27B 19 GB ≈6.71 GB / 100K | Few minor issues | Generation took 4 h and stopped at 187K context tokens. Early on, it needed two short prompts to continue. | Demo | |
|
13Gemini 3.6 Flash | Closed | — | No serious bugs | - | Demo | |
|
14Qwen 3.6 27B ThinkingCap | Open-weight, GGUF | 27B 16.8 GB ≈6.71 GB / 100K | No serious bugs | ThinkingCap modification by Bottlecap. Generated in 17 minutes; no corrective prompt. | Demo | |
|
15Muse Spark 1.2 | Closed | — | Many bugs; freezes after a while | Recorded time: 4 min 19 sec. | Demo | |
|
16GPT-5.6 Terra | Closed | — | Many bugs | Recorded time: 20 min. | Demo | |
|
17Qwen 3.5 397B A17B | Open-weight | 397B 235.7 GB ≈3.27 GB / 100K | Many serious bugs | - | Demo | |
|
18MiniMax M2.7 | Open-weight | 229.9B ≈139.9 GB ≈25.4 GB / 100K | Many serious bugs | Characters barely move. | Demo | |
|
19Qwen 3.8 Max Preview | Closed | 2.4T | Unusual controls, few other issues | Generation stopped unexpectedly before completion. | Demo | |
|
20DeepSeek V4 Pro 0831 | Open-weight | 1.6T 864.7 GB ≈1.01 GB / 100K | Many issues; unusual controls | Generation took more than 2 h and cost USD 6.11. After stopping once, it needed a short prompt to continue. | Demo | |
|
21DeepSeek V4 Flash 0731 DS4 | Open-weight | 284B 164.6 GB ≈1.22 GB / 100K | Unusual camera and many bugs | A more memory-efficient Q2 quantization was used, fitting within 96 GB of RAM. Time: 1 h 49 min. | Demo | |
|
22GLM 5.3 Flash + Claude Code | Open-weight through Claude Code | 320B ≈196.6 GB | Many bugs; reversed controls | GLM 5.3 Flash through Claude Code, max effort. Generated in 2 h 49 min; launch succeeded only after several attempts. | Demo | |
|
23Gemini 3.7 Flash | Closed | — | Many issues, including reversed forward and backward movement | Required 1 short prompt to continue generating the second level. Generation took about 7 min. | Demo | |
|
24GPT-5.5 | Closed | — | Many bugs | - | Demo | |
|
25MiniMax M3 | Open-weight | 428B ≈261.3 GB | Many bugs | - | Demo | |
|
26Claude Sonnet 4.6 | Closed | — | Several issues | Generation took unusually long, over an hour. | Demo | |
|
27GLM 5.2 | Open-weight (OpenRouter) | 744B 464.8 GB ≈9.52 GB / 100K | Many bugs | - | Demo | |
|
28DeepSeek V4 Flash | Open-weight (OpenRouter) | 284B 159.6 GB ≈0.71 GB / 100K | Many bugs | - | Demo | |
|
29Qwen 3.7 Max | Closed | — | Very many bugs | Required 1 corrective prompt. | Demo | |
|
30DeepSeek V4 Pro | Open-weight (OpenRouter) | 1.6T 864.7 GB ≈1.01 GB / 100K | Many bugs | Required 1 corrective prompt. | Demo | |
|
31Big Pickle | Unknown | — | Many bugs | - | Demo | |
|
32Kimi 2.7 Code | Open-weight (OpenRouter) | 1T 595.2 GB | Many bugs | Required 3 corrective prompts. | Demo | |
|
33Qwen 3.6 27B | Open-weight | 27B 19.1 GB ≈6.71 GB / 100K | Many bugs | - | Demo | |
|
34Qwen 3.5 122B A10B | Open-weight | 122B 78.9 GB ≈2.61 GB / 100K | Very many bugs | Both the sheep and dog fly around. | Demo | |
|
35GLM 4.7 Flash | Open-weight (OpenRouter) | 30B ≈19.1 GB ≈5.41 GB / 100K | Many bugs | Required 1 corrective prompt. | Demo | |
|
36Qwen 3.6 35B A3B | Open-weight | 35B ≈21.4 GB ≈2.11 GB / 100K | Many bugs | Required 1 corrective prompt. | Demo | |
|
37Muse Glimmer 30B | Open-weight | 29.6B 16.8 GB ≈1.41 GB / 100K | Very many serious issues; static camera | Generation took 6 minutes. | Demo | |
|
38Gemma 4 31B | Open-weight | 30.7B 17.7 GB ≈9.03 GB / 100K | Many bugs | Required 1 corrective prompt. | Demo | |
|
39Laguna S 2.1 | Open-weight | 118B 71.9 GB ≈4.99 GB / 100K | Very many serious bugs; crashes | Required 5 corrective prompts; the first generation took 1 h 31 min. | Demo | |
|
40GPT-5.6 Luna | Closed | — | A few bugs | Recorded time: 13 min 26 sec. | Demo | |
|
41MiMo v5 | Open-weight | — | Many bugs | Required 1 corrective prompt. | Demo | |
|
42Mistral Small 4 | Open-weight | 119B 70.8 GB ≈2.3 GB / 100K | Very many serious bugs | Required 3 corrective prompts. | Demo | |
|
43Ornith 1.0 35B | Open-weight | 35B 21.2 GB ≈2.11 GB / 100K | Many bugs | Required 2 corrective prompts. | Demo | |
|
44Qwen 3.5 A3B | Open-weight | — | Many bugs | - | Demo | |
| 45Nemotron 3 Ultra | Open-weight | 550B 352.3 GB | — | - | — |
**Shown when available: parameter count, Q4 weight size at 0K, and cache/state at 100K. We use a concrete Q4 checkpoint or a labeled Q4_K_M estimate; 100K assumes one sequence in the documented runtime. Weight quantization does not change the cache while the runtime keeps the same cache layout and data types. Hybrid models can differ substantially in another runtime. Unverifiable values are omitted, and a lone dash means no value is known. Runtime buffers, vision components, and offload are excluded; this is not a universal minimum RAM/VRAM requirement.
Notes refer to corrective prompts needed to launch the game (for example a JavaScript error) and to any run settings such as reasoning effort. No corrective prompt was used to tune the game itself.
On mobile or tablet, Demo opens an arrow and A/B control panel; a game with built-in touch controls opens the original. The original benchmark code remains unchanged.
Method
Assignment used for the comparison
The complete prompt is documented below; where a translation is shown, the original wording remains available. Evaluation is based on actually launching the output, playability, bugs, and constraints of each run.
Prompt used for the comparison
The models received the original Slovak prompt. A faithful English translation is shown below.
I would like to create a simple arcade game that feels a little like a Nintendo game, but in 3D, from a third-person perspective with a somewhat distant camera. The game should be set in mountain meadows, on hilly, rolling terrain where sheep graze. Most of the area should consist of meadows, with mountains, forests, and a few houses in the background. The player should control a sheepdog or another type of dog used for herding sheep, with a choice of several breeds that differ in some of their traits. The player should be able to move with the arrow keys and bark using the space bar or another suitable keyboard key. Barking should also affect the sheep, if that realistically helps with herding, and should scare away other animals—for example other dogs, wolves, and a bear—as well as people, except for the shepherd, who should also be present in the meadow. The sheep should react to the dog: they should move as a flock, while also moving in response to the way the dog herds them. They should bleat (“baa”), and the game should have an arcade feel. The main objective should be to guide the sheep into a pen, from point A to point B, while facing hazards such as occasional attacks by other dogs and, more rarely, wolves or a bear. Wolves and the bear should act as bosses, or simply as larger obstacles in later levels. Come up with smaller, easier obstacles for the early levels. There should be a reasonable number of levels; keep in mind that this is meant to be a short game for fun, lasting a few minutes, at most a few dozen minutes, or perhaps one hour if played all the way through the final level. Other animals could also appear on the map; include, for example, cows, foxes, and the occasional bird—in short, nature. In addition to the meadows, include a dirt road and occasional obstacles such as fences, cliffs, and bushes that the sheep cannot pass through. The graphics should be attractive, colorful, and arcade-like, somewhat reminiscent of a Nintendo game, but do not copy any specific game or its design. Write the game in HTML/JavaScript.
Show original Slovak prompt
Chcel by som vytvorit jednoduchu arkadovu hru, trocha akoby to bola nintendo hra ale v 3D s pohladu z tretej osoby ale trocha vzdialena kamera. Hra bude v prostredi luk v horach na kopcovitom vlnenom terene kde sa pasu ovce, Vacsina plochy budu luky, v pozadi hory a lesy a nejake domy. Hrac bude hrat za ovciarskeho alebo ineho psa ktory sa pouziva pri zahanani oviec, nech si vie vybrat z par plemien ktore sa budu lisit niektorymi vlastnostami. Hrac bude mat moznost sa pohybovat sipkami, medzerou alebo nejakym vhodnym key na klavesnici bude moct brechat co bude moct pouzit aj na ovce teda ak to realne pomaha pri zahanani, ale aj na odplasenie inych zvierat, napr. inych psov, vlkov, medveda, ludi okrem pastiera ktory sa bude tiez nachadzat na luke. Ovce nech reaguju na psa cize budu sa pohybovat v stade ale budu sa pohybovat tak ako ich pes zazenie, nech beekaju (baa), ale nech to je arkadove. Podstata hry ma byt zaviest ovce do kosiara cize, z bodu A do bodu B a celit nastraham ako su obcasne vypady inych psov, vynimocne vlkov, alebo medveda. Vlci a medved nech su nieco ako bosovia alebo proste vacsie prekazky v dalsich levelov. Vymysli si mensie lahsie prekazky v prvych leveloch. Levelov nech je nejaky rozumny pocet, mysli na to ze toto ma byt kratka hra na pobavenie na par minut max par desiatok minut mozno 1 hodina ak by sa to hralo do konca do posledneho levela. Na mape by sa mohli objavovat aj ine zvierata, navhrni napr. kravy, lisky obcas niekde nejake vtaky atd. proste priroda. Nech su okrem luk aj nejaka ta polna cesta, obcas nejake prekazdky ci uz v podobe ohrad, utesov, krikov cez ktore ovce neprejdu. Grafika nech je pekna, farebna, arkadova, trocha nech pripomina akoby to bolo nintendo hra ale nekopiruj ziadnu konkretnu hru respektive jej dizajn. Hru napis v html/javascript/
Standalone demo
Final BAA game
The manually completed playable version is separate from the model ranking and serves as a standalone demo of the original idea.
FAQ
BAA-chmark FAQ
What is BAA-chmark?
It compares AI models on the same assignment: create a short 3D arcade game about a sheepdog herding sheep.
Can the games be played?
Yes. Every result that can be launched includes a direct playable demo.
How is the ranking decided?
The ranking is based on practical evaluation of assignment completion, playability, controls, visual quality, and bugs.
Are the original model outputs modified?
The original benchmark projects remain preserved. The site may provide a separate mobile control wrapper that does not change game rules or logic.
Does the final BAA game count in the ranking?
No. The manually completed version is a separate demo of the original idea and is excluded from the model ranking.