LLM benchmark

BAA benchmark: the same prompt, thirty models.

BAA compares thirty model outputs from the same assignment: a short 3D arcade game about herding sheep.

Results
30
Playable or partial
11
Available demos
29
Method
Same prompt

Full ranking

Results and playable demos

The default ranking reflects the practical result. Filters keep the ranking intact and only hide runs that do not match.

Filters
Playability
Type
Parameters
Benchmark ranking Subjective ranking by practical result in this benchmark
Preview Model Type Parameters / memory** Result Bugs Note Demo
Gameplay screenshot of the BAA game generated by Claude Fable 5 1Claude Fable 5 Closed - Matched the assignment Fully playable Minimal, a few minor issues Reasoning effort 4/6, not maximum.
Small Mac viewport fix.
Demo
Gameplay screenshot of the BAA game generated by Kimi K3 2Kimi K3 Open-weight 2.8T 1.54 TB - 4.90 TB Matched the assignment Fully playable Minimal, minor issues Generation cost: USD 7.31. Demo
Gameplay screenshot of the BAA game generated by Claude Opus 5 3Claude Opus 5 Closed - Matched the assignment Fully playable No issues found - Demo
Gameplay screenshot of the BAA game generated by GPT 5.6 Sol 4GPT 5.6 Sol Closed - Matched the assignment Playable Few issues - Demo
Gameplay screenshot of the BAA game generated by Claude Opus 4.8 5Claude Opus 4.8 Closed - Matched the assignment Playable Few issues - Demo
Gameplay screenshot of the BAA game generated by Grok 4.5 6Grok 4.5 Closed - Matched the assignment Playable No obvious serious issue - Demo
Gameplay screenshot of the BAA game generated by Gemini 3.5 Flash 7Gemini 3.5 Flash Closed - Matched the assignment Playable Few issues Required 1 corrective prompt. Demo
Gameplay screenshot of the BAA game generated by Qwen 3.6 27B ThinkingCap 8Qwen 3.6 27B ThinkingCap Open-weight, GGUF 27B 18 GB - 50 GB Matched the assignment Playable No serious bugs ThinkingCap modification by Bottlecap. Generated in 17 minutes; no corrective prompt. Demo
Gameplay screenshot of the BAA game generated by Gemini 3.6 Flash 9Gemini 3.6 Flash Closed - Matched the assignment Playable No serious bugs - Demo
Gameplay screenshot of the BAA game generated by Qwen 3.5 397B A17B 10Qwen 3.5 397B A17B Open-weight 397B (17B active) 224 GB - 400 GB Partially matched the assignment Playable with caveats Many serious bugs - Demo
Gameplay screenshot of the BAA game generated by MiniMax M2.7 11MiniMax M2.7 Open-weight - Partially matched the assignment Playable with caveats Many serious bugs Characters barely move. Demo
Gameplay screenshot of the BAA game generated by GPT 5.5 12GPT 5.5 Closed - Partially matched the assignment Not playable Many bugs - Demo
Gameplay screenshot of the BAA game generated by MiniMax M3 13MiniMax M3 Open-weight 428B (23B active) 240 GB - 480 GB Partially matched the assignment Not playable Many bugs - Demo
Gameplay screenshot of the BAA game generated by Claude Sonnet 4.6 14Claude Sonnet 4.6 Closed - Partially matched the assignment Not playable Several issues Generation took unusually long, over an hour. Demo
Gameplay screenshot of the BAA game generated by GLM 5.2 15GLM 5.2 Open-weight (OpenRouter) 744B (40B active) 416 GB - 816 GB Partially matched the assignment Not playable Many bugs - Demo
Gameplay screenshot of the BAA game generated by DeepSeek V4 Flash 16DeepSeek V4 Flash Open-weight (OpenRouter) 284B (13B active) 160 GB - 304 GB Partially matched the assignment Not playable Many bugs - Demo
Gameplay screenshot of the BAA game generated by Qwen 3.7 Max 17Qwen 3.7 Max Closed - Partially matched the assignment Not playable Very many bugs Required 1 corrective prompt. Demo
Gameplay screenshot of the BAA game generated by DeepSeek V4 Pro 18DeepSeek V4 Pro Open-weight (OpenRouter) 1.6T (49B active) 896 GB - 1.37 TB Partially matched the assignment Not playable Many bugs Required 1 corrective prompt. Demo
Gameplay screenshot of the BAA game generated by Big Pickle 19Big Pickle Unknown - Partially matched the assignment Not playable Many bugs - Demo
Gameplay screenshot of the BAA game generated by Kimi 2.7 Code 20Kimi 2.7 Code Open-weight (OpenRouter) 1T (32B active) 560 GB - 880 GB Partially matched the assignment Not playable Many bugs Required 3 corrective prompts. Demo
Gameplay screenshot of the BAA game generated by Qwen 3.6 27B 21Qwen 3.6 27B Open-weight 27B 18 GB - 50 GB Partially matched the assignment Not playable Many bugs - Demo
Gameplay screenshot of the BAA game generated by Qwen 3.5 122B A10B 22Qwen 3.5 122B A10B Open-weight 122B (10B active) 72 GB - 176 GB Partially matched the assignment Not playable Very many bugs Both the sheep and dog fly around. Demo
Gameplay screenshot of the BAA game generated by GLM 4.7 Flash 23GLM 4.7 Flash Open-weight (OpenRouter) 31B (3B active) 20 GB - 50 GB Partially matched the assignment Not playable Many bugs Required 1 corrective prompt. Demo
Gameplay screenshot of the BAA game generated by Qwen 3.6 35B A3B 24Qwen 3.6 35B A3B Open-weight 35B (3B active) 22 GB - 52 GB Partially matched the assignment Not playable Many bugs Required 1 corrective prompt. Demo
Gameplay screenshot of the BAA game generated by MiMo v5 25MiMo v5 Open-weight 310B (15B active) 176 GB - 336 GB Partially matched the assignment Not playable Many bugs Required 1 corrective prompt. Demo
Gameplay screenshot of the BAA game generated by Gemma 4 31B 26Gemma 4 31B Open-weight 31B 20 GB - 58 GB Matched the assignment only very partially Not playable Many bugs Required 1 corrective prompt. Demo
Gameplay screenshot of the BAA game generated by Mistral Small 4 27Mistral Small 4 Unknown - Did not match the assignment Completely unplayable Very many serious bugs Required 3 corrective prompts. Demo
Gameplay screenshot of the BAA game generated by Ornith 1.0 35B 28Ornith 1.0 35B Open-weight 35B 22 GB - 64 GB Partially matched the assignment Completely unplayable Many bugs Required 2 corrective prompts. Demo
Gameplay screenshot of the BAA game generated by Qwen 3.5 A3B 29Qwen 3.5 A3B Open-weight 30B (3B active) 20 GB - 50 GB Partially matched the assignment Completely unplayable Many bugs - Demo
30Nemotron 3 Ultra Open-weight 550B (55B active) 320 GB - 864 GB Did not match the assignment Produced nothing -

**Memory is only an approximate estimate for Q4 weights: the first value estimates 0K context, the second 100K context. It is not a measurement or exact sizing for a specific architecture.

Notes refer to corrective prompts needed to launch the game (for example a JavaScript error) and to any run settings such as reasoning effort. No corrective prompt was used to tune the game itself.

On mobile or tablet, Demo opens a version with an added arrow and A/B control panel. The original benchmark code remains unchanged.

Gameplay screenshot of the BAA game generated by Claude Fable 5

1Claude Fable 5

Closed
Parameters / memory**
-
Result
Matched the assignment Fully playable
Bugs
Minimal, a few minor issues
Note
Reasoning effort 4/6, not maximum.
Small Mac viewport fix.
Demo
Gameplay screenshot of the BAA game generated by Kimi K3

2Kimi K3

Open-weight
Parameters / memory**
2.8T 1.54 TB - 4.90 TB
Result
Matched the assignment Fully playable
Bugs
Minimal, minor issues
Note
Generation cost: USD 7.31.
Demo
Gameplay screenshot of the BAA game generated by Claude Opus 5

3Claude Opus 5

Closed
Parameters / memory**
-
Result
Matched the assignment Fully playable
Bugs
No issues found
Note
-
Demo
Gameplay screenshot of the BAA game generated by GPT 5.6 Sol

4GPT 5.6 Sol

Closed
Parameters / memory**
-
Result
Matched the assignment Playable
Bugs
Few issues
Note
-
Demo
Gameplay screenshot of the BAA game generated by Claude Opus 4.8

5Claude Opus 4.8

Closed
Parameters / memory**
-
Result
Matched the assignment Playable
Bugs
Few issues
Note
-
Demo
Gameplay screenshot of the BAA game generated by Grok 4.5

6Grok 4.5

Closed
Parameters / memory**
-
Result
Matched the assignment Playable
Bugs
No obvious serious issue
Note
-
Demo
Gameplay screenshot of the BAA game generated by Gemini 3.5 Flash

7Gemini 3.5 Flash

Closed
Parameters / memory**
-
Result
Matched the assignment Playable
Bugs
Few issues
Note
Required 1 corrective prompt.
Demo
Gameplay screenshot of the BAA game generated by Qwen 3.6 27B ThinkingCap

8Qwen 3.6 27B ThinkingCap

Open-weight, GGUF
Parameters / memory**
27B 18 GB - 50 GB
Result
Matched the assignment Playable
Bugs
No serious bugs
Note
ThinkingCap modification by Bottlecap. Generated in 17 minutes; no corrective prompt.
Demo
Gameplay screenshot of the BAA game generated by Gemini 3.6 Flash

9Gemini 3.6 Flash

Closed
Parameters / memory**
-
Result
Matched the assignment Playable
Bugs
No serious bugs
Note
-
Demo
Gameplay screenshot of the BAA game generated by Qwen 3.5 397B A17B

10Qwen 3.5 397B A17B

Open-weight
Parameters / memory**
397B (17B active) 224 GB - 400 GB
Result
Partially matched the assignment Playable with caveats
Bugs
Many serious bugs
Note
-
Demo
Gameplay screenshot of the BAA game generated by MiniMax M2.7

11MiniMax M2.7

Open-weight
Parameters / memory**
-
Result
Partially matched the assignment Playable with caveats
Bugs
Many serious bugs
Note
Characters barely move.
Demo
Gameplay screenshot of the BAA game generated by GPT 5.5

12GPT 5.5

Closed
Parameters / memory**
-
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
-
Demo
Gameplay screenshot of the BAA game generated by MiniMax M3

13MiniMax M3

Open-weight
Parameters / memory**
428B (23B active) 240 GB - 480 GB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
-
Demo
Gameplay screenshot of the BAA game generated by Claude Sonnet 4.6

14Claude Sonnet 4.6

Closed
Parameters / memory**
-
Result
Partially matched the assignment Not playable
Bugs
Several issues
Note
Generation took unusually long, over an hour.
Demo
Gameplay screenshot of the BAA game generated by GLM 5.2

15GLM 5.2

Open-weight (OpenRouter)
Parameters / memory**
744B (40B active) 416 GB - 816 GB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
-
Demo
Gameplay screenshot of the BAA game generated by DeepSeek V4 Flash

16DeepSeek V4 Flash

Open-weight (OpenRouter)
Parameters / memory**
284B (13B active) 160 GB - 304 GB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
-
Demo
Gameplay screenshot of the BAA game generated by Qwen 3.7 Max

17Qwen 3.7 Max

Closed
Parameters / memory**
-
Result
Partially matched the assignment Not playable
Bugs
Very many bugs
Note
Required 1 corrective prompt.
Demo
Gameplay screenshot of the BAA game generated by DeepSeek V4 Pro

18DeepSeek V4 Pro

Open-weight (OpenRouter)
Parameters / memory**
1.6T (49B active) 896 GB - 1.37 TB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
Required 1 corrective prompt.
Demo
Gameplay screenshot of the BAA game generated by Big Pickle

19Big Pickle

Unknown
Parameters / memory**
-
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
-
Demo
Gameplay screenshot of the BAA game generated by Kimi 2.7 Code

20Kimi 2.7 Code

Open-weight (OpenRouter)
Parameters / memory**
1T (32B active) 560 GB - 880 GB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
Required 3 corrective prompts.
Demo
Gameplay screenshot of the BAA game generated by Qwen 3.6 27B

21Qwen 3.6 27B

Open-weight
Parameters / memory**
27B 18 GB - 50 GB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
-
Demo
Gameplay screenshot of the BAA game generated by Qwen 3.5 122B A10B

22Qwen 3.5 122B A10B

Open-weight
Parameters / memory**
122B (10B active) 72 GB - 176 GB
Result
Partially matched the assignment Not playable
Bugs
Very many bugs
Note
Both the sheep and dog fly around.
Demo
Gameplay screenshot of the BAA game generated by GLM 4.7 Flash

23GLM 4.7 Flash

Open-weight (OpenRouter)
Parameters / memory**
31B (3B active) 20 GB - 50 GB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
Required 1 corrective prompt.
Demo
Gameplay screenshot of the BAA game generated by Qwen 3.6 35B A3B

24Qwen 3.6 35B A3B

Open-weight
Parameters / memory**
35B (3B active) 22 GB - 52 GB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
Required 1 corrective prompt.
Demo
Gameplay screenshot of the BAA game generated by MiMo v5

25MiMo v5

Open-weight
Parameters / memory**
310B (15B active) 176 GB - 336 GB
Result
Partially matched the assignment Not playable
Bugs
Many bugs
Note
Required 1 corrective prompt.
Demo
Gameplay screenshot of the BAA game generated by Gemma 4 31B

26Gemma 4 31B

Open-weight
Parameters / memory**
31B 20 GB - 58 GB
Result
Matched the assignment only very partially Not playable
Bugs
Many bugs
Note
Required 1 corrective prompt.
Demo
Gameplay screenshot of the BAA game generated by Mistral Small 4

27Mistral Small 4

Unknown
Parameters / memory**
-
Result
Did not match the assignment Completely unplayable
Bugs
Very many serious bugs
Note
Required 3 corrective prompts.
Demo
Gameplay screenshot of the BAA game generated by Ornith 1.0 35B

28Ornith 1.0 35B

Open-weight
Parameters / memory**
35B 22 GB - 64 GB
Result
Partially matched the assignment Completely unplayable
Bugs
Many bugs
Note
Required 2 corrective prompts.
Demo
Gameplay screenshot of the BAA game generated by Qwen 3.5 A3B

29Qwen 3.5 A3B

Open-weight
Parameters / memory**
30B (3B active) 20 GB - 50 GB
Result
Partially matched the assignment Completely unplayable
Bugs
Many bugs
Note
-
Demo

30Nemotron 3 Ultra

Open-weight
Parameters / memory**
550B (55B active) 320 GB - 864 GB
Result
Did not match the assignment Produced nothing
Bugs
Note
-

Method

Assignment used for the comparison

The prompt is preserved in its original wording. Evaluation is based on actually launching the output, playability, bugs, and constraints of each run.

Prompt used for the comparison

Chcel by som vytvorit jednoduchu arkadovu hru, trocha akoby to bola nintendo hra ale v 3D s pohladu z tretej osoby ale trocha vzdialena kamera. Hra bude v prostredi luk v horach na kopcovitom vlnenom terene kde sa pasu ovce, Vacsina plochy budu luky, v pozadi hory a lesy a nejake domy. Hrac bude hrat za ovciarskeho alebo ineho psa ktory sa pouziva pri zahanani oviec, nech si vie vybrat z par plemien ktore sa budu lisit niektorymi vlastnostami. Hrac bude mat moznost sa pohybovat sipkami, medzerou alebo nejakym vhodnym key na klavesnici bude moct brechat co bude moct pouzit aj na ovce teda ak to realne pomaha pri zahanani, ale aj na odplasenie inych zvierat, napr. inych psov, vlkov, medveda, ludi okrem pastiera ktory sa bude tiez nachadzat na luke. Ovce nech reaguju na psa cize budu sa pohybovat v stade ale budu sa pohybovat tak ako ich pes zazenie, nech beekaju (baa), ale nech to je arkadove. Podstata hry ma byt zaviest ovce do kosiara cize, z bodu A do bodu B a celit nastraham ako su obcasne vypady inych psov, vynimocne vlkov, alebo medveda. Vlci a medved nech su nieco ako bosovia alebo proste vacsie prekazky v dalsich levelov. Vymysli si mensie lahsie prekazky v prvych leveloch. Levelov nech je nejaky rozumny pocet, mysli na to ze toto ma byt kratka hra na pobavenie na par minut max par desiatok minut mozno 1 hodina ak by sa to hralo do konca do posledneho levela. Na mape by sa mohli objavovat aj ine zvierata, navhrni napr. kravy, lisky obcas niekde nejake vtaky atd. proste priroda. Nech su okrem luk aj nejaka ta polna cesta, obcas nejake prekazdky ci uz v podobe ohrad, utesov, krikov cez ktore ovce neprejdu. Grafika nech je pekna, farebna, arkadova, trocha nech pripomina akoby to bolo nintendo hra ale nekopiruj ziadnu konkretnu hru respektive jej dizajn. Hru napis v html/javascript/

Standalone demo

Final BAA game

The manually completed playable version is separate from the model ranking and serves as a standalone demo of the original idea.

Gameplay screenshot of the final BAA game
  • Playable

BAA - final game

A short 3D arcade game about a sheepdog herding a flock across meadows into a pen. This is the playable version; the benchmark above shows how each model handled the same assignment.