Skip to content

Commit 146d64f

Browse files
authored
feat: add Baba Is AI demo — knowing-doing gap in spatial puzzles (#38)
* feat: add config and template for baba-is-ai demo Ollama config with qwen2.5:7b defaults and YAML template documenting the prompt context for the llm:choose-based action selection. * feat: add baba-is-ai model with rule engine, push system, and LLM integration Complete Baba Is You puzzle demo with 3 levels of increasing difficulty: - Level 1: Navigate (baseline path-finding) - Level 2: Break the Wall (push STOP word-block to disable wall rule) - Level 3: Push and Rearrange (rocks + walls + spatial reasoning) Includes ABM rule engine that scans horizontal/vertical word-block 3-tuples, recursive push chain system, text observation builder for LLM, and win/defeat detection. Uses llm:choose for constrained action selection with accumulating history. * docs: add README for baba-is-ai demo Covers the BALROG knowing-doing gap experiment context, setup prerequisites, 3 level descriptions with expected results, provider swapping instructions, and extension points. * fix: inject game context, persist history, and safe-place permanent rules - Add llm:set-history system message in setup so the LLM knows game mechanics (push word-blocks, break rules, reach the flag) - Remove llm:clear-history from go loop so the agent remembers past moves and learns from failures - Move BABA IS YOU and FLAG IS WIN to row 8 (top edge) across all levels so the agent never accidentally pushes permanent rules - Improve observation builder with directional hints, fixed-width grid map, loop detection, and explicit blocked-move feedback * fix: improve system prompt, seal wall gaps, add rate limit handling - Rewrite system prompt with explicit push mechanics and priority (break STOP first, then head to flag) - Observation now shows exact STOP block coordinates when WALL IS STOP is active - Extend wall columns to y=0-7 in levels 2 and 3 so agent cannot navigate around walls — must break the rule - Cap history to system msg + last 10 exchanges to avoid token bloat - Handle rate limit errors with 2s wait and retry on next tick
1 parent 93630cd commit 146d64f

4 files changed

Lines changed: 970 additions & 0 deletions

File tree

demos/baba-is-ai/README.md

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Baba Is AI
2+
3+
An LLM agent plays a simplified "Baba Is You" puzzle game where the rules of the world are physical word-block objects on a grid. The agent must push word-blocks to change or break rules in order to reach the goal.
4+
5+
## The Experiment
6+
7+
### Background
8+
9+
Based on findings from the [BALROG benchmark](https://arxiv.org/abs/2411.13543) (Benchmarking Agentic LLM and VLM Reasoning On Games), which found that **all tested LLMs score near 0% on Baba Is You** despite understanding the rules perfectly when quizzed in isolation.
10+
11+
### The Knowing-Doing Gap
12+
13+
LLMs can explain that "rules are pushable objects" and "you need to rearrange word-blocks to change rules." But when given a spatial grid and asked to actually execute moves to push specific blocks in specific directions, they fail consistently. They *know* what to do but *cannot do* it.
14+
15+
### What This Demo Tests
16+
17+
Can an LLM agent, given a text observation of a 9x9 grid, figure out that it needs to:
18+
1. Navigate to the right word-block
19+
2. Push it in the right direction
20+
3. Verify the rule changed
21+
4. Navigate through the now-passable area to win
22+
23+
This is **meta-rule discovery** -- the agent doesn't just follow rules, it manipulates them.
24+
25+
### Why It Matters for ABM
26+
27+
Agent-based models traditionally have fixed rules. This demo shows agents operating in a world where **the rules themselves are agents** (word-blocks) that can be moved. The rule engine is pure NetLogo ABM; the decision-making is LLM. Together they create a system where rules are emergent from the spatial arrangement of objects.
28+
29+
## Prerequisites
30+
31+
1. **Ollama** running locally with `qwen2.5:7b` pulled:
32+
```bash
33+
ollama pull qwen2.5:7b
34+
```
35+
2. **NetLogo 7.0.3** with the LLM extension JAR installed
36+
3. Built extension: run `./build.sh` from the extension root if needed
37+
38+
## How to Run
39+
40+
1. Open `baba-is-ai.nlogox` in NetLogo 7.0.3
41+
2. Select a level from the chooser
42+
3. Click **Setup** to initialize
43+
4. Click **Go** to let the LLM agent play autonomously, or **Step** for one move at a time
44+
5. Toggle **show-observations?** to see the text observation sent to the LLM each turn
45+
6. Watch the **Active Rules** monitor update as word-blocks are pushed
46+
47+
## Levels
48+
49+
### Level 1: Navigate (baseline)
50+
```
51+
Rules: BABA IS YOU, FLAG IS WIN
52+
Layout: baba on the left, flag on the right, clear path
53+
Solution: Walk right to the flag (~7 moves)
54+
Expected: Most LLMs solve this consistently
55+
```
56+
57+
### Level 2: Break the Wall (medium)
58+
```
59+
Rules: BABA IS YOU, WALL IS STOP, FLAG IS WIN
60+
Layout: Wall column blocking the path to the flag
61+
Solution: Navigate to the STOP word-block, push it away to break
62+
"WALL IS STOP", then walk through walls to the flag
63+
Expected: Some LLMs occasionally solve this; most get stuck
64+
```
65+
66+
### Level 3: Push and Rearrange (hard)
67+
```
68+
Rules: BABA IS YOU, ROCK IS PUSH, WALL IS STOP, FLAG IS WIN
69+
Layout: Rocks near the WALL IS STOP rule, wall column, flag behind walls
70+
Solution: Navigate around rocks, push STOP away, walk through walls
71+
Expected: Very rarely solved -- demonstrates the knowing-doing gap
72+
```
73+
74+
## Swapping Providers
75+
76+
Edit `config` to try different LLM providers:
77+
78+
```
79+
# OpenAI
80+
provider=openai
81+
model=gpt-4o-mini
82+
api_key=YOUR_KEY_HERE
83+
84+
# Anthropic
85+
provider=anthropic
86+
model=claude-sonnet-4-20250514
87+
api_key=YOUR_KEY_HERE
88+
89+
# Gemini
90+
provider=gemini
91+
model=gemini-2.0-flash
92+
api_key=YOUR_KEY_HERE
93+
```
94+
95+
## Files
96+
97+
| File | Purpose |
98+
|------|---------|
99+
| `baba-is-ai.nlogox` | Main NetLogo model with rule engine, levels, and LLM integration |
100+
| `config` | LLM provider configuration (defaults to local Ollama) |
101+
| `action-template.yaml` | Documents the prompt format (not used at runtime -- `llm:choose` builds its own prompt) |
102+
| `README.md` | This file |
103+
104+
## How It Works
105+
106+
### Rule Engine
107+
Three consecutive word-blocks in a horizontal or vertical line form a rule: `[NOUN] IS [PROPERTY]`. The engine scans all possible 3-tuples every tick and rebuilds the active rule set.
108+
109+
### Push System
110+
Word-blocks are **always pushable**. Other entities (rocks, walls) are pushable only if they have the PUSH property. Push chains propagate: pushing a block into another pushable block pushes both.
111+
112+
### LLM Integration
113+
Each tick, the agent receives a text observation including the grid layout, active rules, adjacent cell descriptions, and feedback from the last action. It chooses from `["up", "down", "left", "right"]` via `llm:choose`. History accumulates across ticks so the agent can learn from failed moves.
114+
115+
## Extension Points
116+
117+
- **New properties**: Add DEFEAT (touching kills baba), SINK (object + baba both destroyed), MELT/HOT combos
118+
- **Multiple YOU entities**: What if two things are YOU and must coordinate?
119+
- **Procedural level generation**: Random placement of word-blocks and obstacles
120+
- **Rule construction**: Levels where the agent must *build* a rule, not just break one
121+
122+
## Reference
123+
124+
- BALROG paper: [arXiv 2411.13543](https://arxiv.org/abs/2411.13543)
125+
- Baba Is You (original game): [hempuli.com/baba](https://hempuli.com/baba/)
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
system: |
2+
You are playing Baba Is AI, a puzzle game where the rules of the world are
3+
physical word-block objects on a grid. Three consecutive word-blocks in a row
4+
or column form a rule: [NOUN] IS [PROPERTY]. For example, "BABA IS YOU" means
5+
you control the baba character, "WALL IS STOP" means walls block movement,
6+
and "FLAG IS WIN" means reaching a flag wins the level.
7+
8+
CRITICAL INSIGHT: You can PUSH word-blocks to change or break rules. If you
9+
push the STOP block away from "WALL IS STOP", walls become passable. Rules
10+
are not fixed -- they are objects you can physically manipulate.
11+
12+
Your goal: reach an entity that has the WIN property.
13+
14+
Respond with exactly one of: up, down, left, right
15+
16+
template: |
17+
{observation}
18+
19+
Think about which rules you need to change to reach the goal.
20+
Choose your move.

0 commit comments

Comments
 (0)