Skip to content

Minecraft

Jump to bottom
Bruce D'Ambrosio edited this page Jan 11, 2026 · 7 revisions

Minecraft Integration

**Standard 'world' disclaimer: Worlds are intended to drive platform research, not as useful places in themselves. As such they are bare skeletons to find where the platform breaks, or as challenges for tool construction. Don't expect this one to do anything useful **

Having said that, current (1/11/2026) tools seem to be the beginning of basic navigation capability in Minecraft. Nothing is ever guaranteed in minecraft, the challenge for the planner is to recover and adapt in a hostile (in the sense of unpredictable) environment.

Minecraft support enables agents to interact with a Minecraft world via the Minescript 4.0 bridge. Minecraft is a large and complex software stack, if you aren't familiar with it, be prepared for learning curve to get all the pieces running.

Setup

Prerequisites

  1. Minecraft Server (1.20.4):

    java -Xmx4G -Xms2G -jar server-1.20.4.jar nogui

  2. Minecraft client:

    • find the minecraft-launcher client somewhere and download it.

  3. Fabric Mod Loader:

    • Install Fabric API: fabric-api-0.97.3+1.20.4.jar
    • Place in ~/.minecraft/mods/

  4. Minescript Bridge:

    • Copy (or better, ln -s) scenarios/minecraft/bridge.py to ~/.minecraft/mods/
    • run minecraft-launcher to launch the client
    • In Minecraft client, type \bridge to start the bridge
    • Bridge runs on http://localhost:3003 (default) (sometimes it hangs on to port on exit, which will cause problems on restart, beware)

  5. Client Configuration:

    • Press F3 + P to prevent pause on focus loss
    • Open chat (/ or '') to free mouse cursor
    • Optional: /op <username> to grant superuser powers

Running the agent

cd src
python3 launcher.py scenarios/jill-minecraft.yaml --ui

World State

Navigation History (world_state.nav)

Navigation state is stored as a list (most recent first):

{
  "nav": [
    {
      "pose": {"x": -123.5, "y": 73.0, "z": -153.5, "yaw": 180},
      "support_here": "solid",
      "fell": false,
      "was_fall": false,
      "timestamp": "2024-01-15T10:30:00"
    },
    ...
  ]
}

Properties:

  • Bounded to 100 entries (oldest discarded)
  • nav[0] is current state
  • nav[1:] is history (for backtracking)
  • Yaw normalized to 0-360 range (never negative)

Initialization

The init tool (minecraft/init) is automatically executed on executor startup:

  1. Gets current status via mc-status
  2. Normalizes pose via nav-turn (block center, cardinal yaw, pitch zero)
  3. Updates world_state.nav[0] with normalized state

Navigation Tools

Atomic Navigation Primitives

nav-turn - Change orientation:

  • Directions: right (+90°), left (-90°), back (+180°), forward (align only)
  • Always results in cardinal yaw (0°, 90°, 180°, 270°)
  • Snaps to block center, sets pitch to 0°
  • Never changes position

nav-move - Single forward step:

  • Moves forward one block (relative to facing)
  • Validates landing with mc-status + mc-observe-blocks
  • Snaps to block center after move
  • Updates world_state.nav history

nav-climb - Climb up one block:

  • Checks clearance above
  • Jumps if needed
  • Validates landing
  • Updates nav history

nav-descend - Descend one block:

  • Checks drop distance
  • Moves forward and down
  • Validates landing
  • Updates nav history

nav-backtrack - Recover previous position:

  • Uses world_state.nav[1:] history
  • Attempts to return to previous safe position
  • Useful after falls or dead ends

Path Planning Tools

path-frontier - Find unexplored directions:

  • Analyzes current cell and adjacent cells
  • Returns directions with unexplored terrain
  • Uses SpatialMap for cell data

path-explore - Systematic exploration:

  • Uses path-frontier to find unexplored directions
  • Moves toward frontier
  • Updates map as it explores

Observation Tools

mc-status - Get agent state:

  • Position, yaw (normalized 0-360), pitch
  • Health, food, vertical state
  • Returns structured data with normalized yaw

mc-observe-blocks - Observe nearby blocks:

  • Scans blocks in a region around agent
  • Returns block types and positions
  • Used by mc-map-update for mapping

mc-observe-entities - Observe entities:

  • Mobs, items, other entities
  • Returns entity types and positions

mc-observe-items - Observe items:

  • Items on ground or in containers
  • Returns item types and positions

Mapping

SpatialMap

Minecraft uses a cell-based spatial memory (SpatialMap) instead of Collection-based logs:

  • Cells keyed by (x, z) coordinates
  • Each cell stores: support, surface, hazards, resources, observability, waypoints
  • Persisted to scenarios/minecraft/resources/Jill_spatial_map.json

mc-map-update - Update spatial map:

  • Checks if current location is already mapped
  • Auto-invokes mc-observe-blocks if unmapped
  • Updates cell data with observation results
  • Auto-saves SpatialMap

mc-map-query - Query spatial map:

  • Query types: cell, nearest, waypoint, unexplored
  • Returns cell data or lists of matching cells
  • Legacy Collection-based queries deprecated

mc-map-visualize - Visualize map:

  • Generates visualization of mapped cells
  • Shows support, hazards, resources

mc-waypoint - Mark waypoint:

  • Stores waypoint name in SpatialMap cell
  • Auto-queries mc-status if coordinates not provided
  • Queryable via mc-map-query with query_type: "waypoint"

Interaction Tools

mc-attack - Attack entity/block mc-dig - Break block mc-place - Place block mc-place-until-supported - Place block with support check mc-pickup - Pick up item mc-drop - Drop item mc-open - Open container/block mc-close - Close container mc-use - Use item/block mc-craft - Craft item mc-equip - Equip item mc-unequip - Unequip item mc-inventory - List inventory mc-respawn - Respawn agent mc-wait - Wait for duration mc-stop - Stop current action mc-say - Send chat message

Tool Conventions

All Minecraft tools:

  • Use executor._create_uniform_return() for results
  • Update world_state.nav on position/orientation changes
  • Normalize yaw to 0-360 before storing
  • Use executor.execute_action_with_log() for nested calls
  • Follow uniform return format (see Tools)

Resources

  • SpatialMap: scenarios/minecraft/resources/Jill_spatial_map.json
  • Documentation: scenarios/minecraft/*.md files loaded into minecraft Collection
  • Bridge: scenarios/minecraft/bridge.py (Minescript 4.0)

Example Tasks

Check status:

{"type": "mc-status", "out": "$status"}

Navigate and map:

{"type": "nav-turn", "direction": "right", "out": "$turn"}
{"type": "nav-move", "out": "$move"}
{"type": "mc-map-update", "out": "$map"}

Explore systematically:

{"type": "path-frontier", "out": "$frontier"}
{"type": "path-explore", "out": "$explore"}

Notes

  • All yaw values normalized to 0-360 (Minescript may return negative values)
  • Navigation history bounded to 100 entries
  • SpatialMap replaces legacy Collection-based map storage
  • init tool auto-executes on startup to normalize initial pose

Clone this wiki locally