Gemini Robotics ER - Live API Examples

live-api/.gitignore

@@ -0,0 +1,31 @@
+# Environment/runtime directories
+.agents/
+.codex/
+.uv-cache/
+.venv/
+venv/
+env/
+node_modules/
+
+# Local config and secrets
+.config
+.env
+.env.*
+
+# Generated files
+__pycache__/
+*.py[oc]
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+apps/**/captures/
+spot/apps/api/captures/
+spot/apps/navigation/waypoints_map.html
+
+# OS/editor local files
+.DS_Store
+Thumbs.db
+*.swp
+*.swo
+.idea/
+.vscode/

live-api/README.md

@@ -0,0 +1,74 @@
+# Gemini Robotics ER - Live API Examples
+
+A repository of examples of connecting Gemini Robotics ER with physical robot embodiments using Live API for task orchestration, voice interactions, etc. The embodiments include Boston Dynamics Spot, Tinybot (a custom stationary robot hardware), and human operators.
+
+---
+
+## Repository Structure & Packages
+
+| Package | Purpose & Features | Environment / Stack | Link |
+| :--- | :--- | :--- | :--- |
+| **`agent`** | **Physical Agent Server**: Core agent server interfacing with the Gemini Live API over WebSockets for real-time audio/video interaction, tool dispatch, and robot control. | Python 3.10+ (FastAPI, WebSockets, `google-genai`, `uv`) | [`./agent`](./agent/README.md) |
+| **`spot`** | **Boston Dynamics Spot SDK Integration**: Suite of CLI tools, REST APIs, object detection/manipulation pipelines, and autonomous delivery applications for Spot. | Python (`bosdyn-client`, `google-genai`), Node.js/React (`apps/hydration`) | [`./spot`](./spot/README.md) |
+| **`tinybot`** | **Compact Robot Controller**: Lightweight server providing camera video streaming and basic REST API controls for compact or custom robot hardware. | Python (FastAPI, OpenCV) | [`./tinybot`](./tinybot) |
+
+---
+
+## Detailed Package Overview
+
+### 1. [Agent Server (`./agent`)](./agent/README.md)
+
+The `agent` directory contains the main orchestration server connecting multimodal AI models to physical hardware.
+
+* **Gemini Live API Integration**: Handles bi-directional audio/video streaming with Gemini Live API using WebSockets.
+* **Embodiment Architecture**: Modular client abstractions (`agent/embodiment/`) to control different physical targets (`spot`, `tinybot`, `human`).
+* **Web UI & Camera Poller**: Includes a built-in web interface and camera polling service (`camera_poller.py`) for real-time visual feeds.
+* **Quickstart**:
+  ```bash
+  cd agent
+  UV_CACHE_DIR=.uv-cache uv sync
+  UV_CACHE_DIR=.uv-cache uv run python server.py --port 8000
+  ```
+
+---
+
+### 2. [Spot Applications & SDK (`./spot`)](./spot/README.md)
+
+The `spot` directory contains Boston Dynamics Spot integrations powered by `bosdyn-client` and Gemini vision tools.
+
+* **Navigation App ([`apps/navigation`](./spot/apps/navigation))**: Manage GraphNav waypoints, register named locations, and command Spot to navigate autonomously via CLI.
+* **Manipulation App ([`apps/manipulation`](./spot/apps/manipulation))**: Arm deployment, Gemini-based 2D/3D object detection, force-change detection, and picking.
+* **FastAPI Server ([`apps/api`](./spot/apps/api))**: Exposes HTTP REST endpoints for Spot movement, leases, arm control, and waypoints (`http://localhost:8000/docs`).
+* **Hydration Delivery Service ([`apps/hydration`](./spot/apps/hydration))**: Full-stack Node/React app and order worker that commands Spot to deliver drinks.
+* **Quickstart**:
+  ```bash
+  cd spot
+  UV_CACHE_DIR=.uv-cache uv sync
+  UV_CACHE_DIR=.uv-cache uv run uvicorn apps.api.main:app --host 127.0.0.1 --port 8000
+  ```
+
+---
+
+### 3. [Tinybot Hardware Controller (`./tinybot`)](./tinybot)
+
+The `tinybot` directory provides lightweight camera streaming and basic hardware control endpoints for smaller physical robot hardware.
+
+* **Camera Streamer ([`src/robot/camera_streamer.py`](./tinybot/src/robot/camera_streamer.py))**: Captures and streams live camera feeds for vision processing.
+* **Robot REST API ([`src/robot/robot_api.py`](./tinybot/src/robot/robot_api.py))**: Exposes REST endpoints for low-level movement execution.
+* **Quickstart**:
+  ```bash
+  cd tinybot
+  ./setup.sh
+  ./run_robot.sh
+  ```
+
+---
+
+## Environment Setup
+
+All Python subpackages use [`uv`](https://github.com/astral-sh/uv) for fast, deterministic dependency management. To keep virtual environments isolated and clean:
+
+```bash
+# Sync dependencies within any subfolder using local cache:
+UV_CACHE_DIR=.uv-cache uv sync
+```

live-api/agent/README.md

@@ -0,0 +1,78 @@
+# Physical Agent Server
+
+Physical Agent Server managed with [`uv`](https://github.com/astral-sh/uv). It exposes a FastAPI server, WebSocket endpoints for live streaming audio/video with Gemini Live API, and robot embodiment integrations (Spot, Human, Tinybot).
+
+---
+
+## Prerequisites
+
+- **Python**: `>=3.10`
+- **`uv`**: Installed locally (e.g. `uv 0.11+`)
+
+---
+
+## Quickstart
+
+### 1. Synchronize Dependencies
+
+Run `uv sync` to set up the virtual environment:
+
+```bash
+UV_CACHE_DIR=.uv-cache uv sync --default-index https://pypi.org/simple
+```
+
+---
+
+### 2. Set API Keys (Optional)
+
+Set your Gemini API key:
+
+```bash
+export GEMINI_API_KEY="your_api_key_here"
+```
+
+---
+
+### 3. Start the Agent Server
+
+#### Basic Launch
+```bash
+UV_CACHE_DIR=.uv-cache uv run --default-index https://pypi.org/simple python server.py --port 8000
+```
+
+#### Launch with Custom Model & Spot Robot Endpoint
+```bash
+UV_CACHE_DIR=.uv-cache uv run --default-index https://pypi.org/simple python server.py \
+    --model gemini-3.1-flash-live-preview \
+    --robot_url http://<spot_pc_ip>:8000 \
+    --port 8000
+```
+
+Once running, access the web UI at:
+- **Local Web UI**: `http://localhost:8000`
+- **API Documentation**: `http://localhost:8000/docs`
+
+---
+
+## Running Tests
+
+Run all unit tests via `uv`:
+
+```bash
+UV_CACHE_DIR=.uv-cache uv run --default-index https://pypi.org/simple python -m unittest discover -p "*_test.py"
+```
+
+---
+
+## Command Line Arguments
+
+| Argument | Description | Default |
+|---|---|---|
+| `--model` | Gemini model name | `gemini-3.1-flash-live-preview` |
+| `--robot_url` | Robot OpenAPI / HTTP base URL | `None` |
+| `--api_key` | Gemini API Key | `GEMINI_API_KEY` env var |
+| `--use_tts` / `--no_tts` | Enable or disable Text-to-Speech | `Disabled` |
+| `--port` | HTTP server port | `8000` |
+| `--media_resolution` | Video input frame resolution (`low`, `medium`, `high`, `ultra_high`) | `low` |
+| `--heartbeat_enabled` | Enable proactive heartbeat | `True` |
+| `--agent_peers` | Comma-separated peer agents (`name=url,name=url`) | `None` |

live-api/agent/__init__.py

@@ -0,0 +1 @@
+

live-api/agent/agent/__init__.py

@@ -0,0 +1 @@
+

live-api/agent/agent/agent.py

@@ -0,0 +1,96 @@
+"""Agent configuration for Proactive Agent.
+
+An Agent is a named composition of system instruction (SI), developer
+instruction (DI), and tool declarations.  It is embodiment-agnostic —
+the same Agent can be used with any embodiment.
+
+Typical usage:
+
+    agent = Agent.from_name("human")
+    print(agent.system_instruction)
+    print(agent.tools)
+"""
+
+import dataclasses
+from typing import Any
+
+from prompt import si_builder
+from tool import tools as tools_lib
+
+
+@dataclasses.dataclass
+class Agent:
+  """A named composition of prompt (system instruction) and tools."""
+
+  name: str
+  system_instruction: str
+  developer_instruction: str
+  tools: list[dict[str, Any]]
+
+  # ---------------------------------------------------------------------------
+  # Pre-built presets
+  # ---------------------------------------------------------------------------
+
+  @classmethod
+  def human(cls) -> "Agent":
+    """Local/browser mode with webcam."""
+    builder = si_builder.SIBuilder()
+    builder.load_instruction_file("human_di.md")
+    return cls(
+        name="human",
+        system_instruction="",
+        developer_instruction=builder.build(),
+        tools=tools_lib.human_tools(),
+    )
+
+  @classmethod
+  def spot(cls) -> "Agent":
+    """Boston Dynamics Spot robot agent."""
+    builder = si_builder.SIBuilder()
+    builder.load_instruction_file("spot_di.md")
+    return cls(
+        name="spot",
+        system_instruction="",
+        developer_instruction=builder.build(),
+        tools=tools_lib.spot_tools(),
+    )
+
+  @classmethod
+  def tinybot(cls) -> "Agent":
+    """Tinybot robot agent."""
+    builder = si_builder.SIBuilder()
+    builder.load_instruction_file("tinybot_di.md")
+    return cls(
+        name="tinybot",
+        system_instruction="",
+        developer_instruction=builder.build(),
+        tools=tools_lib.tinybot_tools(),
+    )
+
+  # ---------------------------------------------------------------------------
+  # Name-based lookup
+  # ---------------------------------------------------------------------------
+
+  @classmethod
+  def from_name(cls, name: str) -> "Agent":
+    """Returns an Agent for the specified name string.
+
+    Args:
+      name: "human", "spot", or "tinybot" is supported for Lite.
+
+    Returns:
+      An Agent instance.
+
+    Raises:
+      ValueError: If the name is unknown.
+    """
+    presets = {
+        "human": cls.human,
+        "spot": cls.spot,
+        "tinybot": cls.tinybot,
+    }
+    if name not in presets:
+      raise ValueError(
+          f"Unknown agent name: {name}. Available: {list(presets.keys())}"
+      )
+    return presets[name]()