fix: handle null base64_image in MCP screenshot responses#476

Open

dzorlu wants to merge 85 commits into

huggingface:mainfrom

fleet-ai:fix/null-screenshot-url

Commits on Dec 12, 2025

fleet integartion step 0

Deniz
authored and
Deniz
committed
updated README

Deniz
authored and
Deniz
committed
readme update

Deniz
authored and
Deniz
committed

Commits on Dec 13, 2025

another iteraton

Deniz
authored and
Deniz
committed

Commits on Dec 18, 2025

readme

Deniz
authored and
Deniz
committed

Commits on Jan 17, 2026

Add FleetTaskEnv for Gymnasium-compatible task environments

authored and
Deniz
committed

Commits on Jan 21, 2026

conb

Deniz
authored and
Deniz
committed

Commits on Jan 22, 2026

Add __init__.py to envs package for pip install compatibility

Deniz
authored and
Deniz
committed
fix: Remove default image_type="mcp" to use standard ECR images

authored and
Deniz
committed
fix: Add data_key and data_version params to from_fleet()

authored and
Deniz
committed
fix: Combine data_key and data_version into Fleet SDK format

authored and
Deniz
committed
fix: Remove seed parameter from reset() call

authored and
Deniz
committed

Commits on Jan 23, 2026

feat: Add request_timeout_s parameter to FleetTaskEnv

authored and
Deniz
committed

Commits on Jan 25, 2026

Fix: make reset() a sync wrapper around reset_async()

authored and
Deniz
committed
Fix: extract text content from MCP CallToolResult

authored and
Deniz
committed
Add tests for tool extraction and reset behavior

authored and
Deniz
committed
feat: fetch tools in __init__, simplify reset_async

authored and
Deniz
committed

Commits on Jan 26, 2026

fix: add detailed logging for verifier execution failures

authored and
Deniz
committed
fix: Use Fleet SDK Task.verify_detailed() for verifier execution

authored and
Deniz
committed
Fix: fetch tools lazily in reset_async to avoid asyncio.run in async context

Deniz
authored and
Deniz
committed

Commits on Jan 28, 2026

fix: add retry with backoff for MCP list_tools and log errors

Deniz
authored and
Deniz
committed
fix: add retry logic to call_tool for connection failures

authored and
Deniz
committed

Commits on Jan 29, 2026

debug: add logging for call_tool to trace success/failure paths

Deniz
authored and
Deniz
committed
fix: unwrap ExceptionGroup to show actual error cause

Deniz
authored and
Deniz
committed
debug: add logging for Fleet instance creation timing

Deniz
authored and
Deniz
committed

Commits on Feb 5, 2026

fix: Add retry logic to Fleet.make() for transient failures

authored and
Deniz
committed

Commits on Feb 8, 2026

feat(fleet_env): add ContextManager for context management tools

authored and
Deniz
committed

Commits on Feb 10, 2026

Use image_type='mcp' for computer_use tasks

Deniz
authored and
Deniz
committed

Commits on Feb 11, 2026

fix: Fetch tools for all modalities (tool_use and computer_use)

authored and
Deniz
committed
Filter to only computer tool for computer_use modality

authored and
Deniz
committed

Commits on Feb 12, 2026

fix: Handle ImageContent in MCP and filter tools for computer_use

authored and
Deniz
committed
Add initial screenshot on reset for computer_use tasks

authored and
Deniz
committed

Commits on Feb 13, 2026

debug: Log actual screenshot result format from MCP

Deniz
authored and
Deniz
committed
fix: Handle Fleet MCP base64_image format for VL models

Deniz
authored and
Deniz
committed
Remove debug logging from task_env.py

Deniz
authored and
Deniz
committed
test: Add tests for base64_image format handling

Deniz
authored and
Deniz
committed

Commits on Feb 16, 2026

Add reset_timeout_s to avoid blocking on broken manager APIs

authored and
Deniz
committed

Commits on Feb 19, 2026

feat: Add env_key to error logs for better debugging

authored and
Deniz
committed

Commits on Feb 22, 2026

fix: Add HTTP-level timeouts to streamablehttp_client calls

authored and
Deniz
committed

Commits on Feb 25, 2026

fix: Raise on exhausted list_tools retries, allow retry on next reset

authored and
Deniz
committed

Commits on Feb 26, 2026

feat: Add Logfire error tracking to fleet env

authored and
Deniz
committed

Commits on Feb 27, 2026

fix(telemetry): consistent schema with env_key, env_version, task_key, modality

authored and
Deniz
committed
feat(telemetry): add fleet_mcp_tool_error for MCP server errors

authored and
Deniz
committed
fix(telemetry): set context BEFORE Fleet.make() so init failures have full context

authored and
Deniz
committed
fix: address bugbot issues in PR #6

authored and
Deniz
committed
fix: add hard timeout to MCP operations to prevent hanging

authored and
Deniz
committed
fix: async Fleet.make() to prevent event loop blocking

authored and
Deniz
committed
fix: enrich fleet_mcp_tool_error with env:version and step info

authored and
Deniz
committed

Commits on Feb 28, 2026

fix: suppress noisy logfire console output and tracebacks

authored and
Deniz
committed
Fix telemetry dashboard: count init failures as rollouts, add total_steps

authored and
Deniz
committed
Add fleet_provisioning_completed telemetry event with provisioning_time_s

authored and
Deniz
committed

Commits on Mar 3, 2026

Fix MCP endpoint routing, telemetry gap, and retry config

authored and
Deniz
committed
fix: call_tool retry used non-existent retry_base_delay attr

authored and
Deniz
committed
feat: auto-select instance TTL based on modality

authored and
Deniz
committed

Commits on Mar 4, 2026

Emit fleet_rollout_completed on close() for orphaned rollouts

authored and
Deniz
committed
Simplify orphaned rollout stop reasons to max_steps / abandoned

authored and
Deniz
committed
Add Fleet telemetry section to README

authored and
Deniz
committed
Increase tool_use TTL from 600s to 900s to reduce 502s from instance expiry

authored and
Deniz
committed
fix: use asyncio.to_thread(Fleet.make()) instead of AsyncFleet.make()

authored and
Deniz
committed
fix: wrap sync blocking calls in asyncio.to_thread() to unblock event loop

authored and
Deniz
committed

Commits on Mar 7, 2026

feat: add trace upload utilities for eval rollouts

authored and
Deniz
committed

Commits on Mar 8, 2026

fix: Convert OpenAI image_url blocks to Fleet ingest format for proper UI rendering

authored and
Deniz
committed
fix: Pass reward as score to ingest API so sessions complete

authored and
Deniz
committed
Add hint-based reward for solver RL training (Options B, C, D)

authored and
Deniz
committed
Revert "Add hint-based reward for solver RL training (Options B, C, D)"

Deniz
authored and
Deniz
committed

Commits on Mar 11, 2026

Commits on Mar 12, 2026

feat: Run verifier at close() for orphaned rollouts

authored and
Deniz
committed
Merge pull request #10 from fleet-ai/feat/close-verifier
dzorlu
authored

Commits on Mar 13, 2026

feat: Add submit_final_answer synthetic tool for carlisle tasks

authored and
Deniz
committed
merge: resolve conflicts with deniz/fleet_client (_reward_computed)

authored and
Deniz
committed
Merge pull request #11 from fleet-ai/deniz/submit-final-answer
dzorlu
authored

Commits on Mar 15, 2026

feat: Add TaskEvaluator for task generation inner loop

authored and
Deniz
committed
refactor: rewrite TaskEvaluator to use Fleet harness (POST /v1/jobs)

authored and
Deniz
committed
Fix model ID format: use provider/model prefix for Fleet harness

authored and
Deniz
committed
Make _poll_job async to avoid blocking the event loop

authored and
Deniz
committed
Fix model ID mismatch between Fleet API and configured models

authored and
Deniz
committed
fix: Remove unused json import, defensive copy DEFAULT_MODELS

authored and
Deniz
committed
Merge pull request #12 from fleet-ai/deniz/task-evaluator
dzorlu
authored

Commits on Mar 16, 2026

Expose verifier feedback properties for hint generation

authored and
Deniz
committed

Commits on Mar 17, 2026

Merge pull request #14 from fleet-ai/deniz/hint-feedback
dzorlu
authored

Commits on Mar 18, 2026

feat: Add DB query methods to FleetEnvClient

authored and
Deniz
committed

Commits on Mar 20, 2026

fix: properly await async describe/query on AsyncFleet env handles

authored and
Deniz
committed

Commits on Mar 29, 2026

fix: handle null base64_image from Fleet MCP screenshot responses

authored and
Deniz
committed