APPENG-6165, APPENG-6166: Gmail & Calendar read-only test cases - #1
Open
csoceanu wants to merge 3 commits into
Open
Conversation
Implement 3 new read-only Gmail evaluation cases with deterministic judges: - case-010: List inbox → extract action-item code from team lead email - case-011: Search by sender with forbidden_codes → locate approval code - case-012: Read 3-message thread → summarize technical decision New judge: score_gmail_read_extraction() validates that agent's response contains all expected_codes and none of forbidden_codes. Proves real reads via response content (needle-in-response pattern), not API side effects. Seeder extension: _seed_gmail() supports messages: [...] for multi-message threads chained via threadId. Single-message path unchanged (backward compat). All 8 cases (001-005 + 010-012) pass 100% with deterministic judges: ✅ used_exec_tool: verified real curl calls (no hallucination) ✅ smolclaw_gmail_read: verified code extraction from responses ✅ response_received: confirmed agents replied ✅ Trajectory: 8-16 exec tool calls per case (real API interactions) ✅ smolclaw state: seeded before execution, agents read from live db Acceptance criteria met: ✅ All cases pass 100% judge pass rate ✅ Trajectory events show real exec tool use (no hallucination) ✅ smolclaw state reflects expected read operations Realistic seed data: - case-010: Team sprint review (internal mgmt email) - case-011: CI/CD alerts with decoys (search filtering test) - case-012: Technical decision thread (multi-person discussion) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…nd fix threading - Add case-013: Test list-mail-folders and list-mail-folder-messages MVP actions * Uses GET /users/me/labels to enumerate all available folders * Uses GET /users/me/messages?labelIds=<id> to list messages in specific folder * Reads message with GET /users/me/messages/<id>?format=full * Extracts project tracking code from labeled email - Fix smolclaw_seed.py: Extract thread subject from last message (messages[-1]) * Matches Gmail API behavior where thread subject comes from final message * Fixes case-012 (Carol's "RE: Database migration plan") and case-013 - Update documentation: * Add case-013 to case table and usage examples * Document threading fix in Part 3 * Add trajectory examples for label operations * Update bash runner with Gmail READ case example All 4 MVP actions (list-messages, list-folders, list-folder-messages, get-message) now have comprehensive test coverage with real exec tool usage verified. Co-authored-by: Cursor <cursoragent@cursor.com>
csoceanu
force-pushed
the
appeng-6165-gmail-operations
branch
from
August 24, 2026 14:27
e19f4a9 to
22a6592
Compare
csoceanu
changed the title
… single-usecase coverage Redesign cases 010-018 so each tests exactly one MVP requirement: Gmail (APPENG-6165): - case-010: list-mail-messages (3 independent emails, check all subjects) - case-011: get-mail-message (read by SMOLCLAW_SEED_MESSAGE_ID) - case-012: list-mail-folders (create custom label, check default + custom) - case-013: list-mail-folder-messages (create label + seed email in it) Calendar (APPENG-6166): - case-014: list-calendar-events (3 events, check all summaries) - case-015: list-calendars (create secondary calendar) - case-016: get-calendar-event (read by SMOLCLAW_SEED_EVENT_ID) - case-017: get-calendar-view (timeMin/timeMax query) - case-018: get-schedule (query + compute free/busy) smolclaw_seed.py: add create_labels, create_calendars, seed_to_label, threaded:false for independent messages, events list for multi-event seeding. smolclaw_score.py: add score_gmail_read_extraction, score_calendar_read_extraction. All 14 cases pass e2e at 100% with real exec tool use, no hallucination. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
csoceanu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
create_labels,create_calendars,seed_to_label,threaded:false, multi-event seedingscore_gmail_read_extraction,score_calendar_read_extractionTest Cases
Gmail (APPENG-6165)
Calendar (APPENG-6166)
Design Principles
E2E Results
All 14 cases (001-005 existing + 010-018 new) pass at 100%:
Files Changed