Repository navigation
test(conformance): executed-case manifests for renderer, agent, and extension harnesses - #3823
Merged
gspencergoog merged 60 commits intoNov 5, 2026
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request implements a comprehensive executed-case manifest system for conformance testing across multiple language harnesses (Dart, Python, Swift, TypeScript, Angular, Lit, and React). It introduces JSON schema validation for manifest records, a Python script to verify executed cases against the suite corpora, and enforces strict key validation within test cases. A critical issue was identified in the Angular test harness where accessing process.env directly in the browser context will result in a ReferenceError.
gspencergoog
force-pushed
the
test/renderer-conformance-manifests
branch
2 times, most recently
from
November 2, 2026 07:42
bb83cb0 to
1d44081
Compare
… dispatch Add suite_registry.dart and suite_registry_test.dart to map every conformance/core/*.yaml suite and action to its test file, record executed cases to A2UI_CONFORMANCE_MANIFEST_DIR, enforce known-key checks across all Dart core conformance harnesses, default matchesErrorFields onUnassertedField to fail, and verify the manifest in Flutter CI.
…ispatch Have the Python core conformance harness emit executed-case manifest records when A2UI_CONFORMANCE_MANIFEST_DIR is set, validate every case and expectation block against known-key tables, verify that UNRUNNABLE_SUITES, UNSUPPORTED, and EXPECTED_FAILURES entries match loaded cases, and check the manifest in Python CI.
…it scaling and skip tables Re-raise UnknownConformanceKeyError from expected-failure wrappers in Python, Dart, and Swift so YAML schema drift is not recorded as xfailed. Verify SKIP_TEST_NAMES in TypeScript verifyHarnessInvariants, add key validation and manifest recording to Swift MessageProcessorScalingConformanceTests, and update the Swift CI exclude reason.
Allow the payload key in Dart message_processor_scaling_conformance_test, handle .integer in ConformanceTestHelper.fromJSONValue, isolate ConformanceManifest.runAsync to MainActor, and unwrap optional action string interpolation in ValidatorConformanceTests.
…rrors Wrap captured RendererError values in a reference box in ExpressionsConformanceTests so outboundListener does not mutate a local array while assertValidateStepError holds an active reference.
Add DataError case to ValidatorConformanceTests.assertErrorCategory so test_error_response_update_data_model_prototype_pollution verifies A2UIDataError.
Why: Wireit cached test:conformance with an empty output list, skipping conformance_test.mjs on cache hits and leaving A2UI_CONFORMANCE_MANIFEST_DIR empty when CI runs check_executed_cases.py. What: Remove the files and output properties from the test:conformance Wireit script so it runs on every test invocation like test and test:timing.
…n the extension harnesses
…esses Add strict key/action dispatch, an executed-case manifest writer, reasoned skip records for unsupported and performance-tagged agent suites, and the CI manifest check for dart/a2ui_agent.
…rnesses - Remove duplicate recordCase call from skipVerdict in the TypeScript agent conformance loader and drop the redundant recordedSkipped deduplication set in manifest.ts. - Assert context.add_activated_extension invocations in the Python A2A try_activate_a2ui_extension conformance runner. - Document the Python A2A and ADK extension conformance harnesses in the harness implementation guide.
gspencergoog
force-pushed
the
test/renderer-conformance-manifests
branch
from
November 2, 2026 16:53
a1d126e to
e89d676
Compare
… cases Upstream added core/agent_function_routing.yaml, so the Swift suite registry must list it, and its cases need executed-case manifest records. The suite also used the 30-second wall-clock call timeout, which a loaded test run can exceed between two steps; the cases settle every call explicitly, so disable it.
…istry suite_registry_test requires every conformance/core/*.yaml file to be listed, and core/agent_function_routing.yaml, which agent_function_routing_conformance_test.dart already runs, was missing.
…istry suite_registry_test requires every conformance/core/*.yaml file to be listed, and core/agent_function_routing.yaml, which agent_function_routing_conformance_test.dart already runs, was missing.
…derer-conformance-manifests
gspencergoog
marked this pull request as ready for review
November 4, 2026 20:43
This was referenced Nov 4, 2026
nan-yu
requested changes
Nov 4, 2026
nan-yu
left a comment
nan-yu
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed PR #3823 (6bca6cc09c) across all 61 changed files (web/Flutter/SwiftUI adapter harnesses, Python/TypeScript/Dart agent harnesses, A2A/ADK extension harnesses, CI workflows, and conformance docs).
Overall the harness extractions, CI manifest checks, and agent_capabilities.json validation are well-structured, and Wireit/workflow triggers were verified clean. Main issues to address before merge:
- Dropped/misreported manifest records:
node:testskipsafterEachwhent.skip()is called (web_coreandlitdropskippedrecords); Angular's Jasmine reporter mapsexcludedspecs (fit/fdescribe) to reasonedskipped; SwiftUI'swithKnownIssue/#expect/if letpaths diverge from manifest statuses (anduser_activation_gated_function_gesture_vs_programmaticcan pass natively by forwardingstep["userActivation"]toNode.action(for:userActivation:)). - Agent harness gaps:
dart-agentandts-agentlist implementedagent/*/scaling.yamlsuites inunsupported_suites_test(masking broken scaling tests and emittingDUPLICATEunder--run-skippedin Dart);ts-agentomitsassertKnownKeys(leaving several YAML fields silently ignored), reversesidvsnameprecedence inloadCases, and tracksactiveCasein module-global mutable state; Python_extract_skip_reasonsynthesizes fallback reasons that bypassUNREASONEDchecks, andxfailwrappers swallow unknown-key failures.
🤖 Agent-generated
gspencergoog
force-pushed
the
test/renderer-conformance-manifests
branch
from
November 4, 2026 22:35
e4d8ea4 to
955365e
Compare
gspencergoog
force-pushed
the
test/renderer-conformance-manifests
branch
from
November 4, 2026 22:41
955365e to
916c56a
Compare
nan-yu
approved these changes
Nov 5, 2026
nan-yu
left a comment
nan-yu
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for addressing all 16 comments from the previous review pass so thoroughly! Verified the fixes across the web/Flutter/SwiftUI adapter harnesses, the Python/TypeScript/Dart agent runners, the A2A/ADK extension runners, the CI workflows, and the upstream/main merge in test_macro_conformance.py. LGTM!
🤖 Agent-generated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Before this change,
conformance/scripts/check_executed_cases.pychecked executed-case manifests only for thecoreconformance corpus. Renderer adapters, agent SDK runners, and framework extension runners could skip a suite or test case without failing CI. Each renderer adapter test file also embedded its own copy of mounting, step execution, and view-tree inspection logic.This pull request extracts shared
adapter_harnessmodules across the web, Flutter, and SwiftUI renderers. It also wires executed-case manifest emission (A2UI_CONFORMANCE_MANIFEST_DIR) andcheck_executed_cases.pyCI checks into theadapter,agent, andextensionscorpora.Reviewer Focus
conformance/conformance_harness_implementation_guide.mdand the CI updates in.github/workflows/web_ci.yml,.github/workflows/flutter_packages_test.yml,.github/workflows/swift_build_and_test.yml, and.github/workflows/python_ci.yml.typescript/renderers/lit/src/tests/adapter_harness.ts,typescript/renderers/react/tests/adapter_harness.tsx,typescript/renderers/angular/src/adapter_harness.ts,dart/a2ui_flutter/test/adapter_harness.dart, andswift/swiftui/Tests/A2UISwiftUITests/AdapterHarness.swift. Pay particular attention totypescript/renderers/angular/karma.conf.js, which bridges browserwindow.__karma__.info({ a2uiConformanceCase })messages to the host manifest JSONL file.python/a2ui_agent/tests/conformance/,typescript/agents/a2ui_agent/tests/conformance/, anddart/a2ui_agent/test/conformance/, pluspython/a2ui_a2a/tests/andpython/a2ui_adk/tests/.test_*.pyfiles underpython/a2ui_agent/tests/conformance/. They replace localload_suiteboilerplate with the shareddiscover_suite_caseshelper andrecord_executed_casefixture.Changes
Renderer Adapter Runners (
typescript/renderers/,dart/a2ui_flutter/,swift/swiftui/)assert_viewop checks into siblingadapter_harnessmodules in Lit, React, Angular, Flutter, and SwiftUI.typescript/core/web_core/tests/conformance/accessibility_conformance.test.ts(web_core_universal) and all five framework adapter runners to emit one JSONL manifest line per case whenA2UI_CONFORMANCE_MANIFEST_DIRis set.a2ui-conformance-manifestKarma reporter plugin intypescript/renderers/angular/karma.conf.jsso headless Chrome tests persist manifest lines on the host filesystem.Agent SDK Conformance Runners (
python/a2ui_agent/,typescript/agents/a2ui_agent/,dart/a2ui_agent/)discover_suite_cases, strict assertion-key validation, and manifest recording acrosspython/a2ui_agent/tests/conformance/conftest.py,typescript/agents/a2ui_agent/tests/conformance/manifest.ts, anddart/a2ui_agent/test/conformance/manifest.dart.unsupported_suitesrunners in Python, TypeScript, and Dart so suites listed underunsupported_suitesinconformance/expected_failures.yamlemitstatus: "skipped"manifest records.Extension Conformance Runners (
python/a2ui_a2a/,python/a2ui_adk/,conformance/extensions/)conftest.py) and strict action/assertion-key dispatch topython/a2ui_a2a/tests/test_a2a_integration.py,python/a2ui_a2a/tests/test_extension.py, andpython/a2ui_adk/tests/test_adk_extensions.py.AgentCardextension capability parameters againstspecification/v1_0/json/agent_capabilities.jsonduringcreate_agent_cardconformance steps.conformance/extensions/a2a/a2a_integration.yamlthat are already covered inconformance/extensions/a2a/extension.yaml.CI Workflows and Conformance Guide (
.github/workflows/,conformance/).github/workflows/web_ci.yml,.github/workflows/flutter_packages_test.yml,.github/workflows/swift_build_and_test.yml, and.github/workflows/python_ci.ymlto setA2UI_CONFORMANCE_MANIFEST_DIRand runconformance/scripts/check_executed_cases.py.conformance/conformance_harness_implementation_guide.md.Impact & Risks
conformance/adapter/,conformance/agent/, orconformance/extensions/must run in a registered runner or appear inconformance/expected_failures.yamlto pass CI.Testing
A2UI_CONFORMANCE_MANIFEST_DIR=<tmpdir>across Lit, React, Angular,web_core_universal, Flutter (a2ui_flutter), and SwiftUI (A2UISwiftUITests), then ranpython3 conformance/scripts/check_executed_cases.py --corpus adapter --manifest-dir <tmpdir>.pytest), TypeScript (vitest), and Dart (dart test)a2ui_agentconformance suites withA2UI_CONFORMANCE_MANIFEST_DIRset and ranpython3 conformance/scripts/check_executed_cases.py --corpus agent --manifest-dir <tmpdir>.pytestonpython/a2ui_a2a/tests/andpython/a2ui_adk/tests/withA2UI_CONFORMANCE_MANIFEST_DIRset and ranpython3 conformance/scripts/check_executed_cases.py --corpus extensions --manifest-dir <tmpdir>.🤖 Agent-generated