Repository navigation
test(e2e): add agent-half and native renderer-half E2E scenario runners - #3879
Closed
gspencergoog wants to merge 69 commits into
Closed
gspencergoog wants to merge 69 commits into
gspencergoog wants to merge 69 commits into
Conversation
… dispatch Add suite_registry.dart and suite_registry_test.dart to map every conformance/core/*.yaml suite and action to its test file, record executed cases to A2UI_CONFORMANCE_MANIFEST_DIR, enforce known-key checks across all Dart core conformance harnesses, default matchesErrorFields onUnassertedField to fail, and verify the manifest in Flutter CI.
…ispatch Have the Python core conformance harness emit executed-case manifest records when A2UI_CONFORMANCE_MANIFEST_DIR is set, validate every case and expectation block against known-key tables, verify that UNRUNNABLE_SUITES, UNSUPPORTED, and EXPECTED_FAILURES entries match loaded cases, and check the manifest in Python CI.
…it scaling and skip tables Re-raise UnknownConformanceKeyError from expected-failure wrappers in Python, Dart, and Swift so YAML schema drift is not recorded as xfailed. Verify SKIP_TEST_NAMES in TypeScript verifyHarnessInvariants, add key validation and manifest recording to Swift MessageProcessorScalingConformanceTests, and update the Swift CI exclude reason.
Allow the payload key in Dart message_processor_scaling_conformance_test, handle .integer in ConformanceTestHelper.fromJSONValue, isolate ConformanceManifest.runAsync to MainActor, and unwrap optional action string interpolation in ValidatorConformanceTests.
…rrors Wrap captured RendererError values in a reference box in ExpressionsConformanceTests so outboundListener does not mutate a local array while assertValidateStepError holds an active reference.
Add DataError case to ValidatorConformanceTests.assertErrorCategory so test_error_response_update_data_model_prototype_pollution verifies A2UIDataError.
Why: Wireit cached test:conformance with an empty output list, skipping conformance_test.mjs on cache hits and leaving A2UI_CONFORMANCE_MANIFEST_DIR empty when CI runs check_executed_cases.py. What: Remove the files and output properties from the test:conformance Wireit script so it runs on every test invocation like test and test:timing.
…n the extension harnesses
…esses Add strict key/action dispatch, an executed-case manifest writer, reasoned skip records for unsupported and performance-tagged agent suites, and the CI manifest check for dart/a2ui_agent.
…rnesses - Remove duplicate recordCase call from skipVerdict in the TypeScript agent conformance loader and drop the redundant recordedSkipped deduplication set in manifest.ts. - Assert context.add_activated_extension invocations in the Python A2A try_activate_a2ui_extension conformance runner. - Document the Python A2A and ADK extension conformance harnesses in the harness implementation guide.
…n Python agent harnesses Add a pytest hookwrapper in python/a2ui_agent/tests/conformance/conftest.py to write executed-case manifest records for the python-agent harness, emit explicit skip marks with reasons for skipped cases, performance-only scaling tests, and unsupported YAML suites, enforce strict expectation key allowlists across the Python agent conformance runners, and verify the python-agent manifest in Python CI.
…ties.json Validate that get_a2ui_agent_extension emits v1.0 AgentCard extension parameters matching the inner v1.0 object schema in specification/v1_0/json/agent_capabilities.json and remains flat without a redundant version key.
…hon-agent-manifest-and-a2a-guard
…derer-conformance-manifests # Conflicts: # typescript/renderers/angular/karma.conf.js # typescript/renderers/angular/src/adapter_conformance.spec.ts
…ftui-hosted-view-driver # Conflicts: # typescript/renderers/angular/karma.conf.js # typescript/renderers/angular/src/adapter_conformance.spec.ts
…pter-conformance-expect-outbound # Conflicts: # typescript/renderers/angular/karma.conf.js # typescript/renderers/angular/src/adapter_conformance.spec.ts
…rmance-manifests # Conflicts: # typescript/renderers/lit/src/tests/adapter_conformance.test.ts
…hosted-view-driver
…ormance-expect-outbound
…formance-manifests
…hosted-view-driver
…ormance-expect-outbound
…-scenario-runners-agent-and-native
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Collaborator
Author
|
Consolidated into #3918. 🤖 Agent-generated |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds golden-transcript E2E scenario conformance runners across the Python, TypeScript, and Dart agent SDKs (
--corpus e2eforpython-agent,ts-agent, anddart-agent) as well as the Flutter and SwiftUI renderer adapters (--corpus e2eforflutterandswiftui).python/a2ui_agent/tests/conformance/test_e2e_conformance.py): Replays each scenario turn'sagent.inputthroughCatalog.from_jsonandA2uiRequestProcessor, validates emitted messages againstexpectServerMessages, and writes.e2e-outcome.jsonwhenA2UI_E2E_OUTCOME_OUTis set.typescript/agents/a2ui_agent/tests/conformance/e2e-scenarios.test.ts): Replays each scenario turn'sagent.inputthroughCatalog.fromJsonandA2uiRequestProcessor, validates emitted messages againstexpectServerMessages, and writes.e2e-outcome.jsonwhenA2UI_E2E_OUTCOME_OUTis set.dart/a2ui_agent/test/conformance/e2e_scenarios_conformance_test.dart): Replays each scenario turn'sagent.inputthroughCatalog.fromJsonandA2uiRequestProcessor, validates emitted messages againstexpectServerMessages, and writes.e2e-outcome.jsonwhenA2UI_E2E_OUTCOME_OUTis set.dart/a2ui_flutter/test/e2e_scenarios_test.dart): Replays each scenario turn'sexpectServerMessagesthroughCatalog.fromJson,MessageProcessor, andA2uiSurfaceView, checksexpectRenderedandsimulatesteps, and validates outbound wire envelopes againstexpectOutbound.swift/swiftui/Tests/A2UISwiftUITests/E2EScenarioTests.swift): Replays each scenario turn'sexpectServerMessagesthroughHostedViewDriver, checksexpectRenderedandsimulatesteps, and validates outbound wire envelopes againstexpectOutbound.Testing
pytest python/a2ui_agent/tests/conformance/test_e2e_conformance.py -qvitest run tests/conformance/e2e-scenarios.test.tsdart test test/conformance/e2e_scenarios_conformance_test.dartflutter test test/e2e_scenarios_test.dartpython3 conformance/scripts/check_executed_cases.py --corpus e2e(python-agent,ts-agent,dart-agent,flutter)