Repository navigation
test(conformance): golden-transcript scenario format, newsletter seed, and outcome tool - #3876
Closed
gspencergoog wants to merge 52 commits into
Closed
gspencergoog wants to merge 52 commits into
gspencergoog wants to merge 52 commits into
Conversation
… dispatch Add suite_registry.dart and suite_registry_test.dart to map every conformance/core/*.yaml suite and action to its test file, record executed cases to A2UI_CONFORMANCE_MANIFEST_DIR, enforce known-key checks across all Dart core conformance harnesses, default matchesErrorFields onUnassertedField to fail, and verify the manifest in Flutter CI.
…ispatch Have the Python core conformance harness emit executed-case manifest records when A2UI_CONFORMANCE_MANIFEST_DIR is set, validate every case and expectation block against known-key tables, verify that UNRUNNABLE_SUITES, UNSUPPORTED, and EXPECTED_FAILURES entries match loaded cases, and check the manifest in Python CI.
…it scaling and skip tables Re-raise UnknownConformanceKeyError from expected-failure wrappers in Python, Dart, and Swift so YAML schema drift is not recorded as xfailed. Verify SKIP_TEST_NAMES in TypeScript verifyHarnessInvariants, add key validation and manifest recording to Swift MessageProcessorScalingConformanceTests, and update the Swift CI exclude reason.
Allow the payload key in Dart message_processor_scaling_conformance_test, handle .integer in ConformanceTestHelper.fromJSONValue, isolate ConformanceManifest.runAsync to MainActor, and unwrap optional action string interpolation in ValidatorConformanceTests.
…rrors Wrap captured RendererError values in a reference box in ExpressionsConformanceTests so outboundListener does not mutate a local array while assertValidateStepError holds an active reference.
Add DataError case to ValidatorConformanceTests.assertErrorCategory so test_error_response_update_data_model_prototype_pollution verifies A2UIDataError.
Why: Wireit cached test:conformance with an empty output list, skipping conformance_test.mjs on cache hits and leaving A2UI_CONFORMANCE_MANIFEST_DIR empty when CI runs check_executed_cases.py. What: Remove the files and output properties from the test:conformance Wireit script so it runs on every test invocation like test and test:timing.
…mance-expect-outbound
… adapter harnesses
…mance-expect-outbound
…ck_executed_cases
…derer-conformance-manifests # Conflicts: # typescript/renderers/angular/karma.conf.js # typescript/renderers/angular/src/adapter_conformance.spec.ts
…ftui-hosted-view-driver # Conflicts: # typescript/renderers/angular/karma.conf.js # typescript/renderers/angular/src/adapter_conformance.spec.ts
…pter-conformance-expect-outbound # Conflicts: # typescript/renderers/angular/karma.conf.js # typescript/renderers/angular/src/adapter_conformance.spec.ts
…rmance-manifests # Conflicts: # typescript/renderers/lit/src/tests/adapter_conformance.test.ts
…hosted-view-driver
…ormance-expect-outbound
…formance-manifests
…hosted-view-driver
…ormance-expect-outbound
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Collaborator
Author
|
Consolidated into #3918. 🤖 Agent-generated |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Defines the golden-transcript end-to-end scenario format (
action: e2e_scenario) inconformance/, adds the multi-turnnewsletter_signupv1.0 seed scenario, and provides the renderer-visible outcome normalizer and comparison CLI (e2e_outcome.pyandcompare_e2e_outcome.py).Changes
conformance/conformance_schema.json: Adds$defs/E2eScenarioTest,E2eTurn,E2eUiOp,E2eViewTrees, andE2eSource, and registerse2e_scenarioin the conformance action enum.conformance/e2e/newsletter_signup.yaml&conformance/tests/test_e2e_scenarios.py: Adds the multi-turn v1.0 newsletter signup scenario (with bothdirect_jsonandexpressstreaming model chunks,expectWire,expectViewTree,uiOps,expectDataModel, andexpectOutbound) and validates every scenario against the protocol JSON schemas and@a2ui/agent/a2ui.agentparsers.conformance/scripts/e2e_outcome.py,conformance/scripts/compare_e2e_outcome.py, &conformance/tests/test_e2e_outcome.py: Computes the renderer-visible outcome of a scenario (normalizing model-chosen component, surface, and function call IDs while ignoring non-UI chat prose changes) and provides a CLI comparison tool for nightly drift detection.conformance/manifest_corpora.json,conformance/README.md, &conformance/conformance_harness_implementation_guide.md: Registers thee2ecorpus and documents the golden-transcript harness specification.Testing
pytest conformance/tests/test_e2e_scenarios.py conformance/tests/test_e2e_outcome.py conformance/tests/test_conformance_yaml.py -q