Add an End-to-End test suite for the iOS SDK - #21
Open
brionmario wants to merge 9 commits into
Open
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
brionmario
force-pushed
the
mobile-e2e-tests
branch
from
September 1, 2026 12:42
7f8eeec to
3552d85
Compare
brionmario
changed the title
brionmario
changed the title
Drive the Quickstart through real authentication against a real ThunderID server using Maestro: sign in and sign out, then register a new account and sign in as it. The suite lives in Tests/e2e and one script, run-e2e.sh, owns the whole run, so CI executes exactly what a contributor runs locally. Maestro rather than XCUITest because the sign-in and sign-up forms are rendered from the flow definition the server returns, not from static native views. The Android and Flutter SDKs tag those fields identically, so one set of selectors works across all three platforms. Also tighten certificate handling in LocalhostPinnedURLSession. It accepted an unevaluated server trust for the configured host whenever no certificate was bundled, and no certificate is bundled in practice, so validation was effectively skipped for every host in every build. It now accepts an unevaluated trust only for loopback, where no network attacker can sit, and defers to the system's own evaluation everywhere else. Refs thunder-id/thunderid#5181 Signed-off-by: Brion <info@brionmario.com>
brionmario
force-pushed
the
mobile-e2e-tests
branch
from
September 1, 2026 15:07
3552d85 to
41a5b76
Compare
CI runners have consistently failed this step at 20s while the local Mac finishes the same flow in well under that. Match the 30s budget already used for the other post-submit waits in this flow. Signed-off-by: Brion <info@brionmario.com>
Debug artifacts from a failing CI run show the credentials POST succeeding (200, ~100ms) with no further requests, yet the app stays on a blank sign-in form and 'Session active' never appears. That matches the flow's credentials-step challenge expiring: CI is slow enough at basic taps and typing that submission lands after the challenge's validity window, and the server silently resets the form rather than authenticating. Wrap the fill-and-submit in a bounded retry so a fresh challenge gets a second try instead of failing the run outright. Signed-off-by: Brion <info@brionmario.com>
Debug artifacts show tapOn's own element-lookup timeout (~17s) is no longer enough on this CI runner: after the sign-up flow's retry re-typed the username, tapOn on thunderid-field-password timed out looking for the element on both retry attempts, failing the run outright. Give it the same explicit extendedWaitUntil the username field already gets, with a 20s budget, instead of relying on tapOn's shorter implicit wait. Signed-off-by: Brion <info@brionmario.com>
run-e2e.sh calls api.github.com unauthenticated to resolve the latest ThunderID release. That call shares GitHub's 60-requests-per-hour limit across the whole GitHub Actions runner IP range, which shared macOS runners exhaust on their own, independent of anything this suite does. Thread the existing THUNDERID_AUTOMATION_BOT secret through as GH_TOKEN so both the PR builder and nightly workflow authenticate that call, lifting the limit to 5,000/hour; GH_TOKEN stays optional so a local run still works unauthenticated. Signed-off-by: Brion <info@brionmario.com>
Fork PRs run without repository secrets, so the GH_TOKEN threaded through in the previous commit is empty for this PR's own CI, and the unauthenticated release-lookup call still hits its rate limit. The nightly workflow already exists specifically to test against whatever ThunderID just released, so pin the PR builder to v1.0.1 instead of resolving 'latest' live - PR builds do not need bleeding-edge, just a fast, reproducible baseline. Signed-off-by: Brion <info@brionmario.com>
Fork PRs run without repository secrets, so this never actually authenticated the call it was meant to fix, and the PR builder is now pinned to a fixed version anyway and no longer makes that call. Keeping it around was dead complexity with no effect. Signed-off-by: Brion <info@brionmario.com>
Every flow starts here, and a bare assertVisible gets Maestro's short default lookup timeout rather than the generous budget the rest of the suite uses. On a loaded CI emulator the landing screen does not always render inside it, which failed the Android sign-up flow 19 seconds in, before the flow had done anything. Wait for it explicitly instead. Signed-off-by: Brion <info@brionmario.com>
CI installed whatever Maestro was newest at the time, so the test runner changed under the suite between runs with nothing in the repository to show it: CI has been on 2.10.0 since it shipped while a contributor following the README gets whatever is current, which makes a CI-only failure impossible to reproduce faithfully. Pin it, and bump deliberately after checking the flows. The JUnit report gives a failed run a machine-readable result instead of a console log to scrape, and CI now collects it with the other debug artifacts. Signed-off-by: Brion <info@brionmario.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The iOS SDK had unit tests but nothing that exercised it the way a person uses it. Unit tests stub the server out, so they cannot catch the failures that matter most to someone integrating the SDK: a flow step the SDK renders but cannot submit, a field the server renames, or an application whose configuration the SDK rejects at run time.
This adds a Maestro suite that drives the Quickstart sample against a real ThunderID server. It signs a user in, signs them out, registers a new account, and signs in as that new account.
It also fixes a certificate validation problem found while building the suite.
LocalhostPinnedURLSessionis installed on every request, and when no certificate is bundled, which is the case in practice, it accepted an unevaluated server trust for the configured host.SecTrustEvaluatewas never called anywhere inSources/. The effect was that certificate validation was skipped for whatever hostbaseUrlpointed at, in release builds, with no opt-in.Approach
Maestro rather than XCUITest. The sign-in and sign-up forms are not static native views: the server returns a flow definition and the SDK renders it. All three mobile SDKs tag the resulting fields the same way,
thunderid-field-<identifier>andthunderid-action-<ref>, so one set of selectors and one set of flows works across iOS, Android and Flutter. No SDK change was needed here, because.accessibilityIdentifieralready reaches the accessibility tree.One script owns the run.
Tests/e2e/run-e2e.shstarts a server, provisions the test application and user, builds and installs the sample, and runs the flows. CI calls the same script through a composite action, so a green run locally and a green run in CI mean the same thing and there is no second implementation to drift. Stages are skippable (--skip-server,--skip-build) and every stage is idempotent.The test application is declarative.
thunderid-config.yamldefines atype: mobileapplication withattestation.devMode: true. A mobile application must normally prove its binary identity before it can initiate a flow, and Apple App Attest does not exist in the Simulator, so the check can never be satisfied on the device the suite drives.devModeis test only.Certificate handling. The delegate now accepts an unevaluated trust only for loopback hosts, where no network attacker can sit, and defers to the system's own evaluation for everything else. Pinning still takes precedence when a certificate is bundled. This keeps the local development workflow working unchanged, which the suite itself demonstrates.
Two behaviours worth knowing about, both documented in
Tests/e2e/README.md:clearState: trueand reinstalling the app, so every flow starts with a subflow that signs out if a session is present.Related Issues
Related PRs
Checklist
Tests/e2e/README.md, plus the SDK Development section of the contributor docsTests/e2e/flows/signin.yaml,Tests/e2e/flows/signup.yamlbreaking changelabel added.Security checks
The credentials in this PR (
e2e_mobile_user/TestPassword@123) belong to a throwaway account the suite creates on a local server. They are not secrets and grant nothing anywhere else.Verification
swift buildandswift testpass, 130 tests, 0 failures.swiftlint lint --strictis clean. The suite was run end to end against a freshly downloaded ThunderID v1.0.1 on an iOS Simulator, 2 of 2 flows passing, both before and after the certificate change.