[Office Hours] SpeziScheduler Unstable UI testing

[dusixian]

Related Discussion

Hi! Previously I created a PR to improve the documentation for SpeziScheduler. However, the UI tests failed twice. Since my PR only updates the documentation and doesn’t touch the core code, it shouldn’t affect the test results. I reviewed the UI test videos, and it seems like the failures are due to existing test instability. Let me explain:

1. UI Test Failure 1.
   • Issue: The test failed at line 90 of TestAppUITests while checking for the "Next trigger, 10 seconds" label.
   • Reason: The app.assertNotificationDetails method checks the notification label approximately every second. However, the interval isn’t perfectly precise — the check might occur just as the label changes from 10 seconds to 9 seconds, causing the assertion to fail. This randomness explains why the test passed on a re-run.
   • Suggestion: We could update assertNotificationDetails to either check more frequently or allow both 9 seconds and 10 seconds as valid matches. Passing the test if either condition is met would make the test more reliable.

2. UI Test Failure 2
   • Issue: The test failed at line 106 of TestAppUITests while checking for the weight notification.
   • Reason: The lab result notification and weight notification are triggered simultaneously , so the lab result notification covers the weight notification, causing the test to fail.
   • Suggestion: I tried solving this by adding app.buttons["Complete Enter Lab Results"].tap() to dismiss the lab result notification. However, I couldn’t run the test locally — after click run this test in the sidebar, a successful build, the test immediately succeeds without actually executing. Could you guide me on how to properly run the test locally, since I don’t have permission to run GitHub Actions?

I hope these insights contribute to more stable UI tests! I’m not sure if I can get everything fixed before the deadline, but I’m eager to improve the test logic. Meanwhile, maybe the PR can be reviewed based on the documentation changes, and I’d be happy to continue working on the test stability afterward.

Reproduction

None.

Additional context

No response

Code of Conduct

• I agree to follow this course's Code of Conduct and Contributing Guidelines

[dusixian]

Involve @lukaskollmer @PSchmiedmayer in this discussion.

[lukaskollmer]

sorry for the late response; i completely missed that you tagged me here.

the UI tests are extremely flaky at times; there sadly isn't anything specific i can recommend here to improve the stability. i've seen this exact issue in the past (it sometimes seems to miss the "in 10 seconds" text, even though it clearly is on screen). the best you can do in a situation like this is to re-run the test case a couple times, until it eventually succeeds.

[dusixian]

No worries! Just some suggestions to improve the stability.