Minimal cross platform UI testing

[4xiom5]

Current Status

Currently, cross-platform UI tests are conducted using Karate and Appium (tests/karate/src/test/java/fe). These tests depend on a mock backend to partially simulate a server. In certain tests, messages exchanged with the server are inspected to ascertain the correct functioning of the client. While this approach allows for protocol adherence verification, it often results in convoluted and hard-to-read tests.

Proposed Approach

First, using a real implementation of the backend instead of a mock server. This should be sufficient for most test cases and simplify greatly the test writing process. Then, establishing a minimal UI specification that all frontends must adhere to. Frontends would be tested as black boxes, using only simulated user inputs and visual feedback. Finally, executed those tests across various devices and platforms, capturing screenshots to facilitate rapid identification of UI bugs. Karate and Appium should provide the necessary feature for the implementation.

Comparison

Pros

• Tests become more intuitive.
• Reduced effort in maintaining a mock backend server.
• Allows testing for (retro)compatibility between frontends and backends.

Cons

• Inability to directly verify client adherence to the protocol.
• Some bugs may stem from server-side issues.
• Unability to test for frontend response to malicious backends.

Proposed Plan

• Test feasability by creating one simple test and the necessary tools for executing it across multiple browsers/emulators on the backend(s) of choice.
• Define minimal UI requirements.
• Implement tests for these minimal UI requirements.
• (Optional) Integrate some tests into the CI pipeline.
• (Optional) Construct a websocket proxy to test more complex scenarios (e.g. malicious backends, multi-server communication, ...).