Sync Integration Tests
The aim of these tests is to check that the synchronization is working correctly between Mobile and Desktop. In order to achieve that these tests are implemented verify that History, Bookmarks, Tabs and Logins items are successfully created on one end and they appear on the other if user is signed in with the same FxA account.
Location of the tests
Tests are in the repo under the Firefox-iOS Integration Test Suite, and the directory to run the tests, SyncIntegrationTests
The test framework is written in Python. It is needed to put together the Desktop and Mobile enviroments in the same test and check the sync process. As well as to show the logs to debug and present the final results.
To run these tests it is necessary to have Python 3 and pipenv installed. Once you have these, make sure you're in the SyncIntegrationTests directory and run the following: $ pipenv install $ pipenv run pytest
That will start running the tests that are defined in test_integration.py file. Let's see how it work with an example. Look at this extract of that file corresponding to a test:
def test_sync_history_from_desktop(tps, xcodebuild):
# Running tests
tps.run('test_history.js')
xcodebuild.test('checkHistoryFromDesktopTest')
Firstly, a newely stage FxA account is created using FxACli tool that will be used to sign in both on Desktop and Firefox-iOS. This account is destroyed when the test finishes.
Secondly, Nightly, latest version on mozilla central, is launched using TPS tool. The test signed in usign previous account and thanks to TPS there is one history entry added to Desktop which is defined in that test_history.js file.
Finally, once that part of the test finishes successfully, the iOS mobile part will begin. That means that the simulator will be launched and the test will start running. The test will sign in using same account on Firefox-iOS and then will go to the History menu to check that the same entry is there.
Once the test finishes there will be a report with the information about test passed/failed and detailed logs in case of failure.
Html report example:
These are the scenarios that are covered by these tests:
- Check bookmark created on Mobile is synced on Desktop
- Check bookmark created on Desktop is synced on Mobile
- Check history created on Mobile is synced on Desktop
- Check history created on Desktop is synced on Mobile
- Check tab created on Mobile is synced on Desktop
- Check tab created on Desktop is synced on Mobile
- Check login created on Desktop is synced on Mobile
- Check login and bookmark created on Deskto are sync on Mobile, sign out and sign in on Mobile to verify the entries are shown
- Check china server configuration
Currently these tests are running daily on Jenkins. They run on latest main and on the branch with latest Application Services version released, which may or may not be the one used in main. The Jenkins job is set up in a Mac Stadium instance.
There is a slack notification on #firefox-ios-alerts with all the information related to each test run. In that notification there is a link to the Jenkins job in Mac Stadium and also a public link to the html report with all the logs.
It would be nice to have more coverage and with that cover more scenarios as well as more devices and/or iOS versions.
For any doubt or comment about these tests, please contact Mobile Test Engineering team, slack: #mobile-testeng