AzureIntegrationTests share one billing profile #1702
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I first converted AzureIntegrationTest to JUnit5. This was necessary in order to create @BeforeAll and @afterall methods to initialize a single Azure billing profile for use in all tests. Expected benefits: - If one test in the suite fails to delete its billing profile on tear-down, all other tests in the suite currently fail fast when trying to create datasets with errors like "Cannot reuse existing application deployment integrationapp from profile … with a different profile …". This change should allow for more precise failure signaling, and increase the efficacy of retries. - Serve as a model for sharing test initialization within a class, not currently used elsewhere in TDR. Overall integration test runtime can take 2+ hours, and sharing set-up may help drive that down while incentivizing writing more granular tests when appropriate rather than overloading existing ones.
okotsopoulos
commented
Jun 5, 2024
| @Category(Integration.class) | ||
| public class AzureIntegrationTest extends UsersBase { | ||
| @Tag(Integration.TAG) | ||
| @TestInstance(TestInstance.Lifecycle.PER_CLASS) |
There was a problem hiding this comment.
This annotation is necessary to use @BeforeAll and @AfterAll annotations on non-static methods.
AzureIntegrationTest had a TestJobWatcher included as a @rule, which is no longer supported in JUnit 5. Instead, we can extend the test class with new TestJobWatcherExtension, which is compatible with JUnit 5. Also refactored two assertions which still used JUnit 4 methods.
To make debugging easier.
|
okotsopoulos
commented
Jun 5, 2024
| - name: "Run integration tests via Gradle" | ||
| uses: broadinstitute/datarepo-actions/actions/[email protected] | ||
| with: | ||
| actions_subcommand: 'gradleinttest' | ||
| pgport: ${{ job.services.postgres.ports[5432] }} | ||
| test_to_run: 'testIntegration' | ||
| test_filter: 'AzureIntegrationTest' ## DO NOT MERGE. |
There was a problem hiding this comment.
I won't check this in, but PSA that this is a supported filter by way of this PR: broadinstitute/datarepo-actions#74
To help triangulate source of flaky integration tests. The valuable part of this work is sharing a billing profile among tests, so removing other possible sources of flakiness.
- Lost a closing quote when converting to multiline string - Typo in docstring
Autowire it into AzureIntegrationTest rather than having the test extend it. Fix an instance where we mistakenly called UsersBase.steward() in a test where we should have used the steward user initialized in test setup.
Previously, the deployed app <-> billing profile association is registered the first time a dataset is created off of the profile. I believe this is producing a race condition / flaky integration tests using the same billing profile. - Added new CreateProfileRegisterDeployedApplicationStep in ProfileCreateFlight - Notably, added this after the profile metadata was saved and made available as a job response in the working map - Expanded docstrings, unit testing of AzureApplicationDeploymentService.getOrRegisterApplicationDeployment
Test failures and flakiness seems to happen… - During dataset creation… - Specifically in CreateDatasetGetOrCreateStorageAccountStep when getting an existing SA (a new condition when billing profiles are shared, previously a new one was created with each test case)… - Specifically in the call to azureBlobStorePdao.enableFileLogging
Otherwise, test tear-down for a different test may unsuccessfully try to delete them -- observed for releaseSnapshotId specifically, which is not initialized in all tests.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I first converted AzureIntegrationTest to JUnit5.
This was necessary in order to create @BeforeAll and @afterall methods to initialize a single Azure billing profile for use in all tests.
Expected benefits: