test(monorepo): adds adhoc package selection for CI system tests - #17931
test(monorepo): adds adhoc package selection for CI system tests#17931chalmerlowe wants to merge 21 commits into
Conversation
chalmerlowe
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces an ad-hoc testing integration into the Kokoro CI pipeline, allowing dynamic package selection based on PR labels and configuration files. The review feedback highlights several crucial robustness improvements, including handling potential grep failures under set -o pipefail in the ad-hoc test runner, authenticating GitHub API requests to prevent rate-limiting, safely parsing the API response in the inline Python snippet, and quoting variables to avoid pathname expansion.
chalmerlowe
added
test:adhoc
yoshi-kokoro
removed
the
kokoro:force-run
60505a8 to
9e72163
Compare
…l parameter confusion
bcb4c15 to
d376408
Compare
chalmerlowe
changed the title
chalmerlowe
added
the
do not merge
|
Changes were made to two packages to ensure that the combination and deduplication of diff'ed packages and ad hoc packages functioned as expected. |
| headers=(-H "User-Agent: Kokoro") | ||
| if [[ -n "${GITHUB_TOKEN:-${GH_TOKEN}}" ]]; then | ||
| headers+=(-H "Authorization: token ${GITHUB_TOKEN:-${GH_TOKEN}}") | ||
| fi |
There was a problem hiding this comment.
Is the GitHub token already being populated in kokoro? Or is this a new secret we need to add for this feature?
| # For this prototype: | ||
| # we use a small inline Python snippet here because parsing JSON in pure Bash is difficult/error-prone, | ||
| # and we cannot guarantee that tools like 'jq' or 'gh' are installed in the test environment. | ||
| # Python and its built-in 'json' module are guaranteed to be available in this repository. |
There was a problem hiding this comment.
we cannot guarantee that tools like 'jq' or 'gh' are installed in the test environment.
Are you sure? It looks like kokoro is using the gcr.io/cloud-devrel-kokoro-resources/python-multi image. I just tested it locally, and it seems to have jq available
| @@ -0,0 +1,40 @@ | |||
| #!/bin/bash | |||
|
|
|||
There was a problem hiding this comment.
looks like the copyright header is missing
| # Ensure we are in the project root if called directly, | ||
| # but usually this is sourced and CWD is already project root. | ||
| # For safety, we can use script location but if sourced $0 might be the parent script. | ||
| # Let's assume CWD is project root as per system.sh behavior. |
There was a problem hiding this comment.
Is this comment still relevant?
3 participants
Problem
Currently, there is no easy way to run tests for specific packages in our continuous integration (CI) pipeline without making temporary changes to those packages (like adding a space to a comment or inserting a blank line) to trigger detection of a
diff. This causes noise in commit history and extra toil to revert the changes used solely for triggering tests prior to merge.Solution
We have implemented an "Ad-Hoc Package Selection" feature for
systemtests. This allows developers to specify which packages or groups of packages they want to test by editing simple configuration files within their Pull Request (PR). The feature is triggered by applying a specific GitHub label (test:adhoc) to the PR.A new script parses these configuration files and merges the ad hoc packages with those that have actual code diffs, ensuring that no modified packages are missed.
Changes
ci/adhoc/.standalone_package_list.txtfor listing individual packagesci/adhoc/.package_groups.txtfor defining groups of packages (e.g.,handwritten,core`).ci/adhoc/adhoc_test_runner.shscript to parse and deduplicate requested packages..kokoro/system.shto integrate this ad-hoc logic when thetest:adhoclabel is present.Note
Notes to Reviewers
ci/adhoc/are intended to be modified in your PR to select target packages and DO NOT require cleanup before merging.google-resumable-mediaand a trigger comment ingoogle-cloud-speech) strictly for verifying the ad-hoc mechanism works with diff'ed packages and to confirm output formatting in successful and failed situations. These will be reverted before final merge.