PYTHON-5389: Add Tags to each buildvariant and augment test skipping/tracking policy. #82
Conversation
Jibola
requested review from
caseyclements,
blink1073,
baileympearson and
Copilot
There was a problem hiding this comment.
Pull Request Overview
This PR adds tagging support to build variants and updates the test notification policy in the documentation.
- Updated README.md to document the new tags field and its effect on Slack notifications.
- Modified .evergreen/config.yml to add language tags for each build variant.
Reviewed Changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| README.md | Updated documentation to include explanation for the new "tags" field. |
| .evergreen/config.yml | Added a "tags" property for each build variant to support language tagging. |
Comments suppressed due to low confidence (1)
README.md:190
- There is an inconsistency in the naming of the notification channel: 'dbx-ai-ml-testing-pipline-notifications' appears to be a typo compared to 'dbx-ai-ml-testing-pipeline-notifications-{language}'. Consider standardizing the channel names.
Tests are run periodically (nightly) and any failures will propagate into both the `dbx-ai-ml-testing-pipline-notifications` and `dbx-ai-ml-testing-pipeline-notifications-{language}` channel. Repo owners of this `ai-ml-testing-pipeline` library are required to join the `dbx-ai-ml-testing-pipeline-notifications`. Pipeline specific implementers must **at least** join `dbx-ai-ml-testing-pipline-notifications-{language}` (e.g. whomever implemented `langchain-js` must at least be a member of `dbx-ai-ml-testing-pipeline-notifications-js`).
| If tests are found to be failing, and cannot be addressed quickly, the responsible team MUST create a JIRA ticket, and disable the relevant tests | ||
| in the `config.yml` file, with a comment about the JIRA ticket that will address it. | ||
|
|
||
| This policy will help ensure that a single failing integration does not cause noise in the `dbx-ai-ml-testing-pipeline-notifications` that would mask other | ||
| This policy will help ensure that a single failing integration does not cause noise in the `dbx-ai-ml-testing-pipeline-notifications` or `dbx-ai-ml-testing-pipeline-notifications-{language}` that would mask other |
There was a problem hiding this comment.
Tests are run periodically (nightly) and any failures will propagate into both the
dbx-ai-ml-testing-pipline-notificationsanddbx-ai-ml-testing-pipeline-notifications-{language}channel.
Won't there still be noise in the shared channel because failures are propagated to both slack channels?
There was a problem hiding this comment.
Yes, the shared channel then becomes an opt-in. If there are several failures in one day across multiple drivers, it's difficult to parse. With individual channels it's easier to pay attention to only the relevant driver team notifications. The only users required to stay within that larger channel are those with a vested interest across driver implementations or the repo owners.
| @@ -110,6 +110,8 @@ Test execution flow is defined in `.evergreen/config.yml`. The test pipeline's c | |||
| - `run_on` -- Specified platform to run on. `rhel87-small` or `ubuntu2204-small` should be used by default. Any other distro may fail Atlas CLI setup. | |||
| - `tasks` -- Tasks to run. See below for more details | |||
| - `cron` -- The tests are run via a cron job on a nightly cadence. This can be modified by setting a different cadence. Cron jobs can be scheduled using [cron syntax](https://crontab.guru/#0_0_*_*_*) | |||
| - `tags` -- This should include the language where the AI/ML is run. i.e. `[python, csharp, golang, javascript]` Any tagged language will populate the | |||
There was a problem hiding this comment.
Is this something we could enforce somehow? Maybe a meta CI task that reads in the config and confirms every variant has a tags field defined that is one of [python, csharp, golang, javascript]?
There was a problem hiding this comment.
I'll look into it. Worst case I can parse the .yaml in python and check everything under "buildvariants" contains tags with at least one referenced language.
That may be best as a follow-up task.
No description provided.