[ci] Allow triggering exhaustive tests from PRs depending on path #18202
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
|
This pull request does not have a backport label. Could you fix it @oakrizan? 🙏
|
oakrizan
changed the title
oakrizan
added
ci
backport-active-all
oakrizan
requested review from
dliappis,
fr4nc1sc0-r4m0n and
v1v
and removed request for
v1v
oakrizan
force-pushed
the
feature/trigger-exhaustive-tests-on-path
branch
from
27f9e6d to
eb8c8e4
Compare
v1v
reviewed
Sep 23, 2025
v1v
reviewed
Sep 24, 2025
|
v1v
reviewed
Sep 25, 2025
v1v
reviewed
Sep 25, 2025
…18121) * noop: add pq compression-codec with no-op implementation * pq: add support for event compression using zstd Adds non-breaking support for event compression to the persisted queue, as configured by a new per-pipeline setting `queue.compression`, which supports: - `none` (default): no compression is performed, but if compressed events are encountered in the queue they will be decompressed - `speed`: compression optimized for speed - `balanced`: compression balancing speed against result size - `size`: compression optimized for maximum reduction of size - `disabled`: compression support entirely disabled; if a pipeline is run in this configuration against a PQ that already contains unacked compressed events, the pipeline WILL crash. To accomplish this, we then provide an abstract base implementation of the CompressionCodec whose decode method is capable of _detecting_ and decoding zstd-encoded payload while letting other payloads through unmodified. The detection is done with an operation on the first four bytes of the payload, so no additional context is needed. An instance of this zstd-aware compression codec is provided with a pass-through encode operation when configured with `queue.compression: none`, which is the default, ensuring that by default logstash is able to decode any event that had previously been written. We provide an additional implementation that is capable of _encoding_ events with a configurable goal: speed, size, or a balance of the two. * license: add notice for `com.github.luben:zstd-jni` * pq: log compression encode/decode from the codec * Apply docs suggestions from code review Co-authored-by: João Duarte <[email protected]> * remove CleanerThreadLocal utility * license: add mapping for com.github.luben:zstd-jni * Apply suggestions from code review Co-authored-by: Rob Bavey <[email protected]> --------- Co-authored-by: João Duarte <[email protected]> Co-authored-by: Rob Bavey <[email protected]>
* metrics: add support for user-defined metrics * Update logstash-core/src/main/java/org/logstash/instrument/metrics/UserMetric.java Co-authored-by: Andrea Selva <[email protected]> --------- Co-authored-by: Andrea Selva <[email protected]>
* metric: improve accuracy of timer metric excludes contention for recording timing from the timing by locking in the completion time before attempting to record it * remove unused imports
* pq-compression: wire through custom user metrics for ratio/spend * add test for queue compression metrics * pq metrics: IORatioMetric edge-case logging, use int at interface By using `int` type in `IORatioMetric#incrementBy(int,int)`, we simplify the failure scenarios while still allowing the desired behaviour, since this is always called in practice with `byte[]#length`. We ensure that attempts to decrement the value are ignored, and result in a log message, and that overflows reduce precision and are also logged. Together, these ensure that long overflows won't ever result in pipeline crashes.
Update the list of logstash settings with the new pipeline.batch.metrics.sampling_mode to control the batch size ones. Co-authored-by: Rob Bavey <[email protected]>
…ce testing (#18181) * Use locally build artifact to build container from public dockerfile Previously we would build an image (which would not actually be used), build dockerfiles, modify dockerfiles to curl from `https://snapshots.elastic.co/downloads/logstash'` then build the image used for testing based on the modified dockerfile. This resulted in testing the last published image to `snapshots`. This presents two problems 1. The test is running against the last published image (not the tip of the branch being tested) 2. this carries a dependency on both a DRA and unified stack release having been run. Therefor acceptance tests will fail in between the time we bump logstash version and a successful run of unified release. This commit modifies the dockerfile to use the artifact prepared in the first step instead of curling the last published one. This solves both issues as the tests run against the code from the tip fo the branch being tested and there is no dependency on an artifact existing as a result of a unified release pipeline. * Remove redundant docker steps from workflows Previously for docker acceptance tests three steps were performed: 1. Build container images (based on local artifacts) 2. Build "public" dockerfiles 3. Build container based on (a modified) file from step 2. The ONLY difference between the dockerfile that ultimately is used to define an image between 1 and 2 is WHERE the logstash source is downloaded from. In acceptance testing we WANT to use the source at the current checkout of logstash (not a remote). Using remote causes a dependency issue when changes are not published. Publishing is tied to unified release and gated on tests so naturally this is a bad fit for that dependency. This commit removes the redundancy by ONLY generating images for testing (step 1 from above). This also firms up our use of LOCAL_ARTIFACTS. Namely, the ONLY time we want that set to `false` is when we build a "public" dockerfile. We explicitly set that in the corresponding DRA script now. Similarly we explicitly set it to `true` when testing. * Remove unused function and argument This commit removes the unused function for building from dockerfiles. It also removes an unused argument for the make task for build_docker.
The regex for finding the latest release (as a fallback when current is not in the file) had a bug. This caused the generated docs to be inserted at the wrong place in the file (the end of the file) instead of the top. This commit fixes the logic such that we find the last release when the current is not found.
* chore: Update logstash version in versions.yml Made with ❤️️ by updatecli * chore: Update logstash-core version in versions.yml Made with ❤️️ by updatecli --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* Downgrade gradle to coninute testing on windows server 2016 * REVERT ME! Temporarily force testing on win 2016 * Revert "REVERT ME! Temporarily force testing on win 2016" This reverts commit 079898b. * update gradle wrapper
* Update release notes for 9.1.5 * finalize release notes * Update docs/release-notes/index.md * Update docs/release-notes/index.md * Update docs/release-notes/index.md --------- (cherry picked from commit f861bb4) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: logstashmachine <[email protected]> Co-authored-by: donoghuc <[email protected]> Co-authored-by: Karen Metts <[email protected]>
* Release notes for 9.0.8 (#18247) * Update release notes for 9.0.8 * finalize release notes * Update docs/release-notes/index.md Co-authored-by: Karen Metts <[email protected]> --------- Co-authored-by: logstashmachine <[email protected]> Co-authored-by: donoghuc <[email protected]> Co-authored-by: Karen Metts <[email protected]> (cherry picked from commit 82ff1f7) * fix merge conflicts --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: logstashmachine <[email protected]> Co-authored-by: donoghuc <[email protected]> Co-authored-by: Karen Metts <[email protected]>
this service will be decommissioned shortly
…ptions to code (#18296) * Set jruby.compile.invokedynamic, log4j2.isThreadContextMapInheritable opts in code * Add invokedynamic to build.gradle settings * Java 11 is no longer supported so don't test for it * Update build.gradle Co-authored-by: Rob Bavey <[email protected]> * test not including log4j setting in build.gradle * Revert "test not including log4j setting in build.gradle" This reverts commit 852f97a. --------- Co-authored-by: Rob Bavey <[email protected]>
…Is. (#18306) * Adds integration test for the _health_report API. * Add _node/plugins API integration test.
Previously the RN generator would look for EVERY plugin in the `logstash-plugins` GH org. The elastic integration plugin is in the `elastic` org. Update the logic to handle this.
Bumps the github-actions group with 2 updates in the / directory: [anchore/scan-action](https://github.com/anchore/scan-action) and [actions/setup-node](https://github.com/actions/setup-node). Updates `anchore/scan-action` from 7.0.0 to 7.0.2 - [Release notes](https://github.com/anchore/scan-action/releases) - [Changelog](https://github.com/anchore/scan-action/blob/main/RELEASE.md) - [Commits](anchore/scan-action@f660128...a5605eb) Updates `actions/setup-node` from 5 to 6 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v5...v6) --- updated-dependencies: - dependency-name: anchore/scan-action dependency-version: 7.0.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: actions/setup-node dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
🔍 Preview links for changed docs |
1 task
|
I did something wrong when rebasing this PR with main:
|
💚 Build Succeeded
History
cc @v1v |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release notes
rn:skip
What does this PR do?
This pull request introduces a new "smart exhaustive tests" pipeline for Logstash, integrating it into the Buildkite CI configuration and documenting it in the service catalog. The main goal is to trigger exhaustive test runs automatically for pull requests that modify files in the
qa/acceptance/directory or when specific GitHub comments are made, improving test coverage and automation for critical changes.Why is it important/What is the impact to the user?
Test the code that will rarely impact the standard PR test suite, but will exercise the acceptance tests.
Checklist
[ ] I have commented my code, particularly in hard-to-understand areas[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration files (and/or docker env variables)[ ] I have added tests that prove my fix is effective or that my feature worksHow to test this PR locally
Add a path to
always_require_ci_on_changedin pull-request.json forlogstash-smart-exhaustive-tests-pipeline, then apply changes in that directory, commit and push. That will trigger the exhaustive tests pipeline in Buildkite.Caveats
The existing BK PR Bot does not support for code changes and GitHub comments, without running the first buidl when a PR is created.
That's the reason for creating the
logstash-smart-exhaustive-tests-pipelineto actually filter:This should help with running faster builds and reduce the overhead of running the exhaustive tests for the first build of each new PR.