Skip to content

[CI] Upload logs as artifacts to BuildKite #3405

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

[CI] Upload logs as artifacts to BuildKite #3405

wants to merge 5 commits into from
+20 −8

Conversation

win5923
Copy link
Contributor

@win5923 win5923 commented Apr 16, 2025 •
edited
Loading

Why are these changes needed?

#3109

I made the test fail to see if any artifacts would be generated — looks good!

Result:
image

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/8147#01963f30-29e8-4225-be66-66a6efa0f516/1102-1161

Related issue number

Closes #3109

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

@win5923 win5923 marked this pull request as draft April 16, 2025 15:01
@win5923 win5923 changed the title [CI] Upload logs as artifacts to BuildKite [Do not merge][CI] Upload logs as artifacts to BuildKite Apr 16, 2025
@win5923 win5923 force-pushed the buildkite/log branch 2 times, most recently from b8537b4 to e225632 Compare April 16, 2025 15:39
@win5923 win5923 changed the title [Do not merge][CI] Upload logs as artifacts to BuildKite [CI] Upload logs as artifacts to BuildKite Apr 16, 2025
@win5923 win5923 marked this pull request as ready for review April 16, 2025 15:42
@win5923
Copy link
Contributor Author

win5923 commented Apr 16, 2025 •
edited
Loading

@MortalHappiness PTAL, thanks!

Also, I noticed that Test E2E Operator Version Upgrade (v1.3.0) requires Go v1.23 to run. Should I open a separate PR for that, or handle it here?

Edited: I create another PR to address this #3406

image

@MortalHappiness
Copy link
Member

We should also upload ${{ env.KUBERAY_TEST_OUTPUT_DIR }}/**/*.log as mentioned in the issue. Please zip them and then upload it as artifact. Thanks.

@win5923 win5923 marked this pull request as draft April 17, 2025 14:07
@win5923 win5923 force-pushed the buildkite/log branch 20 times, most recently from 36e357e to 70e6e2b Compare April 17, 2025 18:06
@win5923 win5923 force-pushed the buildkite/log branch 14 times, most recently from 1e8b01d to 2d50879 Compare April 26, 2025 15:20
@win5923 win5923 marked this pull request as ready for review April 26, 2025 15:26
@win5923
Copy link
Contributor Author

win5923 commented Apr 26, 2025 •
edited
Loading

Hi @MortalHappiness, I've made the necessary fixes, I didn't expect that setting variables in Buildkite requires using $$.

result:

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/8548#0196729a-1020-4297-8809-490e7688731f

image

@MortalHappiness
Copy link
Member

I found a nested folder workdir/ray-operator/tmp in the tar file. These nested folders are not neccessary. KUBERAY_TEST_OUTPUT_DIR should be the top-level folder.

@win5923 win5923 force-pushed the buildkite/log branch 2 times, most recently from 289afb7 to da19dc9 Compare April 26, 2025 17:30
@win5923
Use relative path
56631f0 
Signed-off-by: win5923 <[email protected]>
@win5923
Copy link
Contributor Author

win5923 commented Apr 26, 2025

I found a nested folder workdir/ray-operator/tmp in the tar file. These nested folders are not neccessary. KUBERAY_TEST_OUTPUT_DIR should be the top-level folder.

Fixed in 56631f0

https://buildkite.com/ray-project/ray-ecosystem-ci-kuberay-ci/builds/8554#01967328-05a0-439a-af31-fda817b01d9a

MortalHappiness
MortalHappiness reviewed Apr 27, 2025
View reviewed changes
.buildkite/test-e2e.yml Outdated
- echo "--- END:e2e rayservice (nightly operator) tests finished"
- mkdir -p "$(pwd)/tmp" && export KUBERAY_TEST_OUTPUT_DIR=$(pwd)/tmp
- echo "KUBERAY_TEST_OUTPUT_DIR=$$KUBERAY_TEST_OUTPUT_DIR"
- KUBERAY_TEST_OUTPUT_DIR=$$KUBERAY_TEST_OUTPUT_DIR KUBERAY_TEST_TIMEOUT_SHORT=1m KUBERAY_TEST_TIMEOUT_MEDIUM=5m KUBERAY_TEST_TIMEOUT_LONG=10m go test -timeout 30m -v ./test/e2e 2>&1 | awk -f ../.buildkite/format.awk | tee $$KUBERAY_TEST_OUTPUT_DIR/gotest.log || (kubectl logs --tail -1 -l app.kubernetes.io/name=kuberay | tee $$KUBERAY_TEST_OUTPUT_DIR/kuberay-operator.log && find "$$KUBERAY_TEST_OUTPUT_DIR" -name "*.log" | tar --transform='s|^.*/tmp/||' -cf /artifact-mount/e2e-log.tar -T - && exit 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- KUBERAY_TEST_OUTPUT_DIR=$$KUBERAY_TEST_OUTPUT_DIR KUBERAY_TEST_TIMEOUT_SHORT=1m KUBERAY_TEST_TIMEOUT_MEDIUM=5m KUBERAY_TEST_TIMEOUT_LONG=10m go test -timeout 30m -v ./test/e2e 2>&1 | awk -f ../.buildkite/format.awk | tee $$KUBERAY_TEST_OUTPUT_DIR/gotest.log || (kubectl logs --tail -1 -l app.kubernetes.io/name=kuberay | tee $$KUBERAY_TEST_OUTPUT_DIR/kuberay-operator.log && find "$$KUBERAY_TEST_OUTPUT_DIR" -name "*.log" | tar --transform='s|^.*/tmp/||' -cf /artifact-mount/e2e-log.tar -T - && exit 1)
- KUBERAY_TEST_TIMEOUT_SHORT=1m KUBERAY_TEST_TIMEOUT_MEDIUM=5m KUBERAY_TEST_TIMEOUT_LONG=10m go test -timeout 30m -v ./test/e2e 2>&1 | awk -f ../.buildkite/format.awk | tee $$KUBERAY_TEST_OUTPUT_DIR/gotest.log || (kubectl logs --tail -1 -l app.kubernetes.io/name=kuberay | tee $$KUBERAY_TEST_OUTPUT_DIR/kuberay-operator.log && find "$$KUBERAY_TEST_OUTPUT_DIR" -name "*.log" | tar --transform='s|^.*/tmp/||' -cf /artifact-mount/e2e-log.tar -T - && exit 1)

You've already exported this variable above. Either remove the export above or remove the variable here. Ditto for other commands.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f8694e9, thanks!

@win5923 win5923 marked this pull request as draft April 27, 2025 11:16
@win5923
remove the variable
f8694e9 
Signed-off-by: win5923 <[email protected]>
@win5923 win5923 marked this pull request as ready for review April 27, 2025 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MortalHappiness MortalHappiness MortalHappiness left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@MortalHappiness MortalHappiness

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CI] Upload logs as artifacts to BuildKite
2 participants
@win5923 @MortalHappiness