Skip to content

ROX-29641: Fix k6 tests #15610

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 12, 2025
Merged

Conversation

mtodor
Copy link
Contributor

@mtodor mtodor commented Jun 6, 2025

Description

Initial investigation showed that sensors were properly scheduled and working, but after some additional testing, it turned out that this is not true. It appears that the sensors are not running, which is why the workload is not being created.

Further findings

It looks like the specific node becomes unstable, and sensors scheduled on that node never reach the running state. It's not clear what causes node instability. After additional testing, the most stable solution option is to deploy central without scanner V4 and set sensor memory request higher.

User-facing documentation

Testing and quality

  • the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
  • CI results are inspected

Automated testing

  • no changes on test

How I validated my change

@mtodor mtodor added the ci-performance-tests Runs the K6 based performance tests label Jun 6, 2025
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @mtodor - I've reviewed your changes and they look great!

Here's what I looked at during the review
  • 🟢 General issues: all looks good
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟢 Complexity: all looks good
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@rhacs-bot
Copy link
Contributor

rhacs-bot commented Jun 6, 2025

Images are ready for the commit at 289a5d8.

To use with deploy scripts, first export MAIN_IMAGE_TAG=4.9.x-14-g289a5d8465.

Copy link

codecov bot commented Jun 6, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.80%. Comparing base (0b6ac6d) to head (289a5d8).
Report is 24 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #15610      +/-   ##
==========================================
- Coverage   48.80%   48.80%   -0.01%     
==========================================
  Files        2590     2590              
  Lines      190471   190470       -1     
==========================================
- Hits        92962    92958       -4     
- Misses      90211    90213       +2     
- Partials     7298     7299       +1     
Flag Coverage Δ
go-unit-tests 48.80% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mtodor mtodor force-pushed the mtodor/ROX-29641-workload-creation-failure branch from 1cc31ea to 99d0c82 Compare June 10, 2025 12:53
Copy link
Contributor

github-actions bot commented Jun 10, 2025

Performance test results

Summary

  █ main dashboard

  █ list alerts grpc

    ✓ status is OK

  checks.........................: 100.00% ✓ 50       ✗ 0  
  data_received..................: 4.0 MB  77 kB/s
  data_sent......................: 241 kB  4.7 kB/s
  group_duration.................: avg=470.04ms min=62.15ms  med=522.31ms max=1.03s    p(90)=965.86ms p(95)=975.59ms
  grpc_req_duration..............: avg=75.4ms   min=60.53ms  med=67.03ms  max=280.94ms p(90)=72.65ms  p(95)=83.01ms 
  http_req_blocked...............: avg=139.71µs min=311ns    med=451ns    max=62.64ms  p(90)=681ns    p(95)=751ns   
  http_req_connecting............: avg=67.59µs  min=0s       med=0s       max=30.41ms  p(90)=0s       p(95)=0s      
  http_req_duration..............: avg=95.58ms  min=30.2ms   med=37.16ms  max=649.99ms p(90)=405.42ms p(95)=437.16ms
    { expected_response:true }...: avg=113.98ms min=33.92ms  med=38.37ms  max=649.99ms p(90)=414.84ms p(95)=451.79ms
  ✓ { lib:true }.................: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
  http_req_failed................: 22.22%  ✓ 100      ✗ 350
  http_req_receiving.............: avg=6.65ms   min=23.39µs  med=62.68µs  max=62.73ms  p(90)=59.56ms  p(95)=60.1ms  
  http_req_sending...............: avg=85.91µs  min=38.07µs  med=86.07µs  max=207.01µs p(90)=112.57µs p(95)=126.88µs
  http_req_tls_handshaking.......: avg=71.27µs  min=0s       med=0s       max=32.07ms  p(90)=0s       p(95)=0s      
  http_req_waiting...............: avg=88.83ms  min=30.14ms  med=37.01ms  max=649.84ms p(90)=405.23ms p(95)=436.98ms
  http_reqs......................: 450     8.722079/s
  iteration_duration.............: avg=1.03s    min=928.82ms med=996.41ms max=1.34s    p(90)=1.15s    p(95)=1.19s   
  iterations.....................: 50      0.96912/s
  vus............................: 1       min=1      max=1
  vus_max........................: 1       min=1      max=1

Sources

@mtodor mtodor changed the title ROX-29641: Reduce sensor request in k6 tests ROX-29641: Fix k6 tests Jun 10, 2025
@mtodor mtodor force-pushed the mtodor/ROX-29641-workload-creation-failure branch from 99d0c82 to c1ff529 Compare June 10, 2025 15:53
@mtodor
Copy link
Contributor Author

mtodor commented Jun 11, 2025

/test gke-upgrade-tests

@mtodor mtodor added auto-retest PRs with this label will be automatically retested if prow checks fails and removed auto-retest PRs with this label will be automatically retested if prow checks fails labels Jun 11, 2025
@mtodor mtodor force-pushed the mtodor/ROX-29641-workload-creation-failure branch 2 times, most recently from 27982ef to c1eded0 Compare June 11, 2025 08:28
@mtodor mtodor force-pushed the mtodor/ROX-29641-workload-creation-failure branch from c1eded0 to a79b350 Compare June 11, 2025 10:24
@mtodor mtodor force-pushed the mtodor/ROX-29641-workload-creation-failure branch from a79b350 to 289a5d8 Compare June 11, 2025 12:36
@mtodor
Copy link
Contributor Author

mtodor commented Jun 11, 2025

/retest

@mtodor
Copy link
Contributor Author

mtodor commented Jun 12, 2025

/test gke-qa-e2e-tests

Copy link

openshift-ci bot commented Jun 12, 2025

@mtodor: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/gke-qa-e2e-tests 289a5d8 link false /test gke-qa-e2e-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@mtodor mtodor merged commit 993dbb5 into master Jun 12, 2025
88 of 89 checks passed
@mtodor mtodor deleted the mtodor/ROX-29641-workload-creation-failure branch June 12, 2025 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ci ci-performance-tests Runs the K6 based performance tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants