Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP} Merge Kruize v0.5 to master #1547

Open
wants to merge 175 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
175 commits
Select commit Hold shift + click to select a range
90414b2
Datasource Metaata change and Metadata inclusion
msvinaykumar Jan 31, 2025
381e1ac
initial kafka integration changes
khansaad Feb 4, 2025
f884194
Filter implementation
msvinaykumar Feb 4, 2025
fd56e38
Merge remote-tracking branch 'vinay/bulk_new_format_3' into kafka-2
khansaad Feb 4, 2025
a01017f
Include topic changes based on new bulk format
khansaad Feb 4, 2025
ba3f5cf
add design doc
khansaad Feb 5, 2025
c721a3c
add examples in the design doc
khansaad Feb 6, 2025
c96d88c
Add delete metadataProfile API
shreyabiradar07 Feb 6, 2025
92ba764
Update status code
shreyabiradar07 Feb 6, 2025
30524b3
Update docs
shreyabiradar07 Feb 7, 2025
60f013b
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-1
khansaad Feb 10, 2025
87b544c
move kafka server value to env and corresponding code changes
khansaad Feb 10, 2025
3bb2932
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-2
khansaad Feb 10, 2025
c99d102
move kafka server value to env and corresponding code changes
khansaad Feb 10, 2025
e09c9e3
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-2
khansaad Feb 10, 2025
d760913
Remove redundant checks and set ValidationOutputData message
shreyabiradar07 Feb 12, 2025
8ad91ef
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-2
khansaad Feb 16, 2025
11719b9
Add default metadata profile on initialization
shreyabiradar07 Feb 17, 2025
4734544
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-2
khansaad Feb 17, 2025
02ca965
Refactored the code as per the review
khansaad Feb 17, 2025
17ee6e5
updating namespace e2e
shekhar316 Jan 24, 2025
7169309
updating test for e2e lm
shekhar316 Jan 27, 2025
69eb719
updating sleep time
shekhar316 Feb 5, 2025
845b7cc
updating sleep time
shekhar316 Feb 18, 2025
c99abdb
adding rm=false in del exp
shekhar316 Feb 18, 2025
37a837f
updating measurement duration
shekhar316 Feb 19, 2025
e3485f5
Updated test plan for release 0.4
chandrams Feb 18, 2025
351ef5c
Add metadata profile file path from docker
shreyabiradar07 Feb 19, 2025
0ecbfe6
Remove unused constants
shreyabiradar07 Feb 19, 2025
d17476d
Add default metric profile
shreyabiradar07 Feb 19, 2025
bdd1407
Updated test results
chandrams Feb 20, 2025
3795c91
BulkJob table Database create SQL and md file update
msvinaykumar Mar 2, 2025
71fc960
updating measurement duration and sleep time
shekhar316 Mar 3, 2025
d4926a3
Remove ENV variables
shreyabiradar07 Mar 3, 2025
e603fe7
Check for 'is_ros_enabled' flag
shreyabiradar07 Mar 3, 2025
1e0505f
Mount default file paths in configmap
shreyabiradar07 Mar 4, 2025
5c79a1a
Fix conflicts
shreyabiradar07 Mar 5, 2025
c74060a
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-2
khansaad Mar 5, 2025
9ce24ba
add env for kafka response filters, corresponding file changes
khansaad Mar 5, 2025
090ed26
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-1
khansaad Mar 5, 2025
9f18da5
Merge pull request #1509 from chandrams/test_plan_0.4
dinogun Mar 5, 2025
36bea42
Update docs and error response for delete request
shreyabiradar07 Mar 5, 2025
395d1cf
add missing javadoc and update input topic name
khansaad Mar 5, 2025
73e884a
rebased kafka-1, add missing javadoc in other classes
khansaad Mar 5, 2025
08a37c4
updating print statement
shekhar316 Mar 5, 2025
0c463c0
update design doc, add constants for strings and other review changes
khansaad Mar 6, 2025
f704a2f
Append default profiles only when is_ros_enabled is set
shreyabiradar07 Mar 6, 2025
bf4d260
Fix config values
shreyabiradar07 Mar 6, 2025
c553f50
Merge pull request #1492 from khansaad/kafka-1
dinogun Mar 6, 2025
dea7ede
incorporated review comments
msvinaykumar Mar 6, 2025
ab55d65
Fix conflicts
shreyabiradar07 Mar 6, 2025
4698c0b
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-2
khansaad Mar 6, 2025
baa0c96
Merge pull request #1501 from shreyabiradar07/delete-metadata-profile…
dinogun Mar 6, 2025
2ab1531
Add create metadata profile tests and helper methods
shreyabiradar07 Feb 3, 2025
d88008e
Add test cases for listMetadataProfile API
shreyabiradar07 Feb 4, 2025
7baedb3
Update test docs
shreyabiradar07 Feb 4, 2025
e97ea99
Check for 'verbose' flag
shreyabiradar07 Feb 10, 2025
9f8b2b8
BulkJob table Database create SQL and md file update
msvinaykumar Mar 2, 2025
5381cff
incorporated review comments
msvinaykumar Mar 6, 2025
6fa71a6
Dao and IMPL and create ORM for bulkJOB table
msvinaykumar Mar 2, 2025
8fa988e
add a check for kafka connection failure, update design doc
khansaad Mar 7, 2025
3da62df
incorporated review comments
msvinaykumar Mar 7, 2025
9f9fa59
Merge pull request #1521 from msvinaykumar/bulkjobsave-1
dinogun Mar 7, 2025
05d84b6
Pass metadata profile name as input for the delete function
shreyabiradar07 Mar 7, 2025
c540161
Update mandatory fields
shreyabiradar07 Mar 7, 2025
12f65d6
Clean up temporary file
shreyabiradar07 Mar 10, 2025
caa040e
Incorproated doc update review comment
msvinaykumar Mar 10, 2025
58520fb
Add new marker 'test_bulk_api_ros' for bulk-api-ros testing
shreyabiradar07 Mar 10, 2025
c8dbb53
Fix conflicts
shreyabiradar07 Dec 27, 2024
7474eb7
Remove hardcoded metadata queries
shreyabiradar07 Dec 27, 2024
4c506c7
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-2
khansaad Mar 11, 2025
e474b56
update design doc
khansaad Mar 11, 2025
ef791d2
Add missing dependencies and fix measurement duration
shreyabiradar07 Mar 11, 2025
b6ea980
Add 'metadata_profile' field in bulk API
shreyabiradar07 Mar 11, 2025
aef6cf9
Merge pull request #1522 from msvinaykumar/bulkjobsave-2
dinogun Mar 11, 2025
6dc51a0
Merge pull request #1491 from shreyabiradar07/metadata-profile-tests
dinogun Mar 11, 2025
2b9c02f
Validate if specified metadata_profile exists
shreyabiradar07 Mar 11, 2025
acdcdcb
Merge branch 'mvp_demo' of github.com:kruize/autotune into default-pr…
shreyabiradar07 Mar 12, 2025
0585237
Validate default profiles in bulk testcase
shreyabiradar07 Mar 12, 2025
3ed116f
Updated ubi9 minimal version
chandrams Mar 12, 2025
c9c28d4
Env varaible to control storage of job details and service layer modi…
msvinaykumar Mar 5, 2025
d11e013
Env varaible to control storage of job details and service layer modi…
msvinaykumar Mar 5, 2025
4e3f824
incorproated review comments
msvinaykumar Mar 12, 2025
ea2650b
incorproated review comments
msvinaykumar Mar 12, 2025
eacf272
Fix metadata profile name
shreyabiradar07 Mar 12, 2025
7777aac
calculate replicas with memory metrics first
kusumachalasani Mar 12, 2025
5c94ba4
calculate replicas with memory metrics first
kusumachalasani Mar 12, 2025
31520f8
Check for auto or recreate mode to query instaslice object
bharathappali Mar 12, 2025
c84ed7c
incorproated review comments
msvinaykumar Mar 12, 2025
40a9e81
load recommednations using jobID
msvinaykumar Mar 5, 2025
b10b181
Inclusion of job_id while generating recommendations
msvinaykumar Mar 5, 2025
d1ca91c
Merge pull request #1523 from msvinaykumar/bulkjobsave-3
dinogun Mar 12, 2025
03e5281
Merge pull request #1534 from bharathappali/limit-instaslice-scope
dinogun Mar 12, 2025
890a564
Merge pull request #1532 from chandrams/ubi_ver
dinogun Mar 12, 2025
b90031f
Enable fetching GPU metrics only in non ROS case
bharathappali Mar 13, 2025
7a57c71
Skip testcase without datasource connection
shreyabiradar07 Mar 13, 2025
ffba61e
Merge pull request #1524 from msvinaykumar/bulkjobsave-4
dinogun Mar 13, 2025
d536283
fix pods number
kusumachalasani Mar 13, 2025
e47389c
Add issue number
shreyabiradar07 Mar 13, 2025
d3eea4b
Merge pull request #1525 from msvinaykumar/bulkjobsave-5
dinogun Mar 13, 2025
6778fe7
Merge pull request #1537 from shreyabiradar07/skip-datasource-testcase
chandrams Mar 13, 2025
c9c7755
Refactor jobmanager and call Job save to db
msvinaykumar Mar 5, 2025
64aa82f
incorproated review comments
msvinaykumar Mar 12, 2025
ef8ac29
Add new metadata_profile field in lm experiment schema
shreyabiradar07 Jan 30, 2025
25d20c8
Add CRD context for metadata profile
shreyabiradar07 Feb 4, 2025
5366ab7
Integrate metadataProfile with experiment
shreyabiradar07 Feb 4, 2025
c832ff0
Include missing datasource field
shreyabiradar07 Feb 4, 2025
8281eb8
Fix typo
shreyabiradar07 Mar 13, 2025
54d6ae5
Merge pull request #1526 from msvinaykumar/bulkjobsave-6
dinogun Mar 17, 2025
08e48b4
Merge upstream mvp_demo and fix conflicts
shreyabiradar07 Mar 17, 2025
d634ca5
Misc changes
msvinaykumar Mar 5, 2025
c2d1448
Renamed bulkjobs to kruizebulkJobs
msvinaykumar Mar 7, 2025
ca940d2
misc changes
msvinaykumar Mar 14, 2025
13797eb
incorproated review comments
msvinaykumar Mar 17, 2025
39b9d55
Fix formatting and audit logs
shreyabiradar07 Mar 17, 2025
32273d8
Merge pull request #1527 from msvinaykumar/bulkjobsave-7
dinogun Mar 17, 2025
0c3569c
Merge pull request #1508 from shreyabiradar07/default-profiles
dinogun Mar 17, 2025
623562c
Merge pull request #1536 from bharathappali/no-gpu-ros
dinogun Mar 17, 2025
c43cd92
Merge pull request #1533 from kusumachalasani/pods
dinogun Mar 17, 2025
96bc66e
Merge pull request #1476 from shekhar316/newTests
dinogun Mar 17, 2025
7194ee1
Resolved conflicts
khansaad Mar 17, 2025
81cab68
update docs, add missing kafka logic
khansaad Mar 17, 2025
42ea9ec
Check for invalid values
shreyabiradar07 Mar 17, 2025
1100578
Fix conflicts
shreyabiradar07 Mar 17, 2025
938f594
Update tests to include metadata_profile field
shreyabiradar07 Mar 18, 2025
1ee6372
Update docs
shreyabiradar07 Mar 18, 2025
bc29064
Refactor and add string constants
shreyabiradar07 Mar 18, 2025
f570aef
Refactor the code and handle default values
shreyabiradar07 Mar 18, 2025
0d34911
Audit logs
shreyabiradar07 Mar 18, 2025
b6cac0e
Fix logs
shreyabiradar07 Mar 18, 2025
8641fe5
Update profile name
shreyabiradar07 Mar 18, 2025
f9ca2c3
update the logic to start Kafka producer threads
khansaad Mar 18, 2025
d3d6be0
Update docs
shreyabiradar07 Mar 19, 2025
0bf89de
Add metadata_profile field in create_experiment tests
shreyabiradar07 Mar 19, 2025
45247d0
Add metadata profile in e2e_tests
shreyabiradar07 Mar 19, 2025
9b96231
Remove checks for 'local' flag
shreyabiradar07 Mar 19, 2025
cd19a94
add shutdown hook for kafka executor service
khansaad Mar 19, 2025
353ae7a
Add check for local experiments
shreyabiradar07 Mar 19, 2025
791cb06
remove logs
shreyabiradar07 Mar 19, 2025
1312a56
Merge branch 'mvp_demo' of github.com:kruize/autotune into metadataPr…
shreyabiradar07 Mar 19, 2025
f48771c
Refactor helper functions and update docs
shreyabiradar07 Mar 19, 2025
d75ef0d
Refactor the helper functions
shreyabiradar07 Mar 20, 2025
9940c0a
Add metadata_profile field in bulk experiments
shreyabiradar07 Mar 20, 2025
60465c6
Refactor error handling
shreyabiradar07 Mar 20, 2025
99458db
Merge pull request #1495 from khansaad/kafka-2
dinogun Mar 20, 2025
53d236a
Restore 'local' flag checks
shreyabiradar07 Mar 21, 2025
80455fe
Add string constant
shreyabiradar07 Mar 21, 2025
7314fa1
Resolve conficts
shreyabiradar07 Mar 21, 2025
7719b48
Revert "Resolve conficts"
shreyabiradar07 Mar 21, 2025
c79ee54
Merge upstream mvp_demo branch
shreyabiradar07 Mar 21, 2025
72732cf
Fix the datatype
shreyabiradar07 Mar 21, 2025
62ca04c
Merge pull request #1444 from shreyabiradar07/integrate-metadata-prof…
dinogun Mar 21, 2025
ec193eb
Merge upstream/mvp_demo into metadataProfile-createExp-integration
shreyabiradar07 Mar 21, 2025
f5530b8
Remove duplicate constant
shreyabiradar07 Mar 21, 2025
c45bf18
Merge pull request #1496 from shreyabiradar07/metadataProfile-createE…
dinogun Mar 21, 2025
9e81f1f
Revert kafka flag to false to disable it by default
khansaad Mar 21, 2025
ef18c4a
Merge pull request #1541 from shreyabiradar07/update-dsmetadata-tests
chandrams Mar 21, 2025
8fbd4ef
Merge upstream/mvp_demo into update-create-exp-tests
shreyabiradar07 Mar 21, 2025
0c7cecd
Merge pull request #1548 from khansaad/disable-kafka
dinogun Mar 21, 2025
860d50a
check for 'summary' keyword
shreyabiradar07 Mar 21, 2025
6ba3a30
Merge pull request #1543 from shreyabiradar07/update-create-exp-tests
chandrams Mar 24, 2025
bea9e52
Included metadata profile creation
chandrams Mar 24, 2025
22e0963
Included metadata profile creation in fault tolerant test
chandrams Mar 24, 2025
0c97402
Merge pull request #1550 from chandrams/stress_test_update
chandrams Mar 24, 2025
9738786
For ROS flags isRosEnabled is true and local=false by default via code
msvinaykumar Mar 22, 2025
778f464
modifed need of patch in test functions
msvinaykumar Mar 23, 2025
f17373a
modifed need of patch in test functions
msvinaykumar Mar 23, 2025
22d06fc
modifed need of patch in test functions
msvinaykumar Mar 23, 2025
823faf2
Removing metricprofile from mandatory , and loading default profile…
msvinaykumar Mar 25, 2025
58659ad
Revoked changes related to mandatory feild
msvinaykumar Mar 25, 2025
ce26bb2
invalid image name
msvinaykumar Mar 25, 2025
1c0752e
Formatting fixed
msvinaykumar Mar 25, 2025
785cf0d
added paths for metric and meta data profiles
msvinaykumar Mar 25, 2025
3985bc1
rolled back changes to metric profiles
msvinaykumar Mar 25, 2025
ed78f81
Merge pull request #1549 from msvinaykumar/flags
chandrams Mar 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'upstream/mvp_demo' into kafka-2
khansaad committed Feb 16, 2025
commit 8ad91ef593b4db8f44c3f5b412199adc6aaae8b1
Original file line number Diff line number Diff line change
@@ -173,7 +173,6 @@ public void run() {
if (null == metadataInfo) {
setFinalJobStatus(COMPLETED, String.valueOf(HttpURLConnection.HTTP_OK), NOTHING_INFO, datasource);
} else {

jobData.setMetadata(metadataInfo);
Map<String, CreateExperimentAPIObject> createExperimentAPIObjectMap = getExperimentMap(labelString, jobData, metadataInfo, datasource); //Todo Store this map in buffer and use it if BulkAPI pods restarts and support experiment_type
// TODO: Remove getExperimentMap and instead collect all metadata, process it, and create experiments dynamically during metadata iteration.
Original file line number Diff line number Diff line change
@@ -16,8 +16,6 @@ public class DataSourceMetadataInfo {
* Key: Data Source name
* Value: Associated DataSource object
*/
//@SerializedName(KruizeConstants.DataSourceConstants.DataSourceMetadataInfoJSONKeys.DATASOURCES)
//@JsonProperty(KruizeConstants.DataSourceConstants.DataSourceMetadataInfoJSONKeys.DATASOURCES)
private HashMap<String, DataSource> datasources;

public DataSourceMetadataInfo(HashMap<String, DataSource> datasources) {
You are viewing a condensed version of this merge commit. You can view the full changes here.