DLO: Multi-objective optimization for auto-compaction

[sumedhsakdeo]

Summary

We plan to develop Auto Compaction for for high ROI tables. Our plan is to treat this as a multi-objective optimization problem by aiming to optimizing two objectives -- maximize file count reduction and minimize compute costs. We score and rank the tables, then choose the top-K tables for each iteration of compaction to remain under allocated compute budget.

This will be used by the job scheduler for candidate selection.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

[teamurko]

lgtm, the only question is why returning indexes instead of strategy objects is important

[sumedhsakdeo]

lgtm, the only question is why returning indexes instead of strategy objects is important

I feel returning indexes is more intuitive, but I am fine if you think when you incorporate the module in scheduler you can tell it is not a a good idea and we go to list of strategy objects.