Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add earliest partition date stat to table stats collector #205

Merged

Conversation

chenselena
Copy link
Collaborator

@chenselena chenselena commented Sep 20, 2024

Summary

This PR adds the earliest partition date on a table as a stat to the table stats collector.

For tables where the retention policy has been set on a string type column, the stat is retrieved based on the retention column name, since this stat will be used as a comparison point for Openhouse alerting to identify if older partitions should have been cleaned up through retention jobs.
For tables that are partitioned on timestamp, it searches for the partition column name that is Timestamp. Openhouse only allows a table to be partitioned on 1 timestamp type column so we can identify that as the column set for retention.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Added unit testing.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

@chenselena chenselena force-pushed the selchen/add-earliest-partition-date-stat branch from c76185f to 91ed83a Compare September 20, 2024 16:47
@chenselena chenselena marked this pull request as ready for review September 20, 2024 18:01
Copy link
Collaborator

@rohitkum2506 rohitkum2506 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Added a couple of comments.

@chenselena chenselena merged commit 67a957f into linkedin:main Sep 23, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants