Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: size trigger for image layer generation #11064

Open
skyzh opened this issue Mar 3, 2025 · 0 comments
Open

pageserver: size trigger for image layer generation #11064

skyzh opened this issue Mar 3, 2025 · 0 comments
Assignees
Labels
c/storage/pageserver Component: storage: pageserver t/feature Issue type: feature, for new features or requests

Comments

@skyzh
Copy link
Member

skyzh commented Mar 3, 2025

Currently the criteria for whether to generate an image layer or not is based on the number of delta layers. We should consider to make it based on size.

  1. Larger tenants don't seem to rely as much on image layers as smaller tenants. My assumption is that for larger tenants, they have more full page writes to the pageserver, and therefore we don't need to generate a lot of image layers (that covers the full key range) for them.
  2. L0->L0 compaction alternative. feat(pageserver): l0->l0 compaction #10814 Because we generate image layers based on number of deltas, if we have a tenant that writes a small amount of data during every restart, we will quickly accumulate small L1 delta layers and trigger image generation.

If we change the trigger for image layer generation to size-based, we can solve both the problems. For (1), as L0 compaction gets triggered based on L0 size, we would get a fixed-size L1 levels every time. As larger tenants cover larger keyspaces, each image layer generation partition would contain fewer data, and we will wait for more delta layers before generating images. For (2), as we look at the L1 sizes, we won't generate images if L1 levels are small.

@skyzh skyzh self-assigned this Mar 3, 2025
@skyzh skyzh added t/feature Issue type: feature, for new features or requests c/storage/pageserver Component: storage: pageserver labels Mar 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/pageserver Component: storage: pageserver t/feature Issue type: feature, for new features or requests
Projects
None yet
Development

No branches or pull requests

1 participant