Region per file storage strategy #287
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds the following
enum
so the storage strategy can be selected:The
STORE_REGION_BY_REGION_SINGLE_FILE
is the default strategy. TheSTORE_FLATTENED_REGION_PER_FILE
is the new strategy which stores each region of an object in a separate file. The region size the object is sliced into is decided in:By default it will try to slice the object into regions that are 4 MB in size by halving the largest dimension of the object iteratively until within the <= 4 MB.
This is set here
uint64_t max_bytes_per_file = 4ULL * 1024 * 1024;
within thePDC_shrink_file_dims
function.