We'd like to have zagg work on the atl03 catalog globally; that means planning and testing for the worse case scenario. That scenario is reached near the 88S 'pole hole' , a donut shaped region that follows the 88 degree south line of latitude where all of the ICESat-2 orbits converge. The coverage becomes dense around this region, and is where we should stress test our pipeline.
Ideally, we'd be able to run tdigest aggregation at o9 ; given that this region is particularly demanding, we should start an order up at o10 first to see if we can handle that. Handle here means being able to run on a 2GB lambda worker, without running out of memory or out of time. To make this somewhat easier, we should preemptively bump the timeout from 720 to 900 seconds while we test this.
Picking o9 and o10 shards and querying the catalog + building the shard maps for the 2018 to Jan 2026 will quickly surface how big of a lift this is given the total number of granuals that need to be inspected. We'll want to compare two approaches:
- What we have now (i.e., hierarchical mapping from the geolocation segments to retrieve the index ranges and chunks)
- Processing when we know the chunk offsets ahead of time
Cached chunk boundries
Option 2 doesn't exist yet. It assumes the following:
- At some point, the CMR geometry metadata object is changed such that we can infer chunk boundaries from it
- This would mean instead of a polygon, we would have 6 linestrings / polylines , one per beam
- The linestrings would encode the lat/lon of the first / last locations of the photons on the chunk boundaries
This needs some thought and design. We'd ideally not like to encode both start and stop ends per linestring, since that doubles the size of the geometry and is redundant. Instead, we'd like to encode the start lat/lon of each photon at the chunk boundry, and then infer the end given the start of the next chunk photon recorded. This would give N+1 points per beam, where N is the number of chunks, and the +1 account to the fact that we need to record the last photon to be consistent with metadata requirements that we represent the whole line path and don't truncate the geometry. This is all future refinement-- we can start with recording start/end pairs to prototype for correctness.
We'll need our own lambda function for above; there will be thousands of granules that need to be read for o10 , and it will take took long to run in serial to grab this data. The lambda functions are simple: for each granule, read the datafile chunk by chunk, grab the first+last lat / lon per beam in the heights group, and then accumulate those per beam pairs to write out as a parquet file on an s3 directory.
A second function reads the parquet files, and then emits a geometry. What we'll do is fake a cmr catalog that has these geometries, by copying the real catalog for the o9 and o10 queries, and then replacing the geometry in those files. Then, we'll write helper functions to use that geometry data to determine a priori which chunks need to be read, bypassing the hierarchical reads.
Benchmarks
We'll want to compare 1 & 2 in benchmarks that assess how this impacts runtime and memory usage. The key question is whether we need the geometry --> chunk cache to do the processing at all, or if it's just a speed up on our existing pipeline (which may potentially be able to do the aggs within the timeout).
Memory usage, and streaming
The expectation is that the photon counts will be too large for us to process 88S by holding all the data in memory at once. Rather, we should expect that we'll have to do iterative merges inside the worker as we read granules so that we can release that memory and load more granules. This deserves a fair bit of thought; do we define a fixed size granule buffer (e.g. 50 or 100), or do we literally process sequentially where we create a tdigest per granule that we read, and then continue to update granule by granule. What's the impact on run time with this? We need to stay under the 900 timeout, so this is a joint optimization across both processing time and memory.
We'd like to have zagg work on the atl03 catalog globally; that means planning and testing for the worse case scenario. That scenario is reached near the 88S 'pole hole' , a donut shaped region that follows the 88 degree south line of latitude where all of the ICESat-2 orbits converge. The coverage becomes dense around this region, and is where we should stress test our pipeline.
Ideally, we'd be able to run tdigest aggregation at o9 ; given that this region is particularly demanding, we should start an order up at o10 first to see if we can handle that. Handle here means being able to run on a 2GB lambda worker, without running out of memory or out of time. To make this somewhat easier, we should preemptively bump the timeout from 720 to 900 seconds while we test this.
Picking o9 and o10 shards and querying the catalog + building the shard maps for the 2018 to Jan 2026 will quickly surface how big of a lift this is given the total number of granuals that need to be inspected. We'll want to compare two approaches:
Cached chunk boundries
Option 2 doesn't exist yet. It assumes the following:
This needs some thought and design. We'd ideally not like to encode both start and stop ends per linestring, since that doubles the size of the geometry and is redundant. Instead, we'd like to encode the start lat/lon of each photon at the chunk boundry, and then infer the end given the start of the next chunk photon recorded. This would give N+1 points per beam, where N is the number of chunks, and the +1 account to the fact that we need to record the last photon to be consistent with metadata requirements that we represent the whole line path and don't truncate the geometry. This is all future refinement-- we can start with recording start/end pairs to prototype for correctness.
We'll need our own lambda function for above; there will be thousands of granules that need to be read for o10 , and it will take took long to run in serial to grab this data. The lambda functions are simple: for each granule, read the datafile chunk by chunk, grab the first+last lat / lon per beam in the heights group, and then accumulate those per beam pairs to write out as a parquet file on an s3 directory.
A second function reads the parquet files, and then emits a geometry. What we'll do is fake a cmr catalog that has these geometries, by copying the real catalog for the o9 and o10 queries, and then replacing the geometry in those files. Then, we'll write helper functions to use that geometry data to determine a priori which chunks need to be read, bypassing the hierarchical reads.
Benchmarks
We'll want to compare 1 & 2 in benchmarks that assess how this impacts runtime and memory usage. The key question is whether we need the geometry --> chunk cache to do the processing at all, or if it's just a speed up on our existing pipeline (which may potentially be able to do the aggs within the timeout).
Memory usage, and streaming
The expectation is that the photon counts will be too large for us to process 88S by holding all the data in memory at once. Rather, we should expect that we'll have to do iterative merges inside the worker as we read granules so that we can release that memory and load more granules. This deserves a fair bit of thought; do we define a fixed size granule buffer (e.g. 50 or 100), or do we literally process sequentially where we create a tdigest per granule that we read, and then continue to update granule by granule. What's the impact on run time with this? We need to stay under the 900 timeout, so this is a joint optimization across both processing time and memory.