Automatically iterating through each AOI for grid generation rather than having the users handle it manually

[jtmiclat]

Based on the @joshuacortez suggested #55 (comment)
Some questions:

• Should we iterate on each AOI and deduplicate by default? If yes how do we know when to do and when just do the naive version
• Should this be a separate function or param or just not configurable?

[jtmiclat]

A possible option is to check the unary union of the AOIs and if it is a Polygon do the native method and if MultiPolygon do the iterative method.

[joshuacortez]

Hmm could we just make it iterative by default?

[jtmiclat]

Iterative is an improvement for AOIs that are non-adjacent but i believe if the aoi are adjacent it would be more performance to not iterate through them.

[jtmiclat]

We might need benchmark and compare these scenarios.

[butchtm]

Would it be possible to parallelize this by running the grid computation for each aoi independently in parallel and deduplicating the resulting rows in the end? Maybe by using something like dask-pandas? Of course this should be completely optional and useful only when doing fine-grained grids covering large areas.

[jtmiclat]

Just a heads up, we are using this logic to handle generation of grids for h3 and bing tiles

geowrangler/geowrangler/grids.py

Lines 161 to 166 in 4325da0

	if isinstance(unary_union, Polygon):
	hex_ids.update(self.get_hexes_for_polygon(unary_union))
	else:
	for geom in reprojected_gdf.unary_union.geoms:
	_hexes = self.get_hexes_for_polygon(geom)
	hex_ids.update(_hexes)

[joshuacortez]

awesome thanks!

[jtmiclat]

I made an initial test. For unary_union to make 5km grids off of data/region3.json it took

CPU times: user 918 ms, sys: 12 ms, total: 930 ms
Wall time: 930 ms

while the current way took no iteration took

CPU times: user 38.9 s, sys: 57.8 ms, total: 38.9 s
Wall time: 41.1 s

Will bench mark iterating each aoi vs iterating unary_union!

[jtmiclat]

Benchmark based on gpkg mentioned here #55 (comment)

Iterating through each aoi (300 rows) 
CPU times: user 2min 7s, sys: 1.68 s, total: 2min 8s
Wall time: 2min 10s
Iterating through each unary union (300 rows) 
CPU times: user 42.2 s, sys: 1.15 s, total: 43.3 s
Wall time: 43.6 s

Iterating through each aoi (4k rows)
CPU times: user 52min 34s, sys: 23.3 s, total: 52min 58s
Wall time: 52min 38s

Iterating through each unary union(4k rows)
CPU times: user 41min 17s, sys: 36.5 s, total: 41min 54s
Wall time: 41min 56s

[jtmiclat]

Created #90 to fully implement this!

[joshuacortez]

Cool, so the unary_union shaves off time even if the polygons are mostly adjacent or not

[joshuacortez]

Closing since this is already addressed: using unary_union in Square Grid Generator and Bing Tile Grid Generator