Add Flink Lookup Join support to Iceberg

[Guosmilesmile]

Feature Request / Improvement

Add Flink Lookup Join support to Iceberg with an incremental approach:

1. Memory-only full cache

• Use a heap-based cache with projection and filter pushdown.
• Track the snapshot used to build the cache.
• Load the cache once initially, targeting dimension tables that fit comfortably in TaskManager heap.

2. Periodic full reload
   • Refresh the cache at a configurable interval.
   • Atomically swap in the new cache after a successful reload.
   • Define an explicit failure policy that fails rather than silently serving stale data.
3. Incremental reload via snapshot deltas
   • Use Iceberg's incremental scan between the cached snapshot and the current snapshot.
   • Apply only the changes to the existing cache, so reload cost scales with the change set rather than the entire table.

Large lookup tables can be addressed separately once the second index is available. Potential approaches include RocksDB, FLIP-462 slicing, and async lookup.

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time