Epic: partitioned namespace

[wojiaodoubao]

Code Design

1. ManifestNamespace

• The ManifestNamespace will introduce an extended_properties feature, which supports storing specified properties as __manifest columns. These properties can be specified during create/describe table or namespace operations and are then set to or read from __manifest extended columns.
• ManifestNamespace provides a public insert_into_manifest method to facilitate transactional commits for PartitionedNamespace.

2. PartitionedNamespace
PartitionedNamespace is an impl of trait LanceNamespace, and has a member of ManifestNamespace. PartitionedNamespace exposes two categories of APIs:

• The first category inherits methods from LanceNamespace. All of their implementations directly delegate to ManifestNamespace.
• The second category consists of methods with partition semantics, such as plan_scan and resolve_or_create_partition_table. These methods encapsulate partition-related logic and are exposed to upper-layer engines for integration via public APIs.

Step by Step

• (lance-namespace) Add partition spec to rest.yaml. feat: add partition spec to rest.yaml lance-namespace#297
• (lance-namespace) Add table properties. feat: add table properties lance-namespace#299
• (lance) Support extended properties in ManifestNamespace.
  • Extended properties for namespace related methods. feat: add extended properties to manifest namespace #5738
  • Extended properties for table related methods.
• (lance) Implement PartitionedNamespace. feat: add framework of partitioned namespace #5638

After PartitionedNamespace is implemented, we can start integrating with engines.

Related Links

• Discussion. Introduce Partition Table Spec in Lance Namespace lance-namespace#272