Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion docs/src/client/operations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ See [REST Routes](../../rest/catalog-spec.md#rest-routes) for more details.

## Recommended Basic Operations

To have a functional basic namespace implementation,
To have a functional basic namespace implementation,
the following metadata operations are recommended as a minimum:

**Namespace Metadata Operations:**
Expand All @@ -108,6 +108,19 @@ These operations provide the foundational metadata management capabilities neede
without requiring data or index operation support. With the namespace able to provide basic information about the table,
the Lance SDK can be used to fulfill the other operations.

### Restrictions for Basic Operations

The following restrictions apply to the recommended basic operations to minimize implementation complexity:

**DropNamespace:** Only the `Restrict` behavior mode is required.
This means the namespace must be empty (no tables or child namespaces) before it can be dropped.
The `Cascade` behavior mode, which recursively drops all contents, is not required for basic implementations.

**DescribeTable:** Only `load_detailed_metadata=false` (the default) is required.
This means the implementation only needs to return the table `location` without opening the dataset.
Returning detailed metadata such as `version`, `schema`, and `stats` (which require opening the dataset)
is not required for basic implementations.

### Why Not `CreateTable` and `DropTable`?

`CreateTable` and `DropTable` are common in most catalog systems,
Expand Down
1 change: 1 addition & 0 deletions docs/src/client/operations/models/DescribeTableRequest.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
|**id** | **List<String>** | | [optional] |
|**version** | **Long** | Version of the table to describe. If not specified, server should resolve it to the latest version. | [optional] |
|**withTableUri** | **Boolean** | Whether to include the table URI in the response. Default is false. | [optional] |
|**loadDetailedMetadata** | **Boolean** | Whether to load detailed metadata that requires opening the dataset. When false (default), only `location` is required in the response. When true, the response includes additional metadata such as `version`, `schema`, and `stats` which require reading the dataset. | [optional] |



14 changes: 7 additions & 7 deletions docs/src/client/operations/models/DescribeTableResponse.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@

| Name | Type | Description | Notes |
|------------ | ------------- | ------------- | -------------|
|**table** | **String** | Table name | [optional] |
|**namespace** | **List<String>** | The namespace identifier as a list of parts | [optional] |
|**version** | **Long** | | [optional] |
|**location** | **String** | Table storage location (e.g., S3/GCS path) | [optional] |
|**tableUri** | **String** | Table URI. Unlike location, this field must be a complete and valid URI | [optional] |
|**schema** | [**JsonArrowSchema**](JsonArrowSchema.md) | | [optional] |
|**table** | **String** | Table name. Only populated when `load_detailed_metadata` is true. | [optional] |
|**namespace** | **List<String>** | The namespace identifier as a list of parts. Only populated when `load_detailed_metadata` is true. | [optional] |
|**version** | **Long** | Table version number. Only populated when `load_detailed_metadata` is true. | [optional] |
|**location** | **String** | Table storage location (e.g., S3/GCS path). This is the only required field and is always returned. | |
|**tableUri** | **String** | Table URI. Unlike location, this field must be a complete and valid URI. Only returned when `with_table_uri` is true. | [optional] |
|**schema** | [**JsonArrowSchema**](JsonArrowSchema.md) | Table schema in JSON Arrow format. Only populated when `load_detailed_metadata` is true. | [optional] |
|**storageOptions** | **Map<String, String>** | Configuration options to be used to access storage. The available options depend on the type of storage in use. These will be passed directly to Lance to initialize storage access. | [optional] |
|**stats** | [**TableBasicStats**](TableBasicStats.md) | Table statistics | [optional] |
|**stats** | [**TableBasicStats**](TableBasicStats.md) | Table statistics. Only populated when `load_detailed_metadata` is true. | [optional] |



14 changes: 12 additions & 2 deletions docs/src/integrations/template.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,17 +56,23 @@ For each of the 8 recommended basic operations, provide a detailed subsection. E
The 8 basic operations are:

**Namespace Operations:**

- CreateNamespace
- ListNamespaces
- DescribeNamespace
- DropNamespace
- DropNamespace (only `Restrict` behavior mode required)

**Table Operations:**

- DeclareTable
- ListTables
- DescribeTable
- DescribeTable (only `load_detailed_metadata=false` required)
- DeregisterTable

**Note:** For basic implementations, DropNamespace only needs to support the `Restrict` behavior mode
(namespace must be empty before dropping). DescribeTable only needs to support `load_detailed_metadata=false`
(only return table `location` without opening the dataset).

---

## Template Structure
Expand Down Expand Up @@ -144,6 +150,8 @@ If {condition}, return error code `N` ({ErrorName}).

{Same structure as above}

**Note:** Basic implementations only need to support `Restrict` behavior mode.

### DeclareTable

{Same structure as above}
Expand All @@ -156,6 +164,8 @@ If {condition}, return error code `N` ({ErrorName}).

{Same structure as above}

**Note:** Basic implementations only need to support `load_detailed_metadata=false` (only return table `location`).

### DeregisterTable

{Same structure as above}
Expand Down
49 changes: 43 additions & 6 deletions docs/src/rest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,7 @@ paths:
- $ref: '#/components/parameters/id'
- $ref: '#/components/parameters/delimiter'
- $ref: '#/components/parameters/with_table_uri'
- $ref: '#/components/parameters/load_detailed_metadata'
post:
tags:
- Table
Expand All @@ -398,7 +399,7 @@ paths:
Describe the detailed information for table `id`.

REST NAMESPACE ONLY
REST namespace passes `with_table_uri` as a query parameter instead of in the request body.
REST namespace passes `with_table_uri` and `load_detailed_metadata` as query parameters instead of in the request body.
requestBody:
required: true
content:
Expand Down Expand Up @@ -1835,6 +1836,17 @@ components:
schema:
type: boolean
default: false
load_detailed_metadata:
name: load_detailed_metadata
description: |
Whether to load detailed metadata that requires opening the dataset.
When false (default), only `location` is required in the response.
When true, the response includes additional metadata such as `version`, `schema`, and `stats`.
in: query
required: false
schema:
type: boolean
default: false

schemas:
ErrorResponse:
Expand Down Expand Up @@ -2135,31 +2147,54 @@ components:
Default is false.
type: boolean
default: false
load_detailed_metadata:
description: |
Whether to load detailed metadata that requires opening the dataset.
When false (default), only `location` is required in the response.
When true, the response includes additional metadata such as `version`, `schema`, and `stats`
which require reading the dataset.
type: boolean
default: false

DescribeTableResponse:
type: object
required:
- location
properties:
table:
type: string
description: Table name
description: |
Table name.
Only populated when `load_detailed_metadata` is true.
namespace:
type: array
items:
type: string
description: The namespace identifier as a list of parts
description: |
The namespace identifier as a list of parts.
Only populated when `load_detailed_metadata` is true.
version:
type: integer
format: int64
minimum: 0
description: |
Table version number.
Only populated when `load_detailed_metadata` is true.
location:
type: string
description: Table storage location (e.g., S3/GCS path)
description: |
Table storage location (e.g., S3/GCS path).
This is the only required field and is always returned.
table_uri:
type: string
description: |
Table URI. Unlike location, this field must be a complete and valid URI
Table URI. Unlike location, this field must be a complete and valid URI.
Only returned when `with_table_uri` is true.
schema:
$ref: '#/components/schemas/JsonArrowSchema'
description: |
Table schema in JSON Arrow format.
Only populated when `load_detailed_metadata` is true.
storage_options:
type: object
description: |
Expand All @@ -2171,7 +2206,9 @@ components:
stats:
$ref: '#/components/schemas/TableBasicStats'
nullable: true
description: Table statistics
description: |
Table statistics.
Only populated when `load_detailed_metadata` is true.

TableBasicStats:
type: object
Expand Down
46 changes: 41 additions & 5 deletions java/lance-namespace-apache-client/api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -399,17 +399,19 @@ paths:
- $ref: '#/components/parameters/id'
- $ref: '#/components/parameters/delimiter'
- $ref: '#/components/parameters/with_table_uri'
- $ref: '#/components/parameters/load_detailed_metadata'
post:
description: |
Describe the detailed information for table `id`.

REST NAMESPACE ONLY
REST namespace passes `with_table_uri` as a query parameter instead of in the request body.
REST namespace passes `with_table_uri` and `load_detailed_metadata` as query parameters instead of in the request body.
operationId: DescribeTable
parameters:
- $ref: '#/components/parameters/id'
- $ref: '#/components/parameters/delimiter'
- $ref: '#/components/parameters/with_table_uri'
- $ref: '#/components/parameters/load_detailed_metadata'
requestBody:
content:
application/json:
Expand Down Expand Up @@ -2164,6 +2166,19 @@ components:
default: false
type: boolean
style: form
load_detailed_metadata:
description: |
Whether to load detailed metadata that requires opening the dataset.
When false (default), only `location` is required in the response.
When true, the response includes additional metadata such as `version`, `schema`, and `stats`.
explode: true
in: query
name: load_detailed_metadata
required: false
schema:
default: false
type: boolean
style: form
responses:
ListNamespacesResponse:
content:
Expand Down Expand Up @@ -2871,6 +2886,7 @@ components:
- id
version: 0
with_table_uri: false
load_detailed_metadata: false
properties:
id:
items:
Expand All @@ -2889,6 +2905,14 @@ components:
Whether to include the table URI in the response.
Default is false.
type: boolean
load_detailed_metadata:
default: false
description: |
Whether to load detailed metadata that requires opening the dataset.
When false (default), only `location` is required in the response.
When true, the response includes additional metadata such as `version`, `schema`, and `stats`
which require reading the dataset.
type: boolean
DescribeTableResponse:
example:
schema:
Expand Down Expand Up @@ -2929,23 +2953,33 @@ components:
key: storage_options
properties:
table:
description: Table name
description: |
Table name.
Only populated when `load_detailed_metadata` is true.
type: string
namespace:
description: The namespace identifier as a list of parts
description: |
The namespace identifier as a list of parts.
Only populated when `load_detailed_metadata` is true.
items:
type: string
type: array
version:
description: |
Table version number.
Only populated when `load_detailed_metadata` is true.
format: int64
minimum: 0
type: integer
location:
description: "Table storage location (e.g., S3/GCS path)"
description: |
Table storage location (e.g., S3/GCS path).
This is the only required field and is always returned.
type: string
table_uri:
description: |
Table URI. Unlike location, this field must be a complete and valid URI
Table URI. Unlike location, this field must be a complete and valid URI.
Only returned when `with_table_uri` is true.
type: string
schema:
$ref: '#/components/schemas/JsonArrowSchema'
Expand All @@ -2958,6 +2992,8 @@ components:
passed directly to Lance to initialize storage access.
stats:
$ref: '#/components/schemas/TableBasicStats'
required:
- location
TableBasicStats:
example:
num_deleted_rows: 0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
|**id** | **List<String>** | | [optional] |
|**version** | **Long** | Version of the table to describe. If not specified, server should resolve it to the latest version. | [optional] |
|**withTableUri** | **Boolean** | Whether to include the table URI in the response. Default is false. | [optional] |
|**loadDetailedMetadata** | **Boolean** | Whether to load detailed metadata that requires opening the dataset. When false (default), only `location` is required in the response. When true, the response includes additional metadata such as `version`, `schema`, and `stats` which require reading the dataset. | [optional] |



14 changes: 7 additions & 7 deletions java/lance-namespace-apache-client/docs/DescribeTableResponse.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@

| Name | Type | Description | Notes |
|------------ | ------------- | ------------- | -------------|
|**table** | **String** | Table name | [optional] |
|**namespace** | **List<String>** | The namespace identifier as a list of parts | [optional] |
|**version** | **Long** | | [optional] |
|**location** | **String** | Table storage location (e.g., S3/GCS path) | [optional] |
|**tableUri** | **String** | Table URI. Unlike location, this field must be a complete and valid URI | [optional] |
|**schema** | [**JsonArrowSchema**](JsonArrowSchema.md) | | [optional] |
|**table** | **String** | Table name. Only populated when `load_detailed_metadata` is true. | [optional] |
|**namespace** | **List<String>** | The namespace identifier as a list of parts. Only populated when `load_detailed_metadata` is true. | [optional] |
|**version** | **Long** | Table version number. Only populated when `load_detailed_metadata` is true. | [optional] |
|**location** | **String** | Table storage location (e.g., S3/GCS path). This is the only required field and is always returned. | |
|**tableUri** | **String** | Table URI. Unlike location, this field must be a complete and valid URI. Only returned when `with_table_uri` is true. | [optional] |
|**schema** | [**JsonArrowSchema**](JsonArrowSchema.md) | Table schema in JSON Arrow format. Only populated when `load_detailed_metadata` is true. | [optional] |
|**storageOptions** | **Map<String, String>** | Configuration options to be used to access storage. The available options depend on the type of storage in use. These will be passed directly to Lance to initialize storage access. | [optional] |
|**stats** | [**TableBasicStats**](TableBasicStats.md) | Table statistics | [optional] |
|**stats** | [**TableBasicStats**](TableBasicStats.md) | Table statistics. Only populated when `load_detailed_metadata` is true. | [optional] |



8 changes: 5 additions & 3 deletions java/lance-namespace-apache-client/docs/MetadataApi.md
Original file line number Diff line number Diff line change
Expand Up @@ -1139,11 +1139,11 @@ public class Example {

## describeTable

> DescribeTableResponse describeTable(id, describeTableRequest, delimiter, withTableUri)
> DescribeTableResponse describeTable(id, describeTableRequest, delimiter, withTableUri, loadDetailedMetadata)

Describe information of a table

Describe the detailed information for table `id`. REST NAMESPACE ONLY REST namespace passes `with_table_uri` as a query parameter instead of in the request body.
Describe the detailed information for table `id`. REST NAMESPACE ONLY REST namespace passes `with_table_uri` and `load_detailed_metadata` as query parameters instead of in the request body.

### Example

Expand Down Expand Up @@ -1180,8 +1180,9 @@ public class Example {
DescribeTableRequest describeTableRequest = new DescribeTableRequest(); // DescribeTableRequest |
String delimiter = "delimiter_example"; // String | An optional delimiter of the `string identifier`, following the Lance Namespace spec. When not specified, the `$` delimiter must be used.
Boolean withTableUri = false; // Boolean | Whether to include the table URI in the response
Boolean loadDetailedMetadata = false; // Boolean | Whether to load detailed metadata that requires opening the dataset. When false (default), only `location` is required in the response. When true, the response includes additional metadata such as `version`, `schema`, and `stats`.
try {
DescribeTableResponse result = apiInstance.describeTable(id, describeTableRequest, delimiter, withTableUri);
DescribeTableResponse result = apiInstance.describeTable(id, describeTableRequest, delimiter, withTableUri, loadDetailedMetadata);
System.out.println(result);
} catch (ApiException e) {
System.err.println("Exception when calling MetadataApi#describeTable");
Expand All @@ -1203,6 +1204,7 @@ public class Example {
| **describeTableRequest** | [**DescribeTableRequest**](DescribeTableRequest.md)| | |
| **delimiter** | **String**| An optional delimiter of the `string identifier`, following the Lance Namespace spec. When not specified, the `$` delimiter must be used. | [optional] |
| **withTableUri** | **Boolean**| Whether to include the table URI in the response | [optional] [default to false] |
| **loadDetailedMetadata** | **Boolean**| Whether to load detailed metadata that requires opening the dataset. When false (default), only `location` is required in the response. When true, the response includes additional metadata such as `version`, `schema`, and `stats`. | [optional] [default to false] |

### Return type

Expand Down
Loading