Skip to content

Commit 6e71969

Browse files
CTTYgbrgr
authored andcommitted
doc: Update IO feature flag and examples (apache#2214)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes #. ## What changes are included in this PR? - as title <!-- Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates. --> ## Are these changes tested? NA <!-- Specify what test covers (unit test, integration test, etc.). If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? -->
1 parent 1922432 commit 6e71969

File tree

3 files changed

+104
-39
lines changed

3 files changed

+104
-39
lines changed

crates/iceberg/README.md

Lines changed: 16 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -29,17 +29,24 @@ See the [API documentation](https://docs.rs/iceberg/latest) for examples and the
2929
## Usage
3030

3131
```rust
32+
use std::collections::HashMap;
33+
use std::sync::Arc;
34+
3235
use futures::TryStreamExt;
33-
use iceberg::io::{FileIO, FileIOBuilder};
34-
use iceberg::{Catalog, Result, TableIdent};
35-
use iceberg_catalog_memory::MemoryCatalog;
36+
use iceberg::io::MemoryStorageFactory;
37+
use iceberg::memory::{MemoryCatalogBuilder, MEMORY_CATALOG_WAREHOUSE};
38+
use iceberg::{Catalog, CatalogBuilder, Result, TableIdent};
3639

3740
#[tokio::main]
3841
async fn main() -> Result<()> {
39-
// Build your file IO.
40-
let file_io = FileIOBuilder::new("memory").build()?;
41-
// Connect to a catalog.
42-
let catalog = MemoryCatalog::new(file_io, None);
42+
// Connect to a catalog with a memory storage factory.
43+
let catalog = MemoryCatalogBuilder::default()
44+
.with_storage_factory(Arc::new(MemoryStorageFactory))
45+
.load(
46+
"my_catalog",
47+
HashMap::from([(MEMORY_CATALOG_WAREHOUSE.to_string(), "/tmp/warehouse".to_string())]),
48+
)
49+
.await?;
4350
// Load table from catalog.
4451
let table = catalog
4552
.load_table(&TableIdent::from_strs(["hello", "world"])?)
@@ -58,26 +65,6 @@ async fn main() -> Result<()> {
5865
}
5966
```
6067

61-
## IO Support
62-
63-
Iceberg Rust provides various storage backends through feature flags. Here are the currently supported storage backends:
64-
65-
| Storage Backend | Feature Flag | Status | Description |
66-
| -------------------- | ---------------- | -------------- | --------------------------------------------- |
67-
| Memory | `storage-memory` | ✅ Stable | In-memory storage for testing and development |
68-
| Local Filesystem | `storage-fs` | ✅ Stable | Local filesystem storage |
69-
| Amazon S3 | `storage-s3` | ✅ Stable | Amazon S3 storage |
70-
| Google Cloud Storage | `storage-gcs` | ✅ Stable | Google Cloud Storage |
71-
| Alibaba Cloud OSS | `storage-oss` | 🧪 Experimental | Alibaba Cloud Object Storage Service |
72-
| Azure Datalake | `storage-azdls` | 🧪 Experimental | Azure Datalake Storage v2 |
68+
## Storage Backends
7369

74-
You can enable all stable storage backends at once using the `storage-all` feature flag.
75-
76-
> Note that `storage-oss` and `storage-azdls` are currently experimental and not included in `storage-all`.
77-
78-
Example usage in `Cargo.toml`:
79-
80-
```toml
81-
[dependencies]
82-
iceberg = { version = "x.y.z", features = ["storage-s3", "storage-fs"] }
83-
```
70+
For storage backend support (S3, GCS, local filesystem, etc.), use the [`iceberg-storage-opendal`](https://crates.io/crates/iceberg-storage-opendal) crate. See its [README](../storage/opendal/README.md) for available backends and feature flags.

crates/iceberg/src/io/file_io.rs

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -35,16 +35,10 @@ use crate::Result;
3535
/// All paths passed to `FileIO` must be absolute paths starting with the scheme string
3636
/// appropriate for the storage backend being used.
3737
///
38-
/// Supported storages:
39-
///
40-
/// | Storage | Feature Flag | Expected Path Format | Schemes |
41-
/// |--------------------|-------------------|----------------------------------| ------------------------------|
42-
/// | Local file system | `storage-fs` | `file` | `file://path/to/file` |
43-
/// | Memory | `storage-memory` | `memory` | `memory://path/to/file` |
44-
/// | S3 | `storage-s3` | `s3`, `s3a` | `s3://<bucket>/path/to/file` |
45-
/// | GCS | `storage-gcs` | `gs`, `gcs` | `gs://<bucket>/path/to/file` |
46-
/// | OSS | `storage-oss` | `oss` | `oss://<bucket>/path/to/file` |
47-
/// | Azure Datalake | `storage-azdls` | `abfs`, `abfss`, `wasb`, `wasbs` | `abfs://<filesystem>@<account>.dfs.core.windows.net/path/to/file` or `wasb://<container>@<account>.blob.core.windows.net/path/to/file` |
38+
/// This crate provides native support for local filesystem (`file://`) and
39+
/// memory (`memory://`) storage. For extensive storage backend support (S3, GCS,
40+
/// OSS, Azure, etc.), use the
41+
/// [`iceberg-storage-opendal`](https://crates.io/crates/iceberg-storage-opendal) crate.
4842
///
4943
/// # Example
5044
///

crates/storage/opendal/README.md

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
<!--
2+
~ Licensed to the Apache Software Foundation (ASF) under one
3+
~ or more contributor license agreements. See the NOTICE file
4+
~ distributed with this work for additional information
5+
~ regarding copyright ownership. The ASF licenses this file
6+
~ to you under the Apache License, Version 2.0 (the
7+
~ "License"); you may not use this file except in compliance
8+
~ with the License. You may obtain a copy of the License at
9+
~
10+
~ http://www.apache.org/licenses/LICENSE-2.0
11+
~
12+
~ Unless required by applicable law or agreed to in writing,
13+
~ software distributed under the License is distributed on an
14+
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
~ KIND, either express or implied. See the License for the
16+
~ specific language governing permissions and limitations
17+
~ under the License.
18+
-->
19+
20+
# iceberg-storage-opendal
21+
22+
OpenDAL-based storage backend implementations for [Apache Iceberg Rust](https://rust.iceberg.apache.org/).
23+
24+
## Supported Storage Backends
25+
26+
| Storage Backend | Feature Flag | Status | Description |
27+
| -------------------- | ---------------- | --------------- | --------------------------------------------- |
28+
| Memory | `opendal-memory` | ✅ Stable | In-memory storage for testing and development |
29+
| Local Filesystem | `opendal-fs` | ✅ Stable | Local filesystem storage |
30+
| Amazon S3 | `opendal-s3` | ✅ Stable | Amazon S3 storage |
31+
| Google Cloud Storage | `opendal-gcs` | ✅ Stable | Google Cloud Storage |
32+
| Alibaba Cloud OSS | `opendal-oss` | 🧪 Experimental | Alibaba Cloud Object Storage Service |
33+
| Azure Datalake | `opendal-azdls` | 🧪 Experimental | Azure Datalake Storage v2 |
34+
35+
You can enable all stable storage backends at once using the `opendal-all` feature flag.
36+
37+
> Note that `opendal-oss` and `opendal-azdls` are currently experimental and not included in `opendal-all`.
38+
39+
## Usage
40+
41+
Add the crate to your `Cargo.toml` with the feature flags for the backends you need:
42+
43+
```toml
44+
[dependencies]
45+
iceberg = { version = "x.y.z" }
46+
iceberg-storage-opendal = { version = "x.y.z", features = ["opendal-s3"] }
47+
iceberg-catalog-rest = { version = "x.y.z" }
48+
```
49+
50+
Then pass an `OpenDalStorageFactory` to your catalog builder:
51+
52+
```rust
53+
use std::collections::HashMap;
54+
use std::sync::Arc;
55+
56+
use iceberg::{Catalog, CatalogBuilder, TableIdent};
57+
use iceberg_catalog_rest::{RestCatalogBuilder, REST_CATALOG_PROP_URI};
58+
use iceberg_storage_opendal::OpenDalStorageFactory;
59+
60+
#[tokio::main]
61+
async fn main() -> iceberg::Result<()> {
62+
let catalog = RestCatalogBuilder::default()
63+
.with_storage_factory(Arc::new(OpenDalStorageFactory::S3 {
64+
configured_scheme: "s3".to_string(),
65+
customized_credential_load: None,
66+
}))
67+
.load(
68+
"my_catalog",
69+
HashMap::from([
70+
(REST_CATALOG_PROP_URI.to_string(), "http://localhost:8181".to_string()),
71+
]),
72+
)
73+
.await?;
74+
75+
let table = catalog
76+
.load_table(&TableIdent::from_strs(["my_namespace", "my_table"])?)
77+
.await?;
78+
79+
let scan = table.scan().select_all().build()?;
80+
let stream = scan.to_arrow().await?;
81+
82+
Ok(())
83+
}
84+
```

0 commit comments

Comments
 (0)