Skip to content

OpenDAL datastorage plugin #9332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
josegar74 wants to merge 2 commits into
base: main
Choose a base branch
from
+1,466 −0
Draft

OpenDAL datastorage plugin #9332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4907807
Datastores / Add support for OpenDAL
josegar74 Jun 11, 2026
7a7d172
Datastores / Add support for OpenDAL - update manual
josegar74 Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions core/src/main/resources/config-spring-geonetwork.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,7 @@
default - Default file system store
s3 - AWS S3 storage (see config-store/config-s3.xml for more details)
cmis - CMIS storage (see config-store/config-cmis.xml for more details)
opendal - OpenDAL storage (see config-store/config-opendal.xml for more details)
-->
<import resource="classpath*:/config-store/config-${geonetwork.store.type:default}.xml"/>
</beans>
67 changes: 67 additions & 0 deletions datastorages/opendal/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# OpenDAL Data storage

Extension for data storage using [Apache OpenDAL](https://opendal.apache.org/), which provides a unified data access layer for various storage services.

## Configuration

The OpenDAL datastorage can be configured using environment variables. These variables are mapped to the OpenDAL operator configuration.

### Environment Variables

| Variable | Property | Default | Description |
|----------|----------|---------|-------------|
| `OPENDAL_SCHEME` | `opendal.scheme` | `fs` | The storage scheme (e.g., `fs`, `s3`, `azblob`, `gcs`). |
| `OPENDAL_ROOT` | `opendal.root` | `/tmp/opendal` | The root directory or path for the storage. |
| `OPENDAL_ENDPOINT` | `opendal.endpoint` | | The endpoint URL (required for S3-compatible storage). |
| `OPENDAL_BUCKET` | `opendal.bucket` | | The bucket name (required for S3/GCS/Azblob). |
| `OPENDAL_ACCESS_KEY_ID` | `opendal.access_key_id` | | Access key ID for authentication. |
| `OPENDAL_SECRET_ACCESS_KEY` | `opendal.secret_access_key` | | Secret access key for authentication. |
| `OPENDAL_REGION` | `opendal.region` | | The region for the storage service. |
| `OPENDAL_USERNAME` | `opendal.username` | | Username for authentication (e.g., WebDAV). |
| `OPENDAL_PASSWORD` | `opendal.password` | | Password for authentication (e.g., WebDAV). |

## Examples

### Local Filesystem

To configure OpenDAL to use the local filesystem:

```bash
export OPENDAL_SCHEME=fs
export OPENDAL_ROOT=/path/to/your/storage
```

### Amazon S3

To configure OpenDAL to use Amazon S3:

```bash
export OPENDAL_SCHEME=s3
export OPENDAL_ROOT=my-folder
export OPENDAL_BUCKET=my-bucket
export OPENDAL_REGION=us-east-1
export OPENDAL_ACCESS_KEY_ID=your_access_key
export OPENDAL_SECRET_ACCESS_KEY=your_secret_key
```

### S3 Compatible Storage (e.g., Minio)

```bash
export OPENDAL_SCHEME=s3
export OPENDAL_ENDPOINT=http://localhost:9000
export OPENDAL_BUCKET=my-bucket
export OPENDAL_ACCESS_KEY_ID=minioadmin
export OPENDAL_SECRET_ACCESS_KEY=minioadmin
```

### WebDAV

To configure OpenDAL to use a WebDAV server:

```bash
export OPENDAL_SCHEME=webdav
export OPENDAL_ENDPOINT=http://your-webdav-server.com/dav
export OPENDAL_ROOT=/remote.php/dav/files/user/
export OPENDAL_USERNAME=your_username
export OPENDAL_PASSWORD=your_password
```
273 changes: 273 additions & 0 deletions datastorages/opendal/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,273 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>gn-datastorages</artifactId>
<groupId>org.geonetwork-opensource.datastorage</groupId>
<version>4.4.12-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>gn-datastorage-opendal</artifactId>
<name>OpenDAL datastorage</name>

<dependencies>
<dependency>
<groupId>org.geonetwork-opensource</groupId>
<artifactId>gn-core</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.geonetwork-opensource</groupId>
<artifactId>gn-domain</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.opendal</groupId>
<artifactId>opendal</artifactId>
<version>${opendal.version}</version>
</dependency>

<dependency>
<groupId>org.apache.opendal</groupId>
<artifactId>opendal</artifactId>
<version>${opendal.version}</version>
<classifier>${os.detected.classifier}</classifier>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<id>test-jar</id>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
</resources>
</build>

<profiles>
<profile>
<id>run-static-analysis</id>
<activation>
<property>
<name>!skipTests</name>
</property>
</activation>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</profile>

<!-- Profile to release datastorage bundle -->
<profile>
<id>release</id>
<activation>
<property>
<name>release</name>
</property>
</activation>
<build>
<plugins>
<!-- stage jars for assembly -->
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>stage-libs</id>
<phase>prepare-package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<!-- libs to include are managed here rather than in assembly -->
<configuration>
<silent>true</silent>
<includeScope>runtime</includeScope>
<excludeScope>provided</excludeScope>
<excludeGroupIds>com.google.code.findbugs</excludeGroupIds>
<excludeTransitive>false</excludeTransitive>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>true</overWriteSnapshots>
<excludeClassifiers>${os.detected.classifier}</excludeClassifiers>
</configuration>
</execution>
<execution>
<id>stage-native-libs</id>
<phase>prepare-package</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.apache.opendal</groupId>
<artifactId>opendal</artifactId>
<version>${opendal.version}</version>
<classifier>windows-x86_64</classifier>
<outputDirectory>${project.build.directory}/lib-native/windows-x86_64</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.apache.opendal</groupId>
<artifactId>opendal</artifactId>
<version>${opendal.version}</version>
<classifier>linux-x86_64</classifier>
<outputDirectory>${project.build.directory}/lib-native/linux-x86_64</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.apache.opendal</groupId>
<artifactId>opendal</artifactId>
<version>${opendal.version}</version>
<classifier>osx-aarch_64</classifier>
<outputDirectory>${project.build.directory}/lib-native/osx-aarch_64</outputDirectory>
</artifactItem>
<artifactItem>
<groupId>org.apache.opendal</groupId>
<artifactId>opendal</artifactId>
<version>${opendal.version}</version>
<classifier>osx-x86_64</classifier>
<outputDirectory>${project.build.directory}/lib-native/osx-x86_64</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<!-- generate html contents-->
<plugin>
<groupId>com.ruleoftech</groupId>
<artifactId>markdown-page-generator-plugin</artifactId>
<executions>
<execution>
<id>readme</id>
<phase>process-resources</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<inputDirectory>${project.basedir}/src/main/assembly</inputDirectory>
<outputDirectory>${project.build.directory}/html</outputDirectory>
</configuration>
</execution>
<execution>
<id>licenses</id>
<phase>process-resources</phase>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<inputDirectory>${project.basedir}/../../</inputDirectory>
<outputDirectory>${project.build.directory}/html/license</outputDirectory>
</configuration>
</execution>
</executions>
<configuration>
<recursiveInput>false</recursiveInput>
<transformRelativeMarkdownLinks>true</transformRelativeMarkdownLinks>
<headerHtmlFile>${project.basedir}/../../release/src/markdown/html/header.html</headerHtmlFile>
<footerHtmlFile>${project.basedir}/../../release/src/markdown/html/footer.html</footerHtmlFile>
<pegdownExtensions>TABLES,FENCED_CODE_BLOCKS</pegdownExtensions>
<defaultTitle>true</defaultTitle>
</configuration>
</plugin>
<!-- assemble into zip for release -->
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<id>release-windows-64</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<finalName>${project.artifactId}-${project.version}-windows-x86_64</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/assembly-windows-x86_64.xml</descriptor>
</descriptors>
</configuration>
</execution>
<execution>
<id>release-linux-64</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<finalName>${project.artifactId}-${project.version}-linux-x86_64</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/assembly-linux-x86_64.xml</descriptor>
</descriptors>
</configuration>
</execution>
<execution>
<id>release-osx-aarch_64</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<finalName>${project.artifactId}-${project.version}-osx-aarch_64</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/assembly-osx-aarch_64.xml</descriptor>
</descriptors>
</configuration>
</execution>
<execution>
<id>release-osx-intel-64</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<finalName>${project.artifactId}-${project.version}-osx-x86_64</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/assembly-osx-x86_64.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

</profiles>

<properties>
<rootProjectDir>${basedir}/..</rootProjectDir>
<opendal.version>0.49.0</opendal.version>
</properties>
</project>
46 changes: 46 additions & 0 deletions datastorages/opendal/src/main/assembly/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# OpenDAL data storage provider library

To install the OpenDAL data storage provider in GeoNetwork:

1. Copy the `lib` folder jar files into to `{GEONETWORK_DIR}/WEB-INF/lib` folder.

2. Define the following environmental variables:

- Use OpenDAL data storage provider:

```bash
export GEONETWORK_STORE_TYPE=opendal
```

- Setup OpenDAL connection (check with your setup):

- Filesystem example:

```bash
export OPENDAL_SCHEME=fs
export OPENDAL_ROOT=/tmp/opendal
```

- S3 example:

```bash
export OPENDAL_SCHEME=s3
export OPENDAL_ROOT=/
export OPENDAL_BUCKET=my-bucket
export OPENDAL_ENDPOINT=https://s3.amazonaws.com
export OPENDAL_REGION=us-east-1
export OPENDAL_ACCESS_KEY_ID=access_key
export OPENDAL_SECRET_ACCESS_KEY=secret_key
```

- WebDAV example:

```bash
export OPENDAL_SCHEME=webdav
export OPENDAL_ENDPOINT=http://your-webdav-server.com/dav
export OPENDAL_ROOT=/remote.php/dav/files/user/
export OPENDAL_USERNAME=your_username
export OPENDAL_PASSWORD=your_password
```

3. Start GeoNetwork
Loading
Loading