diff --git a/core/src/main/resources/config-spring-geonetwork.xml b/core/src/main/resources/config-spring-geonetwork.xml index 052e4d6ae6d3..d5a3ed291dea 100644 --- a/core/src/main/resources/config-spring-geonetwork.xml +++ b/core/src/main/resources/config-spring-geonetwork.xml @@ -250,6 +250,7 @@ default - Default file system store s3 - AWS S3 storage (see config-store/config-s3.xml for more details) cmis - CMIS storage (see config-store/config-cmis.xml for more details) + opendal - OpenDAL storage (see config-store/config-opendal.xml for more details) --> diff --git a/datastorages/opendal/README.md b/datastorages/opendal/README.md new file mode 100644 index 000000000000..f01ae9643b6d --- /dev/null +++ b/datastorages/opendal/README.md @@ -0,0 +1,67 @@ +# OpenDAL Data storage + +Extension for data storage using [Apache OpenDAL](https://opendal.apache.org/), which provides a unified data access layer for various storage services. + +## Configuration + +The OpenDAL datastorage can be configured using environment variables. These variables are mapped to the OpenDAL operator configuration. + +### Environment Variables + +| Variable | Property | Default | Description | +|----------|----------|---------|-------------| +| `OPENDAL_SCHEME` | `opendal.scheme` | `fs` | The storage scheme (e.g., `fs`, `s3`, `azblob`, `gcs`). | +| `OPENDAL_ROOT` | `opendal.root` | `/tmp/opendal` | The root directory or path for the storage. | +| `OPENDAL_ENDPOINT` | `opendal.endpoint` | | The endpoint URL (required for S3-compatible storage). | +| `OPENDAL_BUCKET` | `opendal.bucket` | | The bucket name (required for S3/GCS/Azblob). | +| `OPENDAL_ACCESS_KEY_ID` | `opendal.access_key_id` | | Access key ID for authentication. | +| `OPENDAL_SECRET_ACCESS_KEY` | `opendal.secret_access_key` | | Secret access key for authentication. | +| `OPENDAL_REGION` | `opendal.region` | | The region for the storage service. | +| `OPENDAL_USERNAME` | `opendal.username` | | Username for authentication (e.g., WebDAV). | +| `OPENDAL_PASSWORD` | `opendal.password` | | Password for authentication (e.g., WebDAV). | + +## Examples + +### Local Filesystem + +To configure OpenDAL to use the local filesystem: + +```bash +export OPENDAL_SCHEME=fs +export OPENDAL_ROOT=/path/to/your/storage +``` + +### Amazon S3 + +To configure OpenDAL to use Amazon S3: + +```bash +export OPENDAL_SCHEME=s3 +export OPENDAL_ROOT=my-folder +export OPENDAL_BUCKET=my-bucket +export OPENDAL_REGION=us-east-1 +export OPENDAL_ACCESS_KEY_ID=your_access_key +export OPENDAL_SECRET_ACCESS_KEY=your_secret_key +``` + +### S3 Compatible Storage (e.g., Minio) + +```bash +export OPENDAL_SCHEME=s3 +export OPENDAL_ENDPOINT=http://localhost:9000 +export OPENDAL_BUCKET=my-bucket +export OPENDAL_ACCESS_KEY_ID=minioadmin +export OPENDAL_SECRET_ACCESS_KEY=minioadmin +``` + +### WebDAV + +To configure OpenDAL to use a WebDAV server: + +```bash +export OPENDAL_SCHEME=webdav +export OPENDAL_ENDPOINT=http://your-webdav-server.com/dav +export OPENDAL_ROOT=/remote.php/dav/files/user/ +export OPENDAL_USERNAME=your_username +export OPENDAL_PASSWORD=your_password +``` diff --git a/datastorages/opendal/pom.xml b/datastorages/opendal/pom.xml new file mode 100644 index 000000000000..eb4d9abb366d --- /dev/null +++ b/datastorages/opendal/pom.xml @@ -0,0 +1,273 @@ + + + + gn-datastorages + org.geonetwork-opensource.datastorage + 4.4.12-SNAPSHOT + + 4.0.0 + + gn-datastorage-opendal + OpenDAL datastorage + + + + org.geonetwork-opensource + gn-core + ${project.version} + provided + + + + org.geonetwork-opensource + gn-domain + ${project.version} + provided + + + + org.springframework + spring-context + provided + + + + org.apache.opendal + opendal + ${opendal.version} + + + + org.apache.opendal + opendal + ${opendal.version} + ${os.detected.classifier} + + + + + + + maven-jar-plugin + + + test-jar + + test-jar + + + + + + + + src/main/resources + true + + + + + + + run-static-analysis + + + !skipTests + + + + + + org.codehaus.mojo + findbugs-maven-plugin + + + + + + + + release + + + release + + + + + + + maven-dependency-plugin + + + stage-libs + prepare-package + + copy-dependencies + + + + true + runtime + provided + com.google.code.findbugs + false + ${project.build.directory}/lib + false + true + ${os.detected.classifier} + + + + stage-native-libs + prepare-package + + copy + + + + + org.apache.opendal + opendal + ${opendal.version} + windows-x86_64 + ${project.build.directory}/lib-native/windows-x86_64 + + + org.apache.opendal + opendal + ${opendal.version} + linux-x86_64 + ${project.build.directory}/lib-native/linux-x86_64 + + + org.apache.opendal + opendal + ${opendal.version} + osx-aarch_64 + ${project.build.directory}/lib-native/osx-aarch_64 + + + org.apache.opendal + opendal + ${opendal.version} + osx-x86_64 + ${project.build.directory}/lib-native/osx-x86_64 + + + + + + + + + com.ruleoftech + markdown-page-generator-plugin + + + readme + process-resources + + generate + + + ${project.basedir}/src/main/assembly + ${project.build.directory}/html + + + + licenses + process-resources + + generate + + + ${project.basedir}/../../ + ${project.build.directory}/html/license + + + + + false + true + ${project.basedir}/../../release/src/markdown/html/header.html + ${project.basedir}/../../release/src/markdown/html/footer.html + TABLES,FENCED_CODE_BLOCKS + true + + + + + maven-assembly-plugin + + + release-windows-64 + package + + single + + + ${project.artifactId}-${project.version}-windows-x86_64 + false + + src/main/assembly/assembly-windows-x86_64.xml + + + + + release-linux-64 + package + + single + + + ${project.artifactId}-${project.version}-linux-x86_64 + false + + src/main/assembly/assembly-linux-x86_64.xml + + + + + release-osx-aarch_64 + package + + single + + + ${project.artifactId}-${project.version}-osx-aarch_64 + false + + src/main/assembly/assembly-osx-aarch_64.xml + + + + + release-osx-intel-64 + package + + single + + + ${project.artifactId}-${project.version}-osx-x86_64 + false + + src/main/assembly/assembly-osx-x86_64.xml + + + + + + + + + + + + + ${basedir}/.. + 0.49.0 + + diff --git a/datastorages/opendal/src/main/assembly/README.md b/datastorages/opendal/src/main/assembly/README.md new file mode 100644 index 000000000000..4350c02538d8 --- /dev/null +++ b/datastorages/opendal/src/main/assembly/README.md @@ -0,0 +1,46 @@ +# OpenDAL data storage provider library + +To install the OpenDAL data storage provider in GeoNetwork: + +1. Copy the `lib` folder jar files into to `{GEONETWORK_DIR}/WEB-INF/lib` folder. + +2. Define the following environmental variables: + + - Use OpenDAL data storage provider: + + ```bash + export GEONETWORK_STORE_TYPE=opendal + ``` + + - Setup OpenDAL connection (check with your setup): + + - Filesystem example: + + ```bash + export OPENDAL_SCHEME=fs + export OPENDAL_ROOT=/tmp/opendal + ``` + + - S3 example: + + ```bash + export OPENDAL_SCHEME=s3 + export OPENDAL_ROOT=/ + export OPENDAL_BUCKET=my-bucket + export OPENDAL_ENDPOINT=https://s3.amazonaws.com + export OPENDAL_REGION=us-east-1 + export OPENDAL_ACCESS_KEY_ID=access_key + export OPENDAL_SECRET_ACCESS_KEY=secret_key + ``` + + - WebDAV example: + + ```bash + export OPENDAL_SCHEME=webdav + export OPENDAL_ENDPOINT=http://your-webdav-server.com/dav + export OPENDAL_ROOT=/remote.php/dav/files/user/ + export OPENDAL_USERNAME=your_username + export OPENDAL_PASSWORD=your_password + ``` + +3. Start GeoNetwork diff --git a/datastorages/opendal/src/main/assembly/assembly-linux-x86_64.xml b/datastorages/opendal/src/main/assembly/assembly-linux-x86_64.xml new file mode 100644 index 000000000000..0ecdbdcb210b --- /dev/null +++ b/datastorages/opendal/src/main/assembly/assembly-linux-x86_64.xml @@ -0,0 +1,35 @@ + + bin + + zip + + + + ${project.build.directory}/lib + /lib + + + ${project.build.directory}/lib-native/linux-x86_64 + /lib + + + + + + ${project.build.directory}/${project.artifactId}-${project.version}.jar + /lib + + + + ${project.build.directory}/html/README.html + / + + + + ${project.build.directory}/html/license/LICENSE.html + /license + + + diff --git a/datastorages/opendal/src/main/assembly/assembly-osx-aarch_64.xml b/datastorages/opendal/src/main/assembly/assembly-osx-aarch_64.xml new file mode 100644 index 000000000000..312345dc00a9 --- /dev/null +++ b/datastorages/opendal/src/main/assembly/assembly-osx-aarch_64.xml @@ -0,0 +1,35 @@ + + bin + + zip + + + + ${project.build.directory}/lib + /lib + + + ${project.build.directory}/lib-native/osx-aarch_64 + /lib + + + + + + ${project.build.directory}/${project.artifactId}-${project.version}.jar + /lib + + + + ${project.build.directory}/html/README.html + / + + + + ${project.build.directory}/html/license/LICENSE.html + /license + + + diff --git a/datastorages/opendal/src/main/assembly/assembly-osx-x86_64.xml b/datastorages/opendal/src/main/assembly/assembly-osx-x86_64.xml new file mode 100644 index 000000000000..04462d6ca525 --- /dev/null +++ b/datastorages/opendal/src/main/assembly/assembly-osx-x86_64.xml @@ -0,0 +1,35 @@ + + bin + + zip + + + + ${project.build.directory}/lib + /lib + + + ${project.build.directory}/lib-native/osx-x86_64 + /lib + + + + + + ${project.build.directory}/${project.artifactId}-${project.version}.jar + /lib + + + + ${project.build.directory}/html/README.html + / + + + + ${project.build.directory}/html/license/LICENSE.html + /license + + + diff --git a/datastorages/opendal/src/main/assembly/assembly-windows-x86_64.xml b/datastorages/opendal/src/main/assembly/assembly-windows-x86_64.xml new file mode 100644 index 000000000000..5e63bb934538 --- /dev/null +++ b/datastorages/opendal/src/main/assembly/assembly-windows-x86_64.xml @@ -0,0 +1,35 @@ + + bin + + zip + + + + ${project.build.directory}/lib + /lib + + + ${project.build.directory}/lib-native/windows-x86_64 + /lib + + + + + + ${project.build.directory}/${project.artifactId}-${project.version}.jar + /lib + + + + ${project.build.directory}/html/README.html + / + + + + ${project.build.directory}/html/license/LICENSE.html + /license + + + diff --git a/datastorages/opendal/src/main/java/org/fao/geonet/api/records/attachments/OpenDALStore.java b/datastorages/opendal/src/main/java/org/fao/geonet/api/records/attachments/OpenDALStore.java new file mode 100644 index 000000000000..6b6658304a33 --- /dev/null +++ b/datastorages/opendal/src/main/java/org/fao/geonet/api/records/attachments/OpenDALStore.java @@ -0,0 +1,302 @@ +/* + * ============================================================================= + * === Copyright (C) 2001-2026 Food and Agriculture Organization of the + * === United Nations (FAO-UN), United Nations World Food Programme (WFP) + * === and United Nations Environment Programme (UNEP) + * === + * === This program is free software; you can redistribute it and/or modify + * === it under the terms of the GNU General Public License as published by + * === the Free Software Foundation; either version 2 of the License, or (at + * === your option) any later version. + * === + * === This program is distributed in the hope that it will be useful, but + * === WITHOUT ANY WARRANTY; without even the implied warranty of + * === MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * === General Public License for more details. + * === + * === You should have received a copy of the GNU General Public License + * === along with this program; if not, write to the Free Software + * === Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA + * === + * === Contact: Jeroen Ticheler - FAO - Viale delle Terme di Caracalla 2, + * === Rome - Italy. email: geonetwork@osgeo.org + * ============================================================================== + */ +package org.fao.geonet.api.records.attachments; + +import jeeves.server.context.ServiceContext; +import org.apache.opendal.Entry; +import org.apache.opendal.Metadata; +import org.apache.opendal.Operator; +import org.fao.geonet.api.exception.ResourceNotFoundException; +import org.fao.geonet.domain.MetadataResource; +import org.fao.geonet.domain.MetadataResourceContainer; +import org.fao.geonet.domain.MetadataResourceVisibility; +import org.fao.geonet.kernel.setting.SettingManager; +import org.fao.geonet.resources.OpenDALConfiguration; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.core.io.InputStreamResource; +import org.springframework.core.io.Resource; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.file.FileSystems; +import java.nio.file.Path; +import java.nio.file.PathMatcher; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + +public class OpenDALStore extends AbstractStore { + + @Autowired + private OpenDALConfiguration openDALConfiguration; + + @Autowired + private SettingManager settingManager; + + @Override + public List getResources(ServiceContext context, String metadataUuid, MetadataResourceVisibility visibility, String filter, Boolean approved) throws Exception { + final int metadataId = canEdit(context, metadataUuid, approved); + final String path = getMetadataDir(metadataId) + "/" + visibility.toString() + "/"; + + List resourceList = new ArrayList<>(); + if (filter == null) { + filter = FilesystemStore.DEFAULT_FILTER; + } + PathMatcher matcher = FileSystems.getDefault().getPathMatcher("glob:" + filter); + + Operator op = openDALConfiguration.getOperator(); + try { + List entries = op.list(path); + for (Entry entry : entries) { + String entryPath = entry.getPath(); + if (entryPath.endsWith("/")) { + continue; + } + String filename = getFilenameFromPath(entryPath); + Path fileNamePath = Paths.get(filename).getFileName(); + + if (matcher.matches(fileNamePath)) { + Metadata metadata = op.stat(entryPath); + resourceList.add(createResourceDescription(metadataUuid, visibility, filename, metadata.getContentLength(), + new Date(metadata.getLastModified().toEpochMilli()), metadataId, approved)); + } + } + } catch (Exception e) { + // If path doesn't exist, OpenDAL might throw exception depending on service + } + + resourceList.sort(MetadataResourceVisibility.sortByFileName); + return resourceList; + } + + private String getFilenameFromPath(String path) { + if (path.endsWith("/")) { + path = path.substring(0, path.length() - 1); + } + int lastSlash = path.lastIndexOf('/'); + if (lastSlash != -1) { + return path.substring(lastSlash + 1); + } + return path; + } + + private MetadataResource createResourceDescription(final String metadataUuid, + final MetadataResourceVisibility visibility, final String resourceId, long size, Date lastModification, int metadataId, boolean approved) { + return new FilesystemStoreResource(metadataUuid, metadataId, getFilename(metadataUuid, resourceId), + settingManager.getNodeURL() + "api/records/", visibility, size, lastModification, approved); + } + + @Override + public ResourceHolder getResource(ServiceContext context, String metadataUuid, MetadataResourceVisibility visibility, String resourceId, Boolean approved) throws Exception { + int metadataId = canDownload(context, metadataUuid, visibility, approved); + String path = getPath(metadataUuid, metadataId, visibility, resourceId); + + Operator op = openDALConfiguration.getOperator(); + try { + Metadata metadata = op.stat(path); + byte[] data = op.read(path); + return new OpenDALResourceHolder(data, createResourceDescription(metadataUuid, visibility, resourceId, + metadata.getContentLength(), new Date(metadata.getLastModified().toEpochMilli()), metadataId, approved)); + } catch (Exception e) { + throw new ResourceNotFoundException("Resource " + resourceId + " not found in OpenDAL store."); + } + } + + @Override + public MetadataResource getResourceMetadata(ServiceContext context, String metadataUuid, MetadataResourceVisibility visibility, String resourceId, Boolean approved) throws Exception { + int metadataId = canDownload(context, metadataUuid, visibility, approved); + String path = getPath(metadataUuid, metadataId, visibility, resourceId); + + Operator op = openDALConfiguration.getOperator(); + try { + Metadata metadata = op.stat(path); + return createResourceDescription(metadataUuid, visibility, resourceId, + metadata.getContentLength(), new Date(metadata.getLastModified().toEpochMilli()), metadataId, approved); + } catch (Exception e) { + throw new ResourceNotFoundException("Resource " + resourceId + " not found in OpenDAL store."); + } + } + + @Override + public ResourceHolder getResourceWithRange(ServiceContext context, String metadataUuid, MetadataResourceVisibility visibility, String resourceId, Boolean approved, long start, long end) throws Exception { + int metadataId = canDownload(context, metadataUuid, visibility, approved); + String path = getPath(metadataUuid, metadataId, visibility, resourceId); + + Operator op = openDALConfiguration.getOperator(); + try { + Metadata metadata = op.stat(path); + // OpenDAL read with range is possible, but the Java binding API might vary. + // Standard read(path) reads all. For range we might need to use op.read(path).range(start, end) if available. + // As of 0.46.4, op.read(path) returns byte[]. + // In OpenDAL Java, range read might be done via Operator.reader(path) but let's check if it exists. + // For now, let's do a simple implementation reading all and subsetting if needed, + // or better, if the API supports it. + byte[] data = op.read(path); // Simplification + int length = (int) (end - start + 1); + byte[] rangeData = new byte[length]; + System.arraycopy(data, (int) start, rangeData, 0, length); + + return new OpenDALResourceHolder(rangeData, createResourceDescription(metadataUuid, visibility, resourceId, + metadata.getContentLength(), new Date(metadata.getLastModified().toEpochMilli()), metadataId, approved)); + } catch (Exception e) { + throw new ResourceNotFoundException("Resource " + resourceId + " not found in OpenDAL store."); + } + } + + @Override + public MetadataResource putResource(ServiceContext context, String metadataUuid, String filename, InputStream is, Date changeDate, MetadataResourceVisibility visibility, Boolean approved) throws Exception { + int metadataId = canEdit(context, metadataUuid, visibility, approved); + String path = getPath(metadataUuid, metadataId, visibility, filename); + + Operator op = openDALConfiguration.getOperator(); + byte[] data = org.apache.commons.io.IOUtils.toByteArray(is); + op.write(path, data); + + Metadata metadata = op.stat(path); + return createResourceDescription(metadataUuid, visibility, filename, + metadata.getContentLength(), new Date(metadata.getLastModified().toEpochMilli()), metadataId, approved); + } + + @Override + public MetadataResource patchResourceStatus(ServiceContext context, String metadataUuid, String resourceId, MetadataResourceVisibility visibility, Boolean approved) throws Exception { + int metadataId = canEdit(context, metadataUuid, approved); + MetadataResource resource = getResourceMetadata(context, metadataUuid, resourceId, approved); + + if (resource.getVisibility() != visibility) { + String oldPath = getPath(metadataUuid, metadataId, resource.getVisibility(), resourceId); + String newPath = getPath(metadataUuid, metadataId, visibility, resourceId); + + Operator op = openDALConfiguration.getOperator(); + byte[] data = op.read(oldPath); + op.write(newPath, data); + op.delete(oldPath); + + Metadata metadata = op.stat(newPath); + return createResourceDescription(metadataUuid, visibility, resourceId, + metadata.getContentLength(), new Date(metadata.getLastModified().toEpochMilli()), metadataId, approved); + } + return resource; + } + + @Override + public String delResources(ServiceContext context, int metadataId) throws Exception { + String path = getMetadataDir(metadataId) + "/"; + Operator op = openDALConfiguration.getOperator(); + deleteRecursive(op, path); + return "Resources deleted"; + } + + private void deleteRecursive(Operator op, String path) throws Exception { + List entries = op.list(path); + for (Entry entry : entries) { + if (entry.getPath().endsWith("/")) { + deleteRecursive(op, entry.getPath()); + } else { + op.delete(entry.getPath()); + } + } + op.delete(path); + } + + @Override + public String delResource(ServiceContext context, String metadataUuid, String resourceId, Boolean approved) throws Exception { + int metadataId = canEdit(context, metadataUuid, approved); + try { + delResource(context, metadataUuid, MetadataResourceVisibility.PUBLIC, resourceId, approved); + } catch (Exception e) { + // ignore + } + try { + delResource(context, metadataUuid, MetadataResourceVisibility.PRIVATE, resourceId, approved); + } catch (Exception e) { + // ignore + } + return "Resource deleted"; + } + + @Override + public String delResource(ServiceContext context, String metadataUuid, MetadataResourceVisibility visibility, String resourceId, Boolean approved) throws Exception { + int metadataId = canEdit(context, metadataUuid, approved); + String path = getPath(metadataUuid, metadataId, visibility, resourceId); + openDALConfiguration.getOperator().delete(path); + return "Resource deleted"; + } + + @Override + public ResourceHolder getResourceInternal(String metadataUuid, MetadataResourceVisibility visibility, String resourceId, Boolean approved) throws Exception { + return getResource(null, metadataUuid, visibility, resourceId, approved); + } + + @Override + public MetadataResource getResourceDescription(ServiceContext context, String metadataUuid, MetadataResourceVisibility visibility, String filename, Boolean approved) throws Exception { + int metadataId = getAndCheckMetadataId(metadataUuid, approved); + String path = getPath(metadataUuid, metadataId, visibility, filename); + Operator op = openDALConfiguration.getOperator(); + Metadata metadata = op.stat(path); + return createResourceDescription(metadataUuid, visibility, filename, + metadata.getContentLength(), new Date(metadata.getLastModified().toEpochMilli()), metadataId, approved); + } + + @Override + public MetadataResourceContainer getResourceContainerDescription(ServiceContext context, String metadataUuid, Boolean approved) throws Exception { + return new FilesystemStoreResourceContainer(metadataUuid, getAndCheckMetadataId(metadataUuid, approved), + "attachments", settingManager.getNodeURL() + "api/records/", approved); + } + + protected String getPath(String metadataUuid, int metadataId, MetadataResourceVisibility visibility, String resourceId) { + return getMetadataDir(metadataId) + "/" + visibility.toString() + "/" + getFilename(metadataUuid, resourceId); + } + + protected String getMetadataDir(int metadataId) { + return "metadata/" + metadataId; + } + + public static class OpenDALResourceHolder implements ResourceHolder { + private final byte[] data; + private final MetadataResource metadata; + + public OpenDALResourceHolder(byte[] data, MetadataResource metadata) { + this.data = data; + this.metadata = metadata; + } + + @Override + public Resource getResource() { + return new InputStreamResource(new ByteArrayInputStream(data)); + } + + @Override + public MetadataResource getMetadata() { + return metadata; + } + + @Override + public void close() throws IOException { + // Nothing to do + } + } +} diff --git a/datastorages/opendal/src/main/java/org/fao/geonet/resources/OpenDALConfiguration.java b/datastorages/opendal/src/main/java/org/fao/geonet/resources/OpenDALConfiguration.java new file mode 100644 index 000000000000..e1d8c29c57d9 --- /dev/null +++ b/datastorages/opendal/src/main/java/org/fao/geonet/resources/OpenDALConfiguration.java @@ -0,0 +1,75 @@ +/* + * ============================================================================= + * === Copyright (C) 2001-2026 Food and Agriculture Organization of the + * === United Nations (FAO-UN), United Nations World Food Programme (WFP) + * === and United Nations Environment Programme (UNEP) + * === + * === This program is free software; you can redistribute it and/or modify + * === it under the terms of the GNU General Public License as published by + * === the Free Software Foundation; either version 2 of the License, or (at + * === your option) any later version. + * === + * === This program is distributed in the hope that it will be useful, but + * === WITHOUT ANY WARRANTY; without even the implied warranty of + * === MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * === General Public License for more details. + * === + * === You should have received a copy of the GNU General Public License + * === along with this program; if not, write to the Free Software + * === Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA + * === + * === Contact: Jeroen Ticheler - FAO - Viale delle Terme di Caracalla 2, + * === Rome - Italy. email: geonetwork@osgeo.org + * ============================================================================== + */ +package org.fao.geonet.resources; + +import org.apache.opendal.Operator; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.annotation.Nonnull; +import javax.annotation.PostConstruct; +import javax.annotation.PreDestroy; +import java.util.HashMap; +import java.util.Map; + +public class OpenDALConfiguration { + private static final Logger log = LoggerFactory.getLogger(OpenDALConfiguration.class); + + private Operator operator; + private String scheme = "fs"; + private Map options = new HashMap<>(); + + public void setScheme(String scheme) { + this.scheme = scheme; + } + + public void setOptions(Map options) { + if (options != null) { + options.values().removeIf(value -> value == null || value.isEmpty()); + this.options.putAll(options); + } + } + + @PostConstruct + public void init() { + log.info("Initializing OpenDAL operator with scheme: {}", scheme); + this.operator = Operator.of(scheme, options); + } + + @PreDestroy + public void destroy() { + if (operator != null) { + operator.close(); + } + } + + @Nonnull + public Operator getOperator() { + if (operator == null) { + throw new IllegalStateException("OpenDAL Operator not initialized"); + } + return operator; + } +} diff --git a/datastorages/opendal/src/main/java/org/fao/geonet/resources/OpenDALResources.java b/datastorages/opendal/src/main/java/org/fao/geonet/resources/OpenDALResources.java new file mode 100644 index 000000000000..7b4cae94fac5 --- /dev/null +++ b/datastorages/opendal/src/main/java/org/fao/geonet/resources/OpenDALResources.java @@ -0,0 +1,412 @@ +/* + * ============================================================================= + * === Copyright (C) 2001-2026 Food and Agriculture Organization of the + * === United Nations (FAO-UN), United Nations World Food Programme (WFP) + * === and United Nations Environment Programme (UNEP) + * === + * === This program is free software; you can redistribute it and/or modify + * === it under the terms of the GNU General Public License as published by + * === the Free Software Foundation; either version 2 of the License, or (at + * === your option) any later version. + * === + * === This program is distributed in the hope that it will be useful, but + * === WITHOUT ANY WARRANTY; without even the implied warranty of + * === MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * === General Public License for more details. + * === + * === You should have received a copy of the GNU General Public License + * === along with this program; if not, write to the Free Software + * === Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA + * === + * === Contact: Jeroen Ticheler - FAO - Viale delle Terme di Caracalla 2, + * === Rome - Italy. email: geonetwork@osgeo.org + * ============================================================================== + */ +package org.fao.geonet.resources; + +import jeeves.config.springutil.JeevesDelegatingFilterProxy; +import jeeves.server.context.ServiceContext; +import org.apache.commons.io.FileUtils; +import org.apache.commons.io.FilenameUtils; +import org.apache.opendal.Entry; +import org.apache.opendal.Metadata; +import org.apache.opendal.Operator; +import org.fao.geonet.domain.Pair; +import org.fao.geonet.utils.IO; +import org.fao.geonet.utils.Log; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.ApplicationContext; +import org.springframework.context.ConfigurableApplicationContext; + +import javax.annotation.Nonnull; +import javax.annotation.Nullable; +import javax.imageio.ImageIO; +import javax.servlet.ServletContext; +import java.awt.image.BufferedImage; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.file.DirectoryStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardCopyOption; +import java.nio.file.attribute.FileTime; +import java.util.HashSet; +import java.util.List; + +public class OpenDALResources extends Resources { + @Autowired + private OpenDALConfiguration opendal; + + @Override + protected Path getBasePath(final ServiceContext context) { + return Paths.get("/"); + } + + @Override + public Path locateResourcesDir(final ServletContext context, final ApplicationContext applicationContext) { + return Paths.get("/resources"); + } + + @Override + protected Path locateResourcesDir(final ServiceContext context) { + return Paths.get("/resources"); + } + + private String getKey(final Path dir, final String name) { + return getKey(dir.resolve(name)); + } + + private String getKey(final Path path) { + String pathString; + if (path.getFileSystem().getSeparator().equals("/")) { + pathString = path.toString(); + } else { + pathString = path.toString().replace(path.getFileSystem().getSeparator(), "/"); + } + + // Standardize path: remove leading / if present + if (pathString.startsWith("/")) { + pathString = pathString.substring(1); + } + return pathString; + } + + private Path getKeyPath(String key) { + if (!key.startsWith("/")) { + return Paths.get("/" + key); + } + return Paths.get(key); + } + + @Nullable + @Override + protected Path findImagePath(final String imageName, final Path logosDir) throws IOException { + final String key = getKey(logosDir, imageName); + final Operator operator = opendal.getOperator(); + if (imageName.indexOf('.') > -1) { + try { + operator.stat(key); + return getKeyPath(key); + } catch (Exception e) { + // ignore + } + } else { + // If no extension, search for files with the name and an image extension + try { + List entries = operator.list(getKey(logosDir)); + for (Entry entry : entries) { + String entryPath = entry.getPath(); + String name = getFilenameFromPath(entryPath); + if (name.startsWith(imageName + ".")) { + String ext = FilenameUtils.getExtension(name); + if (IMAGE_EXTENSIONS.contains(ext.toLowerCase())) { + return getKeyPath(entryPath); + } + } + } + } catch (Exception e) { + // ignore + } + } + + return null; + } + + @Nullable + @Override + public ResourceHolder getImage(final ServiceContext context, final String imageName, + final Path logosDir) throws IOException { + Path path = findImagePath(imageName, logosDir); + if (path != null) { + String key = getKey(path); + return new OpenDALResourceHolder(key, false); + } else { + return null; + } + } + + @Override + public ResourceHolder getWritableImage(final ServiceContext context, final String imageName, + final Path logosDir) { + return new OpenDALResourceHolder(getKey(logosDir, imageName), true); + } + + @Override + Pair loadResource(final Path resourcesDir, final ServletContext context, + final Path appPath, final String filename, final byte[] defaultValue, + final long loadSince) throws IOException { + final Path file = locateResource(resourcesDir, context, appPath, filename); + final String key = getKey(file); + final Operator operator = opendal.getOperator(); + try { + Metadata metadata = operator.stat(key); + final long lastModified = metadata.getLastModified().toEpochMilli(); + if (loadSince < 0 || lastModified > loadSince) { + byte[] content = operator.read(key); + return Pair.read(content, lastModified); + } else { + return Pair.read(defaultValue, loadSince); + } + } catch (Exception e) { + Log.error(Log.RESOURCES, "Error loading resource: " + key, e); + } + return Pair.read(defaultValue, -1L); + } + + @Override + protected Path locateResource(@Nullable final Path resourcesDir, final ServletContext context, + final Path appPath, @Nonnull String filename) throws IOException { + if (filename.charAt(0) == '/' || filename.charAt(0) == '\\') { + filename = filename.substring(1); + } + + final String key; + if (resourcesDir != null) { + key = getKey(resourcesDir, filename); + } else { + key = filename; + } + + final Operator operator = opendal.getOperator(); + boolean exists = false; + try { + operator.stat(key); + exists = true; + } catch (Exception e) { + exists = false; + } + if (!exists) { + Path webappCopy = null; + if (context != null) { + final String realPath = context.getRealPath(filename); + if (realPath != null) { + webappCopy = IO.toPath(realPath); + } + } + + if (webappCopy == null) { + webappCopy = appPath.resolve(filename); + } + if (!Files.isReadable(webappCopy)) { + if (resourcesDir != null && resourcesDir.equals(Paths.get("/resources"))) { + final ConfigurableApplicationContext applicationContext = + JeevesDelegatingFilterProxy.getApplicationContextFromServletContext(context); + webappCopy = super.locateResourcesDir(context, applicationContext).resolve(filename); + } + } + if (Files.isReadable(webappCopy)) { + try (ResourceHolder holder = new OpenDALResourceHolder(key, true)) { + Log.info(Log.RESOURCES, "Copying " + webappCopy + " to OpenDAL " + key); + Files.copy(webappCopy, holder.getPath(), StandardCopyOption.REPLACE_EXISTING); + } + } else { + final String suffix = FilenameUtils.getExtension(key); + + // find a different format and convert it to our desired format + if (IMAGE_WRITE_SUFFIXES.contains(suffix.toLowerCase())) { + final String suffixless = FilenameUtils.removeExtension(key); + String parentDir = FilenameUtils.getFullPath(key); + String baseName = FilenameUtils.getBaseName(key); + + try { + List entries = operator.list(parentDir); + for (Entry entry : entries) { + String entryPath = entry.getPath(); + String entryName = getFilenameFromPath(entryPath); + if (entryName.startsWith(baseName + ".")) { + final String ext = FilenameUtils.getExtension(entryName).toLowerCase(); + if (IMAGE_READ_SUFFIXES.contains(ext)) { + String entryKey = entryPath; + try (ResourceHolder in = new OpenDALResourceHolder(entryKey, true); + ResourceHolder out = new OpenDALResourceHolder(key, true)) { + try (InputStream inS = IO.newInputStream(in.getPath()); + OutputStream outS = Files.newOutputStream(out.getPath())) { + Log.info(Log.RESOURCES, "Converting " + entryKey + " to " + key); + BufferedImage image = ImageIO.read(inS); + ImageIO.write(image, suffix, outS); + break; + } catch (IOException e) { + if (context != null) { + context.log("Unable to convert image from " + in.getPath() + " to " + + out.getPath(), e); + } else { + Log.warning(Log.RESOURCES, "Unable to convert image from " + + in.getPath() + " to " + out.getPath(), e); + } + } + } + } + } + } + } catch (Exception e) { + // ignore + } + } + } + } + + return getKeyPath(key); + } + + @Override + protected void addFiles(final DirectoryStream.Filter iconFilter, final Path webappDir, + final HashSet result) { + String keyDir = getKey(webappDir); + if (!keyDir.endsWith("/")) { + keyDir += "/"; + } + final Operator operator = opendal.getOperator(); + try { + List entries = operator.list(keyDir); + for (Entry entry : entries) { + String fullKey = entry.getPath(); + if (fullKey.endsWith("/")) { + continue; + } + final Path curPath = getKeyPath(fullKey); + try { + if (iconFilter.accept(curPath)) { + result.add(curPath); + } + } catch (IOException e) { + throw new RuntimeException(e); + } + } + } catch (Exception e) { + Log.error(Log.RESOURCES, "Error listing files in: " + keyDir, e); + } + } + + private String getFilenameFromPath(String path) { + if (path.endsWith("/")) { + path = path.substring(0, path.length() - 1); + } + int lastSlash = path.lastIndexOf('/'); + if (lastSlash != -1) { + return path.substring(lastSlash + 1); + } + return path; + } + + @Nullable + @Override + public FileTime getLastModified(final Path resourcesDir, final ServletContext context, + final Path appPath, final String filename) throws IOException { + final Path file = locateResource(resourcesDir, context, appPath, filename); + final String key = getKey(file); + final Operator operator = opendal.getOperator(); + try { + Metadata metadata = operator.stat(key); + return FileTime.from(metadata.getLastModified()); + } catch (Exception e) { + Log.error(Log.RESOURCES, "Error getting last modified for: " + key, e); + } + return null; + } + + @Override + public void deleteImageIfExists(final String image, final Path dir) throws IOException { + Path icon = findImagePath(image, dir); + if (icon != null) { + opendal.getOperator().delete(getKey(icon)); + } + } + + private class OpenDALResourceHolder implements ResourceHolder { + private final String key; + private Path path = null; + private boolean writeOnClose = false; + + private OpenDALResourceHolder(final String key, boolean writeOnClose) { + this.key = key; + this.writeOnClose = writeOnClose; + } + + @Override + public Path getPath() { + if (path != null) { + return path; + } + final String[] splittedKey = key.split("/"); + try { + path = Files.createTempFile("opendal-res-", splittedKey[splittedKey.length - 1]); + final Operator operator = opendal.getOperator(); + try { + Metadata metadata = operator.stat(key); + byte[] data = operator.read(key); + Files.write(path, data); + } catch (Exception e) { + if (writeOnClose) { + Files.delete(path); + } + } + } catch (IOException e) { + throw new RuntimeException(e); + } + + return path; + } + + @Override + public String getRelativePath() { + return key; + } + + @Override + public FileTime getLastModifiedTime() throws IOException { + Metadata metadata = opendal.getOperator().stat(key); + return FileTime.from(metadata.getLastModified()); + } + + @Override + public void abort() { + writeOnClose = false; + } + + @Override + public void close() throws IOException { + if (path == null) { + return; + } + try { + if (writeOnClose && Files.isReadable(path)) { + byte[] data = Files.readAllBytes(path); + opendal.getOperator().write(key, data); + } + } finally { + FileUtils.deleteQuietly(path.toFile()); + path = null; + } + } + + @Override + protected void finalize() throws Throwable { + super.finalize(); + if (path != null) { + FileUtils.deleteQuietly(path.toFile()); + } + } + } +} diff --git a/datastorages/opendal/src/main/resources/config-store/config-opendal-overrides.properties b/datastorages/opendal/src/main/resources/config-store/config-opendal-overrides.properties new file mode 100644 index 000000000000..d85cdfc4c1d5 --- /dev/null +++ b/datastorages/opendal/src/main/resources/config-store/config-opendal-overrides.properties @@ -0,0 +1,11 @@ +opendal.scheme=${OPENDAL_SCHEME:#{null}} +opendal.root=${OPENDAL_ROOT:/} + +# S3 or other service options +opendal.endpoint=${OPENDAL_ENDPOINT:#{null}} +opendal.bucket=${OPENDAL_BUCKET:#{null}} +opendal.access_key_id=${OPENDAL_ACCESS_KEY_ID:#{null}} +opendal.secret_access_key=${OPENDAL_SECRET_ACCESS_KEY:#{null}} +opendal.region=${OPENDAL_REGION:#{null}} +opendal.username=${OPENDAL_USERNAME:#{null}} +opendal.password=${OPENDAL_PASSWORD:#{null}} diff --git a/datastorages/opendal/src/main/resources/config-store/config-opendal.xml b/datastorages/opendal/src/main/resources/config-store/config-opendal.xml new file mode 100644 index 000000000000..304b41842f25 --- /dev/null +++ b/datastorages/opendal/src/main/resources/config-store/config-opendal.xml @@ -0,0 +1,38 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/datastorages/pom.xml b/datastorages/pom.xml index 15de5b78a5b5..226d5c25c73f 100644 --- a/datastorages/pom.xml +++ b/datastorages/pom.xml @@ -78,6 +78,19 @@ jcloud + + + datastorage-opendal + + + + release + + + + opendal + + diff --git a/docs/manual/docs/install-guide/customizing-data-directory.md b/docs/manual/docs/install-guide/customizing-data-directory.md index 2e6867a083dc..30f51ecf4cdb 100644 --- a/docs/manual/docs/install-guide/customizing-data-directory.md +++ b/docs/manual/docs/install-guide/customizing-data-directory.md @@ -186,6 +186,72 @@ can be used to configure it (convenient in a container environment): - JCLOUD_STORAGEACCOUNTNAME - JCLOUD_STORAGEACCOUNTKEY +## Using OpenDAL storage + +If your infrastructure doesn't have a persistent storage available, you can configure GeoNetwork to use a cloud object storage to store the images and data. The OpenDAL implementation supports the multiple providers. Check the OpenDAL [documentation](https://opendal.apache.org/docs/). + +The OpenDAL datastorage can be configured using environment variables: + + +| Variable | Property | Default | Description | +|----------|----------|---------|-------------| +| `OPENDAL_SCHEME` | `opendal.scheme` | `fs` | The storage scheme (e.g., `fs`, `s3`, `azblob`, `gcs`). | +| `OPENDAL_ROOT` | `opendal.root` | `/tmp/opendal` | The root directory or path for the storage. | +| `OPENDAL_ENDPOINT` | `opendal.endpoint` | | The endpoint URL (required for S3-compatible storage). | +| `OPENDAL_BUCKET` | `opendal.bucket` | | The bucket name (required for S3/GCS/Azblob). | +| `OPENDAL_ACCESS_KEY_ID` | `opendal.access_key_id` | | Access key ID for authentication. | +| `OPENDAL_SECRET_ACCESS_KEY` | `opendal.secret_access_key` | | Secret access key for authentication. | +| `OPENDAL_REGION` | `opendal.region` | | The region for the storage service. | +| `OPENDAL_USERNAME` | `opendal.username` | | Username for authentication (e.g., WebDAV). | +| `OPENDAL_PASSWORD` | `opendal.password` | | Password for authentication (e.g., WebDAV). | + +### Examples + +#### Local Filesystem + +To configure OpenDAL to use the local filesystem: + +```bash +export OPENDAL_SCHEME=fs +export OPENDAL_ROOT=/path/to/your/storage +``` + +#### Amazon S3 + +To configure OpenDAL to use Amazon S3: + +```bash +export OPENDAL_SCHEME=s3 +export OPENDAL_ROOT=my-folder +export OPENDAL_BUCKET=my-bucket +export OPENDAL_REGION=us-east-1 +export OPENDAL_ACCESS_KEY_ID=your_access_key +export OPENDAL_SECRET_ACCESS_KEY=your_secret_key +``` + +#### S3 Compatible Storage (e.g., Minio) + +```bash +export OPENDAL_SCHEME=s3 +export OPENDAL_ENDPOINT=http://localhost:9000 +export OPENDAL_BUCKET=my-bucket +export OPENDAL_ACCESS_KEY_ID=minioadmin +export OPENDAL_SECRET_ACCESS_KEY=minioadmin +``` + +#### WebDAV + +To configure OpenDAL to use a WebDAV server: + +```bash +export OPENDAL_SCHEME=webdav +export OPENDAL_ENDPOINT=http://your-webdav-server.com/dav +export OPENDAL_ROOT=/remote.php/dav/files/user/ +export OPENDAL_USERNAME=your_username +export OPENDAL_PASSWORD=your_password +``` + + ## Structure of the data directory diff --git a/docs/manual/docs/install-guide/plugins.md b/docs/manual/docs/install-guide/plugins.md index 99112c533e3b..03bda1efa2f5 100644 --- a/docs/manual/docs/install-guide/plugins.md +++ b/docs/manual/docs/install-guide/plugins.md @@ -23,6 +23,8 @@ Support for these data storage interfaces is available through the following plu * CMIS: `gn-datastorage-cmis` * JCloud: `gn-datastorage-jcloud` [See the detailed documentation here](./customizing-data-directory.md#using-a-generic-cloud-object-storage-jcloud) +* OpenDAL: `gn-datastorage-opendal` (download the package for your specific platform) + [See the detailed documentation here](./customizing-data-directory.md#using-opendal-storage) ## Datahub integration: `gn-datahub-integration` diff --git a/pom.xml b/pom.xml index f8361ea1f3d8..c317b99d7662 100644 --- a/pom.xml +++ b/pom.xml @@ -350,6 +350,14 @@ + + + + kr.motd.maven + os-maven-plugin + 1.7.0 + + diff --git a/software_development/BUILDING.md b/software_development/BUILDING.md index ad5d6b80208d..5bca27e7bf71 100644 --- a/software_development/BUILDING.md +++ b/software_development/BUILDING.md @@ -83,6 +83,7 @@ The `release` flag above asks the following modukes to produce `zip` bundles for * `datastorage-s3` * `datastorage-jcloud` * `datastorage-cmis` +* `datastorage-opendal` * `plugin-datahub-integration` * `release` diff --git a/web/pom.xml b/web/pom.xml index 60b41a7af4e2..d5084e7ccfaf 100644 --- a/web/pom.xml +++ b/web/pom.xml @@ -1380,6 +1380,17 @@ + + datastorage-opendal + + + org.geonetwork-opensource.datastorage + gn-datastorage-opendal + ${project.version} + + + + datastorage-jcloud