Updated Fcidecomp plugin (v2.0.1) and CharLS library (v2.1.0)

doc/information.rst

@@ -68,22 +68,22 @@ HDF5 filters and compression libraries
 HDF5 compression filters and compression libraries sources were obtained from:
 
 * `LZ4 plugin <https://github.com/nexusformat/HDF5-External-Filter-Plugins>`_ 
-  (commit `d48f960 <https://github.com/nexusformat/HDF5-External-Filter-Plugins/tree/d48f96064cb6e229ede4bf5e5c0e1935cf691036>`_)
+  (commit `49e3b65 <https://github.com/nexusformat/HDF5-External-Filter-Plugins/tree/49e3b65eca772bca77af13ba047d8b577673afba>`_)
   using LZ4.
 * `bitshuffle plugin <https://github.com/kiyo-masui/bitshuffle>`_ (v0.5.1) using LZ4 and ZStd.
 * bzip2 plugin (from `PyTables <https://github.com/PyTables/PyTables/>`_ v3.9.2)
   using `BZip2 <https://sourceware.org/git/bzip2.git>`_ (v1.0.8).
-* `hdf5-blosc plugin <https://github.com/Blosc/hdf5-blosc>`_ (v1.0.0)
+* `hdf5-blosc plugin <https://github.com/Blosc/hdf5-blosc>`_ (v1.0.1)
   using `c-blosc <https://github.com/Blosc/c-blosc>`_ (v1.21.5), LZ4, Snappy, ZLib and ZStd.
 * hdf5-blosc2 plugin (from `PyTables <https://github.com/PyTables/PyTables/>`_ v3.9.2)
   using `c-blosc2 <https://github.com/Blosc/c-blosc2>`_ (v2.13.2), LZ4, ZLib and ZStd.
-* `FCIDECOMP plugin <ftp://ftp.eumetsat.int/pub/OPS/out/test-data/Test-data-for-External-Users/MTG_FCI_Test-Data/FCI_Decompression_Software_V1.0.2>`_ (v1.0.2)
-  using `CharLS <https://github.com/team-charls/charls>`_
-  (1.x branch, commit `25160a4 <https://github.com/team-charls/charls/tree/25160a42fb62e71e4b0ce081f5cb3f8bb73938b5>`_).
+* `FCIDECOMP plugin <https://gitlab.eumetsat.int/open-source/data-tailor-plugins/fcidecomp>`_
+  (`v2.0.1 <https://gitlab.eumetsat.int/open-source/data-tailor-plugins/fcidecomp/-/tree/e88f83c03bafcd0769c167dca14aa7aabf728e1b>`_)
+  using `CharLS <https://github.com/team-charls/charls>`_ (v2.1.0).
 * `SZ plugin <https://github.com/szcompressor/SZ>`_
-  (commit `c25805c12b3 <https://github.com/szcompressor/SZ/commit/c25805c12b339d2cb2f406f95293b9a7313c4fb1>`_)
-* `H5Z-SPERR plugin <https://github.com/NCAR/H5Z-SPERR>`_ (v0.1.3) using `SPERR <https://github.com/NCAR/SPERR>`_ (v0.8.1).
+  (commit `f466775 <https://github.com/szcompressor/SZ/tree/f4667759ead6a902110e80ff838ccdfddbc8dcd7>`_)
   using `SZ <https://github.com/szcompressor/SZ>`_, ZLib and ZStd.
+* `H5Z-SPERR plugin <https://github.com/NCAR/H5Z-SPERR>`_ (v0.1.3) using `SPERR <https://github.com/NCAR/SPERR>`_ (v0.8.1).
 * `SZ3 plugin <https://github.com/szcompressor/SZ3>`_
   (commit `4bbe9df7e4bcb <https://github.com/szcompressor/SZ3/commit/4bbe9df7e4bcb6ae6339fcb3033100da07fe7434>`_)
   using `SZ3 <https://github.com/szcompressor/SZ3>`_ and ZStd.
@@ -95,7 +95,7 @@ HDF5 compression filters and compression libraries sources were obtained from:
 Sources of compression libraries shared accross multiple filters were obtained from:
 
 * `LZ4 v1.9.4 <https://github.com/Blosc/c-blosc2/tree/v2.13.2/internal-complibs/lz4-1.9.4>`_
-* `Snappy v1.1.10 <https://github.com/google/snappy>`_
+* `Snappy v1.2.1 <https://github.com/google/snappy>`_
 * `ZStd v1.5.5 <https://github.com/Blosc/c-blosc2/tree/v2.13.2/internal-complibs/zstd-1.5.5>`_
 * `ZLib v1.2.13 <https://github.com/Blosc/c-blosc/tree/v1.21.5/internal-complibs/zlib-1.2.13>`_
 
@@ -116,7 +116,7 @@ Please read the different licenses:
 * blosc2: See `src/PyTables/LICENSE.txt <https://github.com/silx-kit/hdf5plugin/blob/main/src/PyTables/LICENSE.txt>`_  and `src/c-blosc2/LICENSES/ <https://github.com/silx-kit/hdf5plugin/blob/main/src/c-blosc2/LICENSES/>`_
 * bzip2: See `src/PyTables/LICENSE.txt <https://github.com/silx-kit/hdf5plugin/blob/main/src/PyTables/LICENSE.txt>`_ and `src/bzip2/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/bzip2/LICENSE>`_
 * lz4: See `src/LZ4/COPYING <https://github.com/silx-kit/hdf5plugin/blob/main/src/LZ4/COPYING>`_, `src/LZ4/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/LZ4/LICENSE>`_ and `src/c-blosc/LICENSES/LZ4.txt <https://github.com/silx-kit/hdf5plugin/blob/main/src/c-blosc/LICENSES/LZ4.txt>`_
-* FCIDECOMP: See `src/fcidecomp/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/fcidecomp/LICENSE.txt>`_ and `src/charls/src/License.txt  <https://github.com/silx-kit/hdf5plugin/blob/main/src/charls/src/License.txt>`_
+* FCIDECOMP: See `src/fcidecomp/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/fcidecomp/LICENSE>`_ and `src/charls/LICENSE.md  <https://github.com/silx-kit/hdf5plugin/blob/main/src/charls/LICENSE.md>`_
 * SZ: See `src/SZ/copyright-and-BSD-license.txt <https://github.com/silx-kit/hdf5plugin/blob/main/src/SZ/copyright-and-BSD-license.txt>`_
 * SZ3: See `src/SZ3/copyright-and-BSD-license.txt <https://github.com/silx-kit/hdf5plugin/blob/main/src/SZ3/copyright-and-BSD-license.txt>`_
 * zfp: See `src/H5Z-ZFP/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/H5Z-ZFP/LICENSE>`_ and `src/zfp/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/src/zfp/LICENSE>`_

setup.py

@@ -467,6 +467,7 @@ def finalize_options(self):
             self.distribution.ext_modules = [
                 ext for ext in self.distribution.ext_modules
                 if '-std=c++14' not in ext.extra_compile_args
+                and not (isinstance(ext, HDF5PluginExtension) and ext.cpp14_required)
             ]
 
         if not self.hdf5plugin_config.use_cpp20:
@@ -605,7 +606,17 @@ def build_extensions(self):
 class HDF5PluginExtension(Extension):
     """Extension adding specific things to build a HDF5 plugin"""
 
-    def __init__(self, name, sse2=None, avx2=None, cpp11=None, cpp11_required=False, cpp20_required=False, **kwargs):
+    def __init__(
+            self,
+            name,
+            sse2=None,
+            avx2=None,
+            cpp11=None,
+            cpp11_required=False,
+            cpp14_required=False,
+            cpp20_required=False,
+            **kwargs
+    ):
         Extension.__init__(self, name, **kwargs)
 
         if not self.depends:
@@ -632,6 +643,7 @@ def __init__(self, name, sse2=None, avx2=None, cpp11=None, cpp11_required=False,
         self.avx2 = avx2 if avx2 is not None else {}
         self.cpp11 = cpp11 if cpp11 is not None else {}
         self.cpp11_required = cpp11_required
+        self.cpp14_required = cpp14_required
         self.cpp20_required = cpp20_required
 
     @property
@@ -664,8 +676,9 @@ def get_charls_clib(field=None):
 
     config = dict(
         sources=glob(f'{charls_dir}/*.cpp'),
-        include_dirs=[charls_dir],
-        cflags=['-std=c++11'],
+        include_dirs=["src/charls/include", "src/charls/include/charls", charls_dir],
+        macros=[("CHARLS_STATIC", 1), ("CHARLS_LIBRARY_BUILD", 1)],
+        cflags=['-std=c++14'],
     )
 
     if field is None:
@@ -1100,21 +1113,20 @@ def get_bzip2_plugin():
 
 def get_fcidecomp_plugin():
     """FCIDECOMP plugin build config"""
-    fcidecomp_dir = 'src/fcidecomp/FCIDECOMP_V1.0.2/Software/FCIDECOMP_SOURCES'
+    fcidecomp_dir = 'src/fcidecomp/src/fcidecomp'
 
     extra_compile_args = ['-O3', '-ffast-math', '-std=c99', '-fopenmp']
     extra_compile_args += ['/Ox', '/fp:fast', '/openmp']
-    extra_link_args = ['-fopenmp', '/openmp']
+    extra_link_args = ['-lstdc++', '-fopenmp', '/openmp']
 
     return HDF5PluginExtension(
         "hdf5plugin.plugins.libh5fcidecomp",
         sources=glob(f"{fcidecomp_dir}/fcicomp-*/src/*.c"),
         include_dirs=glob(f"{fcidecomp_dir}/fcicomp-*/include") + get_charls_clib('include_dirs'),
         extra_compile_args=extra_compile_args,
         extra_link_args=extra_link_args,
-        cpp11={'extra_link_args': ['-lstdc++']},
-        cpp11_required=True,
-        define_macros=[('CHARLS_STATIC', 1)],
+        cpp14_required=True,
+        define_macros=[('CHARLS_STATIC', 1), ('LOGGING', 1)],
     )
 
 

src/LZ4/COPYING

@@ -1,3 +1,4 @@
+
 Copyright Notice and License Terms for 
 HDF5 LZ4 compression filter plugin
 -----------------------------------------------------------------------------
@@ -56,4 +57,4 @@ Source code of the HDF5 LZ4 filter is based on h5py implementation found in
            Additional_Legal/h5py_Copyrights_and_Licenses.txt
       Original source:
            http://docs.h5py.org/en/latest/licenses.html
-License type: BSD-style license
+      License type: BSD-style license

src/LZ4/lz4_h5filter.h

@@ -17,7 +17,7 @@
 
 
 #ifndef LZ4_H5FILTER_H
-#define LZ5_H5FILTER_H
+#define LZ4_H5FILTER_H
 
 #define H5Z_class_t_vers 2
 #include "hdf5.h"

src/SZ/sz/CMakeLists.txt

@@ -92,11 +92,23 @@ set_target_properties (SZ PROPERTIES
     OUTPUT_NAME_RELWITHDEBINFO ${LIB_RELEASE_NAME}
   )
 
-target_link_libraries (SZ PUBLIC ${ZLIB_dep} ${ZSTD_dep} m)
+if (WIN32 AND NOT MINGW)
+    target_link_libraries (SZ PUBLIC ${ZLIB_dep} ${ZSTD_dep})
+else ()
+    target_link_libraries (SZ PUBLIC ${ZLIB_dep} ${ZSTD_dep} m)
+endif ()
 
-target_compile_options(SZ
-	PRIVATE $<$<CONFIG:Debug>:-Wall -Wextra -Wpedantic -Wno-unused-parameter>
-	)
+if (WIN32 AND NOT MINGW)
+    message(STATUS "Native windows compiler detected, adding compile flags")
+    target_compile_options(SZ
+        PRIVATE $<$<CONFIG:Debug>:-Wall>
+    )
+else ()
+    message(STATUS "GCC detected, adding compile flags")
+    target_compile_options(SZ
+        PRIVATE $<$<CONFIG:Debug>:-Wall -Wextra -Wpedantic -Wno-unused-parameter>
+    )
+endif()
 
 if(BUILD_OPENMP)
   target_link_libraries(SZ PRIVATE OpenMP::OpenMP_C)

src/SZ/sz/src/sz.c

@@ -485,7 +485,16 @@ unsigned char *SZ_compress_rev(int dataType, void *data, void *reservedValue, si
 
 void *SZ_decompress(int dataType, unsigned char *bytes, size_t byteLength, size_t r5, size_t r4, size_t r3, size_t r2, size_t r1)
 {
-	if(confparams_dec==NULL)
+    //correct dimension if needed
+    size_t _r[5];
+    filterDimension(r5, r4, r3, r2, r1, _r);
+    r5 = _r[4];
+    r4 = _r[3];
+    r3 = _r[2];
+    r2 = _r[1];
+    r1 = _r[0];
+
+    if(confparams_dec==NULL)
 		confparams_dec = (sz_params*)malloc(sizeof(sz_params));
 	memset(confparams_dec, 0, sizeof(sz_params));
 	if(exe_params==NULL)

src/SZ/zlib/CMakeLists.txt

@@ -4,6 +4,10 @@ else ()
   set (BUILD_EXT_LIBS_TYPE "STATIC")
 endif ()
 
+if (WIN32 AND NOT MINGW)
+  set (BUILD_EXT_LIBS_TYPE "STATIC")
+endif ()
+
 add_library(ZLIB ${BUILD_EXT_LIBS_TYPE} 
   ./gzclose.c
   ./uncompr.c

src/SZ/zstd/CMakeLists.txt

@@ -4,6 +4,10 @@ else ()
   set (BUILD_EXT_LIBS_TYPE "STATIC")
 endif ()
 
+if (WIN32 AND NOT MINGW)
+  set (BUILD_EXT_LIBS_TYPE "STATIC")
+endif ()
+
 add_library(zstd ${BUILD_EXT_LIBS_TYPE} 
   ./common/entropy_common.c
   ./common/pool.c

src/charls/.clang-format

@@ -0,0 +1,41 @@
+# Copyright (c) Team CharLS.
+# SPDX-License-Identifier: BSD-3-Clause
+
+BasedOnStyle: LLVM
+Language: Cpp
+
+# Use features of C++11, C++14 and C++17 (e.g. A<A<int>> instead of A<A<int> >)
+Standard: Cpp11
+
+UseTab: Never
+IndentWidth: 4
+ColumnLimit: 0
+PointerAlignment: Left
+
+BreakBeforeBraces: Custom
+BraceWrapping:
+  AfterClass: true
+  AfterControlStatement: true
+  AfterEnum: true
+  AfterFunction: true
+  AfterNamespace: false
+  AfterObjCDeclaration: true
+  AfterStruct: true
+  AfterUnion: true
+  BeforeCatch: true
+  BeforeElse: true
+  IndentBraces: false
+
+AccessModifierOffset: -4
+AlignTrailingComments: true
+AllowShortFunctionsOnASingleLine: true
+AllowShortIfStatementsOnASingleLine: true
+AlwaysBreakTemplateDeclarations: true
+BreakConstructorInitializers: AfterColon
+CompactNamespaces: true
+ConstructorInitializerAllOnOneLineOrOnePerLine: true
+SpaceAfterTemplateKeyword: false
+SpaceBeforeCpp11BracedList: false
+KeepEmptyLinesAtTheStartOfBlocks: false
+MaxEmptyLinesToKeep: 2
+NamespaceIndentation: None

src/charls/.editorconfig

@@ -1,8 +1,57 @@
+# Copyright (c) Team CharLS.
+# SPDX-License-Identifier: BSD-3-Clause
+
 root = true
 
-[{src}/**.{h,cpp}]
+# Default settings
+[*]
 charset = utf-8
+indent_style = space
 trim_trailing_whitespace = true
+
+[*.{h,cpp}]
+indent_size = 4
 insert_final_newline = true
-indent_style = space
-indent_size = 4
+
+[CMakeLists.txt]
+indent_size = 2
+
+[*.cs]
+indent_size = 4
+
+# New line preferences
+csharp_new_line_before_catch = true
+csharp_new_line_before_else = true
+csharp_new_line_before_open_brace = control_blocks, types, methods, object_collection, accessors, properties
+
+# Indentation preferences
+csharp_indent_case_contents = true
+csharp_indent_switch_labels = true
+
+# Space preferences
+csharp_space_after_cast = false
+csharp_space_after_colon_in_inheritance_clause = true
+csharp_space_after_keywords_in_control_flow_statements = true
+csharp_space_before_colon_in_inheritance_clause = true
+csharp_space_between_method_call_empty_parameter_list_parentheses = false
+csharp_space_between_method_call_name_and_opening_parenthesis = false
+csharp_space_between_method_call_parameter_list_parentheses = false
+csharp_space_between_method_declaration_empty_parameter_list_parentheses = false
+csharp_space_between_method_declaration_parameter_list_parentheses = false
+
+csharp_preserve_single_line_blocks = true
+
+csharp_style_expression_bodied_constructors = false:suggestion
+csharp_style_expression_bodied_methods = false:suggestion
+csharp_style_expression_bodied_properties = true:suggestion
+csharp_style_inlined_variable_declaration = false:suggestion
+csharp_style_var_for_built_in_types = true:suggestion
+csharp_style_var_when_type_is_apparent = true:suggestion
+
+dotnet_sort_system_directives_first = true
+
+dotnet_style_predefined_type_for_member_access = true:suggestion
+dotnet_style_predefined_type_for_locals_parameters_members = true:suggestion
+dotnet_style_qualification_for_field = false:suggestion
+dotnet_style_qualification_for_method = false:suggestion
+dotnet_style_qualification_for_property = false:suggestion

src/charls/.gitattributes

@@ -5,6 +5,7 @@
 *.c text diff=cpp
 *.cpp text diff=cpp
 *.h text diff=cpp
+*.cs text diff=csharp
 
 # Explicit exclude binary files from the line ending normalization.
 *.png binary

src/charls/.gitignore

@@ -6,17 +6,29 @@ Win32/
 x64/
 .vscode/
 .vs/
+build/
 [Dd]ebug/
 [Rr]elease/
 [Cc]hecked/
-build/
+TestResults/
+*.opensdf
+*.sdf
 
 # Ignore files created by Visual Studio
 *.suo
 *.user
 *.ipch
-*.opensdf
-*.sdf
 *.opendb
-*.VC.db
-*.aps
+*.db
+*.aps
+*.sqlite
+
+# NuGet Packages
+*.nupkg
+# The packages folder can be ignored because of Package Restore
+**/packages/*
+# except build/, which is used as an MSBuild target.
+!**/packages/build/
+
+# Ignore Coverity build folder
+cov-int/

src/charls/.travis.yml

@@ -1,28 +1,22 @@
+dist: bionic
+
 language: cpp
 
 compiler:
   - gcc
   - clang
 
-install:
-- if [ "$CXX" = "g++" ]; then export CXX="g++-4.8" CC="gcc-4.8"; fi
-- if [ "$CXX" = "clang++" ]; then export CXX="clang++-3.7" CC="clang-3.7"; fi
-
 addons:
   apt:
     sources:
     - ubuntu-toolchain-r-test
     - llvm-toolchain-precise-3.7
-    packages:
-    - gcc-4.8
-    - g++-4.8
-    - clang-3.7
 
 sudo: false
 
 before_script:
   - mkdir debug
   - cd debug
-  - cmake -DCMAKE_BUILD_TYPE=Debug ..
+  - cmake -DCMAKE_BUILD_TYPE=Debug -DCHARLS_PEDANTIC_WARNINGS=On -DCHARLS_THREAT_WARNINGS_AS_ERRORS=On ..
 
-script: make
+script: make