PNG support

[Flydroid]

Hi,

I have been evaluating different options to store images in an HDF5 file.
Because build-in methods come with a large file size, we are storing png files as binary data (u8 array) in HDF5.
What would be really cool is, if h5web would support the decoding of the PNG data to then display the image.

Unfortunately i have don't have typescript experience to really contribute directly.

[axelboc]

Hi @Flydroid, thanks for reaching out!

Until now, we've always considered this to be out of scope of H5Web. It has always seemed like something that should be either done in an "HDF5 way" (RGB image, filters, etc.) or supported natively by HDF5. Adding a PNG/JPEG/... decoding library into H5Web has never seemed worth the cost (maintenance/performance).

However, after some new discussions internally, we think there might be a way for H5Web to support a few basic image formats like PNG and JPEG very cheaply: by simply letting the browser decode the binary data for us! If we detect the PNG/JPEG/... magic numbers at the start of a binary blob, we can pass it straight to an HTML img tag and let the browser deal with it.

We could do that inside the Raw visualization, which shows up when no other visualization is supported. The problem I see is that a uint8 dataset would typically be supported by other visualizations (at least Matrix, Line and Heatmap), so the Raw visualization would not appear. I'm not very keen on changing this behaviour, as the Raw visualization is typically useless and meant as a last resort.

Instead, we think a better solution would be to make use of the H5T_OPAQUE type, which is not supported by any other visualization than Raw and, in fact, seems to be meant for this exact purpose of storing binary data that cannot be otherwise described in HDF5:

• https://davis.lbl.gov/Manuals/HDF5-1.6.1/Datatypes.html
• https://docs.h5py.org/en/stable/special.html#storing-other-types-as-opaque-data

Would you be open to storing your PNGs as opaque datasets? FYI, with h5py, the simplest way to create an opaque dataset is to make use of numpy.void().

[Flydroid]

Hi @axelboc,

Thank you for your openness and elaborate answer!
I had hoped that the display of images would be that simple.

Using the H5T_OPAQUE data type sounds like a good idea and its easy to implement on our side.
They that you describe it would assume that a dataset with this type would be checked for it is a picture.
That would mean one dataset equals to one picture?

I had only imagined to possibly save an array of pictures in a 2d byte array all in one dataset.
But one dataset per picture with many datasets in a group should also work.

[axelboc]

I don't see any major difficulty in supporting nD opaque arrays as long as the visualization need we're talking about is to show a single image at a time, along with sliders on the left to navigate through the dimensions. If, however, we're talking about showing a grid of images, then it's a bit more challenging... and perhaps a bit too niche for a generic viewer.

Could you please expand a bit on your use case for storing PNG images in HDF5? What do the images show, what are they for, how big are they, ...?

[Flydroid]

A slider on the side to scroll through the images would be perfect.

We are acquiring burst of images from high speed cameras for short durations ~1-30ms, so maybe 100 images with 1-3 MB in size.
The images get stored in the HDF5 file together with other measurement data.
We would use h5web probably together with an HSDS backend to preview the experiment data before analysis.

[axelboc]

Closing in favour of issue #1554