replace_fillval_by_nan could not handle int type data

[Beforerr]

Data like the_efw_l1.the_efw is of integer type., so res[res == fill_value] = np.nan raises error.

Since the data is not in speasy cdaweb, I attach a yaml file

the_efw_l1:
  inventory_path: spdf/THEMIS/THE/L1/EFW
  master_cdf: https://spdf.gsfc.nasa.gov/pub/data/themis/the/l1/efw/2021/the_l1_efw_20210102_v01.cdf
  split_frequency: daily
  split_rule: regular
  url_pattern: https://spdf.gsfc.nasa.gov/pub/data/themis/the/l1/efw/{Y}/the_l1_efw_{Y}{M:02d}{D:02d}_v\d+.cdf
  use_file_list: true

[Beforerr]

I tried to fix but somehow numpy does not support NaN Integer, maybe we could log a warning instead?

[jeandet]

@Beforerr, I'm not sure a warning is what we want here since replace_fillval_by_nan is really meant to be used only on floating values. The NaN concept only exists for floating values.
What you can do instead, is always cast your variable to floating type before using replace_fillval_by_nan like this:

import speasy as spz
import numpy as np

np.astype(spz.get_data( spz.inventories.data_tree.cda.Cluster.C1.FGM_SPIN.C1_CP_FGM_SPIN.B_vec_xyz_gse__C1_CP_FGM_SPIN , "2012-03-02", "2012-03-03"),np.float32).replace_fillval_by_nan()

[Beforerr]

@jeandet Thank you for your suggestion! You’re right, explicitly casting the data to a floating type before applying replace_fillval_by_nan() makes sense.

Ideally, I think the user just want to replace the fill value with NaN to clean the data. So a united interface would be helpful. Maybe we could put something like this to sanitized_with_nan (like santize but keep nan)

[jgieseler]

There is a similar issue with discussion for sunpy: sunpy/sunpy#5908

Maybe this helps in finding a solution.

[jeandet]

I have two solutions in mind:

• add a boolean parameter to allow automatic float conversion (False by default)
• add a strategy kw argument that could take 3 values:
  • "default" same as now
  • "ignore" just ignore non float values
  • "cast" cast non float values to float

@Beforerr, @jgieseler, @brenard-irap, @nicolasaunai tell me if you have any preference.

[Beforerr]

Hi, @jeandet I would suggest a "automatic" option that "cast" if nan appears in the data otherwise "default" (and possibly make that default so no error is given).

[jgieseler]

It's complicated. On the one hand, you don't want to mess with the original data more than necessary. On the other hand, I think most users don't care that much about data types, but rather expect that replace_fillval_by_nan does exactly that. So I would favor adding a boolean parameter, so that we have something like .replace_fillval_by_nan(convert_to_float=True).