Bug: loadRun is not working for 2025-1 cycle scans due to beamline codebase refactoring.

[PriyankaKetkarBNL]

Tried to load data for RSoXS energy scan:

scanID = 92202
scan = Loader.loadRun(scanID).unstack('system')

Got AttributeError: 'NoneType' object has no attribute 'round' pointing to md["sam_x"]: https://github.com/usnistgov/PyHyperScattering/blob/main/src/PyHyperScattering/SST1RSoXSDB.py#L1149

The error is likely the result of "RSoXS Sample Outboard-Inboard" being renamed as "manipulator_x". Will try to add new key to md_secondary_lookup.

[PriyankaKetkarBNL]

Longer-term fix (try/except scripts to deal with differently-named or missing metadata) pending. Short-term bypass is reflected in this commit: 7732c9d

The primary md_lookup dictionary was updated to new metadata key names and manually added md["exposure"] value because shutter control was not saved in the baseline.

[PriyankaKetkarBNL]

Major updates made in this branch:

• mdLookup dictionary was updated so that beamline metaddata key names can be entered as a list instead of single value (or two values in the case of secondary_lookup table). Sets up infrastructure for backwards compatibility with respect to key names used historically at the beamline.
• Propagated mdLookup dictionary structure to the code where it is used such that the code can search through all historical metadata key names.
• Added input in loadRun function so that metadata values can be entered manually in case they were not originally written out with the scan.

[pbeaucage]

The last function seems to duplicate the preexisting coords mechanism which is supported in every other loader - how is it different?

[PriyankaKetkarBNL]

@pbeaucage My understanding is that the current coords input cannot take more qualitative single-value entries (e.g., {"sample_notes": "details"}) that are meant to be stored in attrs. If it makes sense, I could think of how to consolidate these two inputs.

[pbeaucage]

So this is correct - coords is intended to take user-provided coordinates not present in metadata and provide an output xarray that is a function of those coordinates.

I'm trying to understand - is your new field simply shorthand for something like:

out = loader.loadFileSeries(whatever)
out.attrs['vibes_during_data_collection'] = 'chill'
figure_2 = do_further_processing(out)

?

And if so, why not just have the attrs updated like that? It seems more straightforward to me.

[PriyankaKetkarBNL]

Makes sense for your example. Below is an example where I think I need to add metadata during loadRun, otherwise I run into an error. Let me know if there is a different/existing workaround for it.

Due to an error on my end while refactoring the beamline codebase, there was a brief period of time when the exposure time was not written out in the baseline. This impacted some scans for one of my users and caused the error below when trying to load their scan.

scan = loader.loadRun(run=92202).unstack('system')

C:\Users\RSoXS User\.conda\envs\20250308_1\Lib\site-packages\PyHyperScattering\SST1RSoXSDB.py:606: UserWarning: Could not find ['RSoXS Shutter Opening Time (ms)'] in either baseline or primary. Setting exposure to None.  Can be entered manually.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[173], line 2
      1 scanID = 92202
----> 2 scan = loader.loadRun(run=scanID).unstack('system')
      3 scan

File ~\.conda\envs\20250308_1\Lib\site-packages\PyHyperScattering\SST1RSoXSDB.py:770, in SST1RSoXSDB.loadRun(self, run, dims, coords, return_dataset, useMonitorShutterThinning, mdManual)
    768 if len(index) != len(data["time"]):
    769     index = index[: len(data["time"])]
--> 770 actual_exposure = md["exposure"] * len(data.dim_0)
    771 mindex_coords = xr.Coordinates.from_pandas_multiindex(index, 'system')
    772 retxr = (
    773     data.sum("dim_0")
    774     .rename({"dim_1": "pix_y", "dim_2": "pix_x"})
    775     .rename({"time": "system"})
    776     .assign_coords(mindex_coords)
    777 )  # ,md['detector']+'_image':'intensity'})

TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

It was resolved after manually adding the exposure time.

scan = loader.loadRun(run=92202, mdManual = {"exposure": 1.0}).unstack('system')

[pbeaucage]

Ah, I get it. That's really unfortunate, a predictable consequence of making serious code changes without testing during production user time...

An alternative way to resolve this, though, would be to just make a default value for exposure. I don't like the idea that users can just input any parameter; it feels like overriding the 'scientific correctness' of the instrument in some way. If a user really wanted to do that "against professional advice", I believe they could still do it with something like:

run = load.c[98765]
run.attrs['exposure'] = 1.0 # the name and location where exposure should be may be different; you might also need to .read() the run or copy.deepcopy or something like that.
load.loadRun(run)

and we wouldn't be encouraging this kind of bad practice as a standard policy.
Independently, we should patch around the exposure issue there. I will put a change on the PR branch that patches around that line.

[PriyankaKetkarBNL]

I like this idea. I haven't looked extensively at how exposure is used down the line, but could a suitable alternative be to have some if statement that determines whether or not the downstream operations are performed? Along with a descriptive warning telling users what parameters are not usable instead of an error that prevents going further?

if md["exposure] is not None: ## [do the downstream operations/calculations]
else: ## [set downstream parameters to none]

Either way, I think it is fine to remove the md_manual input from loadRun.

[pbeaucage]

Sounds great. What I added was essentially that.

The short answer is that exposure is not presently being used. It could/should be used to normalize intensity, but that never was really done. The way this would happen is the corr_mode flag, which has a value for 'exposure', but was never implemented.