first draft of splitting NWSS signals

[dsweber2]

Description

This splits the dataset based on the provider and normalization (not every pair is actually present), and adds the metric signals. The resulting signals are called:

• pcr_conc_smoothed_CDC_VERILY_flow-population
• detect_prop_15d_CDC_VERILY_flow-population
• percentile_CDC_VERILY_flow-population
• ptc_15d_CDC_VERILY_flow-population
• pcr_conc_smoothed_CDC_VERILY_microbial
• detect_prop_15d_CDC_VERILY_microbial
• percentile_CDC_VERILY_microbial
• ptc_15d_CDC_VERILY_microbial
• pcr_conc_smoothed_NWSS_flow-population
• detect_prop_15d_NWSS_flow-population
• percentile_NWSS_flow-population
• ptc_15d_NWSS_flow-population
• pcr_conc_smoothed_NWSS_microbial
• detect_prop_15d_NWSS_microbial
• percentile_NWSS_microbial
• ptc_15d_NWSS_microbial
• pcr_conc_smoothed_WWS_microbial
• detect_prop_15d_WWS_microbial
• percentile_WWS_microbial
• ptc_15d_WWS_microbial

Some of these can have negative values; for e.g. ptc_15d, the values are small enough that I expect these may actually be exponents. Still looking into why the concentration data has negative values, which are too large to make sense as exponents.

Fixes

• Fix generate_weights in the case where some weights are negative, and added a test for it

[dsweber2]

Minh confirmed that this runs on her machine and that the output looks reasonable. A couple of things to do before we merge this:

• figure out the right cut-off for sig digits so that the noise is removed
• seeing what values the formerly negative concentrations became
• write docs similar to this PR

[dsweber2]

Currently not passing because of the same update that made nssp tests fail. Once that's merged and this is rebased it will pass.

[nmdefries]

Some comments about style and comments. The actual functionality here looks fine.

This hasn't been released at all yet, right? You previously did statistical review on this; is there anything else we want to do for these new signals?

[nmdefries]

suggestion (optional): to make this more robust, add assert to make sure that our set of missing column names doesn't include important ones (like geo_id and value). Since this is out of scope, worth making an issue for

[dsweber2]

This hasn't been released at all yet, right? You previously did statistical review on this; is there anything else we want to do for these new signals?

It has not. There's the corresponding docs, which I think Will read through at one point.

[dsweber2]

closing as this is migrating to a new endpoint