using epidata metadata to determine stalenes for nhsn pulling

[aysim319]

Description

using metadata api to determine staleness

Associated Issue(s)

• Addresses nhsn skips run if data updated after scheduled run #2137

[minhkhul]

LGTM. One nit was the utcfromtimestamp thing but everything else looks good.

[melange396]

in theory, i really like the idea of using our own metadata, but in practice, its going to have problems that make it unworkable. The biggest is that there is an indeterminate delay between data getting inserted into the database and when the metadata is updated to include that data (the delay is bounded, but long enough to throw a wrench in this). Another is that patches will affect the metadata, thereby disrupting scheduling.

[melange396]

this is gonna have issues because of DST changes, however this timestamp should be for UTC already.

it shouldnt make too much of a difference because the probability of it biting us should be rare, but i think youll also want a max instead of a min (in case we change signal names or discontinue signals, among other things).

Suggested change

	est = timezone(timedelta(hours=-5))
	last_updated = datetime.fromtimestamp(nhsn_meta_df["last_update"].min(), tz=est)
	last_updated = datetime.fromtimestamp(nhsn_meta_df["last_update"].max(), tz=timezone.utc)

[melange396]

i dont think this math is quite right... why wouldnt we want to proceed any time socrata has a newer timestamp than we do? the form you have here has the potential to delay processing if updates are frequent enough or if RECENTLY_UPDATED_DIFF is too large.

Suggested change

	recently_updated_source = (updated_timestamp - last_updated) > RECENTLY_UPDATED_DIFF
	socrata_ts = updated_timestamp
	delphi_ts = last_updated
	recently_updated_source = socrata_ts > delphi_ts