ENH: New webpoda data gap checker

[tech3371]

Change Summary

Related to #1617

Overview

This checks data gaps by comparing what exists in prod and what exists in WebPODA. This will be turned into Dagster asset in future PR. This is tested locally against prod and WebPODA.

File changes

• queries latest L0 file in production. downloads latest L0 data from WebPODA. compares to see if data has changed. If yes, increment minor version by 1 and upload to production. If no change, skip it. If no file exists in prod yet and data exists, write to new file with default minor version and upload.

Testing

[Copilot]

🟡 Changes recommended

Repoint lookup, upload-failure cleanup, local-file deletion, and outdated tests introduce data-loss and execution risks.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds WebPODA gap detection and versioned L0 generation as groundwork for a future Dagster asset.

Changes:

• Compares WebPODA packets with production L0 files.
• Supports daily and repoint-based data grouping.
• Extends the CLI to select the appropriate workflow.

File summaries

File	Description
imap_data_access/webpoda.py	Adds repoint lookup, comparison, versioning, and upload logic.
imap_data_access/cli.py	Routes instruments through daily or repoint-based downloads.

Review details

Suppressed comments (4)

imap_data_access/webpoda.py:395

• The renamed parameter is not reflected in tests/test_webpoda.py:177 and :195, which still call this function with repointing_file=...; both tests will now fail with TypeError before exercising the implementation. Update those tests to parse the fixture and pass repoint_data, along with coverage for the new comparison paths.

    repoint_data: list,

imap_data_access/webpoda.py:492

• As in the daily path, this deletes the generated file even when _upload_if_requested has caught an upload failure. Keep the local file unless upload success is explicitly communicated, so the recovered packets remain retryable.

        for path in file_paths:
            _upload_if_requested(path, upload_to_server)
            # clean up files after upload to avoid filling up disk space
            path.unlink()

imap_data_access/webpoda.py:790

• imap_data_access.download can return an already-existing local file without downloading it, so this may delete a production L0 file that the user had before running the checker. Only remove files whose temporary ownership is known; the safe behavior here is to remove the newly generated duplicate and leave the production path intact.

        # Clean up the new file since it is a duplicate of the production file
        new_path.unlink()
        prod_l0_path.unlink()

imap_data_access/webpoda.py:452

• The documented return type is a list, and all other no-result paths return an empty list. Returning None here forces callers to handle an undocumented second container type; return the initialized empty list instead.

    if not repoints:
        logger.info(
            f"No repoint IDs found for instrument [{instrument}] between "
            f"{start_time} and {end_time}. Skipping."
        )
        logger.info("-" * 80)
        return None

• Files reviewed: 3/3 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[tmplummer]

I have an overall design question. These changes completely remove the use of ERT in all queries. What is the plan with the Dagster jobs? How will they be triggered and how will they be configured? i.e. what date range will be queried?
I don't have data to back it up, but IMO, the ENA L0 download has been working well and has some advantages. We have two approaches to download data:

Precision Approach:

• Trigger off of some product that we get from each contact. For ENAs, this has been repoint table.
• Query for packets with ERT in time range since last precision query.
• From ERT query, build list of dates/repoints to re-query over based on SCT.

Pros:

• We only download those dates/repoints that we got new data for.
• Triggered off of data downlink. This means that we will download data as soon as it is available.

Cons:

• Relies on POC ingest being completed before our queries run. This is non guaranteed when triggering on repoint table.

Brute Force Approach:

• Run based on cron type schedule
• Query packets with SCT that fall in last N days
• Check downloaded data against existing data and upload files that have changed.

Pros:

• If we miss data or POC is in process of ingesting data during one query, next time the job is run, we will catch that data.

Cons:

• We end up downloading lots of extra data. I don't know if this is an issue for the POC.
• Data is not retrieved as soon as it is available.
• Potential to retrieve data while POC is in the middle of an ingest which means extra reprocessing jobs because data from a single downlink will get split across multiple download jobs.

I am hesitant to abandon the Precision Approach all together because it seems like a more elegant approach to me. I do see value in the Brute Force Approach as a backup that gets run every so often. As I was writing the pros/cons, I was wondering if we could have the POC either deliver an additional "product" to the SDC or hit an API with detailed info about what they just ingested. For example, a JSON file with a list of all unique APIDs that were ingested and the ERT range they belong to as well as the unique SCT dates.

[tech3371]

These were the assumptions I was working with for this work

• take out ERT support and lambda support. Add support in Dagster asset instead.
• query for S/C time for user input date range. date range is the data query date range. Eg. date range of last week to today will query all data within those days and checks againts it’s L0 data.
• If L0 doesn’t exists, create new L0 file.
• If L0 exists, download latest L0 in prod and compare against local file. If data has changed, increment minor version +1 and upload to prod if upload_to_server flag is on.
• Dagster sensor will check the last 7 days once a day. We also have a sensor check a month of data, but maybe only run that one once a week

[tech3371]

Path forward from the meeting:

Keep what's in imap-data-access as it is. Add these supports:

• support to query by S/C for repoint instruments.
• If L0 doesn’t exists, create new L0 file. Make sure we don't create empty files.
• If L0 exists, download latest L0 in prod and compare against local file. If data has changed, increment minor version +1 and upload to prod if upload_to_server flag is on.

Then in dagster, create sensor to check historical data gaps:

• query for S/C time for input date range. date range is the data query date range. Eg. date range of last week to today will query all data within those days and checks againts existing L0 data.
• Dagster sensor check a month of data, but maybe only run that one once a week. Keep current webpoda lambda and it will run as it is.

[Copilot]

🟡 Changes recommended

Version-zero files can bypass comparison, and several date-range behaviors can select stale or out-of-range data.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 5
• Review effort level: Balanced

[tech3371]

I made update based on outcome from the meeting. Then tested the new refactor changes with same hi command and it output same thing. sct support is working now for ENA.

imap-data-access webpoda --start-date 20260801 --end-date 20260910 --instrument hi --query-mode sct

Output:

Data has not changed for imap_hi_l0_raw_20260731-repoint00326_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260801-repoint00327_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260802-repoint00328_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260803-repoint00329_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260804-repoint00330_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260805-repoint00331_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260806-repoint00332_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260807-repoint00333_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260808-repoint00334_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260809-repoint00335_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260810-repoint00336_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260811-repoint00337_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260812-repoint00338_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260813-repoint00339_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_hi_l0_raw_20260814-repoint00340_v001.0001.pkts.
--------------------------------------------------------------------------------

For in-situ, I double checked that it's still working.

imap-data-access webpoda --start-date 20260801 --end-date 20260910 --instrument swapi --query-mode sct

Output:

Data has not changed for imap_swapi_l0_raw_20260801_v001.0004.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260802_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260803_v001.0005.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260804_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260805_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260806_v001.0005.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260807_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260808_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260809_v001.0004.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260810_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260811_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260812_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260813_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260814_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260815_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260816_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260817_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260818_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260819_v001.0005.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260820_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260821_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260822_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260823_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260824_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260825_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260826_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260827_v001.0001.pkts.
--------------------------------------------------------------------------------

imap-data-access webpoda --start-date 20260616 --end-date 20260619 --instrument swapi --query-mode sct

Output:

Data has not changed for imap_swapi_l0_raw_20260616_v001.0002.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260617_v001.0003.pkts.
--------------------------------------------------------------------------------
Data has not changed for imap_swapi_l0_raw_20260618_v001.0001.pkts.
--------------------------------------------------------------------------------
Data has changed
Prod imap_swapi_l0_raw_20260619_v001.0004.pkts: (size: 4.8006 MB)
New     imap_swapi_l0_raw_20260619_v001.0005.pkts: (size: 5.0839 MB)
--------------------------------------------------------------------------------

[tmplummer]

Mostly looks good. There are many print statements that I think should be logger calls. I'm also not sure if checking file size should be strictly less than check. New files being smaller shouldn't happen, but maybe is happening due to POC cleaning up duplicate packets?

[tmplummer]

LGTM, thanks for seeing this through.