Skip to content

Return reprocess ID and tag reprocessing runs with it - #1677

Merged
leowerneck merged 3 commits into
IMAP-Science-Operations-Center:devfrom
vineetbansal:vb/issue1676
Oct 8, 2026
Merged

leowerneck merged 3 commits into
IMAP-Science-Operations-Center:devfrom
vineetbansal:vb/issue1676

Conversation

@vineetbansal

Copy link
Copy Markdown
Collaborator

Partially addresses #1676.

The reprocessing proxy now returns the SQS MessageId as reprocess_id. The reprocess sensor tags each run it creates with imap/reprocess_id, so all runs from one request can be found in Dagster.

This doesn't close the issue completely - a corresponding change in imap-data-access (independent of this) is desired as well, that displays the reprocess ID on screen.

Things should continue to work fine even in the absence of the change in imap-data-access, so that can come later.

@vineetbansal

Copy link
Copy Markdown
Collaborator Author

PR #360 in imap-data-access addresses the other half. There's no requirement that these PRs be merged in a specific order, since each is tolerant of missing keys etc.

@leowerneck leowerneck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

I think this is a very valuable addition for reprocessing workflows.

@leowerneck

Copy link
Copy Markdown
Contributor

I'll add the question here as well: do we want to support the custom tag syntax imap-data-access reprocess .. -tag TAG discussed in #1676?

Comment thread sds_data_manager/orchestration/reprocessing.py

@bryan-harter bryan-harter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, other than my minor comment!

@jaredclaypoole jaredclaypoole left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great!

@leowerneck
leowerneck merged commit 94de5dd into IMAP-Science-Operations-Center:dev Oct 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants