DM-51822: PP-dev upload_from_repo test with LSSTCam #337
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hsinfang
added
the
ignore-for-release
hsinfang
force-pushed
the
tickets/DM-51822
branch
from
6a5a0a5 to
02ebd3a
Compare
Headers of summit LSST instruments have extra comments that are not key-value pairs and hence not JSON serializable. Ignore those. Only LSSTCam-imSim data worked before this fix. For other instruments, we only ran tests with json sidecars available, so we did not notice the issue.
In general, no json files are expected in an archive butler repo. Stop looking for it, and just generate the sidecar based on the fits. This simplifies the upload.
Raws are copied to a temp dir before uploading to the central repo because their metadata need to be modified. The raw copying was done as part of the parallel upload after the exposure. This commit moves it out to a separate function before the exposure, and provides options for it to go serially or parallelly. It introduces a new waiting, as metadata modification and uploading to the central repo only makes sense after the corresponding raw is copied to the temp_dir. This is a prep step before supporting using zipped raws where all detector images of an exposure are stored in one zip file. Before the zip format was introduced, each detector always had its own fits file.
LSSTCam raws are archived as zip files, with one zip for each exposure containing all detector images. /repo/main has LSSTCam raws in zip format, versus /repo/embargo has LSSTCam raws in individual fits format for each detector image.
/repo/embargo_old has been deprecated unintentionally. But using /repo/main works and is actually more proper.
Some input FITS files' headers fail to meet FITS standard according to astropy. These violations can cause failures in downstream fits file handling. This commit allows astropy to correct the fixable issues.
hsinfang
force-pushed
the
tickets/DM-51822
branch
from
02ebd3a to
26991b0
Compare
hsinfang
force-pushed
the
tickets/DM-51822
branch
from
5964cae to
17a0a25
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.