refactor(load): unify legacy and opossum file loading paths#3167
Open
abraemer wants to merge 14 commits into
Open
refactor(load): unify legacy and opossum file loading paths#3167abraemer wants to merge 14 commits into
abraemer wants to merge 14 commits into
Conversation
After the fflate-to-adm-zip migration (PR #3164), the loading code still branched on file format throughout the stack. This commit eliminates that dual-path complexity by ensuring every load produces the same uniform result — a LoadedArchive containing an AdmZip instance, parsed input, and parsed output — regardless of whether the source was a .opossum file or a legacy JSON input. Key changes: - parseFile.ts: replace parseOpossumFile / parseInputJsonFile / parseOutputJsonFile with two format-specific loaders that return a uniform LoadedArchive type: • loadOpossumFile(path) — opens AdmZip, extracts input.json/output.json • loadLegacyFile(path) — reads raw JSON, parses it, reads sidecar _attributions.json if present, then constructs a new in-memory AdmZip with input.json packed in - loadFile.ts: remove all isOpossumFileFormat branching; now takes a LoadedArchive + opossumFilePath (format-agnostic) - dbProcess.ts: split LoadFileMessage into LoadOpossumFileMessage and LoadLegacyFileMessage, route to correct loader, pass uniform result to loadFile - dbProcessClient.ts: split loadFile() into loadOpossumFile() and loadLegacyFile() - importFromFile.ts: split into loadOpossumFileFromPath() and loadLegacyFileFromPath() - listeners.ts: importFileConvertAndLoadListener no longer calls convertToOpossum() (disk write + re-read); it calls loadLegacyFileFromPath() directly. saveFileListener simplified — no more attributionFilePath/inputFileChecksum. initializeGlobalBackendState simplified — no isOpossumFormat boolean. - saveFile.ts: remove attributionFilePath branch; always write via writeOpossumFile - types.ts: remove resourceFilePath/attributionFilePath from GlobalBackendState; remove ParsedOpossumInputAndOutput (replaced by LoadedArchive) - getLoadedFile.ts: remove getLoadedFileType() (dead code); simplify getLoadedFilePath() and isFileLoaded() - Delete enums.ts (only contained LoadedFileFormat), isOpossumFileFormat.ts Signed-off-by: Adrian Braemer <adrian.braemer@tngtech.com>
abraemer
commented
Jun 17, 2026
- Keep inputFileMD5Checksum in save output (always undefined, with comment) - Branch importFileConvertAndLoadListener on fileType: legacy uses loadLegacyFileFromPath, ScanCode/OWASP keep convertToOpossum - Split saveFile into buildOpossumOutputFile (pure, testable) + saveFile - Fix Prettier formatting and FileType import
- Remove unused imports (loadLegacyFileFromPath, LoadFileError, getFilePathWithAppendix, fs) flagged by strict TS - Fix outputEntry possibly-null by using non-null assertion - Fix writeOpossumFile: use addFile when output.json doesn't exist yet (happens when saving an in-memory zip from loadLegacyFile that only contains input.json)
- Unexport OPOSSUM_FILE_EXTENSION (only used internally in write-file-utils) - Unexport LoadFileError (only used internally in loadFile.ts) - Remove FileNotFoundError interface (never imported anywhere)
…date it loadLegacyFile now adds output.json to the in-memory zip when a sidecar _attributions.json exists. writeOpossumFile still needs the getEntry guard because a freshly-opened .opossum file (or a legacy file without sidecar output) may not have the entry yet — the first saveFile call creates it via addFile, subsequent saves use updateFile.
…mplify writeOpossumFile loadFile now adds the output.json entry to the in-memory zip when it creates output from scratch (null-output case). This guarantees the zip always has both entries after loadFile returns, so writeOpossumFile can unconditionally use updateFile — no getEntry guard needed.
SaveFileParams was a wrapper only used to bundle two args that callers already had separately. Replace with explicit (projectId, opossumFilePath, zip) params and rename saveFile -> persistOutputFile to clarify its role.
…ctly persistOutputFile was just buildOpossumOutputFile + writeOpossumFile. The load path already has the output data in hand (resolvedOutputData) so it can pass it directly to writeOpossumFile, avoiding a pointless round-trip through the DB via getSaveFileArgs. The save path in dbProcess now calls buildOpossumOutputFile + writeOpossumFile explicitly, making the global-state dependency visible.
- File now exports buildOpossumOutputFile (main) and getPreferredOverOriginIds (for its own test) - Internal getSaveFileArgs renamed to querySaveFileArgs (private) - Deleted saveFile.ts (was empty after earlier refactor)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After the fflate-to-adm-zip migration (#3164), the loading code still branched on file format throughout the entire stack. Every layer — from listeners through the DB process to
loadFile.ts— hadisOpossumFileFormatchecks that selected different parsing paths, different save targets, and different state fields depending on whether the input was a.opossumfile or a legacy JSON input.This PR eliminates that dual-path complexity by ensuring every load produces the same uniform result — a
LoadedArchivecontaining anAdmZipinstance, parsed input, and parsed output — regardless of whether the source was a.opossumfile or a legacy JSON input. It also simplifies the save path so thatloadFilepasses the output data it already has in hand directly towriteOpossumFile, avoiding a pointless round-trip through the DB.Stacked on top of #3164 (adm-zip migration).
Problem
Before this PR, the import flow for legacy files was:
importFileConvertAndLoadListener→convertToOpossum()writes a.opossumfile to diskinitializeGlobalBackendState(opossumFilePath, true)→ opens it as.opossumloadFile()→ re-reads the file it just wrote, branches onisOpossumFileFormat()This was wasteful (write + immediate re-read) and needlessly complex: the listener already knew the format but threw that information away, forcing
loadFileto re-derive it from the file extension.Meanwhile, the "direct legacy load" path in
loadFile.ts(theelsebranch ofisOpossumFileFormat) was unreachable from the UI and broken — saving would throwCannot save: no input file loadedbecausestoredOpossumZipwasundefined.Additionally,
saveFilecovertly depended on global DB state viagetSaveFileArgs(), even in the initial load path where the output data was already in hand — forcing a pointless DB round-trip.Approach
Unified loading
Two format-specific loader functions produce a uniform
LoadedArchivetype:loadOpossumFile(path)— opensAdmZip, extracts and parsesinput.json/output.jsonloadLegacyFile(path)— reads raw JSON (.json/.json.gz), parses it, reads sidecar_attributions.jsonif present, then constructs a new in-memoryAdmZipwith both entries packed inFrom that point on, all downstream code is format-agnostic: always an
AdmZip, always anopossumFilePathfor saves.The listener layer explicitly picks the right loader (it already knows the format), so
loadFile()no longer needs to infer it.Explicit save paths
loadFilenow addsoutput.jsonto the in-memory zip when it creates output from scratch (the null-output case), then passesresolvedOutputDatadirectly towriteOpossumFile— no round-trip through the DB. The user-initiated save path indbProcess.tsexplicitly callsbuildOpossumOutputFile(which reads from the DB) thenwriteOpossumFile. The global-state dependency is visible at each call site, not hidden inside a wrapper function.Changes
Core loading
parseFile.ts: ReplaceparseOpossumFile/parseInputJsonFile/parseOutputJsonFilewithloadOpossumFile/loadLegacyFilereturning uniformLoadedArchiveloadFile.ts: Remove allisOpossumFileFormatbranching; acceptsLoadedArchive+opossumFilePath. When creating output from scratch, addsoutput.jsonto the zip and writes viawriteOpossumFiledirectly (no DB round-trip)IPC layer
dbProcess.ts: SplitLoadFileMessageintoLoadOpossumFileMessage(format:"opossum") andLoadLegacyFileMessage(format:"legacy"); routes to correct loader, passes uniform result toloadFile. Save handler callsbuildOpossumOutputFile+writeOpossumFileexplicitlydbProcessClient.ts: SplitloadFile()intoloadOpossumFile()andloadLegacyFile();saveFile()takes explicit(projectId, opossumFilePath)instead ofSaveFileParamsimportFromFile.ts: Split intoloadOpossumFileFromPath()andloadLegacyFileFromPath()Listeners
listeners.ts:importFileConvertAndLoadListenerno longer callsconvertToOpossum()(disk write + re-read) for legacy files; callsloadLegacyFileFromPath()directly. ScanCode/OWASP still useconvertToOpossum.saveFileListenersimplified — no moreattributionFilePath/inputFileChecksum.initializeGlobalBackendStatesimplified — noisOpossumFormatboolean.Save path
saveFile.ts—SaveFileParamsandpersistOutputFileremoved; callers usewriteOpossumFiledirectlybuildOpossumOutputFile.ts(renamed fromgetSaveFileArgs.ts): exportsbuildOpossumOutputFile(assemblesOpossumOutputFilefrom DB state) andgetPreferredOverOriginIds(for its own test);querySaveFileArgsis privatewriteOpossumFileinwrite-file.ts: always useszip.updateFile(OUTPUT_FILE_NAME, ...)— the entry is guaranteed to exist becauseloadFileadds it when creating outputTypes & state
types.ts: RemoveresourceFilePath/attributionFilePathfromGlobalBackendState; removeParsedOpossumInputAndOutput(replaced byLoadedArchive)getLoadedFile.ts: RemovegetLoadedFileType()(dead code); simplifygetLoadedFilePath()andisFileLoaded()Deleted files
enums.ts(only containedLoadedFileFormatenum)isOpossumFileFormat.tsopossumArchive.ts(already inlined in previous commit)saveFile.ts(replaced by directwriteOpossumFilecalls)Test plan
parseFile.test.ts— 12 tests pass (updated to testloadOpossumFile/loadLegacyFile; added tests for sidecar output reading,.json.gz, error handling on sidecar)loadFile.test.ts— updated to use new two-step flow (load archive →loadFile); tests thatoutput.jsonis written correctly for legacy inputbuildOpossumOutputFile.test.ts— simplified (only testsbuildOpossumOutputFile, not the deleted wrapper)getLoadedFile.test.ts— simplified (removedLoadedFileFormat.Jsontests)globalBackendState.test.ts— updated (removedresourceFilePath/attributionFilePath)listeners.test.ts/get-save-file-listener.test.ts/importFromFile.test.ts/errorHandling.test.ts— all pass with updated mocksopossum-file.test.ts— updated to useloadOpossumFileinstead ofparseOpossumFiletsc --noEmit).opossumfile.jsonfile via the import dialog