refactor(zip): replace fflate with adm-zip for opossum file handling#3164
refactor(zip): replace fflate with adm-zip for opossum file handling#3164abraemer wants to merge 1 commit into
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
abraemer
force-pushed
the
feat/adm-zip-migration
branch
from
1f92bfa to
083fe2c
Compare
Replace fflate with adm-zip as the zip library for .opossum file reading and writing. This eliminates the need to hold the full decompressed input.json bytes (storedInputFileRaw) in memory for the entire session. Key changes: - opossumArchive.ts: use AdmZip instead of fflate streaming unzip; return the AdmZip instance so it can be reused for in-place saves - write-file.ts: use AdmZip.updateFile() for saves (only output.json is recompressed; input.json compressed bytes pass through unchanged); writeOpossumFile is now synchronous - parseFile.ts: replace fflate.strFromU8 with Buffer.toString - loadFile.ts/dbProcess.ts: replace storedInputFileRaw (Uint8Array) with storedOpossumZip (AdmZip instance) - types.ts: replace inputFileRaw with opossumZip in ParsedOpossumInputAndOutput - LegacyFileConverter.ts/fixtures.ts: remove await from now-sync writeOpossumFile calls - Remove fflate dependency, add adm-zip + @types/adm-zip Signed-off-by: Adrian Braemer <adrian.braemer@tngtech.com>
abraemer
force-pushed
the
feat/adm-zip-migration
branch
from
083fe2c to
1c87228
Compare
|
awesome idea, I was trying to do this but never found the right package. Did not know about |
|
Some numbers would have been nice. On the facebook file with some manual testing I can see that the zipping time goes from ~4s -> ~1s |
| if (output) { | ||
| zip.addFile(OUTPUT_FILE_NAME, toBuffer(output)); | ||
| } | ||
| zip.writeZip(path); |
There was a problem hiding this comment.
Nit: you can move this before the return, since in all branches we want to write the Zip
| if (output) { | ||
| zip.updateFile(OUTPUT_FILE_NAME, toBuffer(output)); | ||
| } | ||
| zip.writeZip(path); |
There was a problem hiding this comment.
Instead of writeZip I would use await zip.writeZipPromise(path), since this is non blocking. Perhaps there is a reason why you decided to go the sync route?
Leaving it async would minimize the blast radius of this PR
|
|
||
| it('writes to opossumFilePath as .opossum format', async () => { | ||
| const inputFileRaw = new Uint8Array([1, 2, 3]); | ||
| const opossumZip = new AdmZip(); |
There was a problem hiding this comment.
Probably no reason, but why did the Uint8Array contain [1, 2, 3]?
Thanks for timing it. I did this on the train and didn't have any sufficiently large opossum file at hand |
2 participants
Summary
Replace fflate with adm-zip as the zip library for
.opossumfile reading and writing.Problem
The full decompressed
input.jsonbytes (storedInputFileRaw) were held in memory for the entire session so they could be written back verbatim on each save. On every save, the entire zip was recreated from scratch viafflate.zip(), recompressing bothinput.jsonandoutput.json.Solution
adm-zip's
updateFile()allows modifying individual zip entries in-place. Unchanged entries (likeinput.json) pass through with their original compressed bytes intact — no decompression or recompression needed.Key changes:
AdmZipinstead of fflate streaming unzip; return theAdmZipinstance for later reuse on saveAdmZip.updateFile()for saves — onlyoutput.jsonis recompressed;input.jsoncompressed bytes pass through unchanged;writeOpossumFileis now synchronousfflate.strFromU8()withBuffer.toString()storedInputFileRaw(Uint8Array) withstoredOpossumZip(AdmZipinstance)inputFileRawwithopossumZipinParsedOpossumInputAndOutputawaitfrom now-synchronouswriteOpossumFilecallsfflatedependency, addadm-zip+@types/adm-zipCompression level note
adm-zip uses Node.js
zlibwith the default compression level (6), whereas fflate was configured at level 5. The practical difference is negligible (~0.1% size difference).What stays
stream-json+bytesAsStream()+parseJsonStream()remain — they are still needed for parsing large input JSON to avoid V8's ~512 MB string-length cap.Checklist