Compression trades speed for space, but we should be able to make that decision on the fly. There are two, perhaps three, action items here:
- Trade space and speed more intelligently
- Automatically prefer
pigz to gzip, etc, if available
- Rate-limit data throughput if we start running out of disk space, pause entirely if we're near critical so we don't crash the job
Compression trades speed for space, but we should be able to make that decision on the fly. There are two, perhaps three, action items here:
pigztogzip, etc, if available