-
Notifications
You must be signed in to change notification settings - Fork 11
/
Copy pathChangeLog
1140 lines (1026 loc) · 52.1 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
lrzip-next ChangeLog
This file is obsolete. See git log.
DECEMBER 2020, version 0.7.27, Peter Hyman, [email protected]
* Revert get_fileinfo function to allow STDIN checks
It just did not work as expected.
* Remove IS_FROM_FILE Definition. It never worked. !STDIN suffices.
* Undo commit 1510f4a. Decompression can use a local buffer if there is ram.
If decompressed file is larger than ram can hold, temp file will fill as
expected. No memory leak. Big speed increase.
* New Version scheme. Major.minor.micro for tags.
* Last ChangeLog Entry. Git makes it obsolete. WHAT'S NEW will be used to
announce new features,
NOVEMBER 2020, version 0.726, Peter Hyman, [email protected]
* New source layout for enhanced readability.
NOVEMBER 2020, version 0.725, Peter Hyman, [email protected]
* revert failure and fatal macro changes.
* Fixup get_fileinfo function to really allow STDIN checks
* Fix issue where lrzip would not fail if receiving piped data (see
FIXME)
OCTOBER 2020, version 0.725, Peter Hyman, [email protected]
* scrap defaul LZMA compression level = 5. All use 7 now.
* add check for corrupt archives
* clean up get_fileinfo function to remove incorrect fatal_ commands where
failure_ should be used.
* remove check for control->library_mode because it kept failure and fatal
from returning properly.
* addresses lrzip issues: 156, 163, 165, by checking for corrupt archives
before decompressing or testing.
* remove beta tag.
AUGUST 2020, version 0.720beta, Peter Hyman, [email protected]
* regression on libzpaq.h get() function overload. Pct counter
did not work with new library.
* Add Makefile.am files for libzpaq and minilzo. Just to be more
proper.
MAY 2020, version 0.715beta, Peter Hyman, [email protected]
* Added Assembler module for Decompression. 30%+ fasrer.
* Updates to configure.ac and lzma/C/Makefile.am to accomodate.
* lzma/C/Makefile.am uses absolute references so debugging can
occur.
* Update help output and options summary printouts.
APRIL 2020, version 0.710beta, Peter Hyman, [email protected]
* zpaq 7.15 integration
* -v will show zpaq settings
* rework logic for open_stream_out to balance threads and
max buffer size for lzma and add zpaq.
* zpaq ARM flags added to prevent JIT compilation that
is not supported on that platform.
* fix --enable-static-bin because some lzo2 libraries don't
include a static one. Add minilzo to source tree.
JANUARY 2020, version 0.704beta, Peter Hyman, [email protected]
* Rework Delta Offset algorithm and FILTER_FLAGS.
* FLAG values now range from 1-7, not 1<<0 to 1<<6.
Offset values are 1-32 on program execution. these are expanded
as follows:
1..17,32,48,64,80,96,112,128,144,160,176,192,208,224,240,256.
These are stored in magic header as 0-31.
JANUARY 2020, version 0.703beta, Peter Hyman, [email protected]
* Apply filters only to stream 1 literrals.
* remove unused best_dlen variable in get_fileinfo.
* Disable threshold testing when using filters.
* Set default Delta offset at 1. No alternative.
* update filters doc.
DECEMBER 2019, verion 0.701beta, Peter Hyman, [email protected]
* Add ARM, ARMT, PPC, SPARC, IA64 filter code.
* Add DELTA filter with default offset 1.
* Update docs.
* Update magic header. 1<<0 .. 1<<6 for filter in byte 16.
DECEMBER 2019, version bump 0.700, Peter Hyman [email protected]
LZMA SDK 19.00
* Add x86 filter. Add one byte to lzma properties.Magic size unchanged.
* All new codebase. Include lzma2 code but not used.
* Multi Threading still handled by p7zip SDK 16.02.
* Add Dictionary Size capability.
* Pass absolute dictionary to lzma compressor since lrzip determines size.
* Pass -1 for threading to lzma encoder so it can choose multi-thread or not.
* Update lrzip.conf to support dictionary size.
* Update setup_overhead() in util.c to handle variable dictionary sizes.
* Update open_stream_out() in stream.c to maximize use of memory and threads.
* Update info display to show actual lzma settings, lc, lp, pb, and
dictionary size.
* Show ETA for each pass of compression (fixed logic in rzip.c).
* Display decompression status every 5 seconds instead of multiples of 10%.
* lrzip can now use nasm or yasm assemblers. AC_CHECK_PROGS in configure.ac.
* LZMA can now use all levels 1-9. Removed 7/9 scaling from early SDK. LZMA
default still 5 and will display on summary screen.
* Update man pages.
* Some code cleanups (align comments, whitespace removal).
NOVEMBER 2019, updates to version 0.631, Peter Hyman
* Fixups to Assembler code in configure.ac lzma/C/Makefile.am by using
optimized code from p7zip LZMA SDK 16.02. Now works for 64 and 32 bit using
one optimized source. Hack to allow libtool to compile and link assembler
code.
JUNE 2016, version 0.630 Con Kolivas, Peter Hyman, Petr Písař, Joel Fredrikson
* checksum.buf should only be changed after the semaphore wait
* Update README
* Add documentation for recursive mode
* Implement gzip compatible -r recursive option
* Add initial argument processing for recursive option
* Tidy
* Add one more verbose for compat mode
* Add support for various combinations in compat mode
* models is array of chars. char's signess is implementation specific. It's
unsigned on ARMv7. Unsigned char cannot represent negative values. GCC 6
complains about it:
* Fix decompression of multiple chunk encrypted archives
* Tidy gotos
* Show correct lengths during testing on big endian and compressed archives
* Update copyright dates
* Allow less than maxram to be malloced for checksum to fix Failed to malloc
ckbuf in hash_search2
* Base temporary output buffer on maximum mallocable, not maxram
* Enable subdir objects for future automake compatibility
* Add support for -m option in lrztar
* Big endian fix for Solaris Sparc courtesy of joelfredrikson.
* Fixed typographical error, changed accomodate to accommodate in README.
* A whitespace fix on lrztar.
* Add sanity check to prevent trying to malloc more ram than a
system/environment is capable of
* Cosmetic help change for compat
* Add rudimentary manpage for lrz
* Fix lrz symbolic linkage
* Do not fail if we are unable to write temporary files, giving a warning only
that it might fail if we don't have enough ram
* Try /tmp/ if none of the temporary environment directories or the current
directory are writeable
* Set STDOUT correctly in compat mode
* Style police
* Fix false warning on decompressing from stdin without keep files
* Fix false warning on compressing from stdin without keep files
* Don't show extra message in compat mode decompress
* Show correct appname when called in compat mode
* Add support for progress, fast and best flags in compat mode
* Add compatibility mode with gzip when called as lrz
* Correct adding slash to control->tmpdir. off-by-one error.
* Update manpage for long options
MARCH 2015, version 0.621 Con Kolivas, Peter Hyman
* Fix libzpaq.cpp warnings
* Fix warnings in LzmaLib.c
* Making the high buffer only one page size is faster for sliding mmap mode
* Fix incompatible log callback in liblrzip
* Use PRId64 instead of lld in fscanf
* Use int64_t for i64
* Fix ISO C warning in lrzip.c
* Fix ISO C warning in lrzip.c
* Fix ISO C warning in runzip.c
* Fix ISO C warnings in rzip.c
* Fix iso c warning in util.c
* Fix EOL to unix on libzpaq
* control->tmpdir fixes
* Null terminate long options to find unhandled options
* Trivial date
* Add long option support
* Brace failure lead to broken dump to stdout
* fflush messages on print_err
* Fix inverse logic
* Honour the proper unix environment variable for temporary directory and then
try variations
* Attempt to decompress file in ram only if we are unable to create temporary
files when decompressing to stdout
* Avoid checking return values repeatedly on compress that only lead to failures
anyway
* Microoptimisation
* Don't check for failure condition that can no longer occur
* Don't check twice for failure conditions that are fatal in hot paths
* Cache the chunk bytes value to avoid setting it on each read_header call
* fake_mremap is only used when defined to mremap
* Remove unused cksem functions
* Fix remaining use of mutexes lock/unlocking in different threads with cksems,
correcting cksem usage on osx
* Update copyright dates
* Make match_len a function completely removing all indirect calls to get_sb,
significantly speeding up the single_get_sb case
* Make full_tag a pointer allowing us to avoid a function call for get_sb
* Call sliding_get_sb directly in sliding_next_tag
* Make next_tag a pointer to allow ordinary mapping to avoid an extra function
call
* fix order of lrzip.conf search
MARCH 2015, version 0.620 Con Kolivas, Peter Hyman, Chris Spiegel, Elmir Jagudin
* Increase maxram when we abandon use of temporary input/output buffers
* Don't delete the tmpinfile when decompressing from stdin before allowing seek
to end to succeed in checking md5
* Use temporary file from read_seekto when STDIN will not fit in the ram input
buffer
* Remove unused read_i64 function
* Add message about issue tracker in BUGS
* Use a common exit path iin lrzip_compress/decompress and fix lr leak on
successful return
* Fix parenthesis placement inside of unlikely().
* Clear sa_mask and sa_handler before calling sigaction().
* Fix for lrzip -i. Decompressed size wrong
* added '-m' command line option
* Fix wrong README file being included in Makefile
* Pass strict sizes to decompress length, rounding up only the amount we're
allocating to not confuse decompression libraries
* Convert the thread locking to use cksems
* Add cksems to util.h
* Fix 'Failed to malloc ckbuf in hash_search2' with very large files.
* Round up compression and decompression buffers to page size since malloc will
allocate them that large anyway.
* Increase the compressed buffer size given to libzpaq in case of incompressible
data since it does not check if it's trying to write beyond the end of the
buffer.
* Provide a helper function to round a value up to the nearest page size for
malloc optimisations.
SEPTEMBER 2013, version 0.616 George Makrydakis
* Making things more homogeneous in checks, some space removed.
* Simplifying and refactoring logic, alignment edits, empty semantic rewire.
* Target directories must always exist and -f should not create them.
* Deferring output directory after -f check has taken place.
* Preserve extraction semantics without resorting to tar stripping.
* Making sure last forward slash is removed from input path.
* Removing pointless forced overwrite point check.
SEPTEMBER 2013, version 0.615 Con Kolivas, George Makrydakis
* Adjusting -O flag semantics, options help update.
* Making -O flag operative for lrztar, whitespace path fix, lrzuntar fix, other
minor.
* Further tighten up ram restrictions with stdin/stdout to prevent running out
of memory with all the buffers involved.
* Massive files fail with -U due to trying to allocate the whole lot in ram
while doing checksums. Do it piecemeal to avoid the problem. Patch and debugging
courtesy of Adam Tkáč.
* We have to run through the clear buffer function even for empty buffers or
corrupt archives with empty match streams.
* MD5 code uses little endian so remove arbitrary SWAP macro and explicitly use
htole32.
* Rewrite the magic if we receive lzma properties and have not yet written them
yet during stdout operation.
* Set the control lzma properties only once.
* Add a control lock mutex for protecting certain control variables.
* Fix stdin fake mremap creating null bytes on osx, patch courtesy of John
Boyle.
* Cache frequently used indirectly referenced variables in the sliding mmap
code.
* Micro-optimise sliding_get_sb_range.
* A fix for a bug where large files containing the same non-zero bytes which
requires a sliding window, courtesy of Serge Belyshev.
* Put vchar should take a 64 bit integer and is used from more than one call
site so uninline it.
* Microoptimise in hash_search.
* Inline rzip functions used from only one caller.
* Check for successful calloc of hash table only after performing it.
* stdin_eof is just a bool
* hash_bits can only be up to 64 so use a char type.
* Check endianness of build with autotools to enable md5 support on more
platforms.
JULY 2012, version 0.614 Con Kolivas
* Fix lrztar failing due to incorrect fix for -S freeing segfault.
JULY 2012, version 0.613 Con Kolivas, Serge Belyshev.
* Fix other non-win32 compilation for libzpaq.
* Fix libzpaq compilation on other non-win32 OSs.
* Fix Types.h missing which breaks typedefs on macosx.
* Cope with missing ffsll() on various platforms.
* Fix get_ram compilation on osx.
* Rename lrzip.h to lrzip_core.h to cope with case insensitive filesystems not
working with Lrzip.h vs lrzip.h
* Include test for pthread.h and include header in lrzip_private.h.
* Use system extensions in configure.ac to allow to build on non-gnu-linux
platforms.
* Fix invalid free error when -o and -S are used together.
* Use no back end compression for blocks smaller than 64 bytes to avoid issues
to do with minimum buffer size and minimum match sizes of the rzip
preprocessing.
* Fix md5 calculaton on blocks larger then 2^32. Bug reproduced and debugged by
Serge Belyshev. Patch by Paul Eggert.
MARCH 2012, version 0.612 Con Kolivas
* Minor formatting improvement to delete misaligned %.
* Implement progress counter on zpaq compress/decompress per thread within the
Reader function.
* Add some lrzip style defines and optimisations to the lrzip functions in
libzpaq.h
* Re-instantiate the lzo compresses test for zpaq compression.
* Remove redundant code.
* Delete now-unused zpipe code.
* Implement virtual write support for bufWrite to speed up zpaq support and
simplify the virtual read function for bufRead.
* Add virtual read() support to the bufRead class to speed up encoding/decoding
via the zpaq library.
* Use libzpaq backend for decompression.
* Move zpaq compression to new libzpaq library back end.
* Merge branch 'master' into zpaq
* Add minimal libzpaq compilation support.
* Move to libzpaq directory.
* Import libzapq files.
* Fix Lrzip.h not being installed into include directory and make distcheck
failing.
MARCH 2012, version 0.611 Con Kolivas.
* Move the update counter to earlier in the hash_search so it still updates even
when there are not matches.
* Perform all checksumming in a separate thread to speed up the hash search in
the rzip phase.
* Optimise the hot function empty_hash.
* Micro-optimise the hot next_tag function.
* Micro-optimise the match_len function.
* Pass one less variable to the get_sb function as an optimisation.
* Cache the hash_entry value passing one less variable to hash_empty as a
micro-optimisation.
* Use ffsl for a faster lesser_bitness function.
* Prevent lrzuntar from automatically overwriting existing directories without
the -f option being specified.
* Calling lrzip via lrzcat would fail as the proper parameters weren't being
passed to the control structure. Fix.
MARCH 2012, version 0.610 Con Kolivas, Michael Blumenkrantz
* Implement complete set of liblrzip libraries, documentation and example uses
with support for simple lrzip_compress() and lrzip_decompress() or complete
fine-grained control over all compression and decompression options.
* Use as much of the low buffer as possible with a single memcopy before going
fine grained byte by byte.
* Preserve the compressed time on decompression where suitable.
* Store a copy of the control struct to be reused on subsequent files to prevent
variables being modified in the control struct on the first file that corrupt
compression/decompression of the 2nd file.
* Explicitly select C99 to avoid certain warnings.
* Generic modifications to silence -Wextra warnings.
* Fix typos.
* Use an array of parameters in lrztar to allow working with directories with
spaces in their names.
SEPTEMBER 2011, version 0.608 Con Kolivas
* get_sb only allows accessing one byte at a time, yet we don't need that
functionality when sliding mmap is not in use. Use different versions of the
function and the larger memcpys depending on whether sliding mmap is in use or
not. This affords a substantial speedup in the rzip phase of files not requiring
sliding mmap.
* A small optimisation of moving the check for remapping the low buffer also
speeds up the sliding mmap version slightly.
* Installation of the included bash completion script was disabled to avoid
conflict with distribution bash completion packages.
* Minor cleanup of sha4.h
* Make mremap use the fake mremap on all platforms not linux.
SEPTEMBER 2011, version 0.607 Con Kolivas
* Updated lzma library to version 920.
* Fixed a rare unable-to-decompress corner case.
* Added lrzip completion script.
* Updated makefile to more portable posix version.
* Hopefully fixed compilation on FreeBSD not supporting memopen.
* Added lots more debugging information about offsets in max verbose mode.
* Removed error and warning messages that are spammy or harmless.
MAY 2011, version 0.606 Con Kolivas
* lrzuntar broke as lrzip doesn't automatically use stdout now, so use lrzcat
in the lrztar script instead.
MAY 2011, version 0.605 Con Kolivas
* Addition of explicit lrzcat symlink which automatically sets "-d -o -"
* Revert automatic output to stdout due to addition of explicit lrzcat
executable, and to be consistent with gzip.
* Fix counter from being zeroed on every loop in rzip.c which was creating
excess progress output.
* Update GNU license.
* Set aside at least page_size of zeroes when compressing a chunk, thus
allowing files that are otherwise too small to compress with lrzip to be
compressed. The zeroes will just be dropped on decompression.
* Add explicit checks to cope with zero sized files as well.
* Remove the arbitrary 128 byte minimum size restriction on archives now that
any size file can be safely archived.
* Set the eof flag when reading older file format files to not hang on zero
byte sized archives.
* Update the percentage counter when getting file info to not overflow with
tiny files.
* Set the maximum file mode to 0666 to avoid setting the executable bit when
permissions are not carried over (such as from STDIO).
APRIL 2011, version 0.604 Con Kolivas
* Detach threads after creating them on the compression side. Not joining them
meant that compressing massive files requiring hundreds of threads would
eventually hit the resource limit of number of threads created even though
the threads themselves would exit.
APRIL 2011, version 0.603 Con Kolivas, George Makrydakis, Jari Aalto.
* lseek in stream.c wasn't being compiled to the lseek64 variant on Apple
due to missing includes, breaking >2GB files. Added includes.
* Detect when stdout is being redirected and automatically direct output to
stdout unless a filename is specified.
* Update lrztar to properly support -S -O and -o, and use new syntax not
requiring '-o -' for stdout.
* Update lrzip.conf to support encryption.
* Do a sanity check to ensure lrzip is not attempting to work on a directory.
* Typo fixes.
APRIL 2011, version 0.602 Con Kolivas
* Fixed the symlinks breaking package generation.
* Made maximum chunk allocable on 32bits 2/3 of a GB again limiting total ram
used to 2GB, regardless of how much crazy ram people have put on such systems
since the rest is unusable.
* Now builds on Cygwin courtesy of Тулебаев Салават.
* Check for podman on ./configure.
* Disable md5 generation and checking on Apple till it's fixed.
* Display full version when -V is invoked.
* Don't restrict file permissions from unknown source, respecting umode
instead.
MARCH 2011, version 0.601 Con Kolivas
* Fixed the symlinks for lrzuntar, lrunzip not being installed.
* Fixed the pod-based manpages not being installed.
* Clearly demonstrate that ASM is not supported on 64bits.
MARCH 2011, version 0.600 Con Kolivas
* Massive rewrite with new file format to accomodate new features.
* Allocate temporary buffers of safely sized ram that can act as temporary
storage for performing de/compression to/from STDIN/STDOUT without requiring
temporary physical files. Files compressed on machines with much larger ram
being decompressed on smaller ram machines may still require temporary files,
but as much as possible is done using in-ram decompression, and minimally
sized temporary files. Information displayed is more verbose and accurate in
STDIN/STDOUT mode.
* The temporary buffers created for decompressing to STDOUT are also used
for decompressing regular files now avoiding multiple write/reads when
re-constructing the file on decompression. This can dramatically speed up
the rzip phase of decompression on complicated files with many small matches,
and always makes testing faster.
* Compress block headers as well now since we know how many bytes can be used
to describe the length of the block, decreasing overall file size.
* Store the rzip chunk size per chunk to make it possible to check total file
size by summating each rzip chunk size when it's not known till the end (as
happens when compressing from STDIN).
* Implement password protected encryption. Import the polarssl code for
sha512 and aes128 routines. Read password without echoing to screen by
disabling echo via terminfo. Take the password then multiply hash it
according to the date it was generated. Inrease the number of hashes according
to Moore's law so it always takes approximately 1 second per password on the
most modern hardware when first encrypted. Hash the password against 8 bytes
of salt which is a combination of the 2 byte encoded loop counter (for how
many times to hash the password) and 6 random bytes. Take random from
/dev/urandom if it's available and fall back to random() if not. Encrypt each
block of compressed data with 8 extra bytes of random salt. Once the headers
are written, go back and encrypt the headers as well. Then encrypt the md5
hash value as well. Anything beyond the initial lrzip magic header should
apppear as random data and no two successive encryptions of the same data with
the same password should generate the same data.
* New build system should be more robust and portable.
* Abstract out functions better into separate files and headers, and remove
all use of global variables. This will make the generation of an lrzip
library possible in the future.
* Prevent testmalloc from coming up with a negative number when determining
how big a block of memory to allocate by decreasing the number of threads to
be used and then aborting to a minimum value should it still be too much
apparent ram.
* Big Endian fixes.
* Change lrztar and lrzuntar to not use temporary files.
* Countless other fixes, documentation and cleanups.
MARCH 2011, version 0.571 Con Kolivas
* Only retry mmaping if it's a memory error, otherwise it may give spurious
errors.
* Check for free space before compression/decompression and abort if there
is inadequate free space if the -f option is not passed.
* Fix the wrong check in rzip.c which was rounding down the page size and
making for one extra small chunk at the end.
* Check the correct stdout when refusing to pipe to a terminal.
* Fix windows EOL on lzma.txt.
* Ignore what stdout is going to when testing from stdin.
* More verbose summary after we know whether we have stdin/out to more
accurately reflect the window that will be used.
* Updated gitignore
* Unlink temporary files immediately to avoid files lying around.
* Check free space AFTER reading magic, and not when decompressing to stdout.
* Don't dump output to stdout when just testing a file.
* Dump the temporary file generated on emulating stdout on decompression after
every chunk is decompressed instead of after the whole file is decompressed.
* Don't attempt to file check decompression when it's not meaningful (test
only or to stdout).
* Clean up and simplify the times displayed component. It could potentially
show thousands of seconds.
* Remove open coded strerror messages when they're to be printed by fatal().
MARCH 2011, Michael Blumenkrantz
* Updated autotools/conf build system.
FEBRUARY 2011, version 0.570 Con Kolivas
* Change the lzo testing to a bool on/off instead of taking a parameter.
* Clean up the messy help output.
* Refuse to read from or write to terminal in stdin/stdout mode.
* Delete temporary files generated when testing from stdin.
* Don't call perror on failures that aren't system related.
* Improve visual output.
* Significant multi-threading speed-ups - as the spawning of threads is
partially serialised, use one more thread than CPUs.
* Don't split up the chunks into multiple threads when no back end
compression is being used.
* Take into account the precise amount of overhead that the memory hungry
back-end compression modes lzma and zpaq use when determining how much ram to
allocate, and limit total back-end compression ram to 1/3.
* Deprecate the -M option as it is no longer of any significant utility
compared to the -U option.
* Strip symbols by default in compilation and add -W -Wall to CXXFLAGS to be
consistent with CFLAGS.
* Update benchmarks.
* Add verbose and max verbose modes for -i information giving a breakdown of
each rzip chunk and compressed block.
* Modify purpose of -T threshold option to mean Disable LZO pre-compression
testing. Threshold level test deprecated.
* Make failure to allocate a buffer in the compress_ functions put out an
error message always rather than just in max_verbose mode.
FEBRUARY 2011, Peter Hyman
* Minor updates to man pages, lrzip.conf example file.
* Update main help screen to include environment settings.
* Update to respect $TMP environment variable for TMP files.
* Updated control structure to include tmpdir pointer.
* Update lrzip.conf parser to respect -U, -H, -T, k, and -c options.
* Update lrzip.conf example to include new parameters.
* Reorder main Switch loop in main.c for readability.
* Have UNLIMITED and control.window be exclusive. UNLIMITED wins.
* Corrects heuristic computation in rzip.c which would override
UNLIMITED if control.window set
* Show heuristically computed control.window when computed.
* Remove display compression level from control.window verbose output.
* Update print_verbose format for Testing for incompressible data in stream.c
to omit extra \n.
FEBRUARY 2011, version 0.560 Con Kolivas
* Rewrote the locking implementation to use pthread_mutexes instead of
semaphores. This makes multi-threading work on OSX.
* Re-use a single malloc in unzip_match instead of multiple malloc/frees.
* Imported the md5 code from coreutils and modified them to be used by lrzip.
* Implemented md5 hash checking on compression and decompression by using
buffers to pass to the existing crc checking and re-using them for the md5
checking.
* Sped up the hash checking on compression by not reading the whole chunk at
once to do the hash check, and instead read one paged size chunk.
* Store the md5 hash value at the end of the archive, maintaining backward
compatibility with existing 0.5 versions of lrzip, and place a flag in the
header denoting its existence.
* Use the header flag for md5's existence to determine what hash check to
perform on decompression.
* Add hash value display to maximum verbose output and in information mode.
* Add -H option to enable hash value displaying without verbose mode.
* Display what hash checking will be used on decompression in verbose mode and
in information displayed with -i.
* Add -c option to perform a hash check on the file generated on decompression,
comparing it to that stored in the lrzip archive, or that generated on
decompression on archives that do not have it stored.
* Delete broken or damaged files generated on interruption or when they fail
integrity testing by default.
* Implement the -k option to keep broken or damaged files.
FEBRUARY 2011, version 0.552 Con Kolivas
* Fix incompressible blocks causing failure with bzip2 and gzip.
* Freebsd doesn't do mremap, use the fake one.
* Update copyright notices.
* Flush writes to disk before allocating ram.
* Relax memory allocation when no back end compression will be used.
* Clean up unnecessary semaphore usage in decompression.
* Fix minor typos.
* Convert mmaps to malloc in runzip as they may silently fail if not a
multiple of page size, thereby leading to silent corruption on decompression.
* Return the correct length by unzip_literal.
* Use wrappers to serialise threads on OSX and bypass use of unnamed semaphores
which aren't supported.
DECEMBER 2010, version 0.551 Con Kolivas
* Fix stdin failing due to getting sizes all wrong.
* Fix stdin compression values not being shown at end.
* Fix inappropriate failure when lzma doesn't compress block.
DECEMBER 2010, version 0.550 Con Kolivas
* Move the threading on compression to higher up in the code, allowing the next
stream to start using compression threads before the previous stream has
finished. This speeds up compression on files that take more than one pass to
compress.
* Limit the number of threads decompressing stream 0 to just 1 since it's always
followed by stream 1 chunks, and it may lead to failure to decompress due to
running out of memory by running too many threads.
* Default compression level and window size on lzma is set to 7 which is the
highest it goes. Scale the 9 lrzip levels into 7, thus making the default lzma
level 5 which uses a lot less memory and is substantially faster at the cost of
some compression.
* Rationalise the memory testing now that the default lzma settings use a lot
less ram by default, and make all systems use no more than 1/3 ram in one mmap.
This allows larger windows to be used by 32 bit at last without memory
allocation errors.
* Revert "Make threads spawn at regular intervals along chunk size thus speeding
up compression" as it actually slowed it down instead of speeding it up.
* Cope with compression/decompression threads failing by waiting till the
previous thread has finished its work, thus serialising the work and using less
ram, making success more likely.
* Fix some dodgy callocs which weren't really allocating enough ram.
* Destroy semaphores used in stream_in on closing the stream.
* Minor output improvements.
DECEMBER 2010, version 0.544 Con Kolivas
* Make multiple stream 0 entry decompression more robust by creating separate
thread groups for stream 0 and stream 1.
* uclibc fails to return valid values for memory size with sysconf so create
a workaround.
* 32 bits has memory allocation errors on large files so be ruthless with
limiting window sizes and use more sliding mmap.
* Split up compression streams according to proportion of hash search thus
evenly distributing work to threads based on raw data length rather than
rzip pre-processed data. This makes for more CPU usage on compression and
smoother write out on decompression with small speed ups.
* Remove unnecessary argument passing.
* Fix -U causing a floating point error with ultra small sized files.
* 32 bits still sometimes fails the lzma compression stage due to internal
memory allocation issues, so use bzip2 as a fallback on blocks that fail to
compress.
* Minor cleanups.
NOVEMBER 2010, version 0.543 Con Kolivas
* Fix for when there are multiple stream 0 entries per chunk, which would
cause a failure on decompression.
* Fix lzma windows being set far too small by mistake at all times.
* Decrease nice level of main process compared to back end threads as it tends
to be the rate limiting step.
NOVEMBER 2010, version 0.542 Con Kolivas
* Choose sane defaults for memory usage since linux ludicriously overcommits.
* Use sliding mmap for any compression windows greater than 2/3 ram.
* Consolidate and simplify testing of allocatable ram.
* Minor tweaks to output.
* Round up the size of the high buffer in sliding mmap to one page.
* Squeeze a little more out of 32 bit compression windows.
* Fix sliding mmap from not working on 2nd pass onwards.
NOVEMBER 2010, version 0.541 Con Kolivas
* Fix wrong number of passes reported.
* Re-fix the off-by-one that wasn't off-by-one.
* Limit lzma compression windows to 300MB as per reports of failures with larger
windows.
NOVEMBER 2010, version 0.540 Con Kolivas
* Massive rewrite of backend decompression phse, implementing multithreading.
This is done by taking each stream of data on read in into separate buffers for
up to as many threads as CPUs. As each thread's data becomes available, feed it
into runzip once it is requests more of the stream. Provided there are enough
chunks in the originally compressed data, this provides a massive speedup
potentially proportional to the number of CPUs. The slower the backend
compression, the better the speed up (i.e. zpaq is the best sped up).
* Fix the output of zpaq compress and decompress from trampling on itself and
racing and consuming a lot of CPU time printing to the console.
* When limiting cwindow to 6 on 32 bits, ensure that control.window is also set.
* When testing for the maximum size of testmalloc, the multiple used was out by
one, so increase it.
* Minor output tweaks.
* Build warning fixes.
* Updated benchmarks.
NOVEMBER 2010, version 0.530 Con Kolivas
* Massive rewrite of backend compression phase. Now the stream is split up
into as many chunks as there are CPUs, of at least 10MB in size, that are
still mallocable. Once the stream has reached a chunk of this size, its buffer
is handed to a new backend compression thread which works while the rzip stream
continues processing. This has the effect of parallelising workloads almost
linearly up to the number of CPUs on the slower compression backends. ZPAQ,
in particular, is effectively 4x faster on quad core now. Decompression is
unchanged.
* Added the -p option to allow the number of processors to be specified to
override the detected number.
* Changed the default level back to 7 as 9 wasn't offering significanly more
compression but was adding time.
* Increased the size of all the buffers to other backends now as well, since
each block adds overhead with its header.
* Numerous alterations to screen output to cope with new threaded compression
phase.
* Deprecated the -P option since not setting the file permissions only
generates a warning now, not a failure.
* Updated docs and benchmarks.
NOVEMBER 2010, version 0.520 Con Kolivas
* Distros don't like 3 point version numbering so just repackaged as 0.520.
NOVEMBER 2010, version 0.5.2 Con Kolivas
* Fixed the Darwin build, again.
* Fixed cases of extreme ram usage on 32 bit failing by limiting zpaq to 600MB
windows as well.
* Check page size if we can instead of assuming it's always 4k.
* Improve the progress output.
* Change failure to chmod and failure to set nice level to warnings only.
* Standardise what's a stderr message and what's output.
NOVEMBER 2010, version 0.5.1 Con Kolivas
* Fix Darwin build - Darwin doesn't support mremap so introduce a fake wrapper
for it.
* Fix the memopen routines, a wrongly implemented wrapper for Darwin equivalents
was also using the faked versions on all builds.
* Fix dodgy ordered includes.
* Clean up excessive use of #ifdefs
* Huge rewrite of buffer reading in rzip.c. We use a wrapper instead of
accessing the buffer directly, thus allowing us to have window sizes larger than
available ram. This is implemented through the use of a "sliding mmap"
implementation. Sliding mmap uses two mmapped buffers, one large one as
previously, and one smaller one. When an attempt is made to read beyond the end
of the large buffer, the small buffer is remapped to the file area that's being
accessed, while the larger one is remapped as the search progresses along the
file. While this implementation is potentially much slower than direct mmapping,
it allows us to implement unlimited sized compression windows.
* Implement the -U option with unlimited sized compression windows.
* Rework the selection of compression windows. Instead of trying to guess how
much ram the machine might be able to access, we try to safely buffer as much
ram as we can, and then use that to determine the file buffer size. Do not
choose an arbitrary upper window limit unless -w is specified.
* Rework the -M option to try to buffer the entire file, reducing the buffer
size until we succeed.
* Align buffer sizes to page size.
* Clean up lots of unneeded variables.
* Fix lots of minor logic issues to do with window sizes accepted/passed to rzip
and the compression backends.
* More error handling.
* Change -L to affect rzip compression level directly as well as backend
compression level and use 9 by default now.
* Fix file size reporting on compressed files generated from stdin.
* More cleanups of information output and more information.
* Add chunk percentage to progress update.
* Reinstated the 2GB buffer limit on 32 bit machines during compression, though
the -U mode can work around it now.
* Code micro-optimisations.
* Use 3 point release numbering in case one minor version has many subversions.
* Numerous minor cleanups and tidying.
* Updated docs, manpages, and benchmarks.
NOVEMBER 2010, version 0.5 Con Kolivas
* Changed offset encoding in rzip stage to use variable byte width offsets
instead of 64 bits wide. Makes for better compression and slightly faster.
* Write the byte width into the file before each block.
* Shrunk match lengths to maximum of 16 bits again as per original rzip as the
larger offsets did not achieve greater compression and made final size larger.
* New file format not backwards compatible due to variable byte widths.
* Rewrote memory initialisation to have a pre-allocation stage to try and
find the maximum memory usable and defragment ram.
* Use reduced window size if allocating memory fails at higher size.
* Change use of malloc to mmap to make it possible to address up to 44 bit
sized offsets even on 32 bit machines on decompression. Still unable to use
greater than 2GB windows on 32 bit machines and unsure if this is fixable.
* Reworked the STDIN code to use an anonymous mmap and read in stdin into this
to make it possible to compress from STDIN without the need for temporary
files. As the file size is not known in advance, memory allocation is set to
large and byte width to equivalent size.
* Reallocation of ram where possible to minimise risk of running out of memory
in the middle of a compression phase, and flushing to disk to empty dirty ram
for the same reason.
* More robust fatal warnings.
* Numerous cleanups and tidying of code and addition of comments.
* Updated documentation to reflect changes.
OCTOBER 2010, version 0.47, Con Kolivas
* Fix the symlinking problem when DESTDIR is in use reported by a billion
people.
MAY 2010, version 0.46, Con Kolivas, Ed Avis.
* Suppress final [OK] message with -q flag EA
* Handle mkstemp() errors correctly EA
* Add lrzuntar manpage
* Update manpages
APRIL 2010, version 0.45, Con Kolivas, Jon Tibble, George Makrydakis
* Fixes the nasm program test (AC_CHECK_PROG doesn't overwrite a
variable that is already set so do it manually) JT
* Fix compiler flags as not all compilers accept -Wall -W (cc on
Solaris/OpenSolaris) JT
* Fix lrztar to not try to compress files already with the .lrz extension GM
* Fix lrztar to decompress files where the pathname is ../* GM
* Add lrzuntar symlink to call lrztar -d
MAR 2010, version 0.45, Con Kolivas, Jari Aalto
* Fixed reported window size
* Fixed 32bit windows being attempted to be larger than contiguous amounts
by taking into account VM kernel/userspace split of 896MB.
* Minor code cleanups
* Added lrztar and lrunzip docs
* Fix minor typos
* Added distclean and maintainer-clean make targets
DEC 2009, version 0.44, Con Kolivas, George Makrydakis
* Added lrztar wrapper to manage whole directories.
* Added -i option to provide information about a compressed file.
* Fixed "nan" showing as Compression speed on very small files.
* Fixed build for old bz library.
* Avoid overwriting output file if input doesn't exist.
* Implement signal handler to delete temporary files.
DEC 2009, version 0.43, Con Kolivas, Jukka Laurila
* Darwin support thanks to Jukka Laurila.
* Finally added stdin/stdout support due to popular demand. This is done
by basically using temporary files so is a low performance way of using
lrzip.
* Added test function. This just uses a temporary file during decompression.
* Config files should now accept zpaq options.
* Minor code style cleanups.
* Updated benchmarks in docs.
* Add a warning when attempting to decompress a file from a newer lrzip
version.
NOV 2009, version 0.42, Con Kolivas
* Changed progress update to show which of 2 chunks are being compressed
in zpaq.
* Fixed progress update in ZPAQ to not update with each byte which was
wasting heaps of CPU time.
NOV 2009, version 0.41, Con Kolivas
* Added zpaq compression backend for extremely good but extremely slow
compression (incompatible with previous versions if used).
* Limited chunk size passed to LZMA to 4GB to avoid library overflows.
* Minor changes to the formatting output
* Changed lower limit of -T threshhold to 0 to allow disabling it.
* Added lzo_compresses check into zpaq and bzip2 as well since they're
slow.
NOV 2009, version 0.40, Con Kolivas
* Massive core code rewrite.
* All code moved to be 64bit based for compression block addressing and length
allowing compression windows to be limited by ram only.
* 64bit userspace should now have no restriction on compression window size,
32bit is still limited to 2GB windows due to userspace limitations.
* New file format using the new addressing and data types, incompatible with
versions prior to 0.40.
* Support for reading and decompressing older formats.
* Minor speedups in read/write routines.
* Countless minor code fixes throughout.
* Code style cleanups and consistency changes in core code.
* Configure script improvements.
NOV 2009, version 0.31, Con Kolivas
* Updated to be in sync with lzma SDK 9.07beta.
* Cleanups and fixes of the configure scripts to use the correct package version
name.
* Massive fixes to the memory management code fixing lots of 32bit overflow
errors. The window size limit is now 2GB on both 32bit and 64bit. While it
appears to be smaller than the old windows, only 900MB was being used on .30
even though it claimed to use more. This can cause huge improvements in the
compression of very large files.
* The offset when mmap()ing was not being set to a multiple of page size so
it would fail if the window size was not a multiple of it.
* Flushing of data to disk between compression windows was implemented to
minimise disk thrashing of read vs write.
NOV 2009, version 0.30, Con Kolivas
* Numerous bugfixes to try and make the most of 64bit environments with huge
memory and to barf less on 32bit environments.
* Executable stacks were fixed.
* Probably other weird and wonderful bugs have been introduced.
* -P option to not set permissions on output files allowing you to write to
braindead filesystems (eg fat32).
JAN 2009, version 0.24, Peter Hyman, [email protected]
Happy New Year!
* Upgrade LZMA SDK to 4.63. Use new C Wrapper. Invalidates
LZMA archives created earlier due to new Magic property
bytes.
* New LZMA logic will automatically determine allow LZMA
code to determine optimal lc, lp, pb, fb, and dictionary
size settings. stream.c will only pass level and thread
information. Compress function will return encoded 5 byte
data with compression settings. This will be stored in lrz
file header.
* add error messages during LZMA compression. There are some
edge cases where LZMA cannot allocate memory. These errors
are reported and the user will be advised to use a lower
compression window setting.
* type changes in rzip_fd function for correctness.
* remove function *Realloc() since it was never used. Cleaned
in rzip.h and util.c.
* apply munmap prior to closing and compressing stream in
function rzip_chunk in rzip.c.
* add realloc function in close_stream_out in stream.c
to reclaim some ram and try and allieviate out of memory
conditions in LZMA compression.
* remove file acconfig.h and include DEFINE in configure.in.
* add lrzip.conf capability.
* add timer for compression including elapsed time and eta.
* add compression and decompression MB/s calculation.
* Updated WHATS-NEW, TODO and created BUGS file.
* Updated lrzip.1 manpage and created lrzip.conf.5 manpage.
* Added lrzip.conf.example file in doc directory.
MAR 2008, Con Kolivas, [email protected]
* Numerous changes all over to place restrictions on window
size to work with 32 bit limitations.
* Various bugfixes with respect to detecting buffer sizes and
likelihood of compressibility.
* Fixed the inappropriate straight copying uncompressed data for
files larger than 4GB.
* Re-initiated the 10MB window limits for non-lzma compression.
I was unable to reproduce any file size savings.
* Allow compression windows larger than ramsize if people really
really want them.
* Decrease thresholds for the test function to a minimum of 5%
compressibility since the hanging in lzma compression bug has been
fixed.
JAN 2008, version 0.22, Peter Hyman, [email protected]
* version update
lzma/LZMALib.cpp
Thanks to Lasse Collin for debugging the problem LZMA
had with hanging on uncompressable files.
Update for control parameters to both compress and
decompress functions.
Makefile.in
* use of @top_srcdir@ (Lasse Collin). Also moved away
more cruft.
main.c stream.c.rzip.h LZMALib.cpp lzmalib.h
* addition of three new control structure members.
control.lc -- literal context bits
control.lp -- literal post state bits
control.pb -- post state bits
These are needed to ensure decompression will work.
These will now be stored along with control.compression_level
in the lrz file beginning at offset 0x16 for three bytes.
These will be passed to the functions lzma_compresses and
lzma_uncompress. Currently, only compression level is
needed or used, but the others are stored for possible future
use.
See magic file for more information.
stream.c
* Change to lzo_compresses function that will reject a chunk
without testing it if the size of the chunk is greater
than the compression window * threshold. This is to avoid
a low probability that lzma would still be passed a chunk
that contains uncompressible data or barely compressible
data. If after rzip hashing the chunk size is still close
to the window size, there is hardly anything worth
compressing. While there is no reason lzma cannot get the
chunk, this will save a lot of time.
magic.headers.txt
* updated file to show new layout that includes lzma
parameters.
README-NOT-BACKWARD-COMPATIBLE
* added warning about using lrzip-0.22 with earlier versions.
WHATS-NEW
* highlight of new features.
DEC 2007, version 0.21. Peter Hyman, [email protected]
* version update.
* Modified to use Assembler routines from lzma SDK for CRC
computation when hashing streams in rzip.c and runzip.c.
Added files 7zCrcT8.c and 7zCrcT8u.s to lzma tree.
Cleaned up source tree. Moved unused files out of the way.
Moved non-core docs to doc directory
configure.in
* correct AC_INIT to set program variables.
* modified to add check for nasm assembler.
* modified syntax of test for errno in error.h to use
echo $ECHO_N/$ECHO_C instead of $ac_n/$ac_c which
was incorrect.
Makefile.in, lzma/Makefile
* modified to add compile instructions for 7zCrcT8.c
and 7zCrcT8U.s and Assembler. Cleaned up to remove
targets that don't exist or sources that don't
exist.
Modified to properly set directories. Added doc install.
Add link command to symlink lrunzip to lrzip.
*main.c
Add CrcGenerateTable() function to init CRC tables.