Don't use libtool convenience libraries to avoid recently
discovered long-standing subtle but somewhat severe bugs
in libtool (at least 1.5.22 and 2.2.6 are affected). It
was found when porting XZ Utils to Windows
<http://lists.gnu.org/archive/html/libtool/2009-06/msg00070.html>
but the problem is significant also e.g. on GNU/Linux.
Unless --disable-shared is passed to configure, static
library built from a set of convenience libraries will
contain PIC objects. That is, while libtool builds non-PIC
objects too, only PIC objects will be used from the
convenience libraries. On 32-bit x86 (tested on mobile XP2400+),
using PIC instead of non-PIC makes the decompressor 10 % slower
with the default CFLAGS.
So while xz was linked against static liblzma by default,
it got the slower PIC objects unless --disable-shared was
used. I tend develop and benchmark with --disable-shared
due to faster build time, so I hadn't noticed the problem
in benchmarks earlier.
This commit also adds support for building Windows resources
into liblzma and executables.
liblzma tries to avoid useless free()/malloc() pairs in
initialization when multiple files are handled using the
same lzma_stream. This didn't work with filter chains
due to comparison of wrong pointers in lzma_next_coder_init(),
making liblzma think that no memory reallocation is needed
even when it actually is.
Easy way to trigger this bug is to decompress two files with
a single xz command. The first file should have e.g. x86+LZMA2
as the filter chain, and the second file just LZMA2.
Now configure.ac will get the version number directly from
src/liblzma/api/lzma/version.h. The intent is to reduce the
number of places where the version number is duplicated. In
future, support for displaying Git commit ID may be added too.
lzma_memlimit_encoder and lzma_memlimit_decoder to
lzma_raw_encoder_memlimit and lzma_raw_decoder_memlimit. :-(
Now it is fixed. Hopefully it doesn't cause too much trouble
to those who already thought API is stable.
Half of developers were already forgetting to use these
functions, which could have caused total breakage in some future
liblzma version or even now if --enable-small was used. Now
liblzma uses pthread_once() to do the initializations unless
it has been built with --disable-threads which make these
initializations thread-unsafe.
When --enable-small isn't used, liblzma currently gets needlessly
linked against libpthread (on systems that have it). While it is
stupid for now, liblzma will need threads in future anyway, so
this stupidity will be temporary only.
When --enable-small is used, different code CRC32 and CRC64 is
now used than without --enable-small. This made the resulting
binary slightly smaller, but the main reason was to clean it up
and to handle the lack of lzma_init_check().
The pkg-config file lzma.pc was renamed to liblzma.pc. I'm not
sure if it works correctly and portably for static linking
(Libs.private includes -pthread or other operating system
specific flags). Hopefully someone complains if it is bad.
lzma_rc_prices[] is now included as a precomputed array even
with --enable-small. It's just 128 bytes now that it uses uint8_t
instead of uint32_t. Smaller array seemed to be at least as fast
as the more bloated uint32_t array on x86; hopefully it's not bad
on other architectures.
- Updated to the latest, probably final file format version.
- Command line tool reworked to not use threads anymore.
Threading will probably go into liblzma anyway.
- Memory usage limit is now about 30 % for uncompression
and about 90 % for compression.
- Progress indicator with --verbose
- Simplified --help and full --long-help
- Upgraded to the last LGPLv2.1+ getopt_long from gnulib.
- Some bug fixes
Use LZMA_PROG_ERROR instead of LZMA_HEADER_ERROR if the Filter ID
is in the reserved range. This allows Block Header encoder to
detect unallowed Filter IDs, which is good for Stream encoder.
code from block_private.h to block_decoder.c. Now the Block
encoder doesn't need compressed_size and uncompressed_size
from lzma_block structure to be initialized.
LZMA_Alone files. Decoding of concatenated LZMA_Alone files is
intentionally not supported, so it is better to put this in
auto decoder than LZMA_Alone decoder.
broken. API has changed a lot and it will still change a
little more here and there. The command line tool doesn't
have all the required changes to reflect the API changes, so
it's easy to get "internal error" or trigger assertions.
specification. Simplify things by removing most of the
support for known uncompressed size in most places.
There are some miscellaneous changes here and there too.
The API of liblzma has got many changes and still some
more will be done soon. While most of the code has been
updated, some things are not fixed (the command line tool
will choke with invalid filter chain, if nothing else).
Subblock filter is somewhat broken for now. It will be
updated once the encoded format of the Subblock filter
has been decided.
encoder and decoder, and put the shared things to
block_private.h. Improved the checks a little so that
they may detect too big Compressed Size at initialization
time if lzma_options_block.total_size or .total_limit is
known.
Allow encoding and decoding Blocks with combinations of
fields that are not allowed by the file format specification.
Doing this requires that the application passes such a
combination in lzma_options_lzma; liblzma doesn't do that,
but it's not impossible that someone could find them useful
in some custom file format.
- Added lzma_memlimit_max() and lzma_memlimit_reached()
API functions.
- Added simple estimation of malloc()'s memory usage
overhead.
- Fixed integer overflow detection in lzma_memlimit_alloc().
- Made some white space cleanups and added more comments.
The description of lzma_memlimit_max() in memlimit.h is bad
and should be improved.
lzma_metadata.header_metadata_size == LZMA_VLI_VALUE_UNKNOWN
is not allowed at all. To indicate missing Header Metadata
Block, header_metadata_size must be set to zero. This is
what Metadata decoder does after this patch too.
Note that other missing fields in lzma_metadata are still
indicated with LZMA_VLI_VALUE_UNKNOWN. This isn't as
illogical as it sounds at first, because missing Size of
Header Metadata Block means that Header Metadata Block is
not present in the Stream. With other Metadata fields,
a missing field means only that the value is unknown.
liblzma as easy as using zlib, because the easy API
don't require developers to know any fancy LZMA options.
Note that Multi-Block Stream encoding is currently broken.
The easy API should be OK, the bug(s) are elsewhere.
function is still shared between encoder and decoder, but the
actual coding is in separate files for encoder and decoder.
There are now separate functions for the actual delta
calculation depending on if Delta is the last filter in the
chain or not. If it is the last, the new code copies the
data from input to output buffer and does the delta
calculation at the same time. The old code first copied the
data, then did the delta in the target buffer, which required
reading through the data twice.
Support for LZMA_SYNC_FLUSH was added to the Delta encoder.
This doesn't change anything in the file format.
table-based version from LZMA SDK 4.57. This should be
fast on most systems.
A simpler and smaller alternative version is also provided.
On some CPUs this can be even a little faster than the
default table-based version (see comments in fastpos.h),
but on most systems the table-based code is faster.
It's not strictly needed there, and just complicates the
code. LZ encoder never even had this feature.
The primary reason to have uncompressed size tracking in
filter encoders was validating that the application
doesn't give different amount of input that it had
promised. A side effect was to validate internal workings
of liblzma.
Uncompressed size tracking is still present in the Block
encoder. Maybe it should be added to LZMA_Alone and raw
encoders too. It's simpler to have one coder just to
validate the uncompressed size instead of having it
in every filter.
Lasse Collin