Here are the list of the most significant issues addressed:
- Avoid using internal common.h header. It's not good to copy the
constants like this but common.h cannot be included for use outside
of liblzma. This is the quickest thing to do that could be fixed later.
- Omit the INIT_FILTER macro. Initialization should be done with just
regular designated initializers.
- Use start_offset = 257 for BCJ tests. It demonstrates that Filter
Flags encoder and decoder don't validate the options thoroughly.
257 is valid only for the x86 filter. This is a bit silly but
not a significant problem in practice because the encoder and
decoder initialization functions will catch bad alignment still.
Perhaps this should be fixed but it's not urgent and doesn't need
to be in 5.4.x.
- Various tweaks to comments such as filter id -> Filter ID
It's not needed in XZ Utils at least for now. It's good to support
it still because if such use is needed later, it wouldn't be
caught on GNU/Linux since malloc(0) from glibc returns non-NULL.
The shell parameter expansion using # and ## is not supported in
Solaris 10 Bourne shell (/bin/sh). Even though this is POSIX, it is not fully
portable, so we should avoid it.
It now tries to test as many files as easily possible.
The exit status indicates skipping if any of the files were
skipped. This way it is easy to notice if something is being
skipped when it isn't expected.
xz (but not xzdec) will normally warn about unsupported check
but since we are testing specifically such a file, it's better
to silence that warning so that it doesn't look suspicious in
test_files.sh.log.
The use of -q and -Q in xzdec is just for consistency and
doesn't affect the result at least for now.
This isn't perfect as the scripts can still fail if only
certain filters are disabled. This is still an improvement
as now "make check" has better behavior when all encoders
or decoders are disabled.
Grepping ../config.h is simple and fairly clean but it only
works if config.h was created. CMake builds don't create
config.h but they don't use these test scripts either.
Thanks to Sebastian Andrzej Siewior for reporting the problem.
Thanks to Jia Tan for the original patch which grepped xz
error messages instead of config.h.
I suspect that I used these in the original version because
Autoconf's manual describes that such a trick is needed in
some specific situations for portability reasons. None of those
situations listed on Autoconf 2.71 manual apply to these test
scripts though so this cleans them up.
Converts test_stream_flags to tuktest. Also the test will now
compile and skip properly if encoders or decoders are disabled.
Thanks to Sebastian Andrzej Siewior.
test_block_header now achieves higher test coverage. Also the
test will now compile and skip properly if encoders or decoders
are disabled.
Thanks to Sebastian Andrzej Siewior.
test_bcj_exact_size, test_check, test_hardware, and test_index will
all now compile and skip properly if encoders or decoders are disabled.
Also fixed a small typo (disabed -> disabled).
Thanks to Sebastian Andrzej Siewior.
This test fails before commit 18d7facd38.
test_files.sh now runs xz -l for bad-3-index-uncomp-overflow.xz
because only then the previously-buggy code path gets tested.
Normal decompression doesn't use lzma_index_append() at all.
Instead, lzma_index_hash functions are used and those already
did the overflow check.
Warnings about unused tuktest_run_test conveniently tell which
test programs haven't been converted to tuktest.h yet but I
silenced that warning too for now anyway.
It is fine to use __attribute__((__unused__)) even when the
function is actually used because the attribute only means
that the function might be unused.
The script uses lcov and genhtml after running the tests
to show the code coverage statistics. The script will create
a coverage directory where it is run. It can be run both in
and out of the source directory.
lzma_vli is unsigned so trying a signed value results in
a compiler warning from -Wsign-conversion. (lzma_vli)-1
equals to LZMA_VLI_UNKNOWN anyway which is the next assertion.
Created tests for all API functions exported in
src/liblzma/api/lzma/hardware.h. The tests are fairly trivial
but are helpful because they will inform users if their machines
cannot support these functions. They also improve the code
coverage metrics.
test_compress.sh now takes one command line argument:
a filename to be tested. If it begins with "compress_generated_"
the file will be created with create_compress_files.
This will allow parallel execution of the slow tests.
If a command line argument is given, then only the test file
of that type is created. It's quite dumb in sense that unknown
names don't give an error but it's good enough here.
Also use EXIT_FAILURE instead of 1 as exit status for errors.
Hiding them makes no sense since normally there's no error
when testing the "good" files. With "bad" files errors are
expected and then it makes sense to keep the messages hidden.
lzma_code() could incorrectly return LZMA_BUF_ERROR if
all of the following was true:
- The caller knows how many bytes of output to expect
and only provides that much output space.
- When the last output bytes are decoded, the
caller-provided input buffer ends right before
the LZMA2 end of payload marker. So LZMA2 won't
provide more output anymore, but it won't know it
yet and thus won't return LZMA_STREAM_END yet.
- A BCJ filter is in use and it hasn't left any
unfiltered bytes in the temp buffer. This can happen
with any BCJ filter, but in practice it's more likely
with filters other than the x86 BCJ.
Another situation where the bug can be triggered happens
if the uncompressed size is zero bytes and no output space
is provided. In this case the decompression can fail even
if the whole input file is given to lzma_code().
A similar bug was fixed in XZ Embedded on 2011-09-19.
The scripts are now made executable in the build tree.
This way the scripts can be run like programs in
test_scripts.sh. Previously test_scripts.sh always
used sh but it's not correct if @POSIX_SHELL@ is set
to something else by configure.
Thanks to Jonathan Nieder for the patch.
xzdiff was clobbering the exit status from diff in a case
statement used to analyze the exit statuses from "xz" when
its operands were two compressed files. Save and restore
diff's exit status to fix this.
The bug is inherited from zdiff in GNU gzip and was fixed
there on 2009-10-09.
Thanks to Jonathan Nieder for the patch and
to Peter Pallinger for reporting the bug.
Most distros want xz linked against shared liblzma, so
it doesn't help much to require --enable-dynamic for that.
Those who want to avoid PIC on x86-32 to get better
performance, can still do it e.g. by using --disable-shared
to compile xz and then another pass to compile shared liblzma.
Part of these static/dynamic tricks were needed for Windows
in the past. Nowadays we rely on GCC and binutils to do the
right thing with auto-import. If the Autotooled build system
needs to support some other toolchain on Windows in the future,
this may need some rethinking.
Testing compression at level -4 now requires 48 MiB of free store at
compression time and 5 MiB at decompression time.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
This breaks API and ABI but most apps are not affected
since most apps don't use this part of the API. You will
get a compile error if you are using anything that got
broken.
Summary of changes:
- Ability to store Stream Flags, which are needed
for random-access reading in multi-Stream files.
- Separate function to set size of Stream Padding.
- Iterator structure makes it possible to read the same
lzma_index from multiple threads at the same time.
- A lot faster code to locate Blocks.
- Removed lzma_index_equal() without adding anything
to replace it. I don't know what it should do exactly
with the new features and what actually needs this
function in the first place other than test_index.c,
which now has its own code to compare lzma_indexes.
Jia Tan
Lasse Collin
Jonathan Nieder