If memory allocation fails and the resulting coder state is reused,
ensure that memory allocation is attempted again. However, coders that
are initialized via lzma_next_filter_init() shouldn't be reinitialized
after failure; they should be cleaned up with lzma_next_end().
Reported-by: GitHub user christos-cantina-security (christos-spearbit)
Worker threads update coder->progress_in while holding coder->mutex, but
the main thread updated the same field without the mutex while decoding
the next Block Header. ThreadSanitizer reported the concurrent writes and
lzma_get_progress() under-reported input progress by one Block.
All other coder->progress_in/progress_out accesses in stream_decode_mt()
are done when the worker threads aren't active, so only the access in
SEQ_BLOCK_HEADER needs to lock coder->mutex.
Co-authored-by: Lasse Collin <lasse.collin@tukaani.org>
Fixes: 4cce3e27f529 ("liblzma: Add threaded .xz decompressor.")
Closes: https://github.com/tukaani-project/xz/pull/243
It wasn't really a bug, but it's good to be more strict.
Side effects ("downsides" would be a too strong word):
- Now the LZMA decoder has to use the slow path at the end of every
LZMA2 chunk even when the read position isn't close to &in[in_size].
This doesn't matter in practice.
- If a chunk header was corrupt exactly so that the compressed size is
too small, now fewer bytes of uncompressed data can be recovered.
Reported-by: GitHub user christos-spearbit
If at least three files are passed to xz --list, the totals might exceed
the range of uint64_t. Previously wrong totals were displayed in this
case. Now an error is shown as soon as an overflow would occur, and the
remaining files aren't processed. In most other cases an error doesn't
prevent xz from processing the remaining files, but in this case it
feels OK because (1) the error should be very rare, (2) it keeps the
code slightly simpler, and (3) with xz --robot --list the caller will
likely ignore the output anyway when xz exits with a non-zero status.
Co-authored-by: Lasse Collin <lasse.collin@tukaani.org>
Fixes: https://github.com/tukaani-project/xz/pull/240
lzma_index_cat() is used by lzma_file_info_decoder() and by a few apps
that were written before lzma_file_info_decoder() existed (it was added
in 5.3.1alpha). The most common use case for these functions is xz --list.
The other use case is random access decoding.
When decoding Index fields from multi-Stream .xz files, lzma_index_cat()
is used to concatenate the decoded lzma_index structures. Such decoding
starts from the last Stream in the file and stops at the first Stream in
the file, so lzma_index_cat() is called with the most recently decoded
Stream as the destination argument while the source argument contains all
other Streams that have been decoded so far. That is, the typical use case
is lzma_index_cat(one_stream, many_streams, NULL).
With the old code, lzma_index_cat() appended Streams from src into dest
one at a time and updated their absolute offsets at the same time.
When a .xz file contained many Streams, the number of Streams in
the source argument grew for each call, which created a performance
problem if the file contained an unusually large number of Streams
(over 10,000). (A large number of Blocks isn't a problem.)
Make lzma_index_cat() prepend the index_streams from dest to src
instead of appending from src to dest. Use the _bias members to adjust
the offsets of the existing index_streams in src.
Change the numbering in index_stream.number from 1, 2, 3, ... to
...., 2, 1, 0. That is, internally the last index_stream is now 0.
Now lzma_index_cat() needs to update all Streams from dest instead from
src, which keeps the normal use case fast. Obviously the performance
problem still exists, but now it's in the opposite situation (repeated
calls of lzma_index_cat(many_streams, one_stream, NULL)). This should
be acceptable because there should be no use cases where doing it would
be useful. A full fix would require a more complex patch.
As an alternative, I tried a rewrite of index.c that replaces the trees
with plain arrays and realloc(). It's smaller and simpler, but it's
still somewhat complex because the API is too flexible compared to what
functionality is actually needed. I didn't merge that version because
a rewrite has a higher risk of regressions, and in some corner cases
the peak memory usage was considerably higher than with the old code
(if a custom lzma_allocator is used, realloc() has to be emulated with
alloc + memcpy + free, and native realloc() might need to do that too),
and large preallocations make it unclear what lzma_index_memusage() and
lzma_index_memused() should return.
Below is an example where xz --list became too slow before this commit.
Pass 18 or 20 instead of 16 to seq, and xz effectively hangs.
true | xz > big.xz
for I in $(seq 16)
do
cat big.xz big.xz > tmp.xz
mv tmp.xz big.xz
echo ====
echo $I
time xz -l big.xz
done
Reported-by: Omkhar Arasaratnam
Previously, if an Index had more than PREALLOC_MAX Records, then
allocations were done in PREALLOC_DEFAULT chunks (512 Records).
That makes little sense in practice because it's likely that one
would run out of address space later. Return LZMA_MEM_ERROR error
pre-emptively if PREALLOC_MAX would be exceeded.
lzma_index_dup() copied the per-Stream flags but not the cached bitmask
for all Streams except the last one. As a result, lzma_index_checks() on
a duplicated multi-Stream Index reported only the Check type of its
final Stream.
Fixes: https://github.com/tukaani-project/xz/pull/238
If *in_pos > 0, the condition became more relaxed than it needed to be.
Fixes: 5005914f1acf ("liblzma: Index decoder: Reject an obviously-bad Number of>
Fixes: 0352dc7929b1 ("liblzma: Silence a compiler warning")
File info decoder knows Backward Size, so pass it to the Index decoder.
Now the Index decoder, when called from the file info decoder, won't
allocate a large amount of memory when Backward Size is tiny and the
Number of Records field contains a huge invalid value.
If a reasonable memory usage limit was set, the old code returned
LZMA_MEMLIMIT_ERROR if the value in Number of Records was too huge.
Now it results in LZMA_DATA_ERROR which is better.
If lzma_stream was initialized using lzma_index_decoder() or
lzma_file_info_decoder(), and lzma_memusage() was called after a few
Records had already been decoded, the return value was too low because
it was calculated based on the number of Records that haven't been
decoded yet. So near the end of a big Index, the return value would be
tiny compared to the actual memory usage.
It worked correctly in the primary use case, that is, after
LZMA_MEMLIMIT_ERROR. Thus, in practice this bug didn't matter much.
There no point to try decoding further when the Index is clearly not
from a valid .xz file.
This also limits what index_decoder_memconfig() can return as the
memory usage. Very huge values aren't possible anymore. This affects
file_info_decoder_memconfig() in file_info.c where very huge values
aren't expected and could result in an integer overflow when not much
memory has actually been allocated. Now the overflow can only happen if
one manages to successfully allocate and fill a few exabytes of memory.
Reported-by: GitHub user jmestwa-coder
Fixes: https://github.com/tukaani-project/xz/pull/226
This is required to handle strings that may contain invalid multibyte
sequences. For example, on GNU/Linux and FreeBSD 14 with UTF-8 locale:
$ printf "\201'A\n" | sed s/./x/ | hexdump -C
00000000 81 78 41 0a |.xA.|
That is, the first byte remains untouched and the ' is replaced with x.
This kind of string would result in inbalanced quotes in xzdiff. In the
C locale it works safely:
$ printf "\201'A\n" | LC_ALL=C sed s/./x/ | hexdump -C
00000000 78 27 41 0a |x'A.|
Similar changes were made in xzgrep in 2022 (see the commits 69d1b3fc2967
and 6a4a4a7d2667). Somehow xzdiff was forgotten even though this issue
was fixed in zdiff in gzip in 2022.
Co-authored-by: rootvector2 <dxbnaveed.k@gmail.com>
Fixes: https://github.com/tukaani-project/xz/pull/228
If --files=FILELIST or --files0=FILELIST was specified via an enrivonment
variable, it worked if there were no errors when reading from the file
FILELIST. However, if an error occurred when reading from FILELIST,
there was a use-after-free bug when printing the error message because
the memory containing the string "FILELIST" had been freed.
printf foo.xz > filelist.txt
XZ_OPT=--files=filelist.txt xz -l
Because filelist.txt doesn't end in a newline, the file list is seen as
invalid/truncated, resulting in an error message like this:
xz: ????????.txt: Unexpected end of input when reading filenames
The question marks are because the garbage string is masked using
tuklib_mask_nonprint(). After this commit, it works:
xz: filelist.txt: Unexpected end of input when reading filenames
Co-authored-by: Lasse Collin <lasse.collin@tukaani.org>
Fixes: https://github.com/tukaani-project/xz/pull/223
For example, the following would run the "id" command:
xzgrep "--a='" "--b=;id;'" "pattern" foo.xz
The bug was inherited from gzip's zgrep. It was reported to bug-gzip by
Leenear. It was fixed in gzip in the commit cd74dc30f8cc by Paul Eggert.
Link: https://lists.gnu.org/archive/html/bug-gzip/2026-04/msg00032.html
This way the newer features are available on new enough kernels
even if the package is built against old <linux/landlock.h>.
Landlock ABI versions 4 and 6 would require an extended
struct landlock_ruleset_attr. No fallback is provided (or planned)
for those features.
Suggested-by: Omkhar Arasaratnam
The extra indentation makes the tables stand out from the text, and
in the English man page it works well. However, some translations
have strings that require more space, and the table might become
too wide. In some translations this has been solved by splitting
the translated string on two lines, for example, Romanian translation
of "4096-byte alignment is best".
On Debian 13 with groff 1.23.0, the "man" command shows a few warnings:
$ LANGUAGE=it MANWIDTH=80 man --warnings xz > /dev/null
<standard input>:477: warning: table wider than line length minus indentation
<standard input>:545: warning: table wider than line length minus indentation
<standard input>:1182: warning: table wider than line length minus indentation
German man page shows those warnings too. On Arch with groff 1.24.1,
there are fewer warnings because the newer groff indents less, and
fewer tables reach or exceed the right margin. Thus, some safety margin
is good to keep the output neat with different man page formatters.
It feels silly to reduce the indentation on the English man page which
has plenty of safety margin, but it's simpler than trying to do it
only for the translated man pages. This doesn't affect readability;
it's just cosmetic.
I tried using .IP instead of .RS + .PP for the tables, but on Solaris 10
it results in an extra empty line before the table. It makes some
difference on AIX 7.3 too. With groff, both are fine. Stick to
.RS + .PP + .TS ... .TE + .RE for slightly better portability.
Co-authored-by: Otto Kekäläinen <otto@debian.org>
Partially-fixes: https://github.com/tukaani-project/xz/pull/220
Don't use "" as an argument to .IP because it makes mandoc insert
an unwanted extra empty line. With the xz man page, this change makes
no difference in plain text output with GNU groff or on AIX or
Solaris 10. The list in the xzgrep man page has a different indentation
now because previously it had two arguments (.IP "" 4), but that's OK.
I don't remember why '.IP' was changed to '.IP ""' in cec0ddc8ec4c.
Maybe it made a difference on some proprietary UNIX over 15 years ago
or maybe it was just a misunderstanding.
While most ARM64 and LoongArch processors support unaligned access,
the code wouldn't have worked on strict-align processors.
Use byte-by-byte method if the input buffer is at most 7 bytes. This
way the code works on strict-align processors too. It might not be
the very best method for tiny buffers, but it shouldn't matter in
practice (the simple implementation for big buffers isn't amazing
either because it cannot execute more than one CRC32 instruction
at a time on superscalar processors).
Reported-by: Guanni Qu
Fixes: 0ed893668554 ("liblzma: ARM64 CRC32: Align the buffer faster")
Fixes: 7baf6835cfbf ("liblzma: Speed up CRC32 calculation on 64-bit LoongArch")
lzma_stream_buffer_decode() would often fail with LZMA_BUF_ERROR
when it should have failed with LZMA_DATA_ERROR.
In debug builds this also resulted in an assertion failure.
In the _buffer_decode() functions, LZMA_BUF_ERROR is only used when
the output buffer is too small. Truncated input is LZMA_DATA_ERROR.
Reported-by: Guanni Qu
Reported-by: Omkhar Arasaratnam
Fixes: c81f13ff2927 ("Added lzma_stream_buffer_decode() and made minor cleanups.")
If --verbose was used when stderr isn't a terminal, the filename
contained a non-printable character, and an error occurred,
a use-after-free occurred when printing the error message.
The bug exists in versions 5.7.1alpha to 5.8.3.
Reported-by: Omkhar Arasaratnam
Fixes: d22f96921fd2 ("xz: Use tuklib_mbstr_nonprint")
If lzma_index_decoder() was used to decode an Index that contained no
Records, the resulting lzma_index had an invalid internal "prealloc"
value. If lzma_index_append() was called on this lzma_index, too
little memory would be allocated and a buffer overflow would occur.
While this combination of the API functions is meant to work, in the
real-world apps this call sequence is rare or might not exist at all.
This bug is older than xz 5.0.0, so all stable releases are affected.
Reported-by: GitHub user christos-spearbit
This requires a filename (or something that pretends to be a filename)
of at least 2 GiB on a 32-bit platform, and that realloc() to
SIZE_MAX / 2 + 1 bytes has succeeded.
Fixes: https://github.com/tukaani-project/xz/pull/218
The __builtin_assume_aligned was applied to the wrong argument which
made aligned writes worse on strict-align archs.
Aligned writes aren't used in liblzma, so this bug didn't matter in
this package; tuklib_integer.h is meant to be usable elsewhere too.
On QNX calling fsync on a directory fails with EINVAL. This causes the
following test failure:
$ cat tests/test_suffix.sh.log
/data/home/root/xz-5.8.2/src/xz/.libs/lt-xz: suffix_temp: Synchronizing the directory of the file failed: Invalid argument
Failed to decompress a file with a suffix set in raw format
FAIL test_suffix.sh (exit status: 1)
Closes: https://github.com/tukaani-project/xz/pull/216
__cpuid from MSVC (and also the old Intel ICC) use int[], while
__get_cpuid from <cpuid.h> in GCC and Clang use unsigned int[].
Adding a cast is the simplest fix.
Link: https://github.com/tukaani-project/xz/pull/208
The "License" property is a pkgconf extension. See pc(5).
pkg-config 0.29.2 (2017) ignores unrecognized properties so this
commit shouldn't create compatibility issues.
pkgconf provides bomtool which uses the "License" property to generate
a software bill of materials. In pkgconf.git there is spdxtool for the
same task but with different output format.
The pkgconf extensions "Copyright" and "Maintainer" would also be
used by bomtool and spdxtool, but I don't want to add those properties
at least for now.
Co-authored-by: Nobuhiro Iwamatsu <iwamatsu@nigauri.org>
Closes: https://github.com/tukaani-project/xz/pull/178
This was already the case in practice because I had forgotten to list
src/xz/sandbox.c in po/POTFILES.in. However, it seems better to never
translate this particular error message. It should almost never occur
and if it does, an untranslated message is should make it easier to
find bug reports about it.
