Don't use "" as an argument to .IP because it makes mandoc insert
an unwanted extra empty line. With the xz man page, this change makes
no difference in plain text output with GNU groff or on AIX or
Solaris 10. The list in the xzgrep man page has a different indentation
now because previously it had two arguments (.IP "" 4), but that's OK.
I don't remember why '.IP' was changed to '.IP ""' in cec0ddc8ec4c.
Maybe it made a difference on some proprietary UNIX over 15 years ago
or maybe it was just a misunderstanding.
While most ARM64 and LoongArch processors support unaligned access,
the code wouldn't have worked on strict-align processors.
Use byte-by-byte method if the input buffer is at most 7 bytes. This
way the code works on strict-align processors too. It might not be
the very best method for tiny buffers, but it shouldn't matter in
practice (the simple implementation for big buffers isn't amazing
either because it cannot execute more than one CRC32 instruction
at a time on superscalar processors).
Reported-by: Guanni Qu
Fixes: 0ed893668554 ("liblzma: ARM64 CRC32: Align the buffer faster")
Fixes: 7baf6835cfbf ("liblzma: Speed up CRC32 calculation on 64-bit LoongArch")
lzma_stream_buffer_decode() would often fail with LZMA_BUF_ERROR
when it should have failed with LZMA_DATA_ERROR.
In debug builds this also resulted in an assertion failure.
In the _buffer_decode() functions, LZMA_BUF_ERROR is only used when
the output buffer is too small. Truncated input is LZMA_DATA_ERROR.
Reported-by: Guanni Qu
Reported-by: Omkhar Arasaratnam
Fixes: c81f13ff2927 ("Added lzma_stream_buffer_decode() and made minor cleanups.")
If --verbose was used when stderr isn't a terminal, the filename
contained a non-printable character, and an error occurred,
a use-after-free occurred when printing the error message.
The bug exists in versions 5.7.1alpha to 5.8.3.
Reported-by: Omkhar Arasaratnam
Fixes: d22f96921fd2 ("xz: Use tuklib_mbstr_nonprint")
If lzma_index_decoder() was used to decode an Index that contained no
Records, the resulting lzma_index had an invalid internal "prealloc"
value. If lzma_index_append() was called on this lzma_index, too
little memory would be allocated and a buffer overflow would occur.
While this combination of the API functions is meant to work, in the
real-world apps this call sequence is rare or might not exist at all.
This bug is older than xz 5.0.0, so all stable releases are affected.
Reported-by: GitHub user christos-spearbit
This requires a filename (or something that pretends to be a filename)
of at least 2 GiB on a 32-bit platform, and that realloc() to
SIZE_MAX / 2 + 1 bytes has succeeded.
Fixes: https://github.com/tukaani-project/xz/pull/218
The __builtin_assume_aligned was applied to the wrong argument which
made aligned writes worse on strict-align archs.
Aligned writes aren't used in liblzma, so this bug didn't matter in
this package; tuklib_integer.h is meant to be usable elsewhere too.
On QNX calling fsync on a directory fails with EINVAL. This causes the
following test failure:
$ cat tests/test_suffix.sh.log
/data/home/root/xz-5.8.2/src/xz/.libs/lt-xz: suffix_temp: Synchronizing the directory of the file failed: Invalid argument
Failed to decompress a file with a suffix set in raw format
FAIL test_suffix.sh (exit status: 1)
Closes: https://github.com/tukaani-project/xz/pull/216
__cpuid from MSVC (and also the old Intel ICC) use int[], while
__get_cpuid from <cpuid.h> in GCC and Clang use unsigned int[].
Adding a cast is the simplest fix.
Link: https://github.com/tukaani-project/xz/pull/208
The "License" property is a pkgconf extension. See pc(5).
pkg-config 0.29.2 (2017) ignores unrecognized properties so this
commit shouldn't create compatibility issues.
pkgconf provides bomtool which uses the "License" property to generate
a software bill of materials. In pkgconf.git there is spdxtool for the
same task but with different output format.
The pkgconf extensions "Copyright" and "Maintainer" would also be
used by bomtool and spdxtool, but I don't want to add those properties
at least for now.
Co-authored-by: Nobuhiro Iwamatsu <iwamatsu@nigauri.org>
Closes: https://github.com/tukaani-project/xz/pull/178
This was already the case in practice because I had forgotten to list
src/xz/sandbox.c in po/POTFILES.in. However, it seems better to never
translate this particular error message. It should almost never occur
and if it does, an untranslated message is should make it easier to
find bug reports about it.
lzma_lzma_decoder_memusage() returns UINT64_MAX if lc/lp/pb aren't
valid. alone_decoder.c and lzip_decoder.c didn't check the return
value because in both it is known that lc/lp/pb are valid. Make them
call the _nocheck() variant instead which skips the validation (it
already existed for LZMA2's internal use).
Fixes: Coverity CID 464658
Fixes: Coverity CID 897069
The partial_update_mode enumeration had three states, _DISABLED,
_START, and _ENABLED. Main thread changed it from _DISABLED to _START
while holding a mutex. Once set to _START, worker thread changed it
to _ENABLED without a mutex. Later main thread read it without a mutex,
so it could see either _START or _ENABLED. However, it made no
difference because the main thread checked for != _DISABLED, so
it didn't matter if it saw _START or _ENABLED.
Nevertheless, such things must not be done. It's clear it was a mistake
because there were two comments that directly contradicted each
other about how the variable was accessed.
Split the enumeration into two booleans:
- partial_update_enabled: A worker thread locks the mutex to read
this variable and the main thread locks the mutex to change the
value. Because only the main thread modifies the variable, the
main thread can read the value without locking the mutex.
This variable replaces the _DISABLED -> _START transition.
- partial_update_started is for worker thread's internal use and thus
needs no mutex. This replaces the _START -> _ENABLED transition.
Fixes: Coverity CID 456025
Fixes: bd93b776c1bd ("liblzma: Fix a deadlock in threaded decoder.")
To make a non-threaded liblzma-only build work with WASI SDK, <signal.h>
and mythread_sigmask() were omitted from mythread.h in the commit
81db3b889830. This broke non-threaded full build with Emscripten because
src/xz/signals.c needs mythread_sigmask() (liblzma-only build was fine).
If __wasm__ is defined, omit <signal.h> and mythread_sigmask() in
non-threaded builds only when __EMSCRIPTEN__ isn't defined.
Reported-by: Marcus Tillmanns
Thanks-to: ChanTsune
Fixes: https://github.com/tukaani-project/xz/issues/161
Fixes: 81db3b889830 ("mythread.h: Disable signal functions in builds targeting Wasm + WASI.")
getauxval() can be available even if HWCAP_CRC32 isn't #defined, so
both have to be checked. HWCAP_CRC32 was added in glibc 2.24 (2016).
Fixes: https://github.com/tukaani-project/xz/issues/190
When no memory usage limits have been set by the user, the default
for multithreaded mode has been 1/4 of total RAM. If this limit is
too high and memory allocation fails, liblzma (and xz) fail. Perhaps
liblzma should handle it better by reducing the number of threads
and continuing with the amount of memory it can allocate, but currently
that isn't the case.
If resource limits were set to about 1/4 of RAM or lower, then xz
could fail for the above reason. This commit makes xz look at
RLIMIT_DATA, RLIMIT_AS, and RLIMIT_VMEM when they are available,
and set the limit 64 MiB below the lowest of those limits. This is
more or less a hack just like the 1/4-of-RAM method is, but this is
simple and quick to implement.
On Linux, there are other limits like cgroup v2 memory.max which
can still make xz fail. The same is likely possible with FreeBSD's
rctl(8).
Co-authored-by: Lasse Collin <lasse.collin@tukaani.org>
Thanks-to: Fangrui Song
Fixes: https://github.com/tukaani-project/xz/issues/195
Closes: https://github.com/tukaani-project/xz/pull/196
It feels better to fix the docs than change the code because this
way newly-written applications will be forced to be compatible with
the lzma_allocator behavior of old liblzma versions. It can matter
if someone builds the application against an older liblzma version.
Fixes: https://github.com/tukaani-project/xz/issues/183
Set also shell because the xz*.in files start with '#!@POSIX_SHELL@'.
SC1003 and SC2016 are only info messages, not warnings. Several other
shellcheck info messages remain. They are safe to ignore, but I didn't
want to disable them now.
Partially-fixes: https://github.com/tukaani-project/xz/issues/174
cap_rights_clear() with no additional arguments acts as a no-op, so
instead of removing all capability rights from STDIN_FILENO, the same
rights were allowed for STDIN_FILENO as were allowed for src_fd.
Fixes: a0eecc235d3b ("xz: Make Capsicum sandbox more strict with stdin and stdout.")
(The commit message says "stdout". It should have said "stderr".)
Don't say that the .lz format allows trailing data. According to the
lzip 1.25 manual, trailing data isn't part of the file format at all.
However, tools are still expected to behave as usefully as possible
when there is trailing data.
Fix the description of lzip >= 1.20 behavior when some of the first
bytes of trailing data match the magic bytes. While the lzip 1.25 manual
recommends that none of the first four bytes in trailing data should
match the magic bytes, the default behavior of lzip 1.25 treats
trailing data as a corrupt member header only if two or three bytes
match the magic bytes; one matching byte isn't enough.
Reported-by: Antonio Diaz Diaz
Link: https://www.mail-archive.com/xz-devel@tukaani.org/msg00702.html
On GitHub runners, VS 2019 16.11 (MSVC 19.29.30158) results in
test failures. VS 2022 17.13 (MSVC 19.43.34808) works.
In xz 5.6.x there was a #pragma-based workaround for MSVC builds for
32-bit x86. Another method was thought to work with the new rewritten
CLMUL CRC. Apparently it doesn't. Keep it simple and disable CLMUL CRC
with any non-recent MSVC when building for 32-bit x86.
Fixes: 54eaea5ea49b ("liblzma: x86 CLMUL CRC: Rewrite")
Fixes: https://github.com/tukaani-project/xz/issues/171
Reported-by: Andrew Murray
Single-shot decoding means calling lzma_code() by giving it the whole
input at once and enough output buffer space to store the uncompressed
data, and combining this with LZMA_FINISH and no timeout
(lzma_mt.timeout = 0). This way the file is decoded with a single
lzma_code() call if possible.
The bug prevented the decoder from starting more than one worker thread
in single-shot mode. The issue was noticed when reviewing the code;
there are no bug reports. Thus maybe few have tried this mode.
Fixes: 64b6d496dc81 ("liblzma: Threaded decoder: Always wait for output if LZMA_FINISH is used.")
Don't set thr->in_size = 0 when returning the thread to the stack of
available threads. Not only is it useless, but the main thread may
read the value in SEQ_BLOCK_THR_RUN. With valid inputs, it made
no difference if the main thread saw the original value or 0. With
invalid inputs (when worker thread stops early), thr->in_size was
no longer modified after the previous commit with the security fix
("Don't free the input buffer too early").
So while the bug appears harmless now, it's important to fix it because
the variable was being modified without proper locking. It's trivial
to fix because there is no need to change the value. Only main thread
needs to set the value in (in SEQ_BLOCK_THR_INIT) when starting a new
Block before the worker thread is activated.
Fixes: 4cce3e27f529 ("liblzma: Add threaded .xz decompressor.")
Reviewed-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Thanks-to: Sam James <sam@gentoo.org>
