Using eight runners seems wasteful. Using only two runners isn't
much slower due to the runner startup overhead.
Also add a comment about the test that fails without b5a5d9e3f702.
On GitHub runners, VS 2019 16.11 (MSVC 19.29.30158) results in
test failures. VS 2022 17.13 (MSVC 19.43.34808) works.
In xz 5.6.x there was a #pragma-based workaround for MSVC builds for
32-bit x86. Another method was thought to work with the new rewritten
CLMUL CRC. Apparently it doesn't. Keep it simple and disable CLMUL CRC
with any non-recent MSVC when building for 32-bit x86.
Fixes: 54eaea5ea49b ("liblzma: x86 CLMUL CRC: Rewrite")
Fixes: https://github.com/tukaani-project/xz/issues/171
Reported-by: Andrew Murray
This makes it easy to crash fuzz_decode_stream_mt when tested
against the code from 5.8.0.
Obviously this might make it harder to reach some other code path now.
The previous code has been in use since 2018 when fuzzing was added
in 106d1a663d4b ("Tests: Add a fuzz test program and a config file
for OSS-Fuzz.").
It doesn't seem possible to trigger the CVE-2025-31115 bug with this
fuzzing target at the moment. It's because the code in fuzz_common.h
passes the whole input buffer to lzma_code() at once.
Single-shot decoding means calling lzma_code() by giving it the whole
input at once and enough output buffer space to store the uncompressed
data, and combining this with LZMA_FINISH and no timeout
(lzma_mt.timeout = 0). This way the file is decoded with a single
lzma_code() call if possible.
The bug prevented the decoder from starting more than one worker thread
in single-shot mode. The issue was noticed when reviewing the code;
there are no bug reports. Thus maybe few have tried this mode.
Fixes: 64b6d496dc81 ("liblzma: Threaded decoder: Always wait for output if LZMA_FINISH is used.")
Don't set thr->in_size = 0 when returning the thread to the stack of
available threads. Not only is it useless, but the main thread may
read the value in SEQ_BLOCK_THR_RUN. With valid inputs, it made
no difference if the main thread saw the original value or 0. With
invalid inputs (when worker thread stops early), thr->in_size was
no longer modified after the previous commit with the security fix
("Don't free the input buffer too early").
So while the bug appears harmless now, it's important to fix it because
the variable was being modified without proper locking. It's trivial
to fix because there is no need to change the value. Only main thread
needs to set the value in (in SEQ_BLOCK_THR_INIT) when starting a new
Block before the worker thread is activated.
Fixes: 4cce3e27f529 ("liblzma: Add threaded .xz decompressor.")
Reviewed-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Thanks-to: Sam James <sam@gentoo.org>
The input buffer must be valid as long as the main thread is writing
to the worker-specific input buffer. Fix it by making the worker
thread not free the buffer on errors and not return the worker thread to
the pool. The input buffer will be freed when threads_end() is called.
With invalid input, the bug could at least result in a crash. The
effects include heap use after free and writing to an address based
on the null pointer plus an offset.
The bug has been there since the first committed version of the threaded
decoder and thus affects versions from 5.3.3alpha to 5.8.0.
As the commit message in 4cce3e27f529 says, I had made significant
changes on top of Sebastian's patch. This bug was indeed introduced
by my changes; it wasn't in Sebastian's version.
Thanks to Harri K. Koskinen for discovering and reporting this issue.
Fixes: 4cce3e27f529 ("liblzma: Add threaded .xz decompressor.")
Reported-by: Harri K. Koskinen <x64nop@nannu.org>
Reviewed-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Thanks-to: Sam James <sam@gentoo.org>
The main thread can directly set THR_IDLE in threads_stop() which is
called when errors are detected. threads_stop() won't return the stopped
threads to the pool or free the memory pointed by thr->in anymore, but
it doesn't matter because the existing workers won't be reused after
an error. The resources will be cleaned up when threads_end() is
called (reinitializing the decoder always calls threads_end()).
Reviewed-by: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Thanks-to: Sam James <sam@gentoo.org>
Oracle Developer Studio 12.6 on Solaris 10 claims C11 support in
__STDC_VERSION__ and supports _Alignas. However, <stdalign.h> is missing.
We only need alignas, so define it to _Alignas with C11/C17 compilers.
If something included <stdalign.h> later, it shouldn't cause problems.
Thanks to Ihsan Dogan for reporting the issue and testing the fix.
Fixes: c0e7eaae8d6eef1e313c9d0da20ccf126ec61f38
POT-Creation-Date is set to match the timestamp in 5.7.2beta which
in the Translation Project is known as 5.8.0-pre1. The strings
haven't changed since 5.7.1alpha but a few comments have.
This is a very noisy commit, but this helps keeping the PO files
similar between the Git repository and stable release tarballs.
Also remove the trivial obsolete messages like man page dates.
This is a noisy commit, but this helps keeping the PO files similar
between the Git repository and stable release tarballs.
Names of environment variables and some other strings must be present
in the original form. The translator couldn't be reached so I'm
changing some of the strings myself. In the "Robot mode" section,
occurrences in the middle of sentences weren't changed to reduce
the chance of grammar breakage, but I kept the translated strings in
parenthesis in the headings. It's not ideal, but now people shouldn't
need to look at the English man page to find the English strings.
SSE2 is supported on every x86-64 processor. The SSE2 code is used on
32-bit x86 if compiler options permit unconditional use of SSE2.
dict_repeat() copies short random-sized unaligned buffers. At least
on glibc, FreeBSD, and Windows (MSYS2, UCRT, MSVCRT), memcpy() is
clearly faster than byte-by-byte copying in this use case. Compared
to the memcpy() version, the new SSE2 version reduces decompression
time by 0-5 % depending on the machine and libc. It should never be
slower than the memcpy() version.
However, on musl 1.2.5 on x86-64, the memcpy() version is the slowest.
Compared to the memcpy() version:
- The byte-by-version takes 6-7 % less time to decompress.
- The SSE2 version takes 16-18 % less time to decompress.
The numbers are from decompressing a Linux kernel source tarball in
single-threaded mode on older AMD and Intel systems. The tarball
compresses well, and thus dict_repeat() performance matters more
than with some other files.
This doesn't make any difference in practice because compilers can
already see that writing through the dict->buf pointer cannot modify
the contents of *dict itself: The LZMA decoder makes a local copy of
the lzma_dict structure, and even if it didn't, the pointer to
lzma_dict in the LZMA decoder is already "restrict".
It's nice to add "restrict" anyway. uint8_t is typically unsigned char
which can alias anything. Without the above conditions or "restrict",
compilers could need to assume that writing through dict->buf might
modify *dict. This would matter in dict_repeat() because the loops
refer to dict->buf and dict->pos instead of making local copies of
those members for the duration of the loops. If compilers had to
assume that writing through dict->buf can affect *dict, then compilers
would need to emit code that reloads dict->buf and dict->pos after
every write through dict->buf.
Revert back to a macro so that list(APPEND CMAKE_REQUIRED_DEFINITIONS)
will affect the calling scope. I had forgotten that while CMake
functions inherit the variables from the parent scope, the changes
to them are local unless using set(... PARENT_SCOPE).
This also means that the commit message in 5bb77d0920dc is wrong. The
commit itself is still fine, making it clearer that -DHAVE_SYS_PARAM_H
is only needed for specific check_c_source_compiles() calls.
Fixes: c1ea7bd0b60eed6ebcdf9a713ca69034f6f07179
Define NetBSD and Darwin/macOS feature test macros. Autoconf defines
these too (and a few others).
Define the macros on Windows except with MSVC. The _GNU_SOURCE macro
makes a difference with mingw-w64.
Use a function instead of a macro. Don't take the TARGET_OR_ALL argument
because there's always global effect because the global variable
CMAKE_REQUIRED_DEFINITIONS is modified.
