This is *NOT* done for security reasons even though the backdoor
relied on the ifunc code. Instead, the reason is that in this
project ifunc provides little benefits but it's quite a bit of
extra code to support it. The only case where ifunc *might* matter
for performance is if the CRC functions are used directly by an
application. In normal compression use it's completely irrelevant.
(cherry picked from commit 689ae24273)
This does the previous commit with CMake.
AC_EGREP_CPP uses AC_REQUIRE so the outermost if-commands must
be changed to AS_IF to ensure that things wont break some day.
See 5a5bd7f871.
(cherry picked from commit 49324b711f)
There are cases when the users want to decide themselves whether
they want to have the generic (even on GNU/Linux) or the linux
(even if we do not recommend that) symbol versioning variant.
The former might be needed to circumvent compiler issues (i.e.
the compiler does not support all features that are required
for the linux versioning), the latter might help in overriding
the assumptions made in the configure script.
(cherry picked from commit f56ed6fac6)
Using __attribute__((__no_profile_instrument_function__)) on the ifunc
resolver works around a bug in GCC -fprofile-generate:
it adds profiling code even to ifunc resolvers which can make
the ifunc resolver crash at program startup. This attribute
was not introduced until GCC 7 and Clang 13, so ifunc won't
be used with prior versions of these compilers.
This bug was brought to our attention by:
https://bugs.gentoo.org/925415
And was reported to upstream GCC by:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=11411
The previous Linux Landlock feature test assumed that having the
linux/landlock.h header file was enough. The new feature tests also
requires that prctl() and the required Landlock system calls are
supported.
Old versions of Clang reported the unsupported function attribute and
__crc32d() function as warnings instead of errors, so the feature test
passed when it shouldn't have, causing a compile error at build time.
-Werror was added to this feature test to fix this. The change is not
needed for CMake because check_c_source_compiles() also performs
linking and the error is caught then.
Thanks to Sebastian Andrzej Siewior for reporting this.
This adds --enable-arm64-crc32/--disable-arm64-crc32 (enabled by
default) for using the ARM64 CRC32 instruction. This can be disabled if
one knows the binary will never need to run on an ARM64 machine
with this instruction extension.
The new Filter ID is 0x0B.
Thanks to Chien Wong <m@xv97.com> for the initial version of the Filter,
the xz CLI updates, and the Autotools build system modifications.
Thanks to Igor Pavlov for his many contributions to the design of
the filter.
A CLMUL-only build will have the crcxx_clmul() inlined into
lzma_crcxx(). Previously a jump to the extern lzma_crcxx_clmul()
was needed. Notes about shared liblzma on ELF platforms:
- On platforms that support ifunc and -fvisibility=hidden, this
was silly because CLMUL-only build would have that single extra
jump instruction of extra overhead.
- On platforms that support neither -fvisibility=hidden nor linker
version script (liblzma*.map), jumping to lzma_crcxx_clmul()
would go via PLT so a few more instructions of overhead (still
not a big issue but silly nevertheless).
There was a downside with static liblzma too: if an application only
needs lzma_crc64(), static linking would make the linker include the
CLMUL code for both CRC32 and CRC64 from crc_x86_clmul.o even though
the CRC32 code wouldn't be needed, thus increasing code size of the
executable (assuming that -ffunction-sections isn't used).
Also, now compilers are likely to inline crc_simd_body()
even if they don't support the always_inline attribute
(or MSVC's __forceinline). Quite possibly all compilers
that build the code do support such an attribute. But now
it likely isn't a problem even if the attribute wasn't supported.
Now all x86-specific stuff is in crc_x86_clmul.h. If other archs
The other archs can then have their own headers with their own
is_clmul_supported() and crcxx_clmul().
Another bonus is that the build system doesn't need to care if
crc_clmul.c is needed.
is_clmul_supported() stays as inline function as it's not needed
when doing a CLMUL-only build (avoids a warning about unused function).
Some compilers support __attribute__((__ifunc__())) even though the
dynamic linker does not. The compiler is able to create the binary
but it will fail on startup. So it is not enough to just test if
the attribute is supported.
The default value for enable_ifunc is now auto, which will attempt
to compile a program using __attribute__((__ifunc__())). There are
additional checks in this program if glibc is being used or if it
is running on FreeBSD.
Setting --enable-ifunc will skip this test and always enable
__attribute__((__ifunc__())), even if is not supported.
Now configure will fail if -fsanitize= is found in CFLAGS
and sanitizer-incompatible ifunc or Landlock sandboxing
would be used. These are incompatible with one or more sanitizers.
It's simpler to reject all -fsanitize= uses instead of trying to
pass those that might not cause problems.
CMake-based build was updated similarly. It lets the configuration
finish (SEND_ERROR instead of FATAL_ERROR) so that both error
messages can be seen at once.
This removes support for FreeBSD 10.0 and 10.1 which used
<sys/capability.h> instead of <sys/capsicum.h>. Support for
FreeBSD 10.1 ended on 2016-12-31. So now FreeBSD >= 10.2 is
required to enable Capsicum support.
This also removes support for Capsicum on Linux (libcaprights)
which seems to have been unmaintained since 2017 and Linux 4.11:
https://github.com/google/capsicum-linux
See the new comment in the code.
This also makes the check for clock_gettime() run with MinGW-w64
with which we don't want to use clock_gettime(). The previous
commit already took care of this situation.
Both crc32_clmul() and crc64_clmul() are now exported from
crc32_clmul.c as lzma_crc32_clmul() and lzma_crc64_clmul(). This
ensures that is_clmul_supported() (now lzma_is_clmul_supported()) is
not duplicated between crc32_fast.c and crc64_fast.c.
Also, it encapsulates the complexity of the CLMUL implementations into a
single file and reduces the complexity of crc32_fast.c and crc64_fast.c.
Before, CLMUL code was present in crc32_fast.c, crc64_fast.c, and
crc_common.h.
During the conversion, various cleanups were applied to code (thanks to
Lasse Collin) including:
- Require using semicolons with MASK_/L/H/LH macros.
- Variable typing and const handling improvements.
- Improvements to comments.
- Fixes to the pragmas used.
- Removed unneeded variables.
- Whitespace improvements.
- Fixed CRC_USE_GENERIC_FOR_SMALL_INPUTS handling.
- Silenced warnings and removed the need for some #pragmas
Now if user-supplied CFLAGS contains -Wall -Wextra -Wpedantic
the two checks that need -Werror will still work.
At CMake side there is add_compile_options(-Wall -Wextra)
but it didn't affect the -Werror tests. So with both Autotools
and CMake only user-supplied CFLAGS could make the checks fail
when they shouldn't.
This is not a full fix as things like -Wunused-macros in
user-supplied CFLAGS will still cause problems with both
GCC and Clang.
There were two uses of AC_COMPILE_IFELSE that didn't use
AC_LANG_SOURCE and Autoconf warned about these. The omission
had been intentional but it turned out that this didn't do
what I thought it would.
Autoconf 2.71 manual gives an impression that AC_LANG_SOURCE
inserts all #defines that have been made with AC_DEFINE so
far (confdefs.h). The idea was that omitting AC_LANG_SOURCE
would mean that only the exact code included in the
AC_COMPILE_IFELSE call would be compiled.
With C programs this is not true: the #defines get added without
AC_LANG_SOURCE too. There seems to be no neat way to avoid this.
Thus, with the C language at least, adding AC_LANG_SOURCE makes
no other difference than silencing a warning from Autoconf. The
generated "configure" remains identical. (Docs of AC_LANG_CONFTEST
say that the #defines have been inserted since Autoconf 2.63b and
that AC_COMPILE_IFELSE uses AC_LANG_CONFTEST. So the behavior is
documented if one also reads the docs of macros that one isn't
calling directly.)
Any extra code, including #defines, can cause problems for
these two tests because these tests must use -Werror.
CC=clang CFLAGS=-Weverything is the most extreme example.
It enables -Wreserved-macro-identifier which warns about
#define __EXTENSIONS__ 1 because it begins with two underscores.
It's possible to write a test file that passes -Weverything but
it becomes impossible when Autoconf inserts confdefs.h.
So this commit adds AC_LANG_SOURCE to silence Autoconf warnings.
A different solution is needed for -Werror tests.
When the compiler supports __attribute__((__constructor__))
mythread_once() is never used, even with --enable-small. A configuration
with win95 threads and --enable-small will compile and be thread safe so
it can be allowed.
This isn't a very common configuration since MSVC does not support
__attribute__((__constructor__)), but MINGW32 and CLANG32 environments
for MSYS2 can use win95 threads and have
__attribute__((__constructor__)) support.
This makes no functional difference in the generated configure
(at least with the Autotools versions I have installed) but this
change might prevent future bugs like the one that was just
fixed in the commit 5a5bd7f871.
This is broken in the releases 5.2.6 to 5.4.2. A workaround
for these releases is to pass EGREP='grep -E' as an argument
to configure in addition to --disable-threads.
The problem appeared when m4/ax_pthread.m4 was updated in
the commit 6629ed929c which
introduced the use of AC_EGREP_CPP. AC_EGREP_CPP calls
AC_REQUIRE([AC_PROG_EGREP]) to set the shell variable EGREP
but this was only executed if POSIX threads were enabled.
Libtool code also has AC_REQUIRE([AC_PROG_EGREP]) but Autoconf
omits it as AC_PROG_EGREP has already been required earlier.
Thus, if not using POSIX threads, the shell variable EGREP
would be undefined in the Libtool code in configure.
ax_pthread.m4 is fine. The bug was in configure.ac which called
AX_PTHREAD conditionally in an incorrect way. Using AS_CASE
ensures that all AC_REQUIREs get always run.
Thanks to Frank Busse for reporting the bug.
Fixes: https://github.com/tukaani-project/xz/issues/45
Instead of having Doxyfile.in configured by Autoconf, the Doxyfile
can have the tags that need to be configured piped into the doxygen
command through stdin with the overrides after Doxyfile's contents.
Going forward, the documentation should be generated in two different
modes: liblzma or internal.
liblzma is useful for most users. It is the documentation for just
the liblzma API header files. This is the default.
internal is for people who want to understand how xz and liblzma work.
It might be useful for people who want to contribute to the project.
On MicroBlaze, GCC 12 is broken in sense that
__has_attribute(__symver__) returns true but it still doesn't
support the __symver__ attribute even though the platform is ELF
and symbol versioning is supported if using the traditional
__asm__(".symver ...") method. Avoiding the traditional method is
good because it breaks LTO (-flto) builds with GCC.
See also: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101766
For now the only extra symbols in liblzma_linux.map are the
compatibility symbols with the patch that spread from RHEL/CentOS 7.
These require the use of __symver__ attribute or __asm__(".symver ...")
in the C code. Compatibility with the patch from CentOS 7 doesn't
seem valuable on MicroBlaze so use liblzma_generic.map on MicroBlaze
instead. It doesn't require anything special in the C code and thus
no LTO issues either.
An alternative would be to detect support for __symver__
attribute in configure.ac and CMakeLists.txt and fall back
to __asm__(".symver ...") but then LTO would be silently broken
on MicroBlaze. It sounds likely that MicroBlaze is a special
case so let's treat it as a such because that is simpler. If
a similar issue exists on some other platform too then hopefully
someone will report it and this can be reconsidered.
(This doesn't do the same fix in CMakeLists.txt. Perhaps it should
but perhaps CMake build of liblzma doesn't matter much on MicroBlaze.
The problem breaks the build so it's easy to notice and can be fixed
later.)
Thanks to Vincent Fazio for reporting the problem and proposing
a patch (in the end that solution wasn't used):
https://github.com/tukaani-project/xz/pull/32
Doxygen is now configurable in autotools only with
--enable-doxygen=[api|all]. The default is "api", which will only
generate HTML output for liblzma API functions. The LaTex documentation
output was also disabled.
It's not that important. It can be annoying in builds that
disable many features since in those cases the tests programs
will correctly trigger this warning with Clang.
-Wstrict-aliasing was removed from the list since it is enabled
by -Wall already.
A normal build is clean with these on GNU/Linux x86-64 with
GCC 12.2.0 and Clang 14.0.6.
Previously, if threading was enabled HAVE_DECL_CLOCK_MONOTONIC would always
be set to 0 or 1. However, this macro was needed in xz so if xz was not
built with threading and HAVE_DECL_CLOCK_MONOTONIC was not defined but
HAVE_CLOCK_GETTIME was, it caused a warning during build. Now,
HAVE_DECL_CLOCK_MONOTONIC has been renamed to HAVE_CLOCK_MONOTONIC and
will only be set if it is 1.
It not only makes no sense to put symbol versions into a static library
but it can also cause breakage.
By default Libtool #defines PIC if building a shared library and
doesn't define it for static libraries. This is documented in the
Libtool manual. It can be overriden using --with-pic or --without-pic.
configure.ac detects if --with-pic or --without-pic is used and then
gives an error if neither --disable-shared nor --disable-static was
used at the same time. Thus, in normal situations it works to build
both shared and static library at the same time on GNU/Linux,
only --with-pic or --without-pic requires that only one type of
library is built.
Thanks to John Paul Adrian Glaubitz from Debian for reporting
the problem that occurred on ia64:
https://www.mail-archive.com/xz-devel@tukaani.org/msg00610.html
It also works on E2K as it supports these intrinsics.
On x86-64 runtime detection is used so the code keeps working on
older processors too. A CLMUL-only build can be done by using
-msse4.1 -mpclmul in CFLAGS and this will reduce the library
size since the generic implementation and its 8 KiB lookup table
will be omitted.
On 32-bit x86 this isn't used by default for now because by default
on 32-bit x86 the separate assembly file crc64_x86.S is used.
If --disable-assembler is used then this new CLMUL code is used
the same way as on 64-bit x86. However, a CLMUL-only build
(-msse4.1 -mpclmul) won't omit the 8 KiB lookup table on
32-bit x86 due to a currently-missing check for disabled
assembler usage.
The configure.ac check should be such that the code won't be
built if something in the toolchain doesn't support it but
--disable-clmul-crc option can be used to unconditionally
disable this feature.
CLMUL speeds up decompression of files that have compressed very
well (assuming CRC64 is used as a check type). It is know that
the CLMUL code is significantly slower than the generic code for
tiny inputs (especially 1-8 bytes but up to 16 bytes). If that
is a real-world problem then there is already a commented-out
variant that uses the generic version for small inputs.
Thanks to Ilya Kurdyukov for the original patch which was
derived from a white paper from Intel [1] (published in 2009)
and public domain code from [2] (released in 2016).
[1] https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
[2] https://github.com/rawrunprotected/crc
Lasse Collin
Sergey Kosukhin
Jia Tan
Dimitri Papadopoulos Orfanos
Hans Jansen