Commit Graph

190 Commits

Author SHA1 Message Date
Lasse Collin bf901dee5d Build: Comment that elf_aux_info(3) will be available on OpenBSD >= 7.6 2024-07-19 20:06:24 +03:00
Lasse Collin f7103c2c2a Revert "liblzma: Add ARM64 CRC32 instruction support detection on OpenBSD"
This reverts commit dc03f6290f.

OpenBSD 7.6 will support elf_aux_info(3), and the detection code used
on FreeBSD will work on OpenBSD 7.6 too. Keep things simpler and drop
the OpenBSD-specific sysctl() method.

Thanks to Christian Weisgerber.
2024-07-19 20:06:24 +03:00
Xi Ruoyao 7baf6835cf liblzma: Speed up CRC32 calculation on 64-bit LoongArch
The crc.w.{b/h/w/d}.w instructions in LoongArch can calculate the CRC32
result for 1/2/4/8 bytes in a single operation. Using these is much
faster compared to the generic method.

Optimized CRC32 is enabled unconditionally on 64-bit LoongArch because
the LoongArch specification says that CRC32 instructions shall be
implemented for 64-bit processors. Optimized CRC32 isn't enabled for
32-bit LoongArch processors because not enough information is available
about them.

Co-authored-by: Lasse Collin <lasse.collin@tukaani.org>

Closes: https://github.com/tukaani-project/xz/pull/86
2024-07-01 17:09:57 +03:00
Lasse Collin 2402e8a1ae Build: Prepend, not append, PTHREAD_CFLAGS to LIBS
It shouldn't make any difference because LIBS should be empty
at that point in configure. But prepending is the correct way
because in general the libraries being added might require other
libraries that come later on the command line.
2024-06-26 15:48:56 +03:00
Lasse Collin 7bb46f2b7b Build: Use AC_LINK_IFELSE to handle implicit function declarations
It's more robust in case the compiler allows pre-C99 implicit function
declarations. If an x86 intrinsic is missing and gets treated as
implicit function, the linking step will very probably fail. This
isn't the only way to workaround implicit function declarations but
it might be the simplest and cleanest.

The problem hasn't been observed in the wild.

There are a couple more AC_COMPILE_IFELSE uses in configure.ac.
Of these, Landlock check calls prctl() and in theory could have
the same problem. In practice it doesn't as the check program
looks for several other things too. However, it was changed to
AC_LINK_IFELSE still to look more correct.

Similarly, m4/tuklib_cpucores.m4 and m4/tuklib_physmem.m4 were
updated although they haven't given any trouble either. They
have worked all these years because those check programs rely
on specific headers and types: if headers or types are missing,
compilation will fail. Using the linker makes these checks more
similar to the ones in cmake/tuklib_*.cmake which always link.
2024-06-26 15:48:56 +03:00
Lasse Collin 35eb57355a Build: Use AC_LINK_IFELSE instead of -Werror
AC_COMPILE_IFELSE needed -Werror because Clang <= 14 would merely
warn about the unsupported attribute and implicit function declaration.
Changing to AC_LINK_IFELSE handles the implicit declaration because
the symbol __crc32d is unlikely to exist in libc.

Note that the other part of the check is that #include <arm_acle.h>
must work. If the header is missing, most compilers give an error
and the linking step won't be attempted.

Avoiding -Werror makes the check more robust in case CFLAGS contains
warning flags that break -Werror anyway (but this isn't the only check
in configure.ac that has this problem). Using AC_LINK_IFELSE also makes
the check more similar to how it is done in CMakeLists.txt.
2024-06-26 15:48:56 +03:00
Lasse Collin 5a728813c3 Build: Sync the compile check changes from CMakeLists.txt
It's nice to keep these in sync. The use of main() will later allow
AC_LINK_IFELSE usage too which may avoid the more fragile -Werror.
2024-06-26 15:48:56 +03:00
Lasse Collin f76837acb6 Build: Define HAVE_CRC_X86_ASM when 32-bit x86 CRC assembly is used
This makes it easier to determine when the CRC tables are needed.
2024-06-23 14:36:44 +03:00
Lasse Collin ead4d15199 Revert "Build: Temporarily disable CRC CLMUL to silence OSS Fuzz"
This reverts commit 9f1a6d6f9a.
2024-06-16 12:56:54 +03:00
Lasse Collin dc03f6290f liblzma: Add ARM64 CRC32 instruction support detection on OpenBSD
The C code is from Christian Weisgerber, I merely reordered the OSes.
Then I added the build system checks without testing them.

Also thanks to Brad Smith who submitted a similar patch on GitHub
a few hours after Christian had sent his via email.

Co-authored-by: Christian Weisgerber <naddy@mips.inka.de>
Closes: https://github.com/tukaani-project/xz/pull/125
2024-06-07 15:06:59 +03:00
Lasse Collin 9f1a6d6f9a Build: Temporarily disable CRC CLMUL to silence OSS Fuzz
The code makes aligned 16-byte reads which may read up to 15 bytes
before the beginning or past the end of the buffer if the buffer
is misaligned. The unneeded bytes are then ignored. It cannot cross
page boundaries and thus cannot cause access violations.

This inherently trips address sanitizer which was already disabled
with __attribute__((__no_sanitize_address__)). However, it also
trips memory sanitizer if the extra bytes are uninitialized because
memory sanitizer doesn't see that those bytes then get ignored by
byte shuffling in the xmm registers.

The plan is to change the code so that all sanitizers pass but it's
not finished yet (performance shouldn't get worse) so as a temporary
measure to keep OSS Fuzz happy, the CLMUL CRC is now disabled even
though I think think the code is fine to use (and easy enough to review
the memory accesses in it too).
2024-05-15 23:14:17 +03:00
Lasse Collin e21efdf96f Build: Add --enable-doxygen to generate and install API docs
It requires Doxygen. This option is disabled by default.
2024-04-30 17:09:08 +03:00
Lasse Collin fc1921b04b Build: Omit useless checks for fcntl.h, limits.h, and sys/time.h 2024-04-21 20:27:50 +03:00
Lasse Collin 308a9af854 Build: If ARM64 feature detection func is found, stop looking for others
This can speed up configure a tiny bit.

Fixes: c5f6d79cc9
2024-04-10 23:12:23 +03:00
Lasse Collin 689ae24273 liblzma: Remove ifunc support.
This is *NOT* done for security reasons even though the backdoor
relied on the ifunc code. Instead, the reason is that in this
project ifunc provides little benefits but it's quite a bit of
extra code to support it. The only case where ifunc *might* matter
for performance is if the CRC functions are used directly by an
application. In normal compression use it's completely irrelevant.
2024-04-09 18:22:27 +03:00
Lasse Collin 17aa2e1a79 Update website URLs back to tukaani.org.
The XZ projects were moved back to their original URLs.
2024-04-09 18:22:27 +03:00
Lasse Collin 49324b711f Build: Use only the generic symbol versioning with NVIDIA HPC Compiler.
This does the previous commit with CMake.

AC_EGREP_CPP uses AC_REQUIRE so the outermost if-commands must
be changed to AS_IF to ensure that things wont break some day.
See 5a5bd7f871.
2024-03-15 18:04:10 +02:00
Lasse Collin 45d33bfc45 Build: Style tweaks to configure.ac.
The AC_MSG_ERROR line is overlong anyway as are a few other
AC_MSG_ERROR lines already.
2024-03-13 22:00:23 +02:00
Sergey Kosukhin f56ed6fac6 Build: Let the users override the symbol versioning variant.
There are cases when the users want to decide themselves whether
they want to have the generic (even on GNU/Linux) or the linux
(even if we do not recommend that) symbol versioning variant.
The former might be needed to circumvent compiler issues (i.e.
the compiler does not support all features that are required
for the linux versioning), the latter might help in overriding
the assumptions made in the configure script.
2024-03-13 21:59:55 +02:00
Jia Tan b93a8d7631 Tests: Replace HAVE_MICROLZMA usage in CMake and Autotools builds.
This reverts commit adaacafde6.
2024-03-09 09:49:55 +08:00
Jia Tan e5faaebbcf Build: Require attribute no_profile_instrument_function for ifunc usage.
Using __attribute__((__no_profile_instrument_function__)) on the ifunc
resolver works around a bug in GCC -fprofile-generate:
it adds profiling code even to ifunc resolvers which can make
the ifunc resolver crash at program startup. This attribute
was not introduced until GCC 7 and Clang 13, so ifunc won't
be used with prior versions of these compilers.

This bug was brought to our attention by:

    https://bugs.gentoo.org/925415

And was reported to upstream GCC by:

    https://gcc.gnu.org/bugzilla/show_bug.cgi?id=11411
2024-03-05 01:54:30 +08:00
Jia Tan 328c52da8a Build: Fix Linux Landlock feature test in Autotools and CMake builds.
The previous Linux Landlock feature test assumed that having the
linux/landlock.h header file was enough. The new feature tests also
requires that prctl() and the required Landlock system calls are
supported.
2024-02-26 23:27:44 +08:00
Jia Tan adaacafde6 Build: Define HAVE_MICROLZMA when it is configured. 2024-02-23 20:57:59 +08:00
Jia Tan 32b0a3ce19 Build: Fix ARM64 CRC32 instruction feature test.
Old versions of Clang reported the unsupported function attribute and
__crc32d() function as warnings instead of errors, so the feature test
passed when it shouldn't have, causing a compile error at build time.
-Werror was added to this feature test to fix this. The change is not
needed for CMake because check_c_source_compiles() also performs
linking and the error is caught then.

Thanks to Sebastian Andrzej Siewior for reporting this.
2024-02-23 16:12:32 +08:00
Lasse Collin 22af94128b Add SPDX license identifier into 0BSD source code files. 2024-02-14 18:31:16 +02:00
Lasse Collin 689e0228ba Change most public domain parts to 0BSD.
Translations and doc/xz-file-format.txt and doc/lzma-file-format.txt
were not touched.

COPYING.0BSD was added.
2024-02-14 18:31:12 +02:00
Jia Tan c5f6d79cc9 Build: Add support for ARM64 CRC32 instruction detection.
This adds --enable-arm64-crc32/--disable-arm64-crc32 (enabled by
default) for using the ARM64 CRC32 instruction. This can be disabled if
one knows the binary will never need to run on an ARM64 machine
with this instruction extension.
2024-02-01 20:09:09 +08:00
Jia Tan 440a2eccb0 liblzma: Add RISC-V BCJ filter.
The new Filter ID is 0x0B.

Thanks to Chien Wong <m@xv97.com> for the initial version of the Filter,
the xz CLI updates, and the Autotools build system modifications.

Thanks to Igor Pavlov for his many contributions to the design of
the filter.
2024-01-23 23:05:41 +08:00
Jia Tan c26812c5b2 Build: Update website URL. 2024-01-19 23:08:14 +08:00
Lasse Collin 419f55f9df liblzma: Avoid extern lzma_crc32_clmul() and lzma_crc64_clmul().
A CLMUL-only build will have the crcxx_clmul() inlined into
lzma_crcxx(). Previously a jump to the extern lzma_crcxx_clmul()
was needed. Notes about shared liblzma on ELF platforms:

  - On platforms that support ifunc and -fvisibility=hidden, this
    was silly because CLMUL-only build would have that single extra
    jump instruction of extra overhead.

  - On platforms that support neither -fvisibility=hidden nor linker
    version script (liblzma*.map), jumping to lzma_crcxx_clmul()
    would go via PLT so a few more instructions of overhead (still
    not a big issue but silly nevertheless).

There was a downside with static liblzma too: if an application only
needs lzma_crc64(), static linking would make the linker include the
CLMUL code for both CRC32 and CRC64 from crc_x86_clmul.o even though
the CRC32 code wouldn't be needed, thus increasing code size of the
executable (assuming that -ffunction-sections isn't used).

Also, now compilers are likely to inline crc_simd_body()
even if they don't support the always_inline attribute
(or MSVC's __forceinline). Quite possibly all compilers
that build the code do support such an attribute. But now
it likely isn't a problem even if the attribute wasn't supported.

Now all x86-specific stuff is in crc_x86_clmul.h. If other archs
The other archs can then have their own headers with their own
is_clmul_supported() and crcxx_clmul().

Another bonus is that the build system doesn't need to care if
crc_clmul.c is needed.

is_clmul_supported() stays as inline function as it's not needed
when doing a CLMUL-only build (avoids a warning about unused function).
2024-01-11 14:29:42 +02:00
Jia Tan 5feb09266f Build: Allow sandbox to be configured for just xzdec.
If xz is disabled, then xzdec can still use the sandbox.
2023-12-20 22:43:44 +08:00
Jia Tan ffb456593d Build: Change --enable-ifunc handling.
Some compilers support __attribute__((__ifunc__())) even though the
dynamic linker does not. The compiler is able to create the binary
but it will fail on startup. So it is not enough to just test if
the attribute is supported.

The default value for enable_ifunc is now auto, which will attempt
to compile a program using __attribute__((__ifunc__())). There are
additional checks in this program if glibc is being used or if it
is running on FreeBSD.

Setting --enable-ifunc will skip this test and always enable
__attribute__((__ifunc__())), even if is not supported.
2023-11-30 20:04:42 +08:00
Jia Tan 148e20607e Build: Fix text wrapping in an output message. 2023-10-31 21:54:11 +08:00
Lasse Collin 88588b1246 Build: Detect -fsanitize= in CFLAGS and incompatible build options.
Now configure will fail if -fsanitize= is found in CFLAGS
and sanitizer-incompatible ifunc or Landlock sandboxing
would be used. These are incompatible with one or more sanitizers.
It's simpler to reject all -fsanitize= uses instead of trying to
pass those that might not cause problems.

CMake-based build was updated similarly. It lets the configuration
finish (SEND_ERROR instead of FATAL_ERROR) so that both error
messages can be seen at once.
2023-10-25 20:18:04 +03:00
Lasse Collin 8276c7f41c xz: Support basic sandboxing with Linux Landlock (ABI versions 1-3).
It is enabled only when decompressing one file to stdout,
similar to how Capsicum is used.

Landlock was added in Linux 5.13.
2023-10-22 19:03:52 +03:00
Lasse Collin 2e2cd11535 Simplify detection of Capsicum support.
This removes support for FreeBSD 10.0 and 10.1 which used
<sys/capability.h> instead of <sys/capsicum.h>. Support for
FreeBSD 10.1 ended on 2016-12-31. So now FreeBSD >= 10.2 is
required to enable Capsicum support.

This also removes support for Capsicum on Linux (libcaprights)
which seems to have been unmaintained since 2017 and Linux 4.11:
https://github.com/google/capsicum-linux
2023-10-22 19:03:52 +03:00
Lasse Collin 1c1a8c3ee4 Build: Check for clock_gettime() even if not using POSIX threads.
See the new comment in the code.

This also makes the check for clock_gettime() run with MinGW-w64
with which we don't want to use clock_gettime(). The previous
commit already took care of this situation.
2023-10-22 18:59:45 +03:00
Jia Tan 8c0f9376f5 liblzma: Create crc_clmul.c.
Both crc32_clmul() and crc64_clmul() are now exported from
crc32_clmul.c as lzma_crc32_clmul() and lzma_crc64_clmul(). This
ensures that is_clmul_supported() (now lzma_is_clmul_supported()) is
not duplicated between crc32_fast.c and crc64_fast.c.

Also, it encapsulates the complexity of the CLMUL implementations into a
single file and reduces the complexity of crc32_fast.c and crc64_fast.c.
Before, CLMUL code was present in crc32_fast.c, crc64_fast.c, and
crc_common.h.

During the conversion, various cleanups were applied to code (thanks to
Lasse Collin) including:

- Require using semicolons with MASK_/L/H/LH macros.
- Variable typing and const handling improvements.
- Improvements to comments.
- Fixes to the pragmas used.
- Removed unneeded variables.
- Whitespace improvements.
- Fixed CRC_USE_GENERIC_FOR_SMALL_INPUTS handling.
- Silenced warnings and removed the need for some #pragmas
2023-10-18 23:54:36 +08:00
Lasse Collin 092d21db2e Build: Update the comment about -Werror usage in checks. 2023-09-26 17:24:15 +03:00
Lasse Collin a37a276338 Build: Fix __attribute__((ifunc(...))) detection with clang -Wall.
Now if user-supplied CFLAGS contains -Wall -Wextra -Wpedantic
the two checks that need -Werror will still work.

At CMake side there is add_compile_options(-Wall -Wextra)
but it didn't affect the -Werror tests. So with both Autotools
and CMake only user-supplied CFLAGS could make the checks fail
when they shouldn't.

This is not a full fix as things like -Wunused-macros in
user-supplied CFLAGS will still cause problems with both
GCC and Clang.
2023-09-26 15:00:43 +03:00
Lasse Collin 9c42f93693 Build: Fix underquoted AC_LANG_SOURCE.
It made no practical difference in this case.
2023-09-26 13:51:31 +03:00
Lasse Collin 9f1444a8a5 Build: Silence two Autoconf warnings.
There were two uses of AC_COMPILE_IFELSE that didn't use
AC_LANG_SOURCE and Autoconf warned about these. The omission
had been intentional but it turned out that this didn't do
what I thought it would.

Autoconf 2.71 manual gives an impression that AC_LANG_SOURCE
inserts all #defines that have been made with AC_DEFINE so
far (confdefs.h). The idea was that omitting AC_LANG_SOURCE
would mean that only the exact code included in the
AC_COMPILE_IFELSE call would be compiled.

With C programs this is not true: the #defines get added without
AC_LANG_SOURCE too. There seems to be no neat way to avoid this.
Thus, with the C language at least, adding AC_LANG_SOURCE makes
no other difference than silencing a warning from Autoconf. The
generated "configure" remains identical. (Docs of AC_LANG_CONFTEST
say that the #defines have been inserted since Autoconf 2.63b and
that AC_COMPILE_IFELSE uses AC_LANG_CONFTEST. So the behavior is
documented if one also reads the docs of macros that one isn't
calling directly.)

Any extra code, including #defines, can cause problems for
these two tests because these tests must use -Werror.
CC=clang CFLAGS=-Weverything is the most extreme example.
It enables -Wreserved-macro-identifier which warns about
#define __EXTENSIONS__ 1 because it begins with two underscores.
It's possible to write a test file that passes -Weverything but
it becomes impossible when Autoconf inserts confdefs.h.

So this commit adds AC_LANG_SOURCE to silence Autoconf warnings.
A different solution is needed for -Werror tests.
2023-09-26 13:50:19 +03:00
Jia Tan be012b8097 Build: Change quoting style from `...' to '...'. 2023-09-24 22:09:16 +08:00
Lasse Collin 74b0e900c9 Build: Omit -Wc99-c11-compat since it warns about _Noreturn. 2023-09-22 20:06:26 +03:00
Jia Tan de574404c4 Build: Conditionally allow win95 threads and --enable-small.
When the compiler supports __attribute__((__constructor__))
mythread_once() is never used, even with --enable-small. A configuration
with win95 threads and --enable-small will compile and be thread safe so
it can be allowed.

This isn't a very common configuration since MSVC does not support
__attribute__((__constructor__)), but MINGW32 and CLANG32 environments
for MSYS2 can use win95 threads and have
__attribute__((__constructor__)) support.
2023-08-09 20:35:16 +08:00
Dimitri Papadopoulos Orfanos 42df7c7aa1
Docs: Fix typos found by codespell 2023-07-31 20:02:21 +08:00
Lasse Collin b4cf7a2822 Minor tweaks to style and comments. 2023-06-27 23:56:06 +08:00
Hans Jansen 23b5c36fb7 Add ifunc check to configure.ac
configure.ac will now verify if __attribute__((__ifunc__())) can be used in
the build system. If so, HAVE_FUNC_ATTRIBUTE_IFUNC will be defined to 1.
2023-06-27 15:33:15 +03:00
Lasse Collin 8be136f667 Build: Add a comment that AC_PROG_CC_C99 is needed for Autoconf 2.69.
It's obsolete in Autoconf >= 2.70 and just an alias for AC_PROG_CC
but Autoconf 2.69 requires AC_PROG_CC_C99 to get a C99 compiler.
2023-03-21 14:07:51 +02:00
Lasse Collin 53cc475f26 Build: configure.ac: Use AS_IF and AS_CASE where required.
This makes no functional difference in the generated configure
(at least with the Autotools versions I have installed) but this
change might prevent future bugs like the one that was just
fixed in the commit 5a5bd7f871.
2023-03-21 14:04:37 +02:00