This is a mess because liblzma DLL outside Cygwin and MSYS2
is liblzma.dll instead of lzma.dll to avoid a conflict with
lzma.dll from LZMA SDK.
On Cygwin the name was "liblzma-5.dll" while "cyglzma-5.dll"
would have been correct (and match what Libtool produces).
MSYS2 likely was broken too as it uses the "msys-" prefix.
This change has no effect with MinGW-w64 because with that
the "lib" prefix was correct already.
With MSVC builds this is a small breaking change that requires developers
to adjust the library name when linking against liblzma. The liblzma.dll
name is kept as is but the import library and static library are now
lzma.lib instead of liblzma.lib. This is helpful when using pkgconf
because "pkgconf --msvc-syntax --libs liblzma" outputs "lzma.lib"
(it's converted from "-llzma" in liblzma.pc). It would be easy to
keep the liblzma.lib naming but the pkgconf compatibility seems worth
it in the long run. The lzma.lib name is compatible with MinGW-w64
too as -llzma will find also lzma.lib.
vcpkg had been patching CMakeLists.txt this way since 2022 but I
learned this only recently. The reasoning for the patch makes sense,
and while this is a small breaking change with MSVC, it seems like
a decent compromise as it keeps the DLL name the same.
2022 patch in vcpkg: 0707a17ecf/ports/liblzma/win_output_name.patch
See the discussion: https://github.com/microsoft/vcpkg/pull/39024
Thanks to Vincent Torri for confirming the naming issue on Cygwin.
One has to pass -DENABLE_X86_ASM=ON to cmake to enable the
CRC assembly code. Autodetection isn't done. Looking at
CMAKE_SYSTEM_PROCESSOR might not work as it comes from uname
unless cross-compilation is done using a CMake toolchain file.
On top of this, if the code is run on modern processors that support
the CLMUL instruction, then the C code should be faster (but then
one should also be using a x86-64 build if possible).
This is disabled by default to match the default in Autotools.
Use -DUSE_DOXYGEN=ON to enable Doxygen usage.
This uses the update-doxygen script, thus this is under if(UNIX)
although Doxygen itself can run on Windows too.
This moves the tests section as is from CMakeLists.txt into
tests/tests.cmake. CMakeLists.txt now includes tests/tests.cmake
if the latter file exists.
Now it's possible to delete the whole "tests" directory and
building with CMake will still work normally, just without
the tests. This way the tests are readily available for those
who want them, and those who won't run the tests anyway have
a straightforward way to ensure that nothing from the "tests"
directory can affect the build process.
This is *NOT* done for security reasons even though the backdoor
relied on the ifunc code. Instead, the reason is that in this
project ifunc provides little benefits but it's quite a bit of
extra code to support it. The only case where ifunc *might* matter
for performance is if the CRC functions are used directly by an
application. In normal compression use it's completely irrelevant.
It doesn't support the __symver__ attribute or __asm__(".symver ...").
The generic symbol versioning can still be used since it only needs
linker support.
Using __attribute__((__no_profile_instrument_function__)) on the ifunc
resolver works around a bug in GCC -fprofile-generate:
it adds profiling code even to ifunc resolvers which can make
the ifunc resolver crash at program startup. This attribute
was not introduced until GCC 7 and Clang 13, so ifunc won't
be used with prior versions of these compilers.
This bug was brought to our attention by:
https://bugs.gentoo.org/925415
And was reported to upstream GCC by:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=11411
The previous Linux Landlock feature test assumed that having the
linux/landlock.h header file was enough. The new feature tests also
requires that prctl() and the required Landlock system calls are
supported.
The original code was good enough for supporting GNU/Linux
and a few others but it wasn't very portable.
CMake doesn't support Solaris Studio's -xldscope=hidden.
If it ever does, things should still work with this commit
as Solaris Studio supports not only its own __global but also
the GNU C __attribute__((visibility("default"))). Support for the
attribute was added in 2007 to Sun Studio 12 compiler version 5.9.
-O3 doesn't seem useful for speed but it makes the code bigger.
CMake makes is difficult for users to simply override the
optimization level: CFLAGS / CMAKE_C_FLAGS aren't helpful because
they go before CMAKE_C_FLAGS_RELEASE. Of course, users can override
CMAKE_C_FLAGS_RELEASE directly but then they have to remember to
add also -DNDEBUG to disable assertions.
This commit changes -O3 to -O2 in CMAKE_C_FLAGS_RELEASE if and only if
CMAKE_C_FLAGS_RELEASE cache variable doesn't already exist. So if
a custom value is passed on the command line (or reconfiguring an
already-configured build), the cache variable won't be modified.
In contrast to Automake, skipping of this test when decoders
are disabled is handled at CMake side instead of test_scripts.sh
because CMake-build doesn't create config.h.
Compared to the Autotools-based build, this has simpler handling
for the shell (@POSIX_SHELL@) and extra PATH entry for the scripts
(configure has --enable-path-for-scripts=PREFIX). The simpler
metho should be enough for non-ancient systems and Solaris.
It helps that cmake_install.cmake doesn't parallelize installation
so symlinks can be created so that the target is always known to
exist (a requirement on Windows in some cases).
This bumps the minimum CMake version from 3.13 to 3.14 to use
file(CREATE_LINK ...). It could be made to work on 3.13 by
calling "cmake -E create_symlink" but it's uglier code and
slower in "make install". 3.14 should be a reasonable version
to require nowadays, especially since the Autotools build
is still the primary build system for most OSes.
If gettext tools are available, the .po files listed in po/LINGUAS
are converted using msgfmt. This allows building with translations
directly from xz.git without Autotools.
If gettext tools aren't available, the Autotools-created .gmo files
in the "po" directory will be used. This allows CMake-based build
to use translations from Autotools-generated tarball.
If translation support is found (Intl_FOUND) but both the
gettext tools and the pre-generated .gmo files are missing,
then "make" will fail.
Landlock is now always used just like pledge(2) is: first in more
permissive mode and later (under certain common conditions) in
a strict mode that doesn't allow opening more files.
I put pledge(2) first in sandbox.c because it's the simplest API
to use and still somewhat fine-grained for basic applications.
So it's the simplest thing to understand for anyone reading sandbox.c.
Even though the proper name for the architecture is aarch64, this
project uses ARM64 throughout. So the rename is for consistency.
Additionally, crc32_arm64.h was slightly refactored for the following
changes:
* Added MSVC, FreeBSD, and macOS support in
is_arch_extension_supported().
* crc32_arch_optimized() now checks the size when aligning the
buffer.
* crc32_arch_optimized() loop conditions were slightly modified to
avoid both decrementing the size and incrementing the buffer
pointer.
* Use the intrinsic wrappers defined in <arm_acle.h> because GCC and
Clang name them differently.
* Minor spacing and comment changes.
The CRC32 instructions in ARM64 can calculate the CRC32 result
for 8 bytes in a single operation, making the use of ARM64
instructions much faster compared to the general CRC32 algorithm.
Optimized CRC32 will be enabled if ARM64 has CRC extension
running on Linux.
Signed-off-by: Chenxi Mao <chenxi.mao2013@gmail.com>
A CLMUL-only build will have the crcxx_clmul() inlined into
lzma_crcxx(). Previously a jump to the extern lzma_crcxx_clmul()
was needed. Notes about shared liblzma on ELF platforms:
- On platforms that support ifunc and -fvisibility=hidden, this
was silly because CLMUL-only build would have that single extra
jump instruction of extra overhead.
- On platforms that support neither -fvisibility=hidden nor linker
version script (liblzma*.map), jumping to lzma_crcxx_clmul()
would go via PLT so a few more instructions of overhead (still
not a big issue but silly nevertheless).
There was a downside with static liblzma too: if an application only
needs lzma_crc64(), static linking would make the linker include the
CLMUL code for both CRC32 and CRC64 from crc_x86_clmul.o even though
the CRC32 code wouldn't be needed, thus increasing code size of the
executable (assuming that -ffunction-sections isn't used).
Also, now compilers are likely to inline crc_simd_body()
even if they don't support the always_inline attribute
(or MSVC's __forceinline). Quite possibly all compilers
that build the code do support such an attribute. But now
it likely isn't a problem even if the attribute wasn't supported.
Now all x86-specific stuff is in crc_x86_clmul.h. If other archs
The other archs can then have their own headers with their own
is_clmul_supported() and crcxx_clmul().
Another bonus is that the build system doesn't need to care if
crc_clmul.c is needed.
is_clmul_supported() stays as inline function as it's not needed
when doing a CLMUL-only build (avoids a warning about unused function).
The sandbox is now enabled for xzdec as well, so it no longer belongs
in just the xz section. xz and xzdec are always built, except for older
MSVC versions, so there isn't a need to conditionally show the sandbox
configuration. CMake will do a little unecessary work on older MSVC
versions that can't build xz or xzdec, but this is a very small
downside.
A CMake option LARGE_FILE_SUPPORT is created if and only if
-D_FILE_OFFSET_BITS=64 affects sizeof(off_t).
This is needed on many 32-bit platforms and even with 64-bit builds
with MinGW-w64 to get support for files larger than 2 GiB.
Autotools based build uses -pthread and thus adds it to Libs.private
in liblzma.pc. CMake doesn't use -pthread at all if pthread functions
are available in libc so Libs.private doesn't get -pthread either.
Now configure will fail if -fsanitize= is found in CFLAGS
and sanitizer-incompatible ifunc or Landlock sandboxing
would be used. These are incompatible with one or more sanitizers.
It's simpler to reject all -fsanitize= uses instead of trying to
pass those that might not cause problems.
CMake-based build was updated similarly. It lets the configuration
finish (SEND_ERROR instead of FATAL_ERROR) so that both error
messages can be seen at once.
Using set(ENABLE_THREADS "posix") is confusing because it sets
a new normal variable and leaves the cache entry with the same
name unchanged. The intent wasn't to change the cache entry so
this switches to a different variable name.
This way typos are caught quickly and compounding error messages
are avoided (a single typo could cause more than one error).
This keeps using SEND_ERROR when the system is lacking a feature
(like threading library or sandboxing method). This way the whole
configuration log will be generated in case someone wishes to
report a problem upstream.
The option is enabled by default, but will only be visible to a user
listing cache variables or using a CMake GUI application if the
immintrin.h header file is found.
This mirrors our Autotools build --disable-clmul-crc functionality.
Both crc32_clmul() and crc64_clmul() are now exported from
crc32_clmul.c as lzma_crc32_clmul() and lzma_crc64_clmul(). This
ensures that is_clmul_supported() (now lzma_is_clmul_supported()) is
not duplicated between crc32_fast.c and crc64_fast.c.
Also, it encapsulates the complexity of the CLMUL implementations into a
single file and reduces the complexity of crc32_fast.c and crc64_fast.c.
Before, CLMUL code was present in crc32_fast.c, crc64_fast.c, and
crc_common.h.
During the conversion, various cleanups were applied to code (thanks to
Lasse Collin) including:
- Require using semicolons with MASK_/L/H/LH macros.
- Variable typing and const handling improvements.
- Improvements to comments.
- Fixes to the pragmas used.
- Removed unneeded variables.
- Whitespace improvements.
- Fixed CRC_USE_GENERIC_FOR_SMALL_INPUTS handling.
- Silenced warnings and removed the need for some #pragmas
CMake doesn't set WIN32 on CYGWIN but the workaround is
probably needed on Cygwin too. Same for MSYS and MSYS2.
The workaround must not be used with Clang that is acting in
MSVC mode. This fixes it by checking for the known environments
that need the workaround instead of using "NOT MSVC".
Thanks to Martin Storsjö.
0570308ddd (commitcomment-129098431)
The Ninja Generator for CMake cannot have a custom target and its
BYPRODUCTS have the same name. This has prevented Ninja builds on
Unix-like systems since the xz symlinks were introduced in
80a1a8bb83.
llvm-windres 17.0.0 has more accurate emulation of GNU windres, so
the hack for GNU windres must now be used with llvm-windres too.
LLVM 16.0.6 has the old behavior and there likely won't be more
16.x releases. So we can simply check for >= 17.0.0.
See also:
2bcc0fdc58
Now if user-supplied CFLAGS contains -Wall -Wextra -Wpedantic
the two checks that need -Werror will still work.
At CMake side there is add_compile_options(-Wall -Wextra)
but it didn't affect the -Werror tests. So with both Autotools
and CMake only user-supplied CFLAGS could make the checks fail
when they shouldn't.
This is not a full fix as things like -Wunused-macros in
user-supplied CFLAGS will still cause problems with both
GCC and Clang.
xzdec might build with VS2013 but it hasn't been tested.
It was never supported before and VS2013 is old anyway
so for simplicity only liblzma is supported with VS2013.
Building the command line tools xz and xzdec with the combination
of CMake + Visual Studio 2015/2017/2019/2022 works now.
VS2013 update 2 should still be able to build liblzma.
VS2013 cannot build the xz command line tool because xz
needs snprintf() that roughly conforms to C99.
VS2013 is old and no extra code will be added to support it.
Thanks to Kelvin Lee and Jia Tan for testing.
There are several new policies. CMP0149 may affect the Windows SDK
version that CMake will choose by default. The new behavior is more
predictable, always choosing the latest SDK version by default.
The other new policies shouldn't affect this package.
If CMake was configured more than once, HAVE_CLOCK_GETTIME and
HAVE_CLOCK_MONOTONIC would not be set as compile definitions. The check
for librt being needed to provide HAVE_CLOCK_GETTIME was also
simplified.
The CMake build will try to create broken symlinks on Unix and Unix-like
platforms. Cygwin and MSYS2 are Unix-like, but may not be able to create
broken symlinks. The value of the CYGWIN or MSYS environment variables
determine if broken symlinks are valid.
The default for MSYS2 does not allow for broken symlinks, so the CMake
build has been broken for MSYS2 since commit
80a1a8bb83.
CMake build system will now verify if __attribute__((__ifunc__())) can be
used in the build system. If so, HAVE_FUNC_ATTRIBUTE_IFUNC will be
defined to 1.
Boost iostream uses `find_package` in quiet mode and then again uses
`find_package` with required. This second call triggers a
`add_library cannot create imported target "ZLIB::ZLIB" because another
target with the same name already exists.`
This can simply be fixed by skipping the alias part on secondary
`find_package` runs.
The thread method is now configurable for the CMake build. It matches
the Autotools build by allowing ON (pick the best threading method),
OFF (no threading), posix, win95, and vista. If both Windows and
posix threading are both available, then ON will choose Windows
threading. Windows threading will also not use:
target_link_libraries(liblzma Threads::Threads)
since on systems like MinGW-w64 it would link the posix threads
without purpose.
This allows users to change the features they build either in
CMakeCache.txt or by using a CMake GUI. The sources built for
liblzma are affected by this too, so only the necessary files
will be compiled.
Lasse Collin
Jia Tan
Chenxi Mao
Hans Jansen
Dimitri Papadopoulos Orfanos
Benjamin Buch