Commit Graph

117 Commits

Author SHA1 Message Date
Lasse Collin 913ddc5572 liblzma: Vaccinate against an ill patch from RHEL/CentOS 7.
RHEL/CentOS 7 shipped with 5.1.2alpha, including the threaded
encoder that is behind #ifdef LZMA_UNSTABLE in the API headers.
In 5.1.2alpha these symbols are under XZ_5.1.2alpha in liblzma.map.
API/ABI compatibility tracking isn't done between development
releases so newer releases didn't have XZ_5.1.2alpha anymore.

Later RHEL/CentOS 7 updated xz to 5.2.2 but they wanted to keep
the exported symbols compatible with 5.1.2alpha. After checking
the ABI changes it turned out that >= 5.2.0 ABI is backward
compatible with the threaded encoder functions from 5.1.2alpha
(but not vice versa as fixes and extensions to these functions
were made between 5.1.2alpha and 5.2.0).

In RHEL/CentOS 7, XZ Utils 5.2.2 was patched with
xz-5.2.2-compat-libs.patch to modify liblzma.map:

  - XZ_5.1.2alpha was added with lzma_stream_encoder_mt and
    lzma_stream_encoder_mt_memusage. This matched XZ Utils 5.1.2alpha.

  - XZ_5.2 was replaced with XZ_5.2.2. It is clear that this was
    an error; the intention was to keep using XZ_5.2 (XZ_5.2.2
    has never been used in XZ Utils). So XZ_5.2.2 lists all
    symbols that were listed under XZ_5.2 before the patch.
    lzma_stream_encoder_mt and _mt_memusage are included too so
    they are listed both here and under XZ_5.1.2alpha.

The patch didn't add any __asm__(".symver ...") lines to the .c
files. Thus the resulting liblzma.so exports the threaded encoder
functions under XZ_5.1.2alpha only. Listing the two functions
also under XZ_5.2.2 in liblzma.map has no effect without
matching .symver lines.

The lack of XZ_5.2 in RHEL/CentOS 7 means that binaries linked
against unpatched XZ Utils 5.2.x won't run on RHEL/CentOS 7.
This is unfortunate but this alone isn't too bad as the problem
is contained within RHEL/CentOS 7 and doesn't affect users
of other distributions. It could also be fixed internally in
RHEL/CentOS 7.

The second problem is more serious: In XZ Utils 5.2.2 the API
headers don't have #ifdef LZMA_UNSTABLE for obvious reasons.
This is true in RHEL/CentOS 7 version too. Thus now programs
using new APIs can be compiled without an extra #define. However,
the programs end up depending on symbol version XZ_5.1.2alpha
(and possibly also XZ_5.2.2) instead of XZ_5.2 as they would
with an unpatched XZ Utils 5.2.2. This means that such binaries
won't run on other distributions shipping XZ Utils >= 5.2.0 as
they don't provide XZ_5.1.2alpha or XZ_5.2.2; they only provide
XZ_5.2 (and XZ_5.0). (This includes RHEL/CentOS 8 as the patch
luckily isn't included there anymore with XZ Utils 5.2.4.)

Binaries built by RHEL/CentOS 7 users get distributed and then
people wonder why they don't run on some other distribution.
Seems that people have found out about the patch and been copying
it to some build scripts, seemingly curing the symptoms but
actually spreading the illness further and outside RHEL/CentOS 7.

The ill patch seems to be from late 2016 (RHEL 7.3) and in 2017 it
had spread at least to EasyBuild. I heard about the events only
recently. :-(

This commit splits liblzma.map into two versions: one for
GNU/Linux and another for other OSes that can use symbol versioning
(FreeBSD, Solaris, maybe others). The Linux-specific file and the
matching additions to .c files add full compatibility with binaries
that have been built against a RHEL/CentOS-patched liblzma. Builds
for OSes other than GNU/Linux won't get the vaccine as they should
be immune to the problem (I really hope that no build script uses
the RHEL/CentOS 7 patch outside GNU/Linux).

The RHEL/CentOS compatibility symbols XZ_5.1.2alpha and XZ_5.2.2
are intentionally put *after* XZ_5.2 in liblzma_linux.map. This way
if one forgets to #define HAVE_SYMBOL_VERSIONS_LINUX when building,
the resulting liblzma.so.5 will have lzma_stream_encoder_mt@@XZ_5.2
since XZ_5.2 {...} is the first one that lists that function.
Without HAVE_SYMBOL_VERSIONS_LINUX @XZ_5.1.2alpha and @XZ_5.2.2
will be missing but that's still a minor problem compared to
only having lzma_stream_encoder_mt@@XZ_5.1.2alpha!

The "local: *;" line was moved to XZ_5.0 so that it doesn't need
to be moved around. It doesn't matter where it is put.

Having two similar liblzma_*.map files is a bit silly as it is,
at least for now, easily possible to generate the generic one
from the Linux-specific file. But that adds extra steps and
increases the risk of mistakes when supporting more than one
build system. So I rather maintain two files in parallel and let
validate_map.sh check that they are in sync when "make mydist"
is run.

This adds .symver lines for lzma_stream_encoder_mt@XZ_5.2.2 and
lzma_stream_encoder_mt_memusage@XZ_5.2.2 even though these
weren't exported by RHEL/CentOS 7 (only @@XZ_5.1.2alpha was
for these two). I added these anyway because someone might
misunderstand the RHEL/CentOS 7 patch and think that @XZ_5.2.2
(@@XZ_5.2.2) versions were exported too.

At glance one could suggest using __typeof__ to copy the function
prototypes when making aliases. However, this doesn't work trivially
because __typeof__ won't copy attributes (lzma_nothrow, lzma_pure)
and it won't change symbol visibility from hidden to default (done
by LZMA_API()). Attributes could be copied with __copy__ attribute
but that needs GCC 9 and a fallback method would be needed anyway.

This uses __symver__ attribute with GCC >= 10 and
__asm__(".symver ...") with everything else. The attribute method
is required for LTO (-flto) support with GCC. Using -flto with
GCC older than 10 is now broken on GNU/Linux and will not be fixed
(can silently result in a broken liblzma build that has dangerously
incorrect symbol versions). LTO builds with Clang seem to work
with the traditional __asm__(".symver ...") method.

Thanks to Boud Roukema for reporting the problem and discussing
the details and testing the fix.
2022-09-08 15:01:29 +03:00
Jia Tan 4d80b463a1 Build: Don't allow empty LIST in --enable-match-finders=LIST.
It's enforced only when a match finder is needed, that is,
when LZMA1 or LZMA2 encoder is enabled.
2022-07-25 18:20:01 +03:00
Lasse Collin 4773608554 Build: Enable Automake's parallel test harness.
It has been the default for quite some time already and
the old serial harness isn't discouraged. The downside is
that with parallel tests one cannot print progress info or
other diagnostics to the terminal; all output from the tests
will be in the log files only. But now that the compression
tests are separated the parallel tests will speed things up.
2022-05-23 21:31:36 +03:00
Lasse Collin feb9c1969b Build: Bump Autoconf and Libtool version requirements.
There is no specific reason for this other than blocking
the most ancient versions. These are still old:

Autoconf 2.69 (2012)
Automake 1.12 (2012)
gettext 0.19.6 (2015)
Libtool 2.4 (2010)
2020-02-15 15:07:11 +02:00
Lasse Collin 3d576cf921 Build: Use AM_GNU_GETTEXT_REQUIRE_VERSION and require 0.19.6.
This bumps the version requirement from 0.19 (from 2014) to
0.19.6 (2015).

Using only the old AM_GNU_GETTEXT_VERSION results in old
gettext infrastructure being placed in the package. By using
both macros we get the latest gettext files while the other
programs in the Autotools family can still see the old macro.
2020-02-15 15:05:35 +02:00
Lasse Collin 986d8c9b52 Build: Workaround a POSIX shell detection problem on Solaris.
I don't know if the problem is in gnulib's gl_POSIX_SHELL macro
or if xzgrep does something that isn't in POSIX. The workaround
adds a special case for Solaris: if /usr/xpg4/bin/sh exists and
gl_cv_posix_shell wasn't overriden on the configure command line,
use that shell for xzgrep and other scripts. That shell is known
to work and exists on most Solaris systems.
2020-02-05 19:33:37 +02:00
Lasse Collin 43ce4ea7c7 Scripts: Put /usr/xpg4/bin to the beginning of PATH on Solaris.
This adds a configure option --enable-path-for-scripts=PREFIX
which defaults to empty except on Solaris it is /usr/xpg4/bin
to make POSIX grep and others available. The Solaris case had
been documented in INSTALL with a manual fix but it's better
to do this automatically since it is needed on most Solaris
systems anyway.

Thanks to Daniel Richard G.
2019-09-24 23:02:40 +03:00
Lasse Collin fc77929e92 configure.ac: Fix a typo in a comment. 2019-06-25 00:16:06 +03:00
Lasse Collin 662b27c417 Update the home page URLs to HTTPS. 2017-04-19 22:17:35 +03:00
Lasse Collin 446e4318fa xz: Fix copying of timestamps on Windows.
xz used to call utime() on Windows, but its result gets lost
on close(). Using _futime() seems to work.

Thanks to Martok for reporting the bug:
http://www.mail-archive.com/xz-devel@tukaani.org/msg00261.html
2016-06-30 20:27:36 +03:00
Lasse Collin ade31a2bfb Build: Bump GNU Gettext version requirement to 0.19.
It silences a few warnings and most people probably have
0.19 even on stable distributions.

Thanks to Christian Kujau.
2016-04-10 20:54:17 +03:00
Lasse Collin ac398c3baf liblzma: Disable external SHA-256 by default.
This is the sane thing to do. The conflict with OpenSSL
on some OSes and especially that the OS-provided versions
can be significantly slower makes it clear that it was
a mistake to have the external SHA-256 support enabled by
default.

Those who want it can now pass --enable-external-sha256 to
configure. INSTALL was updated with notes about OSes where
this can be a bad idea.

The SHA-256 detection code in configure.ac had some bugs that
could lead to a build failure in some situations. These were
fixed, although it doesn't matter that much now that the
external SHA-256 is disabled by default.

MINIX >= 3.2.0 uses NetBSD's libc and thus has SHA256_Init
in libc instead of libutil. Support for the libutil version
was removed.
2016-03-13 20:21:49 +02:00
Lasse Collin 473ef0dc69 Build: Avoid SHA256_Init on FreeBSD and MINIX 3.
On FreeBSD 10 and older, SHA256_Init from libmd conflicts
with libcrypto from OpenSSL. The OpenSSL version has
different sizeof(SHA256_CTX) and it can cause weird
problems if wrong SHA256_Init gets used.

Looking at the source, MINIX 3 seems to have a similar issue but
I'm not sure. To be safe, I disabled SHA256_Init on MINIX 3 too.

NetBSD has SHA256_Init in libc and they had a similar problem,
but they already fixed it in 2009.

Thanks to Jim Wilcoxson for the bug report that helped
in finding the problem.
2016-03-10 20:26:49 +02:00
Lasse Collin c7bc20a6f3 Build: Disable xzdec, lzmadec, and lzmainfo when they cannot be built.
They all need decoder support and if that isn't available,
there's no point trying to build them.
2015-11-03 20:47:07 +02:00
Lasse Collin 5cbca1205d Build: Simplify $enable_{encoders,decoders} usage a bit. 2015-11-03 20:35:19 +02:00
Lasse Collin cb3111e3ed xz: Make xz buildable even when encoders or decoders are disabled.
The patch is quite long but it's mostly about adding new #ifdefs
to omit code when encoders or decoders have been disabled.

This adds two new #defines to config.h: HAVE_ENCODERS and
HAVE_DECODERS.
2015-11-03 20:29:33 +02:00
Lasse Collin 23ed1d4148 Build: Fix configure to handle LZMA1 dependency with LZMA2.
Now it gives an error if LZMA1 encoder/decoder is missing
when LZMA2 encoder/decoder was requested. Even better would
be LZMA2 implicitly enabling LZMA1 but it would need more code.
2015-11-03 17:54:48 +02:00
Lasse Collin cbc9e39bae Build: Minor Cygwin cleanup.
Some tests used "cygwin*" and some used "cygwin". I changed
them all to use "cygwin". Shouldn't affect anything in practice.
2015-08-11 13:23:04 +03:00
Lasse Collin bcacd8ce7a Build: Support building of MSYS2 binaries. 2015-08-11 13:21:52 +03:00
Lasse Collin 6bd0349c58 Revert "xz: Use pipe2() if available."
This reverts commit 7a11c4a8e5.
It is a problem when libc has pipe2() but the kernel is too
old to have pipe2() and thus pipe2() fails. In xz it's pointless
to have a fallback for non-functioning pipe2(); it's better to
avoid pipe2() completely.

Thanks to Michael Fox for the bug report.
2015-04-20 20:17:48 +03:00
Lasse Collin 1238381143 xz: Add support for sandboxing with Capsicum.
The sandboxing is used conditionally as described in main.c.
This isn't optimal but it was much easier to implement than
a full sandboxing solution and it still covers the most common
use cases where xz is writing to standard output. This should
have practically no effect on performance even with small files
as fork() isn't needed.

C and locale libraries can open files as needed. This has been
fine in the past, but it's a problem with things like Capsicum.
io_sandbox_enter() tries to ensure that various locale-related
files have been loaded before cap_enter() is called, but it's
possible that there are other similar problems which haven't
been seen yet.

Currently Capsicum is available on FreeBSD 10 and later
and there is a port to Linux too.

Thanks to Loganaden Velvindron for help.
2015-03-31 22:19:34 +03:00
Lasse Collin 7a11c4a8e5 xz: Use pipe2() if available. 2015-02-22 19:38:48 +02:00
Lasse Collin 7815112153 Build: POSIX shell isn't required if scripts are disabled. 2014-12-26 12:00:05 +02:00
Lasse Collin b13a781833 Build: Replace obsolete AC_HELP_STRING with AS_HELP_STRING. 2014-11-17 18:52:21 +02:00
Lasse Collin 542cac122e Build: Fix Autoconf warnings about escaped backquotes.
Thanks to Daniel Richard G. for pointing out that it's
good to sometimes run autoreconf -fi with -Wall.
2014-11-17 18:43:19 +02:00
Lasse Collin c923b140b2 Build: Prepare to support Automake's subdir-objects.
Due to a bug in Automake, subdir-objects won't be enabled
for now.

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17354

Thanks to Daniel Richard G. for the original patches.
2014-10-29 21:15:35 +02:00
Lasse Collin c4911f2db3 Build: Detect supported compiler warning flags better.
Clang and nowadays also GCC accept any -Wfoobar option
but then may give a warning that an unknown warning option
was specified. To avoid adding unsupported warning options,
the options are now tested with -Werror.

Thanks to Charles Diza.
2014-09-25 18:38:48 +03:00
Lasse Collin e1c8f1d01f liblzma: Add lzma_memcmplen() for fast memory comparison.
This commit just adds the function. Its uses will be in
separate commits.

This hasn't been tested much yet and it's perhaps a bit early
to commit it but if there are bugs they should get found quite
quickly.

Thanks to Jun I Jin from Intel for help and for pointing out
that string comparison needs to be optimized in liblzma.
2014-07-25 20:57:20 +03:00
Lasse Collin 56056571df Build: Add --disable-doc to configure. 2014-04-25 17:44:26 +03:00
Lasse Collin 6b44b4a775 Add native threading support on Windows.
Now liblzma only uses "mythread" functions and types
which are defined in mythread.h matching the desired
threading method.

Before Windows Vista, there is no direct equivalent to
pthread condition variables. Since this package doesn't
use pthread_cond_broadcast(), pre-Vista threading can
still be kept quite simple. The pre-Vista code doesn't
use anything that wasn't already available in Windows 95,
so the binaries should run even on Windows 95 if someone
happens to care.
2013-09-17 11:52:28 +03:00
Lasse Collin ae0ab74a88 Build: Remove a comment about Automake 1.10 from configure.ac.
The previous commit supports silent rules and that requires
Automake 1.11.
2013-09-11 14:40:35 +03:00
Lasse Collin 72975df6c8 Build: Create liblzma.pc in a src/liblzma/Makefile.am.
Previously it was done in configure, but doing that goes
against the Autoconf manual. Autoconf requires that it is
possible to override e.g. prefix after running configure
and that doesn't work correctly if liblzma.pc is created
by configure.

A potential downside of this change is that now e.g.
libdir in liblzma.pc is a standalone string instead of
being defined via ${prefix}, so if one overrides prefix
when running pkg-config the libdir won't get the new value.
I don't know if this matters in practice.

Thanks to Vincent Torri.
2013-09-09 20:37:03 +03:00
Lasse Collin 3e2b198ba3 Build: Fix the detection of missing CRC32.
Thanks to Vincent Torri.
2013-07-15 14:08:02 +03:00
Lasse Collin 9376f5f8f7 Build: Require Automake 1.12 and use serial-tests option.
It should actually still work with Automake 1.10 if
the serial-tests option is removed. Automake 1.13 started
using parallel tests by default and the option to get
the old behavior isn't supported before 1.12.

At least for now, parallel tests don't improve anything
in XZ Utils but they hide the progress output from
test_compress.sh.
2013-06-26 12:17:00 +03:00
Lasse Collin 97379c5ea7 Build: Use -Wvla with GCC if supported.
Variable-length arrays are mandatory in C99 but optional in C11.
The code doesn't currently use any VLAs and it shouldn't in the
future either to stay compatible with C11 without requiring any
optional C11 features.
2013-04-27 22:07:46 +03:00
Lasse Collin ab22562066 A few typo fixes to comments and the xz man page.
Thanks to Jim Meyering.
2012-08-24 16:27:31 +03:00
Lasse Collin d8eaf9d827 Build: Bump gettext version requirement to 0.18.
Otherwise too old version of m4/lib-link.m4 gets included
when autoreconf -fi is run.
2012-08-02 17:13:30 +03:00
Lasse Collin df11317985 Build: Add a comment to configure.ac about symbol versioning. 2012-07-04 17:11:31 +03:00
Lasse Collin 2dcea03712 Fix compiling with IBM XL C on AIX. 2012-02-22 12:00:16 +02:00
Lasse Collin e9ed88126e Run the scripts with the correct shell in test_scripts.sh.
The scripts are now made executable in the build tree.
This way the scripts can be run like programs in
test_scripts.sh. Previously test_scripts.sh always
used sh but it's not correct if @POSIX_SHELL@ is set
to something else by configure.

Thanks to Jonathan Nieder for the patch.
2011-08-06 20:37:28 +03:00
Lasse Collin 492c863455 Build: Make configure print if symbol versioning is enabled or not. 2011-05-28 19:24:56 +03:00
Lasse Collin bd35d903a0 liblzma: Use symbol versioning.
Symbol versioning is enabled by default on GNU/Linux,
other GNU-based systems, and FreeBSD.

I'm not sure how stable this is, so it may need
backward-incompatible changes before the next release.

The idea is that alpha and beta symbols are considered
unstable and require recompiling the applications that
use those symbols. Once a symbol is stable, it may get
extended with new features in ways that don't break
compatibility with older ABI & API.

The mydist target runs validate_map.sh which should
catch some probable problems in liblzma.map. Otherwise
I would forget to update the map file for new releases.
2011-05-28 15:55:39 +03:00
Lasse Collin fe00f95828 Build: Fix checking for system-provided SHA-256. 2011-05-24 00:23:46 +03:00
Lasse Collin b94aa0c838 liblzma: Try to use SHA-256 from the operating system.
If the operating system libc or other base libraries
provide SHA-256, use that instead of our own copy.
Note that this doesn't use OpenSSL or libgcrypt or
such libraries to avoid creating dependencies to
other packages.

This supports at least FreeBSD, NetBSD, OpenBSD, Solaris,
MINIX, and Darwin. They all provide similar but not
identical SHA-256 APIs; everyone is a little different.

Thanks to Wim Lewis for the original patch, improvements,
and testing.
2011-05-21 15:08:44 +03:00
Lasse Collin ec7106309c Build: Fix initialization of enable_check_* variables in configure.ac.
This doesn't matter much in practice since it is unlikely
that anyone would have such environment variable names.

Thanks to Wim Lewis.
2011-05-17 12:01:37 +03:00
Lasse Collin 0b77c4a751 Build: Warn if no supported method to detect the number of CPU cores. 2011-04-19 10:44:48 +03:00
Lasse Collin 9c1b05828a Fix portability problems in mythread.h.
Use gettimeofday() if clock_gettime() isn't available
(e.g. Darwin).

The test for availability of pthread_condattr_setclock()
and CLOCK_MONOTONIC was incorrect. Instead of fixing the
#ifdefs, use an Autoconf test. That way if there exists a
system that supports them but doesn't specify the matching
POSIX #defines, the features will still get detected.

Don't try to use pthread_sigmask() on OpenVMS. It doesn't
have that function.

Guard mythread.h against being #included multiple times.
2011-04-19 09:20:44 +03:00
Lasse Collin de678e0c92 liblzma: Add lzma_stream_encoder_mt() for threaded compression.
This is the simplest method to do threading, which splits
the uncompressed data into blocks and compresses them
independently from each other. There's room for improvement
especially to reduce the memory usage, but nevertheless,
this is a good start.
2011-04-11 22:03:30 +03:00
Lasse Collin 9f0a806aef Revise mythread.h.
This adds:

  - mythread_sync() macro to create synchronized blocks

  - mythread_cond structure and related functions
    and macros for condition variables with timed
    waiting using a relative timeout

  - mythread_create() to create a thread with all
    signals blocked

Some of these wouldn't need to be inline functions,
but I'll keep them this way for now for simplicity.

For timed waiting on a condition variable, librt is
now required on some systems to use clock_gettime().
configure.ac was updated to handle this.
2011-04-10 21:23:21 +03:00
Lasse Collin db33117cc8 Build: Upgrade m4/acx_pthread.m4 to the latest version.
It was renamed to ax_pthread.m4 in Autoconf Archive.
2011-04-05 17:12:20 +03:00