Commit Graph

979 Commits

Author SHA1 Message Date
Lasse Collin 4572d53e16 liblzma: Fix a comment and RC_SYMBOLS_MAX.
The comment didn't match the value of RC_SYMBOLS_MAX and the value
itself was slightly larger than actually needed. The only harm
about this was that memory usage was a few bytes larger.
2020-03-02 13:54:33 +02:00
Lasse Collin b3ed19a55f liblzma: Remove unneeded <sys/types.h> from fastpos_tablegen.c.
This file only generates fastpos_table.c.
It isn't built as a part of liblzma.
2020-02-24 23:23:18 +02:00
Lasse Collin 7b8982b291 Use defined(__GNUC__) before __GNUC__ in preprocessor lines.
This should silence the equivalent of -Wundef in compilers that
don't define __GNUC__.
2020-02-22 14:15:07 +02:00
Lasse Collin 43dfe04e62 liblzma: Add more uses of lzma_memcmplen() to the normal mode of LZMA.
This gives a tiny encoder speed improvement. This could have been done
in 2014 after the commit 544aaa3d13 but
it was forgotten.
2020-02-21 17:40:02 +02:00
Lasse Collin 7fe3ef2eaa xz: Silence a warning when sig_atomic_t is long int.
It can be true at least on z/OS.
2020-02-21 16:10:44 +02:00
Lasse Collin b0a2a77d10 xz: Avoid unneeded access of a volatile variable. 2020-02-21 15:59:26 +02:00
Lasse Collin 57360bb4fd tuklib_exit: Add missing header.
strerror() needs <string.h> which happened to be included via
tuklib_common.h -> tuklib_config.h -> sysdefs.h if HAVE_CONFIG_H
was defined. This wasn't tested without config.h before so it
had worked fine.
2020-02-20 18:54:04 +02:00
Lasse Collin fddd31175e Revert the previous commit and add a comment.
The previous commit broke crc32_tablegen.c.

If the whole package is built without config.h (with defines
set on the compiler command line) this should still work fine
as long as these headers conform to C99 well enough.
2020-02-18 19:12:35 +02:00
Lasse Collin 4e4e9fbb7e Do not check for HAVE_CONFIG_H in tuklib_config.h.
In XZ Utils sysdefs.h takes care of it and the required headers.
2020-02-17 23:37:20 +02:00
Lasse Collin 2d4cef954f sysdefs.h: Omit the conditionals around string.h and limits.h.
string.h is used unconditionally elsewhere in the project and
configure has always stopped if limits.h is missing, so these
headers must have been always available even on the weirdest
systems.
2020-02-16 12:24:13 +02:00
Lasse Collin 6f7211b6bb Build: Add support for translated man pages using po4a.
The dependency on po4a is optional. It's never required to install
the translated man pages when xz is built from a release tarball.
If po4a is missing when building from xz.git, the translated man
pages won't be generated but otherwise the build will work normally.

The translations are only updated automatically by autogen.sh and
by "make mydist". This makes it easy to keep po4a as an optional
dependency and ensures that I won't forget to put updated
translations to a release tarball.

The translated man pages aren't installed if --disable-nls is used.

The installation of translated man pages abuses Automake internals
by calling "install-man" with redefined dist_man_MANS and man_MANS.
This makes the hairy script code slightly less hairy. If it breaks
some day, this code needs to be fixed; don't blame Automake developers.

Also, this adds more quotes to the existing shell script code in
the Makefile.am "-hook"s.
2020-02-07 15:32:21 +02:00
Lasse Collin 15a133b6d1 xz: Make it a fatal error if enabling the sandbox fails.
Perhaps it's too drastic but on the other hand it will let me
learn about possible problems if people report the errors.
This won't be backported to the v5.2 branch.
2020-02-05 20:40:14 +02:00
Lasse Collin af0fb386ef xz: Comment out annoying sandboxing messages. 2020-02-05 20:33:50 +02:00
Lasse Collin 3539705108 xz: Limit --memlimit-compress to at most 4020 MiB for 32-bit xz.
See the code comment for reasoning. It's far from perfect but
hopefully good enough for certain cases while hopefully doing
nothing bad in other situations.

At presets -5 ... -9, 4020 MiB vs. 4096 MiB makes no difference
on how xz scales down the number of threads.

The limit has to be a few MiB below 4096 MiB because otherwise
things like "xz --lzma2=dict=500MiB" won't scale down the dict
size enough and xz cannot allocate enough memory. With
"ulimit -v $((4096 * 1024))" on x86-64, the limit in xz had
to be no more than 4085 MiB. Some safety margin is good though.

This is hack but it should be useful when running 32-bit xz on
a 64-bit kernel that gives full 4 GiB address space to xz.
Hopefully this is enough to solve this:

https://bugzilla.redhat.com/show_bug.cgi?id=1196786

FreeBSD has a patch that limits the result in tuklib_physmem()
to SIZE_MAX on 32-bit systems. While I think it's not the way
to do it, the results on --memlimit-compress have been good. This
commit should achieve practically identical results for compression
while leaving decompression and tuklib_physmem() and thus
lzma_physmem() unaffected.
2020-02-01 19:56:18 +02:00
Lasse Collin ba76d67585 xz: Set the --flush-timeout deadline when the first input byte arrives.
xz --flush-timeout=2000, old version:

  1. xz is started. The next flush will happen after two seconds.
  2. No input for one second.
  3. A burst of a few kilobytes of input.
  4. No input for one second.
  5. Two seconds have passed and flushing starts.

The first second counted towards the flush-timeout even though
there was no pending data. This can cause flushing to occur more
often than needed.

xz --flush-timeout=2000, after this commit:

  1. xz is started.
  2. No input for one second.
  3. A burst of a few kilobytes of input. The next flush will
     happen after two seconds counted from the time when the
     first bytes of the burst were read.
  4. No input for one second.
  5. No input for another second.
  6. Two seconds have passed and flushing starts.
2020-01-26 20:53:25 +02:00
Lasse Collin fd47fd62bb xz: Move flush_needed from mytime.h to file_pair struct in file_io.h. 2020-01-26 20:25:52 +02:00
Lasse Collin 8150356810 xz: coder.c: Make writing output a separate function.
The same code sequence repeats so it's nicer as a separate function.
Note that in one case there was no test for opt_mode != MODE_TEST,
but that was only because that condition would always be true, so
this commit doesn't change the behavior there.
2020-01-26 14:49:22 +02:00
Lasse Collin 5a49e081a0 xz: Fix semi-busy-waiting in xz --flush-timeout.
When input blocked, xz --flush-timeout=1 would wake up every
millisecond and initiate flushing which would have nothing to
flush and thus would just waste CPU time. The fix disables the
timeout when no input has been seen since the previous flush.
2020-01-26 14:13:42 +02:00
Lasse Collin dcca70fe9f xz: Refactor io_read() a bit. 2020-01-26 13:47:31 +02:00
Lasse Collin 4ae9ab70cd xz: Update a comment in file_io.h. 2020-01-26 13:37:08 +02:00
Lasse Collin 3333ba4a67 xz: Move the setting of flush_needed in file_io.c to a nicer location. 2020-01-26 13:27:51 +02:00
Lasse Collin 7136f1735c Rename unaligned_read32ne to read32ne, and similarly for the others. 2019-12-31 00:47:49 +02:00
Lasse Collin 5e78fcbf2e Rename read32ne to aligned_read32ne, and similarly for the others.
Using the aligned methods requires more care to ensure that
the address really is aligned, so it's nicer if the aligned
methods are prefixed. The next commit will remove the unaligned_
prefix from the unaligned methods which in liblzma are used in
more places than the aligned ones.
2019-12-31 00:29:48 +02:00
Lasse Collin 77bc5bc6dd Revise tuklib_integer.h and .m4.
Add a configure option --enable-unsafe-type-punning to get the
old non-conforming memory access methods. It can be useful with
old compilers or in some other less typical situations but
shouldn't normally be used.

Omit the packed struct trick for unaligned access. While it's
best in some cases, this is simpler. If the memcpy trick doesn't
work, one can request unsafe type punning from configure.

Because CRC32/CRC64 code needs fast aligned reads, if no very
safe way to do it is found, type punning is used as a fallback.
This sucks but since it currently works in practice, it seems to
be the least bad option. It's never needed with GCC >= 4.7 or
Clang >= 3.6 since these support __builtin_assume_aligned and
thus fast aligned access can be done with the memcpy trick.

Other things:
  - Support GCC/Clang __builtin_bswapXX
  - Cleaner bswap fallback macros
  - Minor cleanups
2019-12-31 00:18:24 +02:00
Lasse Collin 43ce4ea7c7 Scripts: Put /usr/xpg4/bin to the beginning of PATH on Solaris.
This adds a configure option --enable-path-for-scripts=PREFIX
which defaults to empty except on Solaris it is /usr/xpg4/bin
to make POSIX grep and others available. The Solaris case had
been documented in INSTALL with a manual fix but it's better
to do this automatically since it is needed on most Solaris
systems anyway.

Thanks to Daniel Richard G.
2019-09-24 23:02:40 +03:00
Lasse Collin 6a89e656eb Fix comment typos in tuklib_mbstr* files. 2019-07-12 18:57:43 +03:00
Lasse Collin ac0b421265 Add missing include to tuklib_mbstr_width.c.
It didn't matter in XZ Utils because sysdefs.h
includes string.h anyway.
2019-07-12 18:30:46 +03:00
Lasse Collin 72a443281f Update tuklib base headers to include stdbool.h. 2019-07-12 18:10:57 +03:00
Lasse Collin de1f47b2b4 xz: Automatically align the strings in --info-memory.
This makes it easier to translate the strings.

Also, the string for amount of RAM was shortened.
2019-06-28 00:54:31 +03:00
Lasse Collin 8ce679125d liblzma: Fix a buggy comment. 2019-06-25 23:15:21 +03:00
Lasse Collin d499e467d9 liblzma: Add a comment. 2019-06-24 23:52:17 +03:00
Lasse Collin a12b13c5f0 liblzma: Silence clang -Wmissing-variable-declarations. 2019-06-24 23:45:21 +03:00
Lasse Collin 1b4675cebf Add LZMA_RET_INTERNAL1..8 to lzma_ret and use one for LZMA_TIMED_OUT.
LZMA_TIMED_OUT is *internally* used as a value for lzma_ret
enumeration. Previously it was #defined to 32 and cast to lzma_ret.
That way it wasn't visible in the public API, but this was hackish.

Now the public API has eight LZMA_RET_INTERNALx members and
LZMA_TIMED_OUT is #defined to LZMA_RET_INTERNAL1. This way
the code is cleaner overall although the public API has a few
extra mysterious enum members.
2019-06-24 23:25:41 +03:00
Lasse Collin 159c43875e xz: Silence a warning from clang -Wsign-conversion in main.c. 2019-06-24 22:57:43 +03:00
Lasse Collin 466cfcd3e5 xz: Make "headings" static in list.c.
Caught by clang -Wmissing-variable-declarations.
2019-06-24 22:52:20 +03:00
Lasse Collin 608517b9b7 liblzma: Remove incorrect uses of lzma_attribute((__unused__)).
Caught by clang -Wused-but-marked-unused.
2019-06-24 22:50:36 +03:00
Lasse Collin 2402f7873d xz: Fix an integer overflow with 32-bit off_t.
Or any off_t which isn't very big (like signed 64 bit integer
that most system have). A small off_t could overflow if the
file being decompressed had long enough run of zero bytes,
which would result in corrupt output.
2019-06-24 20:45:49 +03:00
Lasse Collin 4fd3a8dd0b xz: Cleanup io_seek_src() a bit.
lseek() returns -1 on error and checking for -1 is nicer.
2019-06-24 01:24:17 +03:00
Lasse Collin 1d4a904d8f xz: Change io_seek_src and io_pread arguments from off_t to uint64_t.
This helps fixing warnings from -Wsign-conversion and makes the
code look better too.
2019-06-24 00:40:45 +03:00
Lasse Collin 50120deb01 xz: list.c: Fix some warnings from -Wsign-conversion. 2019-06-24 00:12:38 +03:00
Lasse Collin d0a78751eb tuklib_mbstr_width: Fix a warning from -Wsign-conversion. 2019-06-23 23:22:45 +03:00
Lasse Collin 7883d73530 xz: Fix some of the warnings from -Wsign-conversion. 2019-06-23 23:19:34 +03:00
Lasse Collin c2b994fe3d tuklib_cpucores: Silence warnings from -Wsign-conversion. 2019-06-23 22:27:45 +03:00
Lasse Collin 07c4fa9e1a xzdec: Fix warnings from -Wsign-conversion. 2019-06-23 21:40:47 +03:00
Lasse Collin dfac2c9a1d liblzma: Fix warnings from -Wsign-conversion.
Also, more parentheses were added to the literal_subcoder
macro in lzma_comon.h (better style but no functional change
in the current usage).
2019-06-23 21:38:56 +03:00
Lasse Collin 41838dcc26 tuklib_integer: Silence warnings from -Wsign-conversion. 2019-06-23 19:33:55 +03:00
Lasse Collin 3ce05d235f tuklib_integer: Fix usage of conv macros.
Use a temporary variable instead of e.g.
conv32le(unaligned_read32ne(buf)) because the macro can
evaluate its argument multiple times.
2019-06-20 19:40:30 +03:00
Lasse Collin 039a168e8c liblzma: Fix comments.
Thanks to Bruce Stark.
2019-06-03 20:41:54 +03:00
Lasse Collin c460f6defe liblzma: Fix one more unaligned read to use unaligned_read16ne(). 2019-06-02 00:50:59 +03:00
Lasse Collin 386394fc9f liblzma: memcmplen: Use ctz32() from tuklib_integer.h.
The same compiler-specific #ifdefs are already in tuklib_integer.h
2019-06-01 21:36:13 +03:00
Lasse Collin 264ab971ce tuklib_integer: Cleanup MSVC-specific code. 2019-06-01 21:30:03 +03:00
Lasse Collin 33773c6f2a liblzma: Use unaligned_readXXne functions instead of type punning.
Now gcc -fsanitize=undefined should be clean.

Thanks to Jeffrey Walton.
2019-06-01 19:01:21 +03:00
Lasse Collin 3bc112c2d3 tuklib_integer: Improve unaligned memory access.
Now memcpy() or GNU C packed structs for unaligned access instead
of type punning. See the comment in this commit for details.

Avoiding type punning with unaligned access is needed to
silence gcc -fsanitize=undefined.

New functions: unaliged_readXXne and unaligned_writeXXne where
XX is 16, 32, or 64.
2019-06-01 18:41:16 +03:00
Lasse Collin 2a22de439e liblzma: Avoid memcpy(NULL, foo, 0) because it is undefined behavior.
I should have always known this but I didn't. Here is an example
as a reminder to myself:

    int mycopy(void *dest, void *src, size_t n)
    {
        memcpy(dest, src, n);
        return dest == NULL;
    }

In the example, a compiler may assume that dest != NULL because
passing NULL to memcpy() would be undefined behavior. Testing
with GCC 8.2.1, mycopy(NULL, NULL, 0) returns 1 with -O0 and -O1.
With -O2 the return value is 0 because the compiler infers that
dest cannot be NULL because it was already used with memcpy()
and thus the test for NULL gets optimized out.

In liblzma, if a null-pointer was passed to memcpy(), there were
no checks for NULL *after* the memcpy() call, so I cautiously
suspect that it shouldn't have caused bad behavior in practice,
but it's hard to be sure, and the problematic cases had to be
fixed anyway.

Thanks to Jeffrey Walton.
2019-05-13 20:05:17 +03:00
Lasse Collin 4adb8288ab xz: Update xz man page date. 2019-05-11 20:54:12 +03:00
Antoine Cœur 2fb0ddaa55 spelling 2019-05-11 20:52:37 +03:00
Lasse Collin 4ed3396061 xz: In xz -lvv look at the widths of the check names too.
Now the widths of the check names is used to adjust the width
of the Check column. This way there no longer is a need to restrict
the widths of the check names to be at most ten terminal-columns.
2019-05-01 18:43:10 +03:00
Lasse Collin 2f4281a100 xz: Fix xz -lvv column alignment to look at the translated strings. 2019-05-01 18:33:25 +03:00
Lasse Collin a750c35a7d xz: Automatically align column headings in xz -lvv. 2019-03-04 21:20:39 +02:00
Lasse Collin 6cb42e8aa1 xz: Automatically align strings ending in a colon in --list output.
This should avoid alignment errors in translations with these
strings.
2019-03-04 21:16:59 +02:00
Lasse Collin b55d79461d xz: Fix a crash in progress indicator when in passthru mode.
"xz -dcfv not_an_xz_file" crashed (all four options are
required to trigger it). It caused xz to call
lzma_get_progress(&strm, ...) when no coder was initialized
in strm. In this situation strm.internal is NULL which leads
to a crash in lzma_get_progress().

The bug was introduced when xz started using lzma_get_progress()
to get progress info for multi-threaded compression, so the
bug is present in versions 5.1.3alpha and higher.

Thanks to Filip Palian <Filip.Palian@pjwstk.edu.pl> for
the bug report.
2018-12-20 20:39:20 +02:00
Lasse Collin 4ae5526de0 xz: Update man page timestamp. 2018-11-22 17:20:31 +02:00
Pavel Raiskup 6a36d0d5f4 'have have' typos 2018-11-22 17:19:09 +02:00
Lasse Collin a18ae42a79 liblzma: Don't verify header CRC32s if building for fuzz testing.
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is #defined when liblzma
is being built for fuzz testing.

Most fuzzed inputs would normally get rejected because of incorrect
CRC32 and the actual header decoding code wouldn't get fuzzed.
Disabling CRC32 checks avoids this problem. The fuzzer program
must still use LZMA_IGNORE_CHECK flag to disable verification of
integrity checks of uncompressed data.
2018-10-26 22:49:10 +03:00
Lasse Collin f76f7516d6 xzless: Rename unused variables to silence static analysers.
In this particular case I don't see this affecting readability
of the code.

Thanks to Pavel Raiskup.
2018-07-27 18:10:44 +03:00
Lasse Collin 3cbcaeb07e liblzma: Remove an always-true condition from lzma_index_cat().
This should help static analysis tools to see that newg
isn't leaked.

Thanks to Pavel Raiskup.
2018-07-27 16:02:58 +03:00
Lasse Collin 76762ae609 liblzma: Improve lzma_properties_decode() API documentation. 2018-05-19 21:23:25 +03:00
Lasse Collin 2267f5b0d2 Bump the version number to 5.3.1alpha. 2018-04-29 18:58:19 +03:00
Ben Boeckel bc19799169 nothrow: use noexcept for C++11 and newer
In C++11, the `throw()` specifier is deprecated and `noexcept` is
preffered instead.
2018-02-06 18:41:45 +02:00
Lasse Collin fb6d4f83cb liblzma: Remove incorrect #ifdef from range_common.h.
In most cases it was harmless but it could affect some
custom build systems.

Thanks to Pippijn van Steenhoven.
2018-02-06 18:02:48 +02:00
Lasse Collin 713bbc1a80 tuklib_integer: New Intel C compiler needs immintrin.h.
Thanks to Melanie Blower (Intel) for the patch.
2018-01-10 21:54:27 +02:00
Lasse Collin 94e3f986aa Fix or hide warnings from GCC 7's -Wimplicit-fallthrough. 2017-08-14 20:08:33 +03:00
Lasse Collin a015cd1f90 xz: Fix "xz --list --robot missing_or_bad_file.xz".
It ended up printing an uninitialized char-array when trying to
print the check names (column 7) on the "totals" line.

This also changes the column 12 (minimum xz version) to
50000002 (xz 5.0.0) instead of 0 when there are no valid
input files.

Thanks to kidmin for the bug report.
2017-05-23 18:34:43 +03:00
Lasse Collin 8269782283 xz: Use lzma_file_info_decoder() for --list. 2017-04-24 19:48:23 +03:00
Lasse Collin e353d0b1cc liblzma: Add lzma_file_info_decoder(). 2017-04-24 19:48:04 +03:00
Lasse Collin 8c9842c265 liblzma: Rename LZMA_SEEK to LZMA_SEEK_NEEDED and seek_in to seek_pos. 2017-04-21 15:05:16 +03:00
Lasse Collin 662b27c417 Update the home page URLs to HTTPS. 2017-04-19 22:17:35 +03:00
Lasse Collin c28f0b3d00 xz: Add io_seek_src(). 2017-04-05 18:47:22 +03:00
Lasse Collin bba477257d xz: Use POSIX_FADV_RANDOM for in "xz --list" mode.
xz --list is random access so POSIX_FADV_SEQUENTIAL was clearly
wrong.
2017-03-30 22:01:54 +03:00
Lasse Collin 310d19816d liblzma: Make lzma_index_decoder_init() visible to other liblzma funcs.
This is to allow other functions to use it without going
via the public API (lzma_index_decoder()).
2017-03-30 20:03:05 +03:00
Lasse Collin a27920002d liblzma: Add generic support for input seeking (LZMA_SEEK).
Also mention LZMA_SEEK in xz/message.c to silence a warning.
2017-03-30 20:00:09 +03:00
Lasse Collin a0b1dda409 liblzma: Fix lzma_memlimit_set(strm, 0).
The 0 got treated specially in a buggy way and as a result
the function did nothing. The API doc said that 0 was supposed
to return LZMA_PROG_ERROR but it didn't.

Now 0 is treated as if 1 had been specified. This is done because
0 is already used to indicate an error from lzma_memlimit_get()
and lzma_memusage().

In addition, lzma_memlimit_set() no longer checks that the new
limit is at least LZMA_MEMUSAGE_BASE. It's counter-productive
for the Index decoder and was actually needed only by the
auto decoder. Auto decoder has now been modified to check for
LZMA_MEMUSAGE_BASE.
2017-03-30 19:51:14 +03:00
Lasse Collin 84462afaad liblzma: Similar memlimit fix for stream_, alone_, and auto_decoder. 2017-03-30 19:16:55 +03:00
Lasse Collin cbc7401793 liblzma: Fix handling of memlimit == 0 in lzma_index_decoder().
It returned LZMA_PROG_ERROR, which was done to avoid zero as
the limit (because it's a special value elsewhere), but using
LZMA_PROG_ERROR is simply inconvenient and can cause bugs.

The fix/workaround is to treat 0 as if it were 1 byte. It's
effectively the same thing. The only weird consequence is
that then lzma_memlimit_get() will return 1 even when 0 was
specified as the limit.

This fixes a very rare corner case in xz --list where a specific
memory usage limit and a multi-stream file could print the
error message "Internal error (bug)" instead of saying that
the memory usage limit is too low.
2017-03-30 19:10:55 +03:00
Lasse Collin d4a0462abe liblzma: Avoid multiple definitions of lzma_coder structures.
Only one definition was visible in a translation unit.
It avoided a few casts and temp variables but seems that
this hack doesn't work with link-time optimizations in compilers
as it's not C99/C11 compliant.

Fixes:
http://www.mail-archive.com/xz-devel@tukaani.org/msg00279.html
2016-11-21 20:24:50 +02:00
Lasse Collin df8f446e3a tuklib_cpucores: Add support for sched_getaffinity().
It's available in glibc (GNU/Linux, GNU/kFreeBSD). It's better
than sysconf(_SC_NPROCESSORS_ONLN) because sched_getaffinity()
gives the number of cores available to the process instead of
the total number of cores online.

As a side effect, this commit fixes a bug on GNU/kFreeBSD where
configure would detect the FreeBSD-specific cpuset_getaffinity()
but it wouldn't actually work because on GNU/kFreeBSD it requires
using -lfreebsd-glue when linking. Now the glibc-specific function
will be used instead.

Thanks to Sebastian Andrzej Siewior for the original patch
and testing.
2016-10-24 18:51:36 +03:00
Lasse Collin 446e4318fa xz: Fix copying of timestamps on Windows.
xz used to call utime() on Windows, but its result gets lost
on close(). Using _futime() seems to work.

Thanks to Martok for reporting the bug:
http://www.mail-archive.com/xz-devel@tukaani.org/msg00261.html
2016-06-30 20:27:36 +03:00
Lasse Collin 1b0ac0c53c xz: Silence warnings from -Wlogical-op.
Thanks to Evan Nemerson.
2016-06-16 22:46:02 +03:00
Lasse Collin c83b7a0334 Build: Fix = to += for xz_SOURCES in src/xz/Makefile.am.
Thanks to Christian Kujau.
2016-04-10 20:55:49 +03:00
Lasse Collin ac398c3baf liblzma: Disable external SHA-256 by default.
This is the sane thing to do. The conflict with OpenSSL
on some OSes and especially that the OS-provided versions
can be significantly slower makes it clear that it was
a mistake to have the external SHA-256 support enabled by
default.

Those who want it can now pass --enable-external-sha256 to
configure. INSTALL was updated with notes about OSes where
this can be a bad idea.

The SHA-256 detection code in configure.ac had some bugs that
could lead to a build failure in some situations. These were
fixed, although it doesn't matter that much now that the
external SHA-256 is disabled by default.

MINIX >= 3.2.0 uses NetBSD's libc and thus has SHA256_Init
in libc instead of libutil. Support for the libutil version
was removed.
2016-03-13 20:21:49 +02:00
Lasse Collin faf302137e tuklib_physmem: Hopefully silence a warning on Windows. 2015-11-08 20:16:10 +02:00
Lasse Collin 14115f84a3 liblzma: Make Valgrind happier with optimized (gcc -O2) liblzma.
When optimizing, GCC can reorder code so that an uninitialized
value gets used in a comparison, which makes Valgrind unhappy.
It doesn't happen when compiled with -O0, which I tend to use
when running Valgrind.

Thanks to Rich Prohaska. I remember this being mentioned long
ago by someone else but nothing was done back then.
2015-11-04 23:14:00 +02:00
Lasse Collin f4c95ba94b liblzma: Rename lzma_presets.c back to lzma_encoder_presets.c.
It would be too annoying to update other build systems
just because of this.
2015-11-03 20:55:45 +02:00
Lasse Collin cb3111e3ed xz: Make xz buildable even when encoders or decoders are disabled.
The patch is quite long but it's mostly about adding new #ifdefs
to omit code when encoders or decoders have been disabled.

This adds two new #defines to config.h: HAVE_ENCODERS and
HAVE_DECODERS.
2015-11-03 20:29:33 +02:00
Lasse Collin 4cc584985c Build: Build LZMA1/2 presets also when only decoder is wanted.
People shouldn't rely on the presets when decoding raw streams,
but xz uses the presets as the starting point for raw decoder
options anyway.

lzma_encocder_presets.c was renamed to lzma_presets.c to
make it clear it's not used solely by the encoder code.
2015-11-03 18:06:40 +02:00
Lasse Collin b0bc3e0385 Build: Don't omit lzma_cputhreads() unless using --disable-threads.
Previously it was omitted if encoders were disabled
with --disable-encoders. It didn't make sense and
it also broke the build.
2015-11-03 17:41:54 +02:00
Lasse Collin c6bf438ab3 liblzma: Fix a build failure related to external SHA-256 support.
If an appropriate header and structure were found by configure,
but a library with a usable SHA-256 functions wasn't, the build
failed.
2015-11-02 18:16:51 +02:00
Lasse Collin e18adc56f2 xz: Always close the file before trying to delete it.
unlink() can return EBUSY in errno for open files on some
operating systems and file systems.
2015-11-02 15:19:10 +02:00
Lasse Collin 21515d79d7 liblzma: Fix lzma_index_dup() for empty Streams.
Stream Flags and Stream Padding weren't copied from
empty Streams.
2015-10-12 20:45:15 +03:00
Lasse Collin 09f395b6b3 liblzma: Add a note to index.c for those using static analyzers. 2015-10-12 20:31:44 +03:00
Lasse Collin 3bf857edfe liblzma: Fix a memory leak in error path of lzma_index_dup().
lzma_index_dup() calls index_dup_stream() which, in case of
an error, calls index_stream_end() to free memory allocated
by index_stream_init(). However, it illogically didn't
actually free the memory. To make it logical, the tree
handling code was modified a bit in addition to changing
index_stream_end().

Thanks to Evan Nemerson for the bug report.
2015-10-12 20:29:09 +03:00
Lasse Collin fbbb295a91 liblzma: A MSVC-specific hack isn't needed with MSVC 2013 and newer. 2015-07-12 20:48:19 +03:00
Lasse Collin 49c26920d6 xz: Document that threaded decompression hasn't been implemented yet. 2015-05-11 21:26:16 +03:00
Lasse Collin 6bd0349c58 Revert "xz: Use pipe2() if available."
This reverts commit 7a11c4a8e5.
It is a problem when libc has pipe2() but the kernel is too
old to have pipe2() and thus pipe2() fails. In xz it's pointless
to have a fallback for non-functioning pipe2(); it's better to
avoid pipe2() completely.

Thanks to Michael Fox for the bug report.
2015-04-20 20:17:48 +03:00
Lasse Collin fc0df0f8db xz: Fix the Capsicum rights on user_abort_pipe. 2015-04-01 14:45:25 +03:00
Lasse Collin 1238381143 xz: Add support for sandboxing with Capsicum.
The sandboxing is used conditionally as described in main.c.
This isn't optimal but it was much easier to implement than
a full sandboxing solution and it still covers the most common
use cases where xz is writing to standard output. This should
have practically no effect on performance even with small files
as fork() isn't needed.

C and locale libraries can open files as needed. This has been
fine in the past, but it's a problem with things like Capsicum.
io_sandbox_enter() tries to ensure that various locale-related
files have been loaded before cap_enter() is called, but it's
possible that there are other similar problems which haven't
been seen yet.

Currently Capsicum is available on FreeBSD 10 and later
and there is a port to Linux too.

Thanks to Loganaden Velvindron for help.
2015-03-31 22:19:34 +03:00
Lasse Collin 3717885f9e Bump version to 5.3.0alpha and soname to 5.3.99.
The idea of 99 is that it looks a bit weird in this context.
For new features there's no API/ABI stability in devel versions.
2015-03-30 22:44:02 +03:00
Lasse Collin 25263fd9e7 Fix the detection of installed RAM on QNX.
The earlier version compiled but didn't actually work
since sysconf(_SC_PHYS_PAGES) always fails (or so I was told).

Thanks to Ole André Vadla Ravnås for the patch and testing.
2015-03-29 22:13:48 +03:00
Lasse Collin e0ea6737b0 xz: size_t/uint32_t cleanup in options.c. 2015-03-07 22:05:57 +02:00
Lasse Collin 8bcca29a65 xz: Fix a comment and silence a warning in message.c. 2015-03-07 22:04:23 +02:00
Lasse Collin f243f5f44c liblzma: Silence more uint32_t vs. size_t warnings. 2015-03-07 22:01:00 +02:00
Lasse Collin 7f0a4c50f4 xz: Make arg_count an unsigned int to silence a warning.
Actually the value of arg_count cannot exceed INT_MAX
but it's nicer as an unsigned int.
2015-03-07 19:54:00 +02:00
Lasse Collin f6ec468015 liblzma: Fix a warning in index.c. 2015-03-07 19:33:17 +02:00
Lasse Collin dec11497a7 Bump version and soname for 5.2.1. 2015-02-26 16:53:44 +02:00
Lasse Collin 7a11c4a8e5 xz: Use pipe2() if available. 2015-02-22 19:38:48 +02:00
Lasse Collin 117d962685 liblzma: Fix a compression-ratio regression in LZMA1/2 in fast mode.
The bug was added in the commit
f48fce093b and thus
affected 5.1.4beta and 5.2.0. Luckily the bug cannot
cause data corruption or other nasty things.
2015-02-21 23:40:26 +02:00
Lasse Collin ae984e31c1 xz: Fix the fcntl() usage when creating a pipe for the self-pipe trick.
Now it reads the old flags instead of blindly setting O_NONBLOCK.
The old code may have worked correctly, but this is better.
2015-02-21 23:00:19 +02:00
Lasse Collin d935b0cdf3 tuklib_cpucores: Use cpuset_getaffinity() on FreeBSD if available.
In FreeBSD, cpuset_getaffinity() is the preferred way to get
the number of available cores.

Thanks to Rui Paulo for the patch. I edited it slightly, but
hopefully I didn't break anything.
2015-02-10 15:28:30 +02:00
Lasse Collin eb61bc58c2 xzdiff: Make the mktemp usage compatible with FreeBSD's mktemp.
Thanks to Rui Paulo for the fix.
2015-02-09 22:08:37 +02:00
Lasse Collin b9a5b6b7a2 Add a few casts to tuklib_integer.h to silence possible warnings.
I heard that Visual Studio 2013 gave warnings without the casts.

Thanks to Gabi Davar.
2015-02-03 21:45:53 +02:00
Lasse Collin c45757135f liblzma: Set LZMA_MEMCMPLEN_EXTRA depending on the compare method. 2015-01-26 21:24:39 +02:00
Lasse Collin fec88d41e6 liblzma: Silence harmless Valgrind errors.
Thanks to Torsten Rupp for reporting this. I had
forgotten to run Valgrind before the 5.2.0 release.
2015-01-26 20:39:28 +02:00
Lasse Collin a9b45badfe xz: Fix comments. 2015-01-09 21:50:19 +02:00
Lasse Collin 4170edc914 xz: Don't fail if stdout doesn't support O_NONBLOCK.
This is similar to the case with stdin.

Thanks to Brad Smith for the bug report and testing
on OpenBSD.
2015-01-09 21:34:06 +02:00
Lasse Collin 04bbc0c284 xz: Fix a memory leak in DOS-specific code. 2015-01-07 19:18:20 +02:00
Lasse Collin f0f1f6c723 xz: Don't fail if stdin doesn't support O_NONBLOCK.
It's a problem at least on OpenBSD which doesn't support
O_NONBLOCK on e.g. /dev/null. I'm not surprised if it's
a problem on other OSes too since this behavior is allowed
in POSIX-1.2008.

The code relying on this behavior was committed in June 2013
and included in 5.1.3alpha released on 2013-10-26. Clearly
the development releases only get limited testing.
2015-01-07 19:08:06 +02:00
Lasse Collin 6060f7dc76 Bump version and soname for 5.2.0.
I know that soname != app version, but I skip AGE=1
in -version-info to make the soname match the liblzma
version anyway. It doesn't hurt anything as long as
it doesn't conflict with library versioning rules.
2014-12-21 18:11:17 +02:00
Lasse Collin 2cb82ff21c Fix build when --disable-threads is used. 2014-12-21 18:00:38 +02:00
Lasse Collin 42e97a3264 xz: Fix a comment. 2014-12-21 14:07:54 +02:00
Lasse Collin 7f7d093de7 xz: Update the man page about --threads. 2014-12-16 21:00:09 +02:00
Lasse Collin 009823448b xz: Update the man page about --block-size. 2014-12-16 20:57:43 +02:00
Lasse Collin 1190c641af liblzma: Document how lzma_mt.block_size affects memory usage. 2014-12-02 20:04:07 +02:00
Lasse Collin 34f9e40a0a Remove LZMA_UNSTABLE macro. 2014-11-26 20:12:27 +02:00
Lasse Collin 6d9c0ce9f2 liblzma: Update lzma_stream_encoder_mt() API docs. 2014-11-26 20:10:33 +02:00
Lasse Collin 2301f3f05d liblzma: Verify the filter chain in threaded encoder initialization.
This way an invalid filter chain is detected at the Stream
encoder initialization instead of delaying it to the first
call to lzma_code() which triggers the initialization of
the actual filter encoder(s).
2014-11-25 12:32:05 +02:00
Lasse Collin 7b03a15cea xzdiff: Use mkdir if mktemp isn't available. 2014-11-10 18:54:40 +02:00
Lasse Collin f8c13e5e36 xzdiff: Create a temporary directory to hold a temporary file.
This avoids the possibility of "File name too long" when
creating a temp file when the input file name is very long.

This also means that other users on the system can no longer
see the input file names in /tmp (or whatever $TMPDIR is)
since the temporary directory will have a generic name. This
usually doesn't matter since on many systems one can see
the arguments given to all processes anyway.

The number X chars to mktemp where increased from 6 to 10.

Note that with some shells temp files or dirs won't be used at all.
2014-11-10 18:45:01 +02:00
Lasse Collin 7716dcf9df liblzma: Fix lzma_mt.preset in lzma_stream_encoder_mt_memusage().
It read the filter chain from a wrong variable. This is a similar
bug that was fixed in 9494fb6d0f.
2014-11-10 15:38:47 +02:00
Lasse Collin c923b140b2 Build: Prepare to support Automake's subdir-objects.
Due to a bug in Automake, subdir-objects won't be enabled
for now.

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17354

Thanks to Daniel Richard G. for the original patches.
2014-10-29 21:15:35 +02:00
Lasse Collin 076258cc45 Add support for AmigaOS/AROS to tuklib_physmem().
Thanks to Fredrik Wikstrom.
2014-10-09 19:41:51 +03:00
Lasse Collin efa7b0a210 xzgrep: Avoid passing both -q and -l to grep.
The behavior of grep -ql varies:
  - GNU grep behaves like grep -q.
  - OpenBSD grep behaves like grep -l.

POSIX doesn't make it 100 % clear what behavior is expected.
Anyway, using both -q and -l at the same time makes no sense
so both options simply should never be used at the same time.

Thanks to Christian Weisgerber.
2014-10-09 18:42:14 +03:00
Lasse Collin d62028b4c1 liblzma: Fix a portability problem in Makefile.am.
POSIX supports $< only in inference rules (suffix rules).
Using it elsewhere is a GNU make extension and doesn't
work e.g. with OpenBSD make.

Thanks to Christian Weisgerber for the patch.
2014-09-20 19:42:56 +03:00
Lasse Collin c35de31d42 Bump the version number to 5.1.4beta. 2014-09-14 21:54:09 +03:00
Lasse Collin 6b5e3b9eff xz: Add --ignore-check. 2014-08-05 22:32:36 +03:00
Lasse Collin 9adbc2ff37 liblzma: Add support for LZMA_IGNORE_CHECK. 2014-08-05 22:15:07 +03:00
Lasse Collin 0e0f34b8e4 liblzma: Add support for lzma_block.ignore_check.
Note that this slightly changes how lzma_block_header_decode()
has been documented. Earlier it said that the .version is set
to the lowest required value, but now it says that the .version
field is kept unchanged if possible. In practice this doesn't
affect any old code, because before this commit the only
possible .version was 0.
2014-08-05 22:03:30 +03:00
Lasse Collin 71e1437ab5 liblzma: Use lzma_memcmplen() in the BT3 match finder.
I had missed this when writing the commit
5db75054e9.

Thanks to Jun I Jin.
2014-08-04 19:25:58 +03:00
Lasse Collin 5dcffdbcc2 liblzma: SHA-256: Optimize the Maj macro slightly.
The Maj macro is used where multiple things are added
together, so making Maj a sum of two expressions allows
some extra freedom for the compiler to schedule the
instructions.

I learned this trick from
<http://www.hackersdelight.org/corres.txt>.
2014-08-03 21:32:25 +03:00
Lasse Collin a9477d1e0c liblzma: SHA-256: Optimize the way rotations are done.
This looks weird because the rotations become sequential,
but it helps quite a bit on both 32-bit and 64-bit x86:

  - It requires fewer instructions on two-operand
    instruction sets like x86.

  - It requires one register less which matters especially
    on 32-bit x86.

I hope this doesn't hurt other archs.

I didn't invent this idea myself, but I don't remember where
I saw it first.
2014-08-03 21:08:12 +03:00
Lasse Collin 5a76c7c8ee liblzma: SHA-256: Remove the GCC #pragma that became unneeded.
The unrolling in the previous commit should avoid the
situation where a compiler may think that an uninitialized
variable might be accessed.
2014-08-03 20:38:13 +03:00
Lasse Collin 9a096f8e57 liblzma: SHA-256: Unroll a little more.
This way a branch isn't needed for each operation
to choose between blk0 and blk2, and still the code
doesn't grow as much as it would with full unrolling.
2014-08-03 20:33:38 +03:00
Lasse Collin bc7650d87b liblzma: SHA-256: Do the byteswapping without a temporary buffer. 2014-08-03 19:56:43 +03:00
Lasse Collin 544aaa3d13 liblzma: Use lzma_memcmplen() in normal mode of LZMA.
Two locations were not changed yet because the simplest change
assumes that the initial "len" may be greater than "limit".
2014-07-25 22:38:28 +03:00
Lasse Collin f48fce093b liblzma: Simplify LZMA fast mode code by using memcmp(). 2014-07-25 22:30:38 +03:00
Lasse Collin 6bf5308e34 liblzma: Use lzma_memcmplen() in fast mode of LZMA. 2014-07-25 22:29:49 +03:00
Lasse Collin 5db75054e9 liblzma: Use lzma_memcmplen() in the match finders.
This doesn't change the match finder output.
2014-07-25 21:15:07 +03:00
Lasse Collin e1c8f1d01f liblzma: Add lzma_memcmplen() for fast memory comparison.
This commit just adds the function. Its uses will be in
separate commits.

This hasn't been tested much yet and it's perhaps a bit early
to commit it but if there are bugs they should get found quite
quickly.

Thanks to Jun I Jin from Intel for help and for pointing out
that string comparison needs to be optimized in liblzma.
2014-07-25 20:57:20 +03:00
Lasse Collin 17215f751c xz: Update the help message of a few options.
Updated: --threads, --block-size, and --block-list
Added: --flush-timeout
2014-06-29 20:54:14 +03:00
Lasse Collin 96864a6ddf xz: Use lzma_cputhreads() instead of own copy of tuklib_cpucores(). 2014-06-18 22:07:06 +03:00
Lasse Collin a115cc3748 liblzma: Add lzma_cputhreads(). 2014-06-18 22:04:24 +03:00
Lasse Collin 3ce3e79769 xz: Check for filter chain compatibility for --flush-timeout.
This avoids LZMA_PROG_ERROR from lzma_code() with filter chains
that don't support LZMA_SYNC_FLUSH.
2014-06-18 19:11:52 +03:00
Lasse Collin ceca379017 xzgrep: exit 0 when at least one file matches.
Mimic the original grep behavior and return exit_success when
at least one xz compressed file matches given pattern.

Original bugreport:
https://bugzilla.redhat.com/show_bug.cgi?id=1108085

Thanks to Pavel Raiskup for the patch.
2014-06-11 20:43:28 +03:00
Lasse Collin 8c19216bac xz: Force single-threaded mode when --flush-timeout is used. 2014-06-09 21:21:24 +03:00
Lasse Collin da1718f266 liblzma: Use lzma_alloc_zero() in LZ encoder initialization.
This avoids a memzero() call for a newly-allocated memory,
which can be expensive when encoding small streams with
an over-sized dictionary.

To avoid using lzma_alloc_zero() for memory that doesn't
need to be zeroed, lzma_mf.son is now allocated separately,
which requires handling it separately in normalize() too.

Thanks to Vincenzo Innocente for reporting the problem.
2014-05-25 21:45:56 +03:00
Lasse Collin 28af24e9cf liblzma: Add the internal function lzma_alloc_zero(). 2014-05-25 19:25:57 +03:00
Lasse Collin ed9ac85822 xz: Fix uint64_t vs. size_t which broke 32-bit build.
Thanks to Christian Hesse.
2014-05-08 18:03:09 +03:00
Lasse Collin 4d5b7b3fda liblzma: Rename the private API header lzma/lzma.h to lzma/lzma12.h.
It can be confusing that two header files have the same name.
The public API file is still lzma.h.
2014-05-04 11:07:17 +03:00
Lasse Collin 1555a9c566 Build: Fix the combination of --disable-xzdec --enable-lzmadec.
In this case "make install" could fail if the man page directory
didn't already exist at the destination. If it did exist, a
dangling symlink was created there. Now the link is omitted
instead. This isn't the best fix but it's better than the old
behavior.
2014-04-25 17:53:42 +03:00
Lasse Collin 54df428799 xz: Rename a variable to avoid a namespace collision on Solaris.
I don't know the details but I have an impression that there's
no problem in practice if using GCC since people have built xz
with GCC (without patching xz), but renaming the variable cannot
hurt either.

Thanks to Mark Ashley.
2014-04-09 17:26:10 +03:00
Lasse Collin 9494fb6d0f liblzma: Fix lzma_mt.preset not working with lzma_stream_encoder_mt().
It read the filter chain from a wrong variable.
2014-01-29 20:13:51 +02:00
Lasse Collin 673a4cb53d liblzma: Fix typo in a comment. 2014-01-20 11:20:40 +02:00
Lasse Collin 3d5c090872 xz: Fix a comment. 2014-01-12 17:41:14 +02:00
Lasse Collin 69fd4e1c93 Windows: Add MSVC defines for inline and restrict keywords. 2014-01-12 17:04:33 +02:00
Lasse Collin a19d9e8575 liblzma: Avoid C99 compound literal arrays.
MSVC 2013 doesn't like them. Maybe they aren't so good
for readability either since many aren't used to them.
2014-01-12 16:44:52 +02:00
Lasse Collin e28528f1c8 liblzma: Remove a useless C99ism from sha256.c.
Unsurprisingly it makes no difference in compiled output.
2014-01-12 12:50:30 +02:00
Lasse Collin 5ad1effc45 xz: Fix use of wrong variable.
Since the only call to suffix_set() uses optarg
as the argument, fixing this bug doesn't change
the behavior of the program.
2014-01-12 12:17:08 +02:00
Lasse Collin 3e62c68d75 Fix typos in comments. 2014-01-12 12:11:36 +02:00
Lasse Collin b22e94d8d1 liblzma: Document the need for block->check for lzma_block_header_decode().
Thanks to Tomer Chachamu.
2013-11-26 18:20:09 +02:00
Lasse Collin d1cd8b1cb8 xz: Update the man page about --block-size and --block-list. 2013-11-12 16:38:57 +02:00
Lasse Collin dd750acbe2 xz: Make --block-list and --block-size work together in single-threaded.
Previously, --block-list and --block-size only worked together
in threaded mode. Boundaries are specified by --block-list, but
--block-size specifies the maximum size for a Block. Now this
works in single-threaded mode too.

Thanks to James M Leddy for the original patch.
2013-11-12 16:29:48 +02:00
Lasse Collin ae222fe980 Bump the version number to 5.1.3alpha. 2013-10-26 13:26:14 +03:00
Lasse Collin 841da0352d xz: Document behavior of --block-list with threads.
This needs to be updated before 5.2.0.
2013-10-25 22:41:28 +03:00
Lasse Collin 56feb8665b xz: Document --flush-timeout=TIMEOUT on the man page. 2013-10-22 20:03:12 +03:00
Lasse Collin ba413da1d5 xz: Take advantage of LZMA_FULL_BARRIER with --block-list.
Now if --block-list is used in threaded mode, the encoder
won't need to flush at each Block boundary specified via
--block-list. This improves performance a lot, making
threading helpful with --block-list.

The flush timer was reset after LZMA_FULL_FLUSH but since
LZMA_FULL_BARRIER doesn't flush, resetting the timer is
no longer done.
2013-10-22 19:51:55 +03:00
Lasse Collin 0cd45fc2bc liblzma: Support LZMA_FULL_FLUSH and _BARRIER in threaded encoder.
Now --block-list=SIZES works with in the threaded mode too,
although the performance is still bad due to the use of
LZMA_FULL_FLUSH instead of the new LZMA_FULL_BARRIER.
2013-10-02 20:05:23 +03:00
Lasse Collin 97bb38712f liblzma: Add LZMA_FULL_BARRIER support to single-threaded encoder.
In the single-threaded encoder LZMA_FULL_BARRIER is simply
an alias for LZMA_FULL_FLUSH.
2013-10-02 12:55:11 +03:00
Lasse Collin fef0c6b410 liblzma: Add block_buffer_encoder.h into Makefile.inc.
This should have been in b465da5988.
2013-09-17 11:57:51 +03:00
Lasse Collin 8083e03291 xz: Add a missing test for TUKLIB_DOSLIKE. 2013-09-17 11:55:38 +03:00
Lasse Collin 6b44b4a775 Add native threading support on Windows.
Now liblzma only uses "mythread" functions and types
which are defined in mythread.h matching the desired
threading method.

Before Windows Vista, there is no direct equivalent to
pthread condition variables. Since this package doesn't
use pthread_cond_broadcast(), pre-Vista threading can
still be kept quite simple. The pre-Vista code doesn't
use anything that wasn't already available in Windows 95,
so the binaries should run even on Windows 95 if someone
happens to care.
2013-09-17 11:52:28 +03:00
Lasse Collin 72975df6c8 Build: Create liblzma.pc in a src/liblzma/Makefile.am.
Previously it was done in configure, but doing that goes
against the Autoconf manual. Autoconf requires that it is
possible to override e.g. prefix after running configure
and that doesn't work correctly if liblzma.pc is created
by configure.

A potential downside of this change is that now e.g.
libdir in liblzma.pc is a standalone string instead of
being defined via ${prefix}, so if one overrides prefix
when running pkg-config the libdir won't get the new value.
I don't know if this matters in practice.

Thanks to Vincent Torri.
2013-09-09 20:37:03 +03:00
Lasse Collin 1c2b6e7e83 Fix the previous commit which broke the build.
Apparently I didn't even compile-test the previous commit.

Thanks to Christian Hesse.
2013-08-04 15:24:09 +03:00
Lasse Collin 124eb69c78 Windows: Add Windows support to tuklib_cpucores().
It is used for Cygwin too. I'm not sure if that is
a good or bad idea.

Thanks to Vincent Torri.
2013-08-03 13:52:58 +03:00
Lasse Collin dee6ad3d59 xz: Add preliminary support for --flush-timeout=TIMEOUT.
When --flush-timeout=TIMEOUT is used, xz will use
LZMA_SYNC_FLUSH if read() would block and at least
TIMEOUT milliseconds has elapsed since the previous flush.

This can be useful in realtime-like use cases where the
data is simultanously decompressed by another process
(possibly on a different computer). If new uncompressed
input data is produced slowly, without this option xz could
buffer the data for a long time until it would become
decompressible from the output.

If TIMEOUT is 0, the feature is disabled. This is the default.

This commit affects the compression side. Using xz for
the decompression side for the above purpose doesn't work
yet so well because there is quite a bit of input and
output buffering when decompressing.

The --long-help or man page were not updated yet.
The details of this feature may change.
2013-07-04 14:18:46 +03:00
Lasse Collin fa381acaf9 xz: Don't set src_eof=true after an I/O error because it's useless. 2013-07-04 13:41:03 +03:00
Lasse Collin ea00545bea xz: Fix the test when to read more input.
Testing for end of file was no longer correct after full flushing
became possible with --block-size=SIZE and --block-list=SIZES.
There was no bug in practice though because xz just made a few
unneeded zero-byte reads.
2013-07-04 13:25:11 +03:00
Lasse Collin 736903c64b xz: Move some of the timing code into mytime.[hc].
This switches units from microseconds to milliseconds.

New clock_gettime(CLOCK_MONOTONIC) will be used if available.
There is still a fallback to gettimeofday().
2013-07-04 12:51:57 +03:00
Lasse Collin c0627b3fce xz: Silence a warning seen with _FORTIFY_SOURCE=2.
Thanks to Christian Hesse.
2013-07-01 14:34:11 +03:00
Lasse Collin a37ae8b5eb Man pages: Use similar syntax for synopsis as in xz.
The man pages of lzmainfo, xzmore, and xzdec had similar
constructs as the man page of xz had before the commit
eb6ca9854b. Eric S. Raymond
didn't mention these man pages in his bug report, but
it's nice to be consistent.
2013-06-30 18:02:27 +03:00
Lasse Collin cdba9ddd87 xz: Use non-blocking I/O for the output file.
Now both reading and writing should be without
race conditions with signals.

They might still be signal handling issues left.
Signals are blocked during many operations to avoid
EINTR but it may cause problems e.g. if writing to
stderr blocks when trying to display an error message.
2013-06-29 15:59:13 +03:00
Lasse Collin e61a5c95da xz: Fix return value type in io_write_buf().
It didn't affect the behavior of the code since -1
becomes true anyway.
2013-06-28 23:56:17 +03:00
Lasse Collin 9dc319eabb xz: Use the self-pipe trick to avoid a race condition with signals.
It is possible that a signal to set user_abort arrives right
before a blocking system call is made. In this case the call
may block until another signal arrives, while the wanted
behavior is to make xz clean up and exit as soon as possible.

After this commit, the race condition is avoided with the
input side which already uses non-blocking I/O. The output
side still uses blocking I/O and thus has the race condition.
2013-06-28 23:48:05 +03:00
Lasse Collin 3541bc79d0 xz: Use non-blocking I/O for the input file. 2013-06-28 22:51:02 +03:00
Lasse Collin 78673a08be xz: Remove an outdated NetBSD-specific comment.
Nowadays errno == EFTYPE is documented in open(2).
2013-06-28 18:46:13 +03:00
Lasse Collin a616fdad34 xz: Fix error detection of fcntl(fd, F_SETFL, flags) calls.
POSIX says that fcntl(fd, F_SETFL, flags) returns -1 on
error and "other than -1" on success. This is how it is
documented e.g. on OpenBSD too. On Linux, success with
F_SETFL is always 0 (at least accorinding to fcntl(2)
from man-pages 3.51).
2013-06-28 18:09:47 +03:00
Lasse Collin 4a08a6e4c6 xz: Fix use of wrong variable in a fcntl() call.
Due to a wrong variable name, when writing a sparse file
to standard output, *all* file status flags were cleared
(to the extent the operating system allowed it) instead of
only clearing the O_APPEND flag. In practice this worked
fine in the common situations on GNU/Linux, but I didn't
check how it behaved elsewhere.

The original flags were still restored correctly. I still
changed the code to use a separate boolean variable to
indicate when the flags should be restored instead of
relying on a special value in stdout_flags.
2013-06-28 17:36:47 +03:00
Lasse Collin b790b435da xz: Fix assertion related to posix_fadvise().
Input file can be a FIFO or something else that doesn't
support posix_fadvise() so don't check the return value
even with an assertion. Nothing bad happens if the call
to posix_fadvise() fails.
2013-06-28 14:55:37 +03:00
Lasse Collin 84d2da6c9d xz: Check the value of lzma_stream_flags.version in --list.
It is a no-op for now, but if an old xz version is used
together with a newer liblzma that supports something new,
then this check becomes important and will stop the old xz
from trying to parse files that it won't understand.
2013-06-26 13:30:57 +03:00
Lasse Collin 46540e4c10 liblzma: Avoid a warning about a shadowed variable.
On Mac OS X wait() is declared in <sys/wait.h> that
we include one way or other so don't use "wait" as
a variable name.

Thanks to Christian Kujau.
2013-06-23 18:57:23 +03:00
Lasse Collin ebb501ec73 xz: Validate Uncompressed Size from Block Header in list.c.
This affects only "xz -lvv". Normal decompression with xz
already detected if Block Header and Index had mismatched
Uncompressed Size fields. So this just makes "xz -lvv"
show such files as corrupt instead of showing the
Uncompressed Size from Index.
2013-06-23 17:36:47 +03:00
Lasse Collin eb6ca9854b xz: Make the man page more friendly to doclifter.
Thanks to Eric S. Raymond.
2013-06-21 22:04:45 +03:00
Lasse Collin 0c0a1947e6 xz: A couple of man page fixes.
Now the interaction of presets and custom filter chains
is described correctly. Earlier it contradicted itself.

Thanks to DevHC who reported these issues on IRC to me
on 2012-12-14.
2013-06-21 21:54:59 +03:00
Lasse Collin 2fcda89939 xz: Fix interaction between preset and custom filter chains.
There was somewhat illogical behavior when --extreme was
specified and mixed with custom filter chains.

Before this commit, "xz -9 --lzma2 -e" was equivalent
to "xz --lzma2". After it is equivalent to "xz -6e"
(all earlier preset options get forgotten when a custom
filter chain is specified and the default preset is 6
to which -e is applied). I find this less illogical.

This also affects the meaning of "xz -9e --lzma2 -7".
Earlier it was equivalent to "xz -7e" (the -e specified
before a custom filter chain wasn't forgotten). Now it
is "xz -7". Note that "xz -7e" still is the same as "xz -e7".

Hopefully very few cared about this in the first place,
so pretty much no one should even notice this change.

Thanks to Conley Moorhous.
2013-06-21 21:50:26 +03:00
Lasse Collin 8957c58609 xzdec: Improve the --help message.
The options are now ordered in the same order as in xz's help
message.

Descriptions were added to the options that are ignored.
I left them in parenthesis even if it looks a bit weird
because I find it easier to spot the ignored vs. non-ignored
options from the list that way.
2013-04-15 19:29:09 +03:00
Jeff Bastian 5019413a05 xzgrep: make the '-h' option to be --no-filename equivalent
* src/scripts/xzgrep.in: Accept the '-h' option in argument parsing.
2013-04-05 19:14:50 +03:00
Lasse Collin 5ea900cb5a liblzma: Be less picky in lzma_alone_decoder().
To avoid false positives when detecting .lzma files,
rare values in dictionary size and uncompressed size fields
were rejected. They will still be rejected if .lzma files
are decoded with lzma_auto_decoder(), but when using
lzma_alone_decoder() directly, such files will now be accepted.
Hopefully this is an OK compromise.

This doesn't affect xz because xz still has its own file
format detection code. This does affect lzmadec though.
So after this commit lzmadec will accept files that xz or
xz-emulating-lzma doesn't.

NOTE: lzma_alone_decoder() still won't decode all .lzma files
because liblzma's LZMA decoder doesn't support lc + lp > 4.

Reported here:
http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/7068827
2013-03-23 22:25:15 +02:00
Lasse Collin bb117fffa8 liblzma: Use lzma_block_buffer_bound64() in threaded encoder.
Now it uses lzma_block_uncomp_encode() if the data doesn't
fit into the space calculated by lzma_block_buffer_bound64().
2013-03-23 21:55:13 +02:00
Lasse Collin e572e123b5 liblzma: Fix another deadlock in the threaded encoder.
This race condition could cause a deadlock if lzma_end() was
called before finishing the encoding. This can happen with
xz with debugging enabled (non-debugging version doesn't
call lzma_end() before exiting).
2013-03-23 21:51:38 +02:00
Lasse Collin b465da5988 liblzma: Add lzma_block_uncomp_encode().
This also adds a new internal function
lzma_block_buffer_bound64() which is similar to
lzma_block_buffer_bound() but uses uint64_t instead
of size_t.
2013-03-23 19:17:33 +02:00
Lasse Collin 9e6dabcf22 Avoid unneeded use of awk in xzless.
Use "read" instead of "awk" in xzless to get the version
number of "less". The need for awk was introduced in
the commit db5c1817fa.

Thanks to Ariel P for the patch.
2013-03-05 19:14:50 +02:00
Lasse Collin e7b424d267 Make the progress indicator smooth in threaded mode.
This adds lzma_get_progress() to liblzma and takes advantage
of it in xz.

lzma_get_progress() collects progress information from
the thread-specific structures so that fairly accurate
progress information is available to applications. Adding
a new function seemed to be a better way than making the
information directly available in lzma_stream (like total_in
and total_out are) because collecting the information requires
locking mutexes. It's waste of time to do it more often than
the up to date information is actually needed by an application.
2012-12-14 20:13:32 +02:00
Lasse Collin 2ebbb994e3 liblzma: Fix mythread_sync for nested locking. 2012-12-14 11:01:41 +02:00
Lasse Collin 4c7e28705f xz: Mention --threads in --help.
Thanks to Olivier Delhomme for pointing out that this
was still missing.
2012-12-13 21:05:36 +02:00
Jonathan Nieder db5c1817fa xzless: Make "less -V" parsing more robust
In v4.999.9beta~30 (xzless: Support compressed standard input,
2009-08-09), xzless learned to parse ‘less -V’ output to figure out
whether less is new enough to handle $LESSOPEN settings starting
with “|-”.  That worked well for a while, but the version string from
‘less’ versions 448 (June, 2012) is misparsed, producing a warning:

	$ xzless /tmp/test.xz; echo $?
	/usr/bin/xzless: line 49: test: 456 (GNU regular expressions): \
	integer expression expected
	0

More precisely, modern ‘less’ lists the regexp implementation along
with its version number, and xzless passes the entire version number
with attached parenthetical phrase as a number to "test $a -gt $b",
producing the above confusing message.

	$ less-444 -V | head -1
	less 444
	$ less -V | head -1
	less 456 (no regular expressions)

So relax the pattern matched --- instead of expecting "less <number>",
look for a line of the form "less <number>[ (extra parenthetical)]".
While at it, improve the behavior when no matching line is found ---
instead of producing a cryptic message, we can fall back on a LESSPIPE
setting that is supported by all versions of ‘less’.

The implementation uses "awk" for simplicity.  Hopefully that’s
portable enough.

Reported-by: Jörg-Volker Peetz <jvpeetz@web.de>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
2012-11-21 19:19:44 +02:00
Lasse Collin 65536214a3 xz: Fix the note about --rsyncable on the man page. 2012-10-03 15:54:24 +03:00
Lasse Collin 3d93b63549 xz: Improve handling of failed realloc in xrealloc.
Thanks to Jim Meyering.
2012-09-28 20:11:09 +03:00
Lasse Collin ab22562066 A few typo fixes to comments and the xz man page.
Thanks to Jim Meyering.
2012-08-24 16:27:31 +03:00
Lasse Collin f3c1ec69d9 xz: Add a warning to --help about alpha and beta versions. 2012-08-13 21:40:09 +03:00
Lasse Collin 3778db1be5 liblzma: Make the use of lzma_allocator const-correct.
There is a tiny risk of causing breakage: If an application
assigns lzma_stream.allocator to a non-const pointer, such
code won't compile anymore. I don't know why anyone would do
such a thing though, so in practice this shouldn't cause trouble.

Thanks to Jan Kratochvil for the patch.
2012-07-17 18:19:59 +03:00
Lasse Collin d6e0b23d46 Build: Include validate_map.sh in the distribution.
It's required by "make mydist".

Fix also the location of EXTRA_DIST+= so that those files
get distributed also if symbol versioning isn't enabled.
2012-07-05 07:28:53 +03:00
Lasse Collin cafb523ada xz: Document --block-list better.
Thanks to Jonathan Nieder.
2012-07-04 22:31:58 +03:00
Lasse Collin c7ff218528 Bump the version number to 5.1.2alpha. 2012-07-04 20:01:49 +03:00
Lasse Collin 0d5fa05466 xz: Fix the version number printed by xz -lvv.
The decoder bug was fixed in 5.0.2 instead of 5.0.3.
2012-07-04 19:58:23 +03:00
Lasse Collin 88ccf47205 xz: Add incomplete support for --block-list.
It's broken with threads and when also --block-size is used.
2012-07-03 21:16:39 +03:00
Lasse Collin 972179cdcd xz: Update the man page about the new field in --robot -lvv. 2012-07-01 18:44:33 +03:00
Lasse Collin 1403707fc6 liblzma: Check that the first byte of range encoded data is 0x00.
It is just to be more pedantic and thus perhaps catch broken
files slightly earlier.
2012-06-28 10:47:49 +03:00
Lasse Collin 2e6754eac2 xz: Update man page date to match the latest update. 2012-06-22 14:34:03 +03:00
Lasse Collin d8db706acb liblzma: Fix possibility of incorrect LZMA_BUF_ERROR.
lzma_code() could incorrectly return LZMA_BUF_ERROR if
all of the following was true:

  - The caller knows how many bytes of output to expect
    and only provides that much output space.

  - When the last output bytes are decoded, the
    caller-provided input buffer ends right before
    the LZMA2 end of payload marker. So LZMA2 won't
    provide more output anymore, but it won't know it
    yet and thus won't return LZMA_STREAM_END yet.

  - A BCJ filter is in use and it hasn't left any
    unfiltered bytes in the temp buffer. This can happen
    with any BCJ filter, but in practice it's more likely
    with filters other than the x86 BCJ.

Another situation where the bug can be triggered happens
if the uncompressed size is zero bytes and no output space
is provided. In this case the decompression can fail even
if the whole input file is given to lzma_code().

A similar bug was fixed in XZ Embedded on 2011-09-19.
2012-05-28 20:42:11 +03:00
Lasse Collin 7769ea051d xz: Don't show a huge number in -vv when memory limit is disabled. 2012-05-28 15:37:43 +03:00
Lasse Collin ec92110572 xz: Document the "summary" lines of --robot -lvv.
This documents only the columns that are in v5.0.
The new columns added in the master branch aren't
necessarily stable yet.
2012-05-27 22:30:17 +03:00
Lasse Collin 27d24eb0a9 xz: Fix output of verbose --robot --list modes.
It printed the filename in "filename (x/y)" format
which it obviously shouldn't do in robot mode.
2012-05-27 21:53:20 +03:00
Lasse Collin aac1b31ea4 liblzma: Remove outdated comments. 2012-04-19 15:25:26 +03:00
Lasse Collin 03ed742a3a liblzma: Fix Libs.private in liblzma.pc to include -lrt when needed. 2012-04-19 14:02:25 +03:00
Lasse Collin cff070aba6 Fix exit status of xzgrep when grepping binary files.
When grepping binary files, grep may exit before it has
read all the input. In this case, gzip -q returns 2 (eating
SIGPIPE), but xz and bzip2 show SIGPIPE as the exit status
(e.g. 141). This causes wrong exit status when grepping
xz- or bzip2-compressed binary files.

The fix checks for the special exit status that indicates SIGPIPE.
It uses kill -l which should be supported everywhere since it
is in both SUSv2 (1997) and POSIX.1-2008.

Thanks to James Buren for the bug report.
2012-02-22 14:02:34 +02:00
Lasse Collin 418fe668b3 xz: Show minimum required XZ Utils version in xz -lvv.
Man page wasn't updated yet.
2011-11-07 13:07:52 +02:00
Lasse Collin 7081d82c37 xz: Fix a typo in a comment.
Thanks to Bela Lubkin.
2011-11-04 17:57:16 +02:00
Lasse Collin 74d2bae4d3 xz: Fix xz on EBCDIC systems.
Thanks to Chris Donawa.
2011-11-03 17:07:22 +02:00
Lasse Collin ab50ae3ef4 liblzma: Fix invalid free() in the threaded encoder.
It was triggered if initialization failed e.g. due to
running out of memory.

Thanks to Arkadiusz Miskiewicz.
2011-10-23 17:08:14 +03:00
Lasse Collin 6b620a0f08 liblzma: Fix a deadlock in the threaded encoder.
It was triggered when reinitializing the encoder,
e.g. when encoding two files.
2011-10-23 17:05:55 +03:00
Lasse Collin 5b1e1f1074 Workaround unusual SIZE_MAX on SCO OpenServer. 2011-08-09 21:16:44 +03:00
Lasse Collin 1c673e5681 Fix exit status of "xzdiff foo.xz bar.xz".
xzdiff was clobbering the exit status from diff in a case
statement used to analyze the exit statuses from "xz" when
its operands were two compressed files. Save and restore
diff's exit status to fix this.

The bug is inherited from zdiff in GNU gzip and was fixed
there on 2009-10-09.

Thanks to Jonathan Nieder for the patch and
to Peter Pallinger for reporting the bug.
2011-07-31 11:01:47 +03:00