Commit Graph

1416 Commits

Author SHA1 Message Date
Lasse Collin ba413da1d5 xz: Take advantage of LZMA_FULL_BARRIER with --block-list.
Now if --block-list is used in threaded mode, the encoder
won't need to flush at each Block boundary specified via
--block-list. This improves performance a lot, making
threading helpful with --block-list.

The flush timer was reset after LZMA_FULL_FLUSH but since
LZMA_FULL_BARRIER doesn't flush, resetting the timer is
no longer done.
2013-10-22 19:51:55 +03:00
Lasse Collin 0cd45fc2bc liblzma: Support LZMA_FULL_FLUSH and _BARRIER in threaded encoder.
Now --block-list=SIZES works with in the threaded mode too,
although the performance is still bad due to the use of
LZMA_FULL_FLUSH instead of the new LZMA_FULL_BARRIER.
2013-10-02 20:05:23 +03:00
Lasse Collin 97bb38712f liblzma: Add LZMA_FULL_BARRIER support to single-threaded encoder.
In the single-threaded encoder LZMA_FULL_BARRIER is simply
an alias for LZMA_FULL_FLUSH.
2013-10-02 12:55:11 +03:00
Lasse Collin fef0c6b410 liblzma: Add block_buffer_encoder.h into Makefile.inc.
This should have been in b465da5988.
2013-09-17 11:57:51 +03:00
Lasse Collin 8083e03291 xz: Add a missing test for TUKLIB_DOSLIKE. 2013-09-17 11:55:38 +03:00
Lasse Collin 6b44b4a775 Add native threading support on Windows.
Now liblzma only uses "mythread" functions and types
which are defined in mythread.h matching the desired
threading method.

Before Windows Vista, there is no direct equivalent to
pthread condition variables. Since this package doesn't
use pthread_cond_broadcast(), pre-Vista threading can
still be kept quite simple. The pre-Vista code doesn't
use anything that wasn't already available in Windows 95,
so the binaries should run even on Windows 95 if someone
happens to care.
2013-09-17 11:52:28 +03:00
Lasse Collin ae0ab74a88 Build: Remove a comment about Automake 1.10 from configure.ac.
The previous commit supports silent rules and that requires
Automake 1.11.
2013-09-11 14:40:35 +03:00
Lasse Collin 72975df6c8 Build: Create liblzma.pc in a src/liblzma/Makefile.am.
Previously it was done in configure, but doing that goes
against the Autoconf manual. Autoconf requires that it is
possible to override e.g. prefix after running configure
and that doesn't work correctly if liblzma.pc is created
by configure.

A potential downside of this change is that now e.g.
libdir in liblzma.pc is a standalone string instead of
being defined via ${prefix}, so if one overrides prefix
when running pkg-config the libdir won't get the new value.
I don't know if this matters in practice.

Thanks to Vincent Torri.
2013-09-09 20:37:03 +03:00
Lasse Collin 1c2b6e7e83 Fix the previous commit which broke the build.
Apparently I didn't even compile-test the previous commit.

Thanks to Christian Hesse.
2013-08-04 15:24:09 +03:00
Lasse Collin 124eb69c78 Windows: Add Windows support to tuklib_cpucores().
It is used for Cygwin too. I'm not sure if that is
a good or bad idea.

Thanks to Vincent Torri.
2013-08-03 13:52:58 +03:00
Anders F Bjorklund eada8a875c macosx: separate liblzma package 2013-08-03 13:15:32 +03:00
Anders F Bjorklund be0100d01c macosx: set minimum to leopard 2013-08-03 13:15:32 +03:00
Anders F Bjorklund 416729e2d7 move configurables into variables 2013-08-03 13:15:32 +03:00
Lasse Collin 16581080e5 Update THANKS. 2013-07-15 14:08:41 +03:00
Lasse Collin 3e2b198ba3 Build: Fix the detection of missing CRC32.
Thanks to Vincent Torri.
2013-07-15 14:08:02 +03:00
Lasse Collin dee6ad3d59 xz: Add preliminary support for --flush-timeout=TIMEOUT.
When --flush-timeout=TIMEOUT is used, xz will use
LZMA_SYNC_FLUSH if read() would block and at least
TIMEOUT milliseconds has elapsed since the previous flush.

This can be useful in realtime-like use cases where the
data is simultanously decompressed by another process
(possibly on a different computer). If new uncompressed
input data is produced slowly, without this option xz could
buffer the data for a long time until it would become
decompressible from the output.

If TIMEOUT is 0, the feature is disabled. This is the default.

This commit affects the compression side. Using xz for
the decompression side for the above purpose doesn't work
yet so well because there is quite a bit of input and
output buffering when decompressing.

The --long-help or man page were not updated yet.
The details of this feature may change.
2013-07-04 14:18:46 +03:00
Lasse Collin fa381acaf9 xz: Don't set src_eof=true after an I/O error because it's useless. 2013-07-04 13:41:03 +03:00
Lasse Collin ea00545bea xz: Fix the test when to read more input.
Testing for end of file was no longer correct after full flushing
became possible with --block-size=SIZE and --block-list=SIZES.
There was no bug in practice though because xz just made a few
unneeded zero-byte reads.
2013-07-04 13:25:11 +03:00
Lasse Collin 736903c64b xz: Move some of the timing code into mytime.[hc].
This switches units from microseconds to milliseconds.

New clock_gettime(CLOCK_MONOTONIC) will be used if available.
There is still a fallback to gettimeofday().
2013-07-04 12:51:57 +03:00
Lasse Collin 24edf8d807 Update THANKS. 2013-07-01 14:35:03 +03:00
Lasse Collin c0627b3fce xz: Silence a warning seen with _FORTIFY_SOURCE=2.
Thanks to Christian Hesse.
2013-07-01 14:34:11 +03:00
Lasse Collin 1936718bb3 Update NEWS for 5.0.5. 2013-06-30 19:40:11 +03:00
Lasse Collin a37ae8b5eb Man pages: Use similar syntax for synopsis as in xz.
The man pages of lzmainfo, xzmore, and xzdec had similar
constructs as the man page of xz had before the commit
eb6ca9854b. Eric S. Raymond
didn't mention these man pages in his bug report, but
it's nice to be consistent.
2013-06-30 18:02:27 +03:00
Lasse Collin cdba9ddd87 xz: Use non-blocking I/O for the output file.
Now both reading and writing should be without
race conditions with signals.

They might still be signal handling issues left.
Signals are blocked during many operations to avoid
EINTR but it may cause problems e.g. if writing to
stderr blocks when trying to display an error message.
2013-06-29 15:59:13 +03:00
Lasse Collin e61a5c95da xz: Fix return value type in io_write_buf().
It didn't affect the behavior of the code since -1
becomes true anyway.
2013-06-28 23:56:17 +03:00
Lasse Collin 9dc319eabb xz: Use the self-pipe trick to avoid a race condition with signals.
It is possible that a signal to set user_abort arrives right
before a blocking system call is made. In this case the call
may block until another signal arrives, while the wanted
behavior is to make xz clean up and exit as soon as possible.

After this commit, the race condition is avoided with the
input side which already uses non-blocking I/O. The output
side still uses blocking I/O and thus has the race condition.
2013-06-28 23:48:05 +03:00
Lasse Collin 3541bc79d0 xz: Use non-blocking I/O for the input file. 2013-06-28 22:51:02 +03:00
Lasse Collin 78673a08be xz: Remove an outdated NetBSD-specific comment.
Nowadays errno == EFTYPE is documented in open(2).
2013-06-28 18:46:13 +03:00
Lasse Collin a616fdad34 xz: Fix error detection of fcntl(fd, F_SETFL, flags) calls.
POSIX says that fcntl(fd, F_SETFL, flags) returns -1 on
error and "other than -1" on success. This is how it is
documented e.g. on OpenBSD too. On Linux, success with
F_SETFL is always 0 (at least accorinding to fcntl(2)
from man-pages 3.51).
2013-06-28 18:09:47 +03:00
Lasse Collin 4a08a6e4c6 xz: Fix use of wrong variable in a fcntl() call.
Due to a wrong variable name, when writing a sparse file
to standard output, *all* file status flags were cleared
(to the extent the operating system allowed it) instead of
only clearing the O_APPEND flag. In practice this worked
fine in the common situations on GNU/Linux, but I didn't
check how it behaved elsewhere.

The original flags were still restored correctly. I still
changed the code to use a separate boolean variable to
indicate when the flags should be restored instead of
relying on a special value in stdout_flags.
2013-06-28 17:36:47 +03:00
Lasse Collin b790b435da xz: Fix assertion related to posix_fadvise().
Input file can be a FIFO or something else that doesn't
support posix_fadvise() so don't check the return value
even with an assertion. Nothing bad happens if the call
to posix_fadvise() fails.
2013-06-28 14:55:37 +03:00
Lasse Collin 84d2da6c9d xz: Check the value of lzma_stream_flags.version in --list.
It is a no-op for now, but if an old xz version is used
together with a newer liblzma that supports something new,
then this check becomes important and will stop the old xz
from trying to parse files that it won't understand.
2013-06-26 13:30:57 +03:00
Lasse Collin 9376f5f8f7 Build: Require Automake 1.12 and use serial-tests option.
It should actually still work with Automake 1.10 if
the serial-tests option is removed. Automake 1.13 started
using parallel tests by default and the option to get
the old behavior isn't supported before 1.12.

At least for now, parallel tests don't improve anything
in XZ Utils but they hide the progress output from
test_compress.sh.
2013-06-26 12:17:00 +03:00
Lasse Collin b7e200d7bd Update THANKS. 2013-06-23 18:59:13 +03:00
Lasse Collin 46540e4c10 liblzma: Avoid a warning about a shadowed variable.
On Mac OS X wait() is declared in <sys/wait.h> that
we include one way or other so don't use "wait" as
a variable name.

Thanks to Christian Kujau.
2013-06-23 18:57:23 +03:00
Lasse Collin ebb501ec73 xz: Validate Uncompressed Size from Block Header in list.c.
This affects only "xz -lvv". Normal decompression with xz
already detected if Block Header and Index had mismatched
Uncompressed Size fields. So this just makes "xz -lvv"
show such files as corrupt instead of showing the
Uncompressed Size from Index.
2013-06-23 17:36:47 +03:00
Lasse Collin c09e91dd23 Update THANKS. 2013-06-21 22:08:11 +03:00
Lasse Collin eb6ca9854b xz: Make the man page more friendly to doclifter.
Thanks to Eric S. Raymond.
2013-06-21 22:04:45 +03:00
Lasse Collin 0c0a1947e6 xz: A couple of man page fixes.
Now the interaction of presets and custom filter chains
is described correctly. Earlier it contradicted itself.

Thanks to DevHC who reported these issues on IRC to me
on 2012-12-14.
2013-06-21 21:54:59 +03:00
Lasse Collin 2fcda89939 xz: Fix interaction between preset and custom filter chains.
There was somewhat illogical behavior when --extreme was
specified and mixed with custom filter chains.

Before this commit, "xz -9 --lzma2 -e" was equivalent
to "xz --lzma2". After it is equivalent to "xz -6e"
(all earlier preset options get forgotten when a custom
filter chain is specified and the default preset is 6
to which -e is applied). I find this less illogical.

This also affects the meaning of "xz -9e --lzma2 -7".
Earlier it was equivalent to "xz -7e" (the -e specified
before a custom filter chain wasn't forgotten). Now it
is "xz -7". Note that "xz -7e" still is the same as "xz -e7".

Hopefully very few cared about this in the first place,
so pretty much no one should even notice this change.

Thanks to Conley Moorhous.
2013-06-21 21:50:26 +03:00
Lasse Collin 97379c5ea7 Build: Use -Wvla with GCC if supported.
Variable-length arrays are mandatory in C99 but optional in C11.
The code doesn't currently use any VLAs and it shouldn't in the
future either to stay compatible with C11 without requiring any
optional C11 features.
2013-04-27 22:07:46 +03:00
Lasse Collin 8957c58609 xzdec: Improve the --help message.
The options are now ordered in the same order as in xz's help
message.

Descriptions were added to the options that are ignored.
I left them in parenthesis even if it looks a bit weird
because I find it easier to spot the ignored vs. non-ignored
options from the list that way.
2013-04-15 19:29:09 +03:00
Lasse Collin ed886e1a92 Update THANKS. 2013-04-05 19:25:40 +03:00
Jeff Bastian 5019413a05 xzgrep: make the '-h' option to be --no-filename equivalent
* src/scripts/xzgrep.in: Accept the '-h' option in argument parsing.
2013-04-05 19:14:50 +03:00
Lasse Collin 5ea900cb5a liblzma: Be less picky in lzma_alone_decoder().
To avoid false positives when detecting .lzma files,
rare values in dictionary size and uncompressed size fields
were rejected. They will still be rejected if .lzma files
are decoded with lzma_auto_decoder(), but when using
lzma_alone_decoder() directly, such files will now be accepted.
Hopefully this is an OK compromise.

This doesn't affect xz because xz still has its own file
format detection code. This does affect lzmadec though.
So after this commit lzmadec will accept files that xz or
xz-emulating-lzma doesn't.

NOTE: lzma_alone_decoder() still won't decode all .lzma files
because liblzma's LZMA decoder doesn't support lc + lp > 4.

Reported here:
http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/7068827
2013-03-23 22:25:15 +02:00
Lasse Collin bb117fffa8 liblzma: Use lzma_block_buffer_bound64() in threaded encoder.
Now it uses lzma_block_uncomp_encode() if the data doesn't
fit into the space calculated by lzma_block_buffer_bound64().
2013-03-23 21:55:13 +02:00
Lasse Collin e572e123b5 liblzma: Fix another deadlock in the threaded encoder.
This race condition could cause a deadlock if lzma_end() was
called before finishing the encoding. This can happen with
xz with debugging enabled (non-debugging version doesn't
call lzma_end() before exiting).
2013-03-23 21:51:38 +02:00
Lasse Collin b465da5988 liblzma: Add lzma_block_uncomp_encode().
This also adds a new internal function
lzma_block_buffer_bound64() which is similar to
lzma_block_buffer_bound() but uses uint64_t instead
of size_t.
2013-03-23 19:17:33 +02:00
Lasse Collin 9e6dabcf22 Avoid unneeded use of awk in xzless.
Use "read" instead of "awk" in xzless to get the version
number of "less". The need for awk was introduced in
the commit db5c1817fa.

Thanks to Ariel P for the patch.
2013-03-05 19:14:50 +02:00
Lasse Collin e7b424d267 Make the progress indicator smooth in threaded mode.
This adds lzma_get_progress() to liblzma and takes advantage
of it in xz.

lzma_get_progress() collects progress information from
the thread-specific structures so that fairly accurate
progress information is available to applications. Adding
a new function seemed to be a better way than making the
information directly available in lzma_stream (like total_in
and total_out are) because collecting the information requires
locking mutexes. It's waste of time to do it more often than
the up to date information is actually needed by an application.
2012-12-14 20:13:32 +02:00