Commit Graph

2177 Commits

Author SHA1 Message Date
Lasse Collin f644473a21 liblzma: Add fast CRC64 for 32/64-bit x86 using SSSE3 + SSE4.1 + CLMUL.
It also works on E2K as it supports these intrinsics.

On x86-64 runtime detection is used so the code keeps working on
older processors too. A CLMUL-only build can be done by using
-msse4.1 -mpclmul in CFLAGS and this will reduce the library
size since the generic implementation and its 8 KiB lookup table
will be omitted.

On 32-bit x86 this isn't used by default for now because by default
on 32-bit x86 the separate assembly file crc64_x86.S is used.
If --disable-assembler is used then this new CLMUL code is used
the same way as on 64-bit x86. However, a CLMUL-only build
(-msse4.1 -mpclmul) won't omit the 8 KiB lookup table on
32-bit x86 due to a currently-missing check for disabled
assembler usage.

The configure.ac check should be such that the code won't be
built if something in the toolchain doesn't support it but
--disable-clmul-crc option can be used to unconditionally
disable this feature.

CLMUL speeds up decompression of files that have compressed very
well (assuming CRC64 is used as a check type). It is know that
the CLMUL code is significantly slower than the generic code for
tiny inputs (especially 1-8 bytes but up to 16 bytes). If that
is a real-world problem then there is already a commented-out
variant that uses the generic version for small inputs.

Thanks to Ilya Kurdyukov for the original patch which was
derived from a white paper from Intel [1] (published in 2009)
and public domain code from [2] (released in 2016).

[1] https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
[2] https://github.com/rawrunprotected/crc
2022-11-14 23:05:46 +02:00
Lasse Collin 3b466bc796 Translations: Update the Swedish translation one more time. 2022-11-14 20:14:34 +02:00
Lasse Collin e963379a86 Translations: Update the Swedish translation again. 2022-11-14 19:34:15 +02:00
Lasse Collin a4bc689a82 Translations: Update the Swedish translation. 2022-11-14 19:07:45 +02:00
Lasse Collin bbf2073d82 Translations: Update the Ukrainian translation. 2022-11-14 18:58:09 +02:00
Lasse Collin ac10b1b362 Build: Omit x86_64 from --enable-assembler.
It didn't do anything. There are only 32-bit x86 assembly files
and it feels likely that new files won't be added as intrinsics
in C are more portable across toolchains and OSes.
2022-11-14 17:58:07 +02:00
Lasse Collin eb0f1450ad liblzma: Use __attribute__((__constructor__)) if available.
This uses it for CRC table initializations when using --disable-small.
It avoids mythread_once() overhead. It also means that then
--disable-small --disable-threads is thread-safe if this attribute
is supported.
2022-11-14 16:00:52 +02:00
Lasse Collin 6553f49b11 Translations: Update the Romanian translation. 2022-11-12 21:19:52 +02:00
Lasse Collin db97e69e12 Translations: Update the Hungarian translation. 2022-11-12 21:17:45 +02:00
Lasse Collin 2bbb9c0f38 Translations: Update the Finnish translation. 2022-11-11 17:58:57 +02:00
Lasse Collin 3c8cbb8137 Translations: Update the Croatian translation. 2022-11-11 17:58:18 +02:00
Lasse Collin 26c3359eac Translations: Update the Polish translation. 2022-11-11 17:57:18 +02:00
Lasse Collin 577e467b13 Translations: Update the Spanish translation. 2022-11-11 17:56:44 +02:00
Lasse Collin f9b4ff6e9a Update THANKS. 2022-11-11 17:16:03 +02:00
Lasse Collin a39961ef21 liblzma: Fix building with Intel ICC (the classic compiler).
It claims __GNUC__ >= 10 but doesn't support __symver__ attribute.

Thanks to Stephen Sachs.
2022-11-11 17:15:25 +02:00
Lasse Collin c715f683dc liblzma: Fix incorrect #ifdef for x86 SSE2 support.
__SSE2__ is the correct macro for SSE2 support with GCC, Clang,
and ICC. __SSE2_MATH__ means doing floating point math with SSE2
instead of 387. Often the latter macro is defined if the first
one is but it was still a bug.
2022-11-11 14:35:58 +02:00
Lasse Collin bd334ae56a Add NEWS for 5.2.7 (forgotten cherry-pick from v5.2). 2022-11-11 13:27:06 +02:00
Lasse Collin 3c7860cf49 xzdiff: Add support for .lz files.
The other scripts don't need changes for .lz support because
in those scripts it is enough that xz supports .lz.
2022-11-11 13:16:21 +02:00
Lasse Collin d76c752a6d Scripts: Ignore warnings from xz.
In practice this means making the scripts work when
the input files have an unsupported check type which
isn't a problem in practice unless support for
some check types has been disabled at build time.
2022-11-11 12:23:58 +02:00
Lasse Collin 6552535afd Translations: Rename poa4/fr_FR.po to po4a/fr.po.
That's how it is preferred at the Translation Project.
On my system /usr/share/man/fr_FR doesn't contain any
other man pages than XZ Utils while /usr/share/man/fr
has quite a few, so this will fix that too.

Thanks to Benno Schulenberg from the Translation Project.
2022-11-10 12:39:08 +02:00
Lasse Collin 0918159ce4 xz: Update the man page about BCJ filters, including upcoming --arm64.
The --arm64 isn't actually implemented yet in the form
described in this commit.

Thanks to Jia Tan.
2022-11-09 19:09:26 +02:00
Lasse Collin ba2ae3596f xz: Add --arm64 to --long-help and omit endianness from ARM(-Thumb).
Modern 32-bit ARM in big endian mode use little endian for
instruction encoding still, so the filters work on such
executables too. It's likely less confusing for users this way.

The --arm64 option hasn't been implemented yet (there is
--experimental-arm64 but it's different). The --arm64 option
is added now anyway because this is the likely result and the
strings need to be ready for translators.

Thanks to Jia Tan.
2022-11-09 18:14:14 +02:00
Lasse Collin 802d57d921 Windows: Update the VS project files for ARM64 and .lz support. 2022-11-09 15:12:13 +02:00
Lasse Collin 5846aeda05 DOS: Update Makefile and config.h to include ARM64 and .lz support. 2022-11-09 14:57:48 +02:00
Lasse Collin 781da8d6c4 CMake: Add lzip decoder files and #define to the build. 2022-11-09 14:45:05 +02:00
Lasse Collin df8ad4af65 Docs: Update INSTALL and also add new prohibited options to PACKAGERS. 2022-11-09 14:41:56 +02:00
Lasse Collin c8ef089c14 Tests: Test the .lz files in test_files.sh. 2022-11-09 14:32:33 +02:00
Lasse Collin c8f70ebb46 Tests: Add .lz (lzip) test files. 2022-11-09 14:31:21 +02:00
Lasse Collin 731db13e6f xz: Remove the commented-out FORMAT_GZIP, gzip, .gz, and .tgz. 2022-11-09 14:31:10 +02:00
Lasse Collin 3176f992c5 xz: Add .lz (lzip) decompression support.
If configured with --disable-lzip-decoder then --long-help will
still list `lzip' in --format but I left it like that since
due to translations it would be messy to have two help strings.
Features are disabled only in special situations so wrong help
in such a situation shouldn't matter much.

Thanks to Michał Górny for the original patch.
2022-11-09 14:28:41 +02:00
Lasse Collin 034086e1ae liblzma: Add .lz support to lzma_auto_decoder().
Thanks to Michał Górny for the original patch.
2022-11-09 14:25:26 +02:00
Lasse Collin 0538db038f liblzma: Add .lz (lzip) decompression support (format versions 0 and 1).
Support for format version 0 was removed from lzip 1.18 for some
reason. .lz format version 0 files are rare (and old) but some
source packages were released in this format, and some people might
have personal files in this format too. It's very little extra code
to support it along side format version 1 so this commits adds
support for both.

The Sync Flush marker extentension to the original .lz format
version 1 isn't supported. It would require changes to the
LZMA decoder itself. Such files are very rare anyway.

See the API doc for lzma_lzip_decoder() for more details about
the .lz format support.

Thanks to Michał Górny for the original patch.
2022-11-09 14:24:20 +02:00
Lasse Collin 633d48a075 liblzma: Add the missing Makefile.inc change for --disable-microlzma.
This was forgotten from commit 59c4d6e139.
2022-11-09 14:17:23 +02:00
Lasse Collin 724285dadb xz: Add comments about stdin and src_st.st_size.
"xz -v < regular_file > out.xz" doesn't display the percentage
and estimated remaining time because it doesn't even try to
check the input file size when input is read from stdin.
This could be improved but for now there's just a comment
to remind about it.
2022-11-09 14:10:52 +02:00
Lasse Collin f723eec68b xz: Fix displaying of file sizes in progress indicator in passthru mode.
It worked for one input file since the counters are zero when
xz starts but they weren't reset when starting a new file in
passthru mode. For example, if files A, B, and C are one byte each,
then "xz -dcvf A B C" would show file sizes as 1, 2, and 3 bytes
instead of 1, 1, and 1 byte.
2022-11-09 12:48:22 +02:00
Lasse Collin 69265d0f22 xz: Add a comment why --to-stdout is not in --help.
It is on the man page still.
2022-11-09 11:27:20 +02:00
Lasse Collin fe6b8852a3 xz: Make xz -lvv show that the upcoming --arm64 needs 5.4.0 to decompress. 2022-11-08 23:05:37 +02:00
Lasse Collin fb3f05ac9f Docs: Update faq.txt a little. 2022-11-08 22:26:54 +02:00
Lasse Collin 05331f091e Translations: Update Turkish translation. 2022-11-08 16:59:12 +02:00
Lasse Collin ed3a482296 Translations: Update Croatian translation. 2022-11-08 16:59:12 +02:00
Lasse Collin 4746f5ec72 liblzma: Update API docs about decoder flags. 2022-11-08 14:13:03 +02:00
Lasse Collin 8779a9db5d liblzma: Use the return_if_error() macro in alone_decoder.c. 2022-11-08 14:01:50 +02:00
Lasse Collin 3f4990b682 liblzma: Fix a comment in auto_decoder.c. 2022-11-08 14:00:58 +02:00
Lasse Collin 026a5897c7 xz: Initialize the pledge(2) sandbox at the very beginning of main().
It feels better that the initializations are sandboxed too.
They don't do anything that the pledge() call wouldn't allow.
2022-11-08 13:43:19 +02:00
Lasse Collin 49a59f6ca0 xz: Extend --robot --info-memory output.
Now it includes everything that the human-readable --info-memory shows.
2022-11-07 22:51:16 +02:00
Lasse Collin 5e2450c75c liblzma: Include cached memory in reported memusage in threaded decoder.
This affects lzma_memusage() and lzma_memlimit_set() when used
with the threaded decompressor. Now all allocations are reported
by lzma_memusage() (so it's not misleading) and lzma_memlimit_set()
cannot lower the limit below that value.

The alternative would have been to allow lowering the limit if
doing so is possible by freeing the cached memory but since
the primary use case of lzma_memlimit_set() is to increase
memlimit after LZMA_MEMLIMIT_ERROR this simple approach
was selected.

The cached memory was always included when enforcing
the memory usage limit while decoding.

Thanks to Jia Tan.
2022-11-07 17:22:04 +02:00
Jia Tan 1fc6e7dd1f xz: Avoid a compiler warning in progress_speed() in message.c.
This should be smaller too since it avoids the string constants.
2022-11-07 16:24:56 +02:00
Lasse Collin cf118c0146 Build: Clarify comment in configure.ac about SSE2. 2022-10-31 16:26:05 +02:00
Lasse Collin aad3c609ff Build: Remove obsolete commented-out lines from configure.ac. 2022-10-31 16:16:37 +02:00
Lasse Collin e53e0e2186 Windows: Fix mythread_once() macro with Vista threads.
Don't call InitOnceComplete() if initialization was already done.

So far mythread_once() has been needed only when building
with --enable-small. windows/build.bash does this together
with --disable-threads so the Vista-specific mythread_once()
is never needed by those builds. VS project files or
CMake-builds don't support HAVE_SMALL builds at all.
2022-10-31 13:31:58 +02:00