Commit Graph

1351 Commits

Author SHA1 Message Date
Lasse Collin 85b081f5d4 liblzma: Make 32-bit x86 CRC assembly co-exist with CLMUL
Now runtime detection of CLMUL support can pick between the CLMUL and
the generic assembly implementations. Whatever overhead this has for
builds that omit CLMUL completely isn't important because builds for
any non-ancient system is likely to include the CLMUL code too.

Handle the CRC tables in crcXX_fast.c files because now these files
are built even when assembly code is used.

If 32-bit x86 assembly is enabled then it will always be built even
if compiler flags were such that CLMUL would be allowed unconditionally.
That is, runtime detection will be used anyway. This keeps the build
rules simpler.

In LZ encoder, build and use lzma_lz_hash_table[256] if CLMUL CRC
is used without runtime detection. Previously this wasn't needed
because crc32_table.c included the lzma_crc32_table[][] in the build
unless encoder support had been disabled. Including an 8 KiB table
was silly when only 1 KiB is actually used. So now liblzma is 7 KiB
smaller if CLMUL is enabled without runtime detection.
2024-06-23 14:36:44 +03:00
Lasse Collin 6667d503b5 liblzma: CRC: Rename crcXX_generic to lzma_crcXX_generic
This prepares for the possibility that lzma_crc32_generic and
lzma_crc64_generic are extern functions.
2024-06-23 14:36:44 +03:00
Lasse Collin 6a3c4aaa43 Windows: Drop Visual Studio 2013 support
This simplifies things a little. Building liblzma with VS2013 probably
still worked but building the command line tools was not supported.

Microsoft ended support for VS2013 on 2024-04.
2024-06-20 21:53:07 +03:00
Lasse Collin 30a2d5d510 liblzma: CRC CLMUL: Omit is_arch_extension_supported() when not needed
On E2K the function compiles only due to compiler emulation but the
function is never used. It's cleaner to omit the function when it's
not needed even though it's a "static inline" function.

Thanks to Ilya Kurdyukov.
2024-06-17 15:00:55 +03:00
Lasse Collin 54eaea5ea4 liblzma: x86 CLMUL CRC: Rewrite
It's faster with both tiny and large buffers and doesn't require
disabling any sanitizers. With large buffers the extra speed is
from folding four 16-byte chunks in parallel.

The 32-bit x86 with MSVC reportedly still needs a workaround.
Now the simpler "__asm mov ebx, ebx" trick is enough but it
needs to be in lzma_crc64() instead of crc64_arch_optimized().
Thanks to Iouri Kharon for testing and the fix.

Thanks to Ilya Kurdyukov for testing the speed with aligned and
unaligned buffers on a few x86 processors and on E2K v6.

Thanks to Sam James for general feedback.

Fixes: https://github.com/tukaani-project/xz/issues/112
Fixes: https://github.com/tukaani-project/xz/issues/122
2024-06-17 15:00:49 +03:00
Lasse Collin c0e7eaae8d sysdefs.h: Add alignas 2024-06-16 12:59:20 +03:00
Lasse Collin 20014c2614 liblzma: Use a single macro to select CLMUL CRC to build
This way it's clearer that two things cannot be selected
at the same time.
2024-06-16 12:59:17 +03:00
Lasse Collin d8fb098617 liblzma: CRC32 CLMUL: Refactor the constants and simplify
By using modulus scaled constants, the final reduction can
be simplified.
2024-06-16 12:56:54 +03:00
Lasse Collin ef652ac391 liblzma: CRC64 CLMUL: Refactor the constants
Now it refers to crc_clmul_consts_gen.c. vfold8 was renamed to mu_p
and the p no longer has the lowest bit set (it makes no difference
as the output bits it affects are ignored).
2024-06-16 12:56:54 +03:00
Lasse Collin 9f5fc17e32 liblzma: Add crc_clmul_consts_gen.c
It's a standalone program that prints the required constants.
It's won't be a part of the normal build of the package.
2024-06-16 12:56:54 +03:00
Lasse Collin 71b147aab7 liblzma: Remove CRC_USE_GENERIC_FOR_SMALL_INPUTS
It was already commented out.
2024-06-16 12:56:54 +03:00
Lasse Collin f99a7be406 liblzma: Remove crc_attr_no_sanitize_address
It's not enough to silence the address sanitizer. Also memory and
thread sanitizers would need to be silenced. They, at least currently,
aren't smart enough to see that the extra bytes are discarded from
the xmm registers by later instructions.

Valgrind is smarter, possibly because this kind of code isn't weird
to write in assembly. Agner Fog's optimizing_assembly.pdf even mentions
this idea of doing an aligned read and then discarding the extra
bytes. The sanitizers don't instrument assembly code but Valgrind
checks all code.

It's better to change the implementation to avoid the sanitization
attributes which also look scary in the code. (Somehow they can look
more scary than __asm__ which is implictly unsanitized.)

See also:
https://github.com/tukaani-project/xz/issues/112
https://github.com/tukaani-project/xz/issues/122
2024-06-16 12:56:54 +03:00
Lasse Collin c7164b1927 xz: Fix white space 2024-06-11 22:42:26 +03:00
Lasse Collin 0a32d2072c liblzma: Fix a typo in a comment
Thanks to Sam James for spotting it.

Fixes: f644473a21
2024-06-11 22:42:04 +03:00
Lasse Collin afd9b4d282 liblzma: Fix a comment indentation 2024-06-10 23:19:27 +03:00
Lasse Collin 50e6bff274 liblzma: Fix white space 2024-06-10 23:19:27 +03:00
Lasse Collin caea7844d3 tuklib: __STDC_VERSION__ in C23 is 202311 2024-06-10 23:19:27 +03:00
RainRat 9e73918a4f Fix typos
Closes: https://github.com/tukaani-project/xz/pull/124
2024-06-07 16:01:27 +03:00
Lasse Collin 04b23addf3 tuklib_integer: Fix building on OpenBSD/sparc64 that uses GCC 4.2
GCC 4.2 doesn't have __builtin_bswap16() and friends so tuklib_integer.h
tries to use OS-specific byte swap methods instead. On OpenBSD those
macros are swap16/32/64 instead of bswap16/32/64 like on other *BSDs
and Darwin.

An alternative to "#ifdef __OpenBSD__" could be "#ifdef swap16" as it
is a macro. But since OpenBSD seems to be a special case under this
special case of "*BSDs and Darwin", checking for __OpenBSD__ seems
the more conservative choice now.

Thanks to Christian Weisgerber and Brad Smith who both submitted
the same patch a few hours apart.

Co-authored-by: Christian Weisgerber <naddy@mips.inka.de>
Co-authored-by: Brad Smith <brad@comstyle.com>
Closes: https://github.com/tukaani-project/xz/pull/126
2024-06-07 15:47:20 +03:00
Lasse Collin dc03f6290f liblzma: Add ARM64 CRC32 instruction support detection on OpenBSD
The C code is from Christian Weisgerber, I merely reordered the OSes.
Then I added the build system checks without testing them.

Also thanks to Brad Smith who submitted a similar patch on GitHub
a few hours after Christian had sent his via email.

Co-authored-by: Christian Weisgerber <naddy@mips.inka.de>
Closes: https://github.com/tukaani-project/xz/pull/125
2024-06-07 15:06:59 +03:00
Sam James b69768c8bd xz: list: suppress -Wformat-nonliteral for Solaris
Solaris' GCC can't understand that our use is fine, unlike modern compilers:
```
list.c: In function 'print_totals_basic':
list.c:1191:4: error: format not a string literal, argument types not checked [-Werror=format-nonliteral]
  uint64_to_str(totals.files, 0));
  ^~~~~~~~~~~~~
cc1: all warnings being treated as errors
```

It's presumably because of older gettext missing format attributes.

This is with `gcc (GCC) 7.3.0`.
2024-06-03 12:32:34 +03:00
Lasse Collin 4e9023857d Fix typos
Thanks to xx on #tukaani.
2024-05-18 00:34:07 +03:00
Lasse Collin b14d08fbbc liblzma: Fix white space
Thanks to xx on #tukaani.
2024-05-18 00:24:50 +03:00
Lasse Collin 142e670a41 xz: Document the static function get_chains_memusage() 2024-05-13 18:00:41 +03:00
Lasse Collin 78e984399a xz: Rename filters_memusage_max() to get_chains_memusage() 2024-05-13 18:00:41 +03:00
Lasse Collin 54c3db0a83 xz: Rename filter_memusages to chains_memusages 2024-05-13 18:00:41 +03:00
Lasse Collin d9e1ae79ec xz: Simplify the memory usage scaling code
This is closer to what it was before the --filtersX support was added,
just extended to support for scaling all filter chains. The method
before this commit was an extended version of the original too but
it was done in a more complex way for no clear reason. In case of
an error, the complex version printed fewer informative messages
(a good thing) but it's not a sigificant benefit.

In the limit is too low even for single-threaded mode, the required
amount of memory is now reported like in 5.4.x instead of like in
5.5.1alpha - 5.6.1 which showed the original non-scaled usage. It
had been a FIXME in the old code but it's not clear what message
makes the most sense.

Fixes: 5f0c5a0438
2024-05-13 18:00:41 +03:00
Lasse Collin 0ee56983d1 xz: Edit comments 2024-05-13 18:00:41 +03:00
Lasse Collin ec82a49c35 xz: Rename chain_idx to chain_num 2024-05-13 18:00:41 +03:00
Lasse Collin a731a6993c xz: Edit coding style 2024-05-13 18:00:41 +03:00
Lasse Collin 32eb176b89 xz: Edit comments
Fixes: 5f0c5a0438
2024-05-13 15:41:48 +03:00
Lasse Collin b90339f4da xz: Fix grammar in a comment
Fixes: cb3111e3ed
2024-05-13 15:41:48 +03:00
Lasse Collin 4c0bdaf13d xz: Rename filter_memusages to encoder_memusages 2024-05-13 15:41:46 +03:00
Lasse Collin b54aa023e0 xz: Edit coding style 2024-05-13 15:41:05 +03:00
Lasse Collin 49f67d3d3f xz: Rename filters_index to chain_num
The reason is the same as in bd0782c1f13e52cd0fd8415208e30e47004a4c68.
2024-05-13 15:41:05 +03:00
Lasse Collin ff9e8b3d06 xz: Replace a few uint32_t with "unsigned" to reduce the number of casts
These hold only tiny values.
2024-05-13 15:41:05 +03:00
Lasse Collin b5e6c1113b xz: Rename filters_used_mask to chains_used_mask
The reason is the same as in bd0782c1f13e52cd0fd8415208e30e47004a4c68.
2024-05-13 15:41:05 +03:00
Lasse Collin 32500dfaad xz: Move the setting of "check" in coder_set_compression_settings()
It's more logical to do it in the beginning instead of in the middle
of the filter chain handling.

Fixes: d6af7f3470
2024-05-13 15:41:05 +03:00
Lasse Collin ad146b1f42 xz: Rename "filters" to "chains"
The convention is that

    lzma_filter filters[LZMA_FILTERS_MAX + 1];

contains the filters of a single filter chain.
It was so here as well before the commit
d6af7f3470.
It changes "filters" to a ten-element array of filter chains.
It's clearer to call this array-of-arrays "chains".

This also renames "filter_idx" to "chain_idx" which is used
as an index as in chains[chain_idx].
2024-05-13 15:40:58 +03:00
Lasse Collin 5a4ae4e4d0 xz: Clean up a comment 2024-05-13 15:39:39 +03:00
Lasse Collin 2de80494ed xz: Add clarifying assertions 2024-05-13 15:39:39 +03:00
Lasse Collin 1eaad004bf xz: Add a clarifying assertion
Fixes: 5f0c5a0438
2024-05-13 15:39:39 +03:00
Lasse Collin 605094329b xz: Clarify a comment 2024-05-13 15:39:39 +03:00
Lasse Collin 8fac2577f2 xz: Use the info collected in parse_block_list()
This is slightly simpler and it avoids looping through
the opt_block_list array.
2024-05-13 15:39:39 +03:00
Lasse Collin 81d350dab8 xz: Remember the filter chains and the largest Block in parse_block_list() 2024-05-13 15:39:39 +03:00
Lasse Collin 46ab56968f xz: Update a comment and initialization of filters_used_mask 2024-05-13 15:39:39 +03:00
Lasse Collin e89293a0ba xz: parse_block_list: Edit integer type casting 2024-05-13 15:39:39 +03:00
Lasse Collin 87011e40c1 xz: Make filter_memusages a local variable 2024-05-13 15:39:12 +03:00
Lasse Collin 347b412a93 xz: Remove unused code and simplify
opt_mode == MODE_COMPRESS isn't possible when HAVE_ENCODERS isn't
defined. Thus, when *encoding*, the message about *decoder* memory
usage is possible to show only when both encoder and decoder have
been built.

Since the message is shown only at V_DEBUG, skip the memusage
calculation if verbosity level isn't high enough.

Fixes: 5f0c5a0438
2024-05-13 15:31:15 +03:00
Lasse Collin 31358c057c xz: Fix integer type from uint64_t to uint32_t
lzma_options_lzma.dict_size is uint32_t so use it here too.

Fixes: 5f0c5a0438
2024-05-11 00:29:24 +03:00