Commit Graph

2186 Commits

Author SHA1 Message Date
Jia Tan a165d7df19 xz: Add a new --filters-help option.
The --filters-help can be used to help create filter chains with the
--filters and --filtersX options. The message in --long-help is too
short to fully explain the syntax to construct complex filter chains.

In --robot mode, xz will only print the output from liblzma function
lzma_str_list_filters.
2023-07-17 23:34:55 +08:00
Jia Tan 95f1a414b1 xz: Update the man page for --block-list and --filtersX
The --block-list option description needed updating since the new
--filtersX option changes how it can be used. The new entry for
--filters1=FILTERS ... --filter9=FILTERS was created right after
the --filters option.
2023-07-17 23:34:55 +08:00
Jia Tan 47a63cad2a xz: Update --long-help for the new --filtersX option. 2023-07-17 23:34:55 +08:00
Jia Tan 8b9913a13d xz: Ignore filter chains that are set but never used in --block-list.
If a filter chain is set but not used in --block-list, it introduced
unexpected behavior such as requiring an unneeded amount of memory to
compress, reducing the number of threads in multi-threaded encoding, and
printing an incorrect amount of memory needed to decompress.

This also renames filters_init_mask => filters_used_mask. A filter is
assumed to be used if it is specified in --filtersX until
coder_set_compression_settings() determines which filters are referenced
in --block-list.
2023-07-17 23:34:55 +08:00
Jia Tan 183819bfd9 xz: Set the Block size for mt encoding correctly.
When opt_block_size is not used, the Block size for mt encoder is
derived from the minimum of the largest Block specified by
--block-list and the recommended Block size on all filter chains
calculated by lzma_mt_block_size(). This avoids using unnecessary
memory and ensures that all Blocks are large enough for the most memory
needy filter chain.
2023-07-17 23:34:55 +08:00
Jia Tan afb2dbec3d xz: Validate --flush-timeout for all specified filter chains. 2023-07-17 23:34:55 +08:00
Jia Tan 5f0c5a0438 xz: Allows --block-list filters to scale down memory usage.
Previously, only the default filter chain could have its memory usage
adjusted. The filter chains specified with --filtersX were not checked
for memory usage. Now, all used filter chains will be adjusted if
necessary.
2023-07-17 23:34:55 +08:00
Jia Tan 479fd58d60 xz: Do not include block splitting if encoders are disabled.
The block splitting logic and split_block() function are not needed if
encoders are disabled. This will help slightly reduce the binary size
when built without encoders and allow split_block() to use functions
that require encoders being enabled.
2023-07-17 23:34:55 +08:00
Jia Tan f86ede2250 xz: Free filters[] in debug mode.
This will only free filter chains created with --filters1-9 since the
default filter chain may be set from a static function variable. The
complexity to free the default filter chain is not worth the burden on
code maintenance.
2023-07-17 23:34:55 +08:00
Jia Tan f281cd0d69 xz: Add a message if --block-list is used outside of xz compresssion.
--block-list is only supported with compression in xz format. This avoids
silently ignoring when --block-list is unused.
2023-07-17 23:34:55 +08:00
Jia Tan d6af7f3470 xz: Create command line options for filters[1-9].
The new command line options are meant to be combined with --block-list.
They work as an optional extension to --block-list to specify a custom
filter chain for each block listed. The new options allow the creation
of up to 9 reusable filter chains. For instance:

xz --block-list=1:10MiB,3:5MiB,,2:5MiB,1:0 --filters1=delta--lzma2 \
--filters2=x86--lzma2 --filters3=arm64--lzma2

Will create the following blocks:
1. A block of size 10 MiB with filter chain delta, lzma2.
2. A block of size 5 MiB with filter chain arm64, lzma2.
3. A block of size 5 MiB with filter chain arm64, lzma2.
4. A block of size 5 MiB with filter chain x86, lzma2.
5. A block containing the rest of the file contents with filter chain
   delta, lzma2.
2023-07-17 23:34:55 +08:00
Jia Tan 072d292501 xz: Use lzma_filters_free() in forget_filter_chain().
This is a little cleaner than the previous implementation of
forget_filter_chain(). It is also more consistent since
lzma_str_to_filters() will always terminate the filter chain so there
is no need to terminate it later in coder_set_compression_settings().
2023-07-17 23:34:55 +08:00
Jia Tan 3d21da5cff xz: Separate string to filter conversion into a helper function.
Converting from string to filter will also need to be done for block
specific filter chains.
2023-07-17 23:34:55 +08:00
Jia Tan a6583726e5 Tests: Use new --filters option in test_compress.sh 2023-07-17 23:34:55 +08:00
Jia Tan 5f3b898d07 xz: Update --long-help and man page for new --filters option. 2023-07-17 23:34:55 +08:00
Jia Tan 9ded880a02 xz: Add --filters option to CLI.
The --filters option uses the new lzma_str_to_filters() function
to convert a string into a full filter chain. Using this option
will reset all previous filters set by --preset, --[filter], or
--filters.
2023-07-17 23:34:55 +08:00
Jia Tan 2c189bb00a Tests: Improve feature testing for skipping.
Fixed a bug where test_compress_* would all fail if arm64 or armthumb
filters were enabled for compression but arm was disabled. Since the
grep tests only checked for "define HAVE_ENCODER_ARM", this would match
on HAVE_ENCODER_ARM64 or HAVE_ENCODER_ARMTHUMB.

Now the config.h feature test requires " 1" at the end to prevent the
prefix problem. have_feature() was also updated for this even though
there were known current bugs affecting it. This is just in case future
features have a similar prefix problem.
2023-07-14 21:30:25 +08:00
Jia Tan 80a6b9bcad Translations: Update the Chinese (traditional) translation. 2023-07-10 20:56:28 +08:00
Jia Tan 17f8844e6f liblzma: Remove non-portable empty initializer.
Commit 78704f36e7 added an empty
initializer {} to prevent a warning. The empty initializer is a GNU
extension and results in a build failure on MSVC. The -wpedantic flag
warns about empty initializers.
2023-07-08 21:24:19 +08:00
Jia Tan 3aca4f629c Translations: Update the Vietnamese translation. 2023-07-08 20:03:59 +08:00
Jia Tan 66bdcfa85f Tests: Fix memory leaks in test_index.
Several tests were missing calls to lzma_index_end() to clean up the
lzma_index structs. The memory leaks were discovered by using
-fsanitize=address with GCC.
2023-06-29 00:07:16 +08:00
Jia Tan fe3bd438fb Tests: Fix memory leaks in test_block_header.
test_block_header was not properly freeing the filter options between
calls to lzma_block_header_decode(). The memory leaks were discovered by
using -fsanitize=address with GCC.
2023-06-29 00:07:05 +08:00
Jia Tan 78704f36e7 liblzma: Prevent uninitialzed warning in mt stream encoder.
This change only impacts the compiler warning since it was impossible
for the wait_abs struct in stream_encode_mt() to be used before it was
initialized since mythread_condtime_set() will always be called before
mythread_cond_timedwait().

Since the mythread.h code is different between the POSIX and
Windows versions, this warning was only present on Windows builds.

Thanks to Arthur S for reporting the warning and providing an initial
patch.
2023-06-29 00:06:16 +08:00
Jia Tan e3356a204c liblzma: Prevent warning for MSYS2 Windows build.
In lzma_memcmplen(), the <intrin.h> header file is only included if
_MSC_VER and _M_X64 are both defined but _BitScanForward64() was
previously used if _M_X64 was defined. GCC for MSYS2 defines _M_X64 but
not _MSC_VER so _BitScanForward64() was used without including
<intrin.h>.

Now, lzma_memcmplen() will use __builtin_ctzll() for MSYS2 GCC builds as
expected.
2023-06-28 23:59:51 +08:00
Jia Tan 45e250a9e9 CI: Add test with -fsanitize=address,undefined.
ci_build.sh was updated to accept disabling of __attribute__ ifunc
and CLMUL. This will allow -fsanitize=address to pass because ifunc
is incompatible with -fsanitize=address. The CLMUL implementation has
optimizations that potentially read past the buffer and mask out the
unwanted bytes.

This test will only run on Autotools Linux.
2023-06-28 23:59:47 +08:00
Jia Tan 596ee722cd CI: Upgrade checkout action from v2 to v3. 2023-06-28 20:48:49 +08:00
Jia Tan 86118ea320 Update THANKS. 2023-06-27 23:56:06 +08:00
Jia Tan 3d1fdddf92 Docs: Document the configure option --disable-ifunc in INSTALL. 2023-06-27 23:56:06 +08:00
Lasse Collin b4cf7a2822 Minor tweaks to style and comments. 2023-06-27 23:56:06 +08:00
Lasse Collin 23fb9e3a32 CMake: Rename CHECK_ATTR_IFUNC to ALLOW_ATTR_IFUNC.
It's so that there's a clear difference in wording compared
to liblzma's integrity check types.
2023-06-27 23:56:06 +08:00
Lasse Collin ee44863ae8 liblzma: Add ifunc implementation to crc64_fast.c.
The ifunc method avoids indirection via the function pointer
crc64_func. This works on GNU/Linux and probably on FreeBSD too.
The previous __attribute((__constructor__)) method is kept for
compatibility with ELF platforms which do support ifunc.

The ifunc method has some limitations, for example, building
liblzma with -fsanitize=address will result in segfaults.
The configure option --disable-ifunc must be used for such builds.

Thanks to Hans Jansen for the original patch.
Closes: https://github.com/tukaani-project/xz/pull/53
2023-06-27 23:55:59 +08:00
Hans Jansen b72d212024 Add ifunc check to CMakeLists.txt
CMake build system will now verify if __attribute__((__ifunc__())) can be
used in the build system. If so, HAVE_FUNC_ATTRIBUTE_IFUNC will be
defined to 1.
2023-06-27 15:33:15 +03:00
Hans Jansen 23b5c36fb7 Add ifunc check to configure.ac
configure.ac will now verify if __attribute__((__ifunc__())) can be used in
the build system. If so, HAVE_FUNC_ATTRIBUTE_IFUNC will be defined to 1.
2023-06-27 15:33:15 +03:00
Jia Tan dbb3a536ed CI: Add apt update command before installing dependencies.
Without the extra command, all of the CI tests were automatically
failing because the Ubuntu servers could not be reached properly.
2023-06-07 00:23:02 +08:00
Jia Tan 6bcd516812 Update THANKS. 2023-06-07 00:10:38 +08:00
Benjamin Buch 0d94ba6922
CMake: Protects against double find_package
Boost iostream uses `find_package` in quiet mode and then again uses
`find_package` with required. This second call triggers a 
`add_library cannot create imported target "ZLIB::ZLIB" because another
target with the same name already exists.`

This can simply be fixed by skipping the alias part on secondary
`find_package` runs.
2023-06-06 21:32:45 +08:00
Jia Tan 045d7aae28 Translations: Update the Esperanto translation. 2023-05-31 23:57:57 +08:00
Jia Tan b0cc7c2dce Translations: Update the Croatian translation. 2023-05-31 23:57:57 +08:00
Jia Tan af045ef6f8 Translations: Update the Chinese (simplified) translation. 2023-05-31 20:15:53 +08:00
Jia Tan e6b92d5817 Translations: Update German translation of man pages. 2023-05-17 23:12:13 +08:00
Jia Tan 592961ccdb Translations: Update the German translation. 2023-05-17 23:09:18 +08:00
Jia Tan 13572cb2c3 Translations: Update the Croatian translation. 2023-05-17 20:30:01 +08:00
Jia Tan 4e6e425ea8 Translations: Update Korean translation of man pages. 2023-05-17 20:26:54 +08:00
Jia Tan d5ef1f6faf Translations: Update the Korean translation. 2023-05-17 20:13:01 +08:00
Jia Tan e22d0b0f2e Translations: Update the Spanish translation. 2023-05-16 23:49:09 +08:00
Jia Tan f50da74d52 Translations: Update the Romanian translation. 2023-05-16 23:47:23 +08:00
Jia Tan 4b9ad60a73 Translations: Update Romanian translation of man pages. 2023-05-16 23:45:43 +08:00
Jia Tan cb6fd57f88 Translations: Update Ukrainian translation of man pages. 2023-05-16 23:44:45 +08:00
Jia Tan c3e8fcbc2d Translations: Update the Ukrainian translation. 2023-05-16 23:44:45 +08:00
Jia Tan 27b81b84fc Translations: Update the Polish translation. 2023-05-16 23:44:36 +08:00