Commit Graph

2504 Commits

Author SHA1 Message Date
Lasse Collin d28a4b2520 license-check.sh: Use '--' with slightly untrusted filenames
Names from git ls-files should be safe but if one runs it on
a tree without the .git dir and there are extra files, it's
safer to have the end of arguments marked with '--'.
2024-05-28 12:18:09 +03:00
Lasse Collin fda0ec862a license-check.sh: Use xargs -0 instead of -d
Neither are in POSIX but -0 is much more portable in practice.

Despite the old comment, the grep usage should be portable already.
2024-05-28 12:18:09 +03:00
Lasse Collin 9114267038 Translations: Omit man page line numbers from .pot and .po files 2024-05-28 01:17:45 +03:00
Lasse Collin 093490b582 Translations: Use the xgettext option --add-location=file 2024-05-28 01:06:30 +03:00
Lasse Collin fccebe2b4f Translations: Use the msgmerge option --add-location=file
This way the PO file diffs are less noisy but the locations of the
strings are still present at file level, just without line numbers.

The option is available since gettext 0.19 (2014).
configure.ac requires 0.19.6.
2024-05-28 00:43:53 +03:00
Lasse Collin f361d9ae85 Build: Use $(SHELL) instead of sh to run scripts in Makefile.am 2024-05-27 12:22:08 +03:00
Lasse Collin a26dece347 Translations: Change the home page URLs in man page translations
Since the source strings have changed, these would get marked as
fuzzy and the original string would be used instead. The original
and translated strings are identical in this case so it wouldn't
matter. But patching the translations helps still because then
po4a will show the correct translation percentage.
2024-05-23 17:25:13 +03:00
Lasse Collin 24387c234b CMake: Add manual support for 32-bit x86 assembly files
One has to pass -DENABLE_X86_ASM=ON to cmake to enable the
CRC assembly code. Autodetection isn't done. Looking at
CMAKE_SYSTEM_PROCESSOR might not work as it comes from uname
unless cross-compilation is done using a CMake toolchain file.

On top of this, if the code is run on modern processors that support
the CLMUL instruction, then the C code should be faster (but then
one should also be using a x86-64 build if possible).
2024-05-23 15:32:43 +03:00
Lasse Collin 0fb3c9c3f6 CMake: Rename USE_DOXYGEN to ENABLE_DOXYGEN
It's more consistent with the other option() uses.
2024-05-23 14:26:45 +03:00
Lasse Collin 6bbec3bda0 Mention license-check.sh in COPYING 2024-05-22 15:21:53 +03:00
Lasse Collin 62733592a1 Use more confident language in COPYING 2024-05-22 15:21:53 +03:00
Lasse Collin a119a4209e Build: Run license-check.sh in "mydist" and "dist-hook"
In mydist the point is to check using the file list from the Git
repository. In dist-hook it is to check that the TARBALL_IGNORE
patterns work when the .git dir or the "git" command aren't available.

Refuse to create a distribution tarball if license issues are found.
2024-05-22 15:21:53 +03:00
Lasse Collin f3434ecfcb Add build-aux/license-check.sh
This helps in spotting files that lack SPDX license identifier
and which haven't been explicitly white listed either. The script
requires the .git directory to be present as only the files that
are in the Git repository are checked.

XZ Utils isn't FSFE REUSE compliant for now.
2024-05-22 15:21:53 +03:00
Lasse Collin 9ae2ebc1e5 Add SPDX license identifiers to files under tests/ossfuzz 2024-05-22 15:18:42 +03:00
Lasse Collin 9000d70eb9 Add SPDX license identifier to .codespellrc 2024-05-22 15:18:42 +03:00
Lasse Collin 903c16fcfa Move entries po4a/.gitignore to the top level .gitignore
The po4a directory is in EXTRA_DIST and thus all files there
are included in the package. .gitignore doesn't belong in the
package so keep that file out of the po4a directory.
2024-05-22 15:12:09 +03:00
Lasse Collin 56f1d5ed68 Tests: Make the config.h grep patterns Meson compatible
Now the test scripts detect both

    #define HAVE_DECODER_ARM
    #define HAVE_DECODER_ARM 1

as support for the ARM filter without confusing it with these:

    #define HAVE_DECODER_ARM64
    #define HAVE_DECODER_ARM64 1

Previously only the ones ending with " 1" were accepted for
the macros where this kind of confusion was possible.

This should help with Meson support because Meson's built-in
features produce config.h entries that are either

    #define FOO 1
    #define FOO 0

or:

    #define FOO
    #undef FOO

The former method has a benefit that one can use "#if FOO" and -Wundef
will catch if a #define is missing (for example, it helps catching
typos). But XZ Utils has to use the latter since it has been
convenient with Autoconf's default behavior.[*] While it's easy to
emulate the Autoconf style (#define FOO 1 vs. no #define at all)
in Meson, it results in clumsy code. Thus it's better to change
the few places in the tests where this difference matters.

[*] While most checks in Autoconf default to the second style above,
    a few things use the first style (like AC_CHECK_DECLS). The mix
    of both styles is the most confusing as one has to remember which
    macro needs #ifdef and which #if. Currently HAVE_VISIBILITY is
    only such config.h entry that is 1 or 0. It comes unmodified
    from Gnulib's visibility.m4.
2024-05-20 16:55:00 +03:00
Lasse Collin 9d997d6f9d CMake: Add comments 2024-05-20 16:55:00 +03:00
Lasse Collin d35368b33e CMake: Remove the note that some tests aren't run
They are now in the common build configurations.
2024-05-20 16:55:00 +03:00
Lasse Collin dc232d5846 CMake: Add support for test_files.sh 2024-05-20 16:55:00 +03:00
Lasse Collin a7e9230af9 Tests: Make test_files.sh more flexible
Add a new optional argument to specify the directory of the xz and
xzdec executables.

If ../config.h doesn't exist, assume that all encoders and decoders
are available.
2024-05-20 16:55:00 +03:00
Lasse Collin b40e6efbb4 CMake: Add support for test_compress.sh tests 2024-05-20 16:55:00 +03:00
Lasse Collin ac3222d2cb Tests: Make test_compress.sh more flexible
Add a new optional second argument: directory of the xz and xzdec
executables. This is need with the CMake build where the binaries
end up in the top-level build directory.

If ../config.h doesn't exist, assume that all encoders and decoders
are available. This will make this script usable from CMake in the
most common build configuration.

NOTE: Since the existence of ../config.h is checked, the working
directory of the test script must be a subdir in the build tree!
Otherwise ../config.h would look outside the build tree.

Use the default check type instead of forcing CRC32 or CRC64.
Now the script doesn't need to check if CRC64 is available.
2024-05-20 16:55:00 +03:00
Lasse Collin 006040b29c CMake: Prepare to support the test_*.sh tests
This is a bit hacky since the scripts grep config.h to know which
features were built but the CMake build doesn't create config.h.
So instead those test scripts will be run only when all relevant
features have been enabled.
2024-05-20 16:55:00 +03:00
Lasse Collin 6167607a6e Tests: test_suffix.sh: Add a comment 2024-05-20 16:55:00 +03:00
Lasse Collin 4e9023857d Fix typos
Thanks to xx on #tukaani.
2024-05-18 00:34:07 +03:00
Lasse Collin b14d08fbbc liblzma: Fix white space
Thanks to xx on #tukaani.
2024-05-18 00:24:50 +03:00
Lasse Collin 9f1a6d6f9a Build: Temporarily disable CRC CLMUL to silence OSS Fuzz
The code makes aligned 16-byte reads which may read up to 15 bytes
before the beginning or past the end of the buffer if the buffer
is misaligned. The unneeded bytes are then ignored. It cannot cross
page boundaries and thus cannot cause access violations.

This inherently trips address sanitizer which was already disabled
with __attribute__((__no_sanitize_address__)). However, it also
trips memory sanitizer if the extra bytes are uninitialized because
memory sanitizer doesn't see that those bytes then get ignored by
byte shuffling in the xmm registers.

The plan is to change the code so that all sanitizers pass but it's
not finished yet (performance shouldn't get worse) so as a temporary
measure to keep OSS Fuzz happy, the CLMUL CRC is now disabled even
though I think think the code is fine to use (and easy enough to review
the memory accesses in it too).
2024-05-15 23:14:17 +03:00
Lasse Collin 142e670a41 xz: Document the static function get_chains_memusage() 2024-05-13 18:00:41 +03:00
Lasse Collin 78e984399a xz: Rename filters_memusage_max() to get_chains_memusage() 2024-05-13 18:00:41 +03:00
Lasse Collin 54c3db0a83 xz: Rename filter_memusages to chains_memusages 2024-05-13 18:00:41 +03:00
Lasse Collin d9e1ae79ec xz: Simplify the memory usage scaling code
This is closer to what it was before the --filtersX support was added,
just extended to support for scaling all filter chains. The method
before this commit was an extended version of the original too but
it was done in a more complex way for no clear reason. In case of
an error, the complex version printed fewer informative messages
(a good thing) but it's not a sigificant benefit.

In the limit is too low even for single-threaded mode, the required
amount of memory is now reported like in 5.4.x instead of like in
5.5.1alpha - 5.6.1 which showed the original non-scaled usage. It
had been a FIXME in the old code but it's not clear what message
makes the most sense.

Fixes: 5f0c5a0438
2024-05-13 18:00:41 +03:00
Lasse Collin 0ee56983d1 xz: Edit comments 2024-05-13 18:00:41 +03:00
Lasse Collin ec82a49c35 xz: Rename chain_idx to chain_num 2024-05-13 18:00:41 +03:00
Lasse Collin a731a6993c xz: Edit coding style 2024-05-13 18:00:41 +03:00
Lasse Collin 32eb176b89 xz: Edit comments
Fixes: 5f0c5a0438
2024-05-13 15:41:48 +03:00
Lasse Collin b90339f4da xz: Fix grammar in a comment
Fixes: cb3111e3ed
2024-05-13 15:41:48 +03:00
Lasse Collin 4c0bdaf13d xz: Rename filter_memusages to encoder_memusages 2024-05-13 15:41:46 +03:00
Lasse Collin b54aa023e0 xz: Edit coding style 2024-05-13 15:41:05 +03:00
Lasse Collin 49f67d3d3f xz: Rename filters_index to chain_num
The reason is the same as in bd0782c1f13e52cd0fd8415208e30e47004a4c68.
2024-05-13 15:41:05 +03:00
Lasse Collin ff9e8b3d06 xz: Replace a few uint32_t with "unsigned" to reduce the number of casts
These hold only tiny values.
2024-05-13 15:41:05 +03:00
Lasse Collin b5e6c1113b xz: Rename filters_used_mask to chains_used_mask
The reason is the same as in bd0782c1f13e52cd0fd8415208e30e47004a4c68.
2024-05-13 15:41:05 +03:00
Lasse Collin 32500dfaad xz: Move the setting of "check" in coder_set_compression_settings()
It's more logical to do it in the beginning instead of in the middle
of the filter chain handling.

Fixes: d6af7f3470
2024-05-13 15:41:05 +03:00
Lasse Collin ad146b1f42 xz: Rename "filters" to "chains"
The convention is that

    lzma_filter filters[LZMA_FILTERS_MAX + 1];

contains the filters of a single filter chain.
It was so here as well before the commit
d6af7f3470.
It changes "filters" to a ten-element array of filter chains.
It's clearer to call this array-of-arrays "chains".

This also renames "filter_idx" to "chain_idx" which is used
as an index as in chains[chain_idx].
2024-05-13 15:40:58 +03:00
Lasse Collin 5a4ae4e4d0 xz: Clean up a comment 2024-05-13 15:39:39 +03:00
Lasse Collin 2de80494ed xz: Add clarifying assertions 2024-05-13 15:39:39 +03:00
Lasse Collin 1eaad004bf xz: Add a clarifying assertion
Fixes: 5f0c5a0438
2024-05-13 15:39:39 +03:00
Lasse Collin 605094329b xz: Clarify a comment 2024-05-13 15:39:39 +03:00
Lasse Collin 8fac2577f2 xz: Use the info collected in parse_block_list()
This is slightly simpler and it avoids looping through
the opt_block_list array.
2024-05-13 15:39:39 +03:00
Lasse Collin 81d350dab8 xz: Remember the filter chains and the largest Block in parse_block_list() 2024-05-13 15:39:39 +03:00