Commit Graph

1364 Commits

Author SHA1 Message Date
Lasse Collin 46ee006162 Windows: Embed an application manifest in the EXE files
IMPORTANT: This includes a security fix to command line tool
           argument handling.

Some toolchains embed an application manifest by default to declare
UAC-compliance. Some also declare compatibility with Vista/8/8.1/10/11
to let the app access features newer than those of Vista.

We want all the above but also two more things:

  - Declare that the app is long path aware to support paths longer
    than 259 characters (this may also require a registry change).

  - Force the code page to UTF-8. This allows the command line tools
    to access files whose names contain characters that don't exist
    in the current legacy code page (except unpaired surrogates).
    The UTF-8 code page also fixes security issues in command line
    argument handling which can be exploited with malicious filenames.
    See the new file w32_application.manifest.comments.txt.

Thanks to Orange Tsai and splitline from DEVCORE Research Team
for discovering this issue.

Thanks to Vijay Sarvepalli for reporting the issue to me.

Thanks to Kelvin Lee for testing with MSVC and helping with
the required build system fixes.
2024-10-01 12:10:23 +03:00
Lasse Collin dad1530915 Windows: Set DLL name accurately in StringFileInfo on Cygwin and MSYS2
Now the information in the "Details" tab in the file properties
dialog matches the naming convention of Cygwin and MSYS2. This
is only a cosmetic change.
2024-09-30 16:55:23 +03:00
Lasse Collin 8940ecb96f common_w32res.rc: White space edits
LANGUAGE and VS_VERSION_INFO begin new statements so put an empty line
between them.
2024-09-29 01:27:16 +03:00
Tobias Stoeckmann 76cfd0a9bb lzmainfo: Avoid integer overflow
The MB output can overflow with huge numbers. Most likely these are
invalid .lzma files anyway, but let's avoid garbage output.

lzmadec was adapted from LZMA Utils. The original code with this bug
was written in 2005, over 19 years ago.

Co-authored-by: Lasse Collin <lasse.collin@tukaani.org>
Closes: https://github.com/tukaani-project/xz/pull/144
2024-09-17 01:26:02 +03:00
Tobias Stoeckmann 78355aebb7 xzdec: Remove unused short option -M
"xzdec -M123" exited with exit status 1 without printing
any messages. The "M:" entry should have been removed when
the memory usage limiter support was removed from xzdec.

Fixes: 792331bdee
Closes: https://github.com/tukaani-project/xz/pull/143
[ Lasse: Commit message edits ]
2024-09-16 23:33:29 +03:00
Yifeng Li 6cd7c86078 liblzma: Fix x86-64 movzw compatibility in range_decoder.h
Support for instruction "movzw" without suffix in "GNU as" was
added in commit [1] and stabilized in binutils 2.27, released
in August 2016. Earlier systems don't accept this instruction
without a suffix, making range_decoder.h's inline assembly
unable to build on old systems such as Ubuntu 16.04, creating
error messages like:

    lzma_decoder.c: Assembler messages:
    lzma_decoder.c:371: Error: no such instruction: `movzw 2(%r11),%esi'
    lzma_decoder.c:373: Error: no such instruction: `movzw 4(%r11),%edi'
    lzma_decoder.c:388: Error: no such instruction: `movzw 6(%r11),%edx'
    lzma_decoder.c:398: Error: no such instruction: `movzw (%r11,%r14,4),%esi'

Change "movzw" to "movzwl" for compatibility.

[1] https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=c07315e0c610e0e3317b4c02266f81793df253d2

Suggested-by: Lasse Collin <lasse.collin@tukaani.org>
Tested-by: Yifeng Li <tomli@tomli.me>
Signed-off-by: Yifeng Li <tomli@tomli.me>
Fixes: 3182a330c1
Fixes: https://github.com/tukaani-project/xz/issues/121
Closes: https://github.com/tukaani-project/xz/pull/136
2024-08-22 10:59:08 +03:00
Lasse Collin f7103c2c2a Revert "liblzma: Add ARM64 CRC32 instruction support detection on OpenBSD"
This reverts commit dc03f6290f.

OpenBSD 7.6 will support elf_aux_info(3), and the detection code used
on FreeBSD will work on OpenBSD 7.6 too. Keep things simpler and drop
the OpenBSD-specific sysctl() method.

Thanks to Christian Weisgerber.
2024-07-19 20:06:24 +03:00
Lasse Collin 7c292dd0bf liblzma: Tweak a comment 2024-07-13 22:10:37 +03:00
Lasse Collin baecfa1426 xz: Remove the TODO comment about --recursive
It won't be implemented. find + xargs is more flexible, for example,
it allows compressing small files in parallel. An example for that
has been included in the xz man page since 2010.
2024-07-06 14:04:48 +03:00
Xi Ruoyao 7baf6835cf liblzma: Speed up CRC32 calculation on 64-bit LoongArch
The crc.w.{b/h/w/d}.w instructions in LoongArch can calculate the CRC32
result for 1/2/4/8 bytes in a single operation. Using these is much
faster compared to the generic method.

Optimized CRC32 is enabled unconditionally on 64-bit LoongArch because
the LoongArch specification says that CRC32 instructions shall be
implemented for 64-bit processors. Optimized CRC32 isn't enabled for
32-bit LoongArch processors because not enough information is available
about them.

Co-authored-by: Lasse Collin <lasse.collin@tukaani.org>

Closes: https://github.com/tukaani-project/xz/pull/86
2024-07-01 17:09:57 +03:00
Lasse Collin 0ed8936685 liblzma: ARM64 CRC32: Align the buffer faster
Instead of doing it byte by byte, use the 1/2/4-byte CRC32 instructions.
2024-06-28 14:20:49 +03:00
Lasse Collin fe77c4e130 liblzma: Tidy up crc_common.h
Prefix ARM64_RUNTIME_DETECTION with CRC_ and reorder it to be with
the other ARM64-specific lines. That macro isn't used outside this
file.

ARM64 CLMUL implementation doesn't exist yet and thus CRC64_ARM64_CLMUL
isn't used anywhere yet.

It's not ideal that the single-letter CRC utility macros are here
as they pollute the namespace of the LZ encoder files. Those could
be moved their own crc_macros.h like they were in 5.2.x but in practice
this is fine enough already.
2024-06-23 23:09:14 +03:00
Lasse Collin 7484d37538 liblzma: Move lzma_crcXX_table[][] declarations to crc_common.h
LZ encoder needs lzma_crc32_table[0] but otherwise those tables
are private to the CRC code. In contrast, the other things in
check.h are needed in several places.
2024-06-23 15:37:46 +03:00
Lasse Collin 85b081f5d4 liblzma: Make 32-bit x86 CRC assembly co-exist with CLMUL
Now runtime detection of CLMUL support can pick between the CLMUL and
the generic assembly implementations. Whatever overhead this has for
builds that omit CLMUL completely isn't important because builds for
any non-ancient system is likely to include the CLMUL code too.

Handle the CRC tables in crcXX_fast.c files because now these files
are built even when assembly code is used.

If 32-bit x86 assembly is enabled then it will always be built even
if compiler flags were such that CLMUL would be allowed unconditionally.
That is, runtime detection will be used anyway. This keeps the build
rules simpler.

In LZ encoder, build and use lzma_lz_hash_table[256] if CLMUL CRC
is used without runtime detection. Previously this wasn't needed
because crc32_table.c included the lzma_crc32_table[][] in the build
unless encoder support had been disabled. Including an 8 KiB table
was silly when only 1 KiB is actually used. So now liblzma is 7 KiB
smaller if CLMUL is enabled without runtime detection.
2024-06-23 14:36:44 +03:00
Lasse Collin 6667d503b5 liblzma: CRC: Rename crcXX_generic to lzma_crcXX_generic
This prepares for the possibility that lzma_crc32_generic and
lzma_crc64_generic are extern functions.
2024-06-23 14:36:44 +03:00
Lasse Collin 6a3c4aaa43 Windows: Drop Visual Studio 2013 support
This simplifies things a little. Building liblzma with VS2013 probably
still worked but building the command line tools was not supported.

Microsoft ended support for VS2013 on 2024-04.
2024-06-20 21:53:07 +03:00
Lasse Collin 30a2d5d510 liblzma: CRC CLMUL: Omit is_arch_extension_supported() when not needed
On E2K the function compiles only due to compiler emulation but the
function is never used. It's cleaner to omit the function when it's
not needed even though it's a "static inline" function.

Thanks to Ilya Kurdyukov.
2024-06-17 15:00:55 +03:00
Lasse Collin 54eaea5ea4 liblzma: x86 CLMUL CRC: Rewrite
It's faster with both tiny and large buffers and doesn't require
disabling any sanitizers. With large buffers the extra speed is
from folding four 16-byte chunks in parallel.

The 32-bit x86 with MSVC reportedly still needs a workaround.
Now the simpler "__asm mov ebx, ebx" trick is enough but it
needs to be in lzma_crc64() instead of crc64_arch_optimized().
Thanks to Iouri Kharon for testing and the fix.

Thanks to Ilya Kurdyukov for testing the speed with aligned and
unaligned buffers on a few x86 processors and on E2K v6.

Thanks to Sam James for general feedback.

Fixes: https://github.com/tukaani-project/xz/issues/112
Fixes: https://github.com/tukaani-project/xz/issues/122
2024-06-17 15:00:49 +03:00
Lasse Collin c0e7eaae8d sysdefs.h: Add alignas 2024-06-16 12:59:20 +03:00
Lasse Collin 20014c2614 liblzma: Use a single macro to select CLMUL CRC to build
This way it's clearer that two things cannot be selected
at the same time.
2024-06-16 12:59:17 +03:00
Lasse Collin d8fb098617 liblzma: CRC32 CLMUL: Refactor the constants and simplify
By using modulus scaled constants, the final reduction can
be simplified.
2024-06-16 12:56:54 +03:00
Lasse Collin ef652ac391 liblzma: CRC64 CLMUL: Refactor the constants
Now it refers to crc_clmul_consts_gen.c. vfold8 was renamed to mu_p
and the p no longer has the lowest bit set (it makes no difference
as the output bits it affects are ignored).
2024-06-16 12:56:54 +03:00
Lasse Collin 9f5fc17e32 liblzma: Add crc_clmul_consts_gen.c
It's a standalone program that prints the required constants.
It's won't be a part of the normal build of the package.
2024-06-16 12:56:54 +03:00
Lasse Collin 71b147aab7 liblzma: Remove CRC_USE_GENERIC_FOR_SMALL_INPUTS
It was already commented out.
2024-06-16 12:56:54 +03:00
Lasse Collin f99a7be406 liblzma: Remove crc_attr_no_sanitize_address
It's not enough to silence the address sanitizer. Also memory and
thread sanitizers would need to be silenced. They, at least currently,
aren't smart enough to see that the extra bytes are discarded from
the xmm registers by later instructions.

Valgrind is smarter, possibly because this kind of code isn't weird
to write in assembly. Agner Fog's optimizing_assembly.pdf even mentions
this idea of doing an aligned read and then discarding the extra
bytes. The sanitizers don't instrument assembly code but Valgrind
checks all code.

It's better to change the implementation to avoid the sanitization
attributes which also look scary in the code. (Somehow they can look
more scary than __asm__ which is implictly unsanitized.)

See also:
https://github.com/tukaani-project/xz/issues/112
https://github.com/tukaani-project/xz/issues/122
2024-06-16 12:56:54 +03:00
Lasse Collin c7164b1927 xz: Fix white space 2024-06-11 22:42:26 +03:00
Lasse Collin 0a32d2072c liblzma: Fix a typo in a comment
Thanks to Sam James for spotting it.

Fixes: f644473a21
2024-06-11 22:42:04 +03:00
Lasse Collin afd9b4d282 liblzma: Fix a comment indentation 2024-06-10 23:19:27 +03:00
Lasse Collin 50e6bff274 liblzma: Fix white space 2024-06-10 23:19:27 +03:00
Lasse Collin caea7844d3 tuklib: __STDC_VERSION__ in C23 is 202311 2024-06-10 23:19:27 +03:00
RainRat 9e73918a4f Fix typos
Closes: https://github.com/tukaani-project/xz/pull/124
2024-06-07 16:01:27 +03:00
Lasse Collin 04b23addf3 tuklib_integer: Fix building on OpenBSD/sparc64 that uses GCC 4.2
GCC 4.2 doesn't have __builtin_bswap16() and friends so tuklib_integer.h
tries to use OS-specific byte swap methods instead. On OpenBSD those
macros are swap16/32/64 instead of bswap16/32/64 like on other *BSDs
and Darwin.

An alternative to "#ifdef __OpenBSD__" could be "#ifdef swap16" as it
is a macro. But since OpenBSD seems to be a special case under this
special case of "*BSDs and Darwin", checking for __OpenBSD__ seems
the more conservative choice now.

Thanks to Christian Weisgerber and Brad Smith who both submitted
the same patch a few hours apart.

Co-authored-by: Christian Weisgerber <naddy@mips.inka.de>
Co-authored-by: Brad Smith <brad@comstyle.com>
Closes: https://github.com/tukaani-project/xz/pull/126
2024-06-07 15:47:20 +03:00
Lasse Collin dc03f6290f liblzma: Add ARM64 CRC32 instruction support detection on OpenBSD
The C code is from Christian Weisgerber, I merely reordered the OSes.
Then I added the build system checks without testing them.

Also thanks to Brad Smith who submitted a similar patch on GitHub
a few hours after Christian had sent his via email.

Co-authored-by: Christian Weisgerber <naddy@mips.inka.de>
Closes: https://github.com/tukaani-project/xz/pull/125
2024-06-07 15:06:59 +03:00
Sam James b69768c8bd xz: list: suppress -Wformat-nonliteral for Solaris
Solaris' GCC can't understand that our use is fine, unlike modern compilers:
```
list.c: In function 'print_totals_basic':
list.c:1191:4: error: format not a string literal, argument types not checked [-Werror=format-nonliteral]
  uint64_to_str(totals.files, 0));
  ^~~~~~~~~~~~~
cc1: all warnings being treated as errors
```

It's presumably because of older gettext missing format attributes.

This is with `gcc (GCC) 7.3.0`.
2024-06-03 12:32:34 +03:00
Lasse Collin 4e9023857d Fix typos
Thanks to xx on #tukaani.
2024-05-18 00:34:07 +03:00
Lasse Collin b14d08fbbc liblzma: Fix white space
Thanks to xx on #tukaani.
2024-05-18 00:24:50 +03:00
Lasse Collin 142e670a41 xz: Document the static function get_chains_memusage() 2024-05-13 18:00:41 +03:00
Lasse Collin 78e984399a xz: Rename filters_memusage_max() to get_chains_memusage() 2024-05-13 18:00:41 +03:00
Lasse Collin 54c3db0a83 xz: Rename filter_memusages to chains_memusages 2024-05-13 18:00:41 +03:00
Lasse Collin d9e1ae79ec xz: Simplify the memory usage scaling code
This is closer to what it was before the --filtersX support was added,
just extended to support for scaling all filter chains. The method
before this commit was an extended version of the original too but
it was done in a more complex way for no clear reason. In case of
an error, the complex version printed fewer informative messages
(a good thing) but it's not a sigificant benefit.

In the limit is too low even for single-threaded mode, the required
amount of memory is now reported like in 5.4.x instead of like in
5.5.1alpha - 5.6.1 which showed the original non-scaled usage. It
had been a FIXME in the old code but it's not clear what message
makes the most sense.

Fixes: 5f0c5a0438
2024-05-13 18:00:41 +03:00
Lasse Collin 0ee56983d1 xz: Edit comments 2024-05-13 18:00:41 +03:00
Lasse Collin ec82a49c35 xz: Rename chain_idx to chain_num 2024-05-13 18:00:41 +03:00
Lasse Collin a731a6993c xz: Edit coding style 2024-05-13 18:00:41 +03:00
Lasse Collin 32eb176b89 xz: Edit comments
Fixes: 5f0c5a0438
2024-05-13 15:41:48 +03:00
Lasse Collin b90339f4da xz: Fix grammar in a comment
Fixes: cb3111e3ed
2024-05-13 15:41:48 +03:00
Lasse Collin 4c0bdaf13d xz: Rename filter_memusages to encoder_memusages 2024-05-13 15:41:46 +03:00
Lasse Collin b54aa023e0 xz: Edit coding style 2024-05-13 15:41:05 +03:00
Lasse Collin 49f67d3d3f xz: Rename filters_index to chain_num
The reason is the same as in bd0782c1f13e52cd0fd8415208e30e47004a4c68.
2024-05-13 15:41:05 +03:00
Lasse Collin ff9e8b3d06 xz: Replace a few uint32_t with "unsigned" to reduce the number of casts
These hold only tiny values.
2024-05-13 15:41:05 +03:00
Lasse Collin b5e6c1113b xz: Rename filters_used_mask to chains_used_mask
The reason is the same as in bd0782c1f13e52cd0fd8415208e30e47004a4c68.
2024-05-13 15:41:05 +03:00