If at least three files are passed to xz --list, the totals might exceed
the range of uint64_t. Previously wrong totals were displayed in this
case. Now an error is shown as soon as an overflow would occur, and the
remaining files aren't processed. In most other cases an error doesn't
prevent xz from processing the remaining files, but in this case it
feels OK because (1) the error should be very rare, (2) it keeps the
code slightly simpler, and (3) with xz --robot --list the caller will
likely ignore the output anyway when xz exits with a non-zero status.
Co-authored-by: Lasse Collin <lasse.collin@tukaani.org>
Fixes: https://github.com/tukaani-project/xz/pull/240
lzma_index_cat() is used by lzma_file_info_decoder() and by a few apps
that were written before lzma_file_info_decoder() existed (it was added
in 5.3.1alpha). The most common use case for these functions is xz --list.
The other use case is random access decoding.
When decoding Index fields from multi-Stream .xz files, lzma_index_cat()
is used to concatenate the decoded lzma_index structures. Such decoding
starts from the last Stream in the file and stops at the first Stream in
the file, so lzma_index_cat() is called with the most recently decoded
Stream as the destination argument while the source argument contains all
other Streams that have been decoded so far. That is, the typical use case
is lzma_index_cat(one_stream, many_streams, NULL).
With the old code, lzma_index_cat() appended Streams from src into dest
one at a time and updated their absolute offsets at the same time.
When a .xz file contained many Streams, the number of Streams in
the source argument grew for each call, which created a performance
problem if the file contained an unusually large number of Streams
(over 10,000). (A large number of Blocks isn't a problem.)
Make lzma_index_cat() prepend the index_streams from dest to src
instead of appending from src to dest. Use the _bias members to adjust
the offsets of the existing index_streams in src.
Change the numbering in index_stream.number from 1, 2, 3, ... to
...., 2, 1, 0. That is, internally the last index_stream is now 0.
Now lzma_index_cat() needs to update all Streams from dest instead from
src, which keeps the normal use case fast. Obviously the performance
problem still exists, but now it's in the opposite situation (repeated
calls of lzma_index_cat(many_streams, one_stream, NULL)). This should
be acceptable because there should be no use cases where doing it would
be useful. A full fix would require a more complex patch.
As an alternative, I tried a rewrite of index.c that replaces the trees
with plain arrays and realloc(). It's smaller and simpler, but it's
still somewhat complex because the API is too flexible compared to what
functionality is actually needed. I didn't merge that version because
a rewrite has a higher risk of regressions, and in some corner cases
the peak memory usage was considerably higher than with the old code
(if a custom lzma_allocator is used, realloc() has to be emulated with
alloc + memcpy + free, and native realloc() might need to do that too),
and large preallocations make it unclear what lzma_index_memusage() and
lzma_index_memused() should return.
Below is an example where xz --list became too slow before this commit.
Pass 18 or 20 instead of 16 to seq, and xz effectively hangs.
true | xz > big.xz
for I in $(seq 16)
do
cat big.xz big.xz > tmp.xz
mv tmp.xz big.xz
echo ====
echo $I
time xz -l big.xz
done
Reported-by: Omkhar Arasaratnam
This can matter on Windows:
- MSVC puts the binaries into subdirectories based on build type
(like "Debug" or "Release"), and then the test executables won't
find liblzma.dll because it's not in the working directory from
which the tests are run.
- With mingw-w64, liblzma.dll was in the working directory so the test
executables worked. However, the if the system happened to have
liblzma.dll in, for example, the Windows directory, then the DLL
search order might pick that instead.
Putting the shared library and test executables in the same directory
avoids these problems.
Reported-by: Frank <mail@franku.nl>
Fixes: https://github.com/tukaani-project/xz/pull/239
Previously, if an Index had more than PREALLOC_MAX Records, then
allocations were done in PREALLOC_DEFAULT chunks (512 Records).
That makes little sense in practice because it's likely that one
would run out of address space later. Return LZMA_MEM_ERROR error
pre-emptively if PREALLOC_MAX would be exceeded.
lzma_index_dup() copied the per-Stream flags but not the cached bitmask
for all Streams except the last one. As a result, lzma_index_checks() on
a duplicated multi-Stream Index reported only the Check type of its
final Stream.
Fixes: https://github.com/tukaani-project/xz/pull/238
If *in_pos > 0, the condition became more relaxed than it needed to be.
Fixes: 5005914f1acf ("liblzma: Index decoder: Reject an obviously-bad Number of>
Fixes: 0352dc7929b1 ("liblzma: Silence a compiler warning")
File info decoder knows Backward Size, so pass it to the Index decoder.
Now the Index decoder, when called from the file info decoder, won't
allocate a large amount of memory when Backward Size is tiny and the
Number of Records field contains a huge invalid value.
If a reasonable memory usage limit was set, the old code returned
LZMA_MEMLIMIT_ERROR if the value in Number of Records was too huge.
Now it results in LZMA_DATA_ERROR which is better.
If lzma_stream was initialized using lzma_index_decoder() or
lzma_file_info_decoder(), and lzma_memusage() was called after a few
Records had already been decoded, the return value was too low because
it was calculated based on the number of Records that haven't been
decoded yet. So near the end of a big Index, the return value would be
tiny compared to the actual memory usage.
It worked correctly in the primary use case, that is, after
LZMA_MEMLIMIT_ERROR. Thus, in practice this bug didn't matter much.
The headings are RTL but in some tables the rows were LTR because the
first strong char was LTR, like the "e" in "-0e". Add {RLM}{LRM}...{RLM}
to those columns ("-0" to "-9", and "-0e" to "-9e"):
sed -ri $'/^msgstr /{s/"(-[0-9]e?)"/"\u200f\u200e\\1\u200f"/}' ar.po
It's not perfect because groff doesn't treat RLM and LRM as zero-width
chars, and thus the columns are slightly misaligned. It's good enough
in practice still.
Thanks-to: Zayed Al-Saidi <zayed.alsaidi@gmail.com>
These workflows don't use any secrets other than GITHUB_TOKEN, which
only has the 'contents: read' permission. That is, if an action was
compromised, it shouldn't be able to modify the repository or steal any
secrets. I recognize that a compromised action isn't completely harmless
still. For example, the runners on GitHub have Internet access which the
action could abuse until the workflow timeout (currently 10 or 20 minutes)
expires.
Now we always use the most recent actions instead of ones that are
sometimes several months old. The hassle of updating the pinned
commit IDs goes away. The benefits feel bigger than the risks.
Link: https://github.com/tukaani-project/xz/pull/227
GitHub's actions/checkout@v6 is documented to need 'contents: read' but
it works without it too, probably because the repository is public.
Reported-by: GitHub user Alb3e3
Fixes: https://github.com/tukaani-project/xz/pull/230
There no point to try decoding further when the Index is clearly not
from a valid .xz file.
This also limits what index_decoder_memconfig() can return as the
memory usage. Very huge values aren't possible anymore. This affects
file_info_decoder_memconfig() in file_info.c where very huge values
aren't expected and could result in an integer overflow when not much
memory has actually been allocated. Now the overflow can only happen if
one manages to successfully allocate and fill a few exabytes of memory.
Reported-by: GitHub user jmestwa-coder
Fixes: https://github.com/tukaani-project/xz/pull/226
