Filter coders are initialized using lzma_next_filter_init(). If filter
chain initialization fails, the entire filter chain should be cleaned up
with lzma_next_end(). lzma_raw_encoder_init() and lzma_raw_decoder_init()
have always done this via lzma_raw_coder_init() in filter_common.c.
(Separate cleanup is weird, but it must have made sense to the young me.)
The following decoders initialize LZMA1 filter directly without using the
raw filter chain API. This avoids needlessly pulling in all other filters
when a program is linked against static liblzma. These functions didn't
call lzma_next_end() after an initialization error, which left the state
available for later reinitialization.
- lzma_alone_decoder()
- lzma_lzip_decoder()
- lzma_auto_decoder() [*]
- lzma_microlzma_decoder()
[*] lzma_auto_decoder() is only indirectly affected due to the first two
functions. lzma_auto_decoder() itself doesn't need a fix.
There are also encoder functions that initialize the LZMA1 encoder
directly. They don't have this issue because the whole lzma_stream
is cleaned up when encoder initialization fails.
Reported-by: GitHub user christos-cantina-security (christos-spearbit)
Turns out that this is needed for .lzma files as the spec in
LZMA SDK says that end marker may be present even if the size
is stored in the header. Such files are rare but exist in the
real world. The code in liblzma is so old that the spec didn't
exist in LZMA SDK back then and I had understood that such
files weren't possible (the lzma tool in LZMA SDK didn't
create such files).
This modifies the internal API so that LZMA decoder can be told
if EOPM is allowed even when the uncompressed size is known.
It's allowed with .lzma and not with other uses.
Thanks to Karl Beldan for reporting the problem.
