.xz, .lzma, and .lz Test Files ------------------------------ 0. Introduction This directory contains bunch of files to test handling of .xz, .lzma (LZMA_Alone), and .lz (lzip) files in decoder implementations. Many of the files have been created by hand with a hex editor, thus there is no better "source code" than the files themselves. All the test files and this README may be distributed under the terms of the BSD Zero Clause License (0BSD). 1. File Types Good files (good-*) must decode successfully without requiring a lot of CPU time or RAM. Unsupported files (unsupported-*) are good files, but headers indicate features not supported by the current file format specification. Bad files (bad-*) must cause the decoder to give an error. Like with the good files, these files must not require a lot of CPU time or RAM before they get detected to be broken. 2. Descriptions of Individual .xz Files 2.1. Good Files good-0-empty.xz has one Stream with no Blocks. good-0pad-empty.xz has one Stream with no Blocks followed by four-byte Stream Padding. good-0cat-empty.xz has two zero-Block Streams concatenated without Stream Padding. good-0catpad-empty.xz has two zero-Block Streams concatenated with four-byte Stream Padding between the Streams. good-1-check-none.xz has one Stream with one Block with two uncompressed LZMA2 chunks and no integrity check. good-1-check-crc32.xz has one Stream with one Block with two uncompressed LZMA2 chunks and CRC32 check. good-1-check-crc64.xz is like good-1-check-crc32.xz but with CRC64. good-1-check-sha256.xz is like good-1-check-crc32.xz but with SHA256. good-2-lzma2.xz has one Stream with two Blocks with one uncompressed LZMA2 chunk in each Block. good-1-block_header-1.xz has both Compressed Size and Uncompressed Size in the Block Header. This has also four extra bytes of Header Padding. good-1-block_header-2.xz has known Compressed Size. good-1-block_header-3.xz has known Uncompressed Size. good-1-delta-lzma2.tiff.xz is an image file that compresses better with Delta+LZMA2 than with plain LZMA2. good-1-arm64-lzma2-1.xz uses the ARM64 filter and LZMA2. The uncompressed data is constructed so that it tests integer wrap around and sign extension. To recreate the file, compress using XZ Utils 5.4.x (newer may or may not work too): ./debug/testfilegen-arm64 \ | xz -T1 -Ccrc32 --arm64 \ --lzma2=dict=64KiB,lp=2,lc=2 \ > good-1-arm64-lzma2-1.xz good-1-arm64-lzma2-2.xz is like good-1-arm64-lzma2-1.xz but with non-zero start offset. XZ Embedded doesn't support this file. To recreate the file, compress using XZ Utils 5.4.x (newer may or may not work too): ./debug/testfilegen-arm64 \ | xz -T1 -Ccrc32 --arm64=start=4294963200 \ --lzma2=dict=64KiB,lp=2,lc=2 \ > good-1-arm64-lzma2-2.xz good-1-lzma2-1.xz has two LZMA2 chunks, of which the second sets new properties. good-1-lzma2-2.xz has two LZMA2 chunks, of which the second resets the state without specifying new properties. good-1-lzma2-3.xz has two LZMA2 chunks, of which the first is uncompressed and the second is LZMA. The first chunk resets dictionary and the second sets new properties. good-1-lzma2-4.xz has three LZMA2 chunks: First is LZMA, second is uncompressed with dictionary reset, and third is LZMA with new properties but without dictionary reset. good-1-lzma2-5.xz has an empty LZMA2 stream with only the end of payload marker. XZ Utils 5.0.1 and older incorrectly see this file as corrupt. good-1-3delta-lzma2.xz has three Delta filters and LZMA2. good-1-empty-bcj-lzma2.xz has an empty Block that uses PowerPC BCJ and LZMA2. liblzma from XZ Utils 5.0.1 and older may incorrectly return LZMA_BUF_ERROR in some cases. See commit message d8db706acb8316f9861abd432cfbe001dd6d0c5c for the details. 2.2. Unsupported Files unsupported-check.xz uses Check ID 0x02 which isn't supported by the current version of the file format. It is implementation-defined how this file handled (it may reject it, or decode it possibly with a warning). unsupported-block_header.xz has a non-null byte in Header Padding, which may indicate presence of a new unsupported field. unsupported-filter_flags-1.xz has unsupported Filter ID 0x7F. unsupported-filter_flags-2.xz specifies only Delta filter in the List of Filter Flags, but Delta isn't allowed as the last filter in the chain. It could be a little more correct to detect this file as corrupt instead of unsupported, but saying it is unsupported is simpler in case of liblzma. unsupported-filter_flags-3.xz specifies two LZMA2 filters in the List of Filter Flags. LZMA2 is allowed only as the last filter in the chain. It could be a little more correct to detect this file as corrupt instead of unsupported, but saying it is unsupported is simpler in case of liblzma. 2.3. Bad Files bad-0pad-empty.xz has one Stream with no Blocks followed by five-byte Stream Padding. Stream Padding must be a multiple of four bytes, thus this file is corrupt. bad-0catpad-empty.xz has two zero-Block Streams concatenated with five-byte Stream Padding between the Streams. bad-0cat-alone.xz is good-0-empty.xz concatenated with an empty LZMA_Alone file. bad-0cat-header_magic.xz is good-0cat-empty.xz but with one byte wrong in the Header Magic Bytes field of the second Stream. liblzma gives LZMA_DATA_ERROR for this. (LZMA_FORMAT_ERROR is used only if the first Stream of a file has invalid Header Magic Bytes.) bad-0-header_magic.xz is good-0-empty.xz but with one byte wrong in the Header Magic Bytes field. liblzma gives LZMA_FORMAT_ERROR for this. bad-0-footer_magic.xz is good-0-empty.xz but with one byte wrong in the Footer Magic Bytes field. liblzma gives LZMA_DATA_ERROR for this. bad-0-empty-truncated.xz is good-0-empty.xz without the last byte of the file. bad-0-nonempty_index.xz has no Blocks but Index claims that there is one Block. bad-0-backward_size.xz has wrong Backward Size in Stream Footer. bad-1-stream_flags-1.xz has different Stream Flags in Stream Header and Stream Footer. bad-1-stream_flags-2.xz has wrong CRC32 in Stream Header. bad-1-stream_flags-3.xz has wrong CRC32 in Stream Footer. bad-1-vli-1.xz has two-byte variable-length integer in the Uncompressed Size field in Block Header while one-byte would be enough for that value. It's important that the file gets rejected due to too big integer encoding instead of due to Uncompressed Size not matching the value stored in the Block Header. That is, the decoder must not try to decode the Compressed Data field. bad-1-vli-2.xz has ten-byte variable-length integer as Uncompressed Size in Block Header. It's important that the file gets rejected due to too big integer encoding instead of due to Uncompressed Size not matching the value stored in the Block Header. That is, the decoder must not try to decode the Compressed Data field. bad-1-block_header-1.xz has Block Header that ends in the middle of the Filter Flags field. bad-1-block_header-2.xz has Block Header that has Compressed Size and Uncompressed Size but no List of Filter Flags field. bad-1-block_header-3.xz has wrong CRC32 in Block Header. bad-1-block_header-4.xz has too big Compressed Size in Block Header (2^63 - 1 bytes while maximum is a little less, because the whole Block must stay smaller than 2^63). It's important that the file gets rejected due to invalid Compressed Size value; the decoder must not try decoding the Compressed Data field. bad-1-block_header-5.xz has zero as Compressed Size in Block Header. bad-1-block_header-6.xz has corrupt Block Header which may crash xz -lvv in XZ Utils 5.0.3 and earlier. It was fixed in the commit c0297445064951807803457dca1611b3c47e7f0f. bad-2-index-1.xz has wrong Unpadded Sizes in Index. bad-2-index-2.xz has wrong Uncompressed Sizes in Index. bad-2-index-3.xz has non-null byte in Index Padding. bad-2-index-4.xz wrong CRC32 in Index. bad-2-index-5.xz has zero as Unpadded Size. It is important that the file gets rejected specifically due to Unpadded Size having an invalid value. bad-3-index-uncomp-overflow.xz has Index whose Uncompressed Size fields have huge values whose sum exceeds the maximum allowed size of 2^63 - 1 bytes. In this file the sum is exactly 2^64. lzma_index_append() in liblzma <= 5.2.6 lacks the integer overflow check for the uncompressed size and thus doesn't catch the error when decoding the Index field in this file. This makes "xz -l" not detect the error and will display 0 as the uncompressed size. Note that regular decompression isn't affected by this bug because it uses lzma_index_hash_append() instead. bad-2-compressed_data_padding.xz has non-null byte in the padding of the Compressed Data field of the first Block. bad-1-check-crc32.xz has wrong Check (CRC32). bad-1-check-crc32-2.xz has Compressed Size and Uncompressed Size in Block Header but wrong Check (CRC32) in the actual data. This file differs by one byte from good-1-block_header-1.xz: the last byte of the Check field is wrong. This file is useful for testing error detection in the threaded decoder when a worker thread is configured to pass input one byte at a time to the Block decoder. bad-1-check-crc64.xz has wrong Check (CRC64). bad-1-check-sha256.xz has wrong Check (SHA-256). bad-1-lzma2-1.xz has LZMA2 stream whose first chunk (uncompressed) doesn't reset the dictionary. bad-1-lzma2-2.xz has two LZMA2 chunks, of which the second chunk indicates dictionary reset, but the LZMA compressed data tries to repeat data from the previous chunk. bad-1-lzma2-3.xz sets new invalid properties (lc=8, lp=0, pb=0) in the middle of Block. bad-1-lzma2-4.xz has two LZMA2 chunks, of which the first is uncompressed and the second is LZMA. The first chunk resets dictionary as it should, but the second chunk tries to reset state without specifying properties for LZMA. bad-1-lzma2-5.xz is like bad-1-lzma2-4.xz but doesn't try to reset anything in the header of the second chunk. bad-1-lzma2-6.xz has reserved LZMA2 control byte value (0x03). bad-1-lzma2-7.xz has EOPM at LZMA level. bad-1-lzma2-8.xz is like good-1-lzma2-4.xz but doesn't set new properties in the third LZMA2 chunk. bad-1-lzma2-9.xz has LZMA2 stream that is truncated at the end of a LZMA2 chunk (no end marker). The uncompressed size of the partial LZMA2 stream exceeds the value stored in the Block Header. bad-1-lzma2-10.xz has LZMA2 stream that, from point of view of a LZMA2 decoder, extends past the end of Block (and even the end of the file). Uncompressed Size in Block Header is bigger than the invalid LZMA2 stream may produce (even if a decoder reads until the end of the file). The Check type is None to nullify certain simple size-based sanity checks in a Block decoder. bad-1-lzma2-11.xz has LZMA2 stream that lacks the end of payload marker. When Compressed Size bytes have been decoded, Uncompressed Size bytes of output will have been produced but the LZMA2 decoder doesn't indicate end of stream. 3. Descriptions of Individual .lzma Files 3.1. Good Files good-unknown_size-with_eopm.lzma has unknown size in the header and end of payload marker at the end. good-known_size-without_eopm.lzma has a known size in the header and no end of payload marker at the end. good-known_size-with_eopm.lzma has a known size in the header and end of payload marker at the end. XZ Utils 5.2.5 and older will give an error at the end of the file after producing the correct uncompressed output. 3.2. Bad Files bad-unknown_size-without_eopm.lzma has unknown size in the header but no end of payload marker at the end. This file might be seen by a decoder as if it were truncated. bad-too_big_size-with_eopm.lzma has too big uncompressed size in the header and the end of payload marker will be detected before the specified number of bytes have been decoded. bad-too_small_size-without_eopm-1.lzma has too small uncompressed size in the header. The decoder will look for end of payload marker but instead find a literal that would produce more output. bad-too_small_size-without_eopm-2.lzma is like -1 above but instead of a literal the problem occurs with a short repeated match. bad-too_small_size-without_eopm-3.lzma is like -1 above but instead of a literal the problem occurs in the middle of a match. 4. Descriptions of Individual .lz (lzip) Files 4.1. Good Files good-1-v0.lz contains a single version 0 member. lzip 1.17 and *older* can decompress this; support for version 0 was removed in lzip 1.18. good-1-v0-trailing-1.lz is like good-1-v0.lz but contains trailing data that the decompressor must ignore. good-1-v1.lz contains a single version 1 member. lzip 1.3 and newer can decompress this. good-1-v1-trailing-1.lz is like good-1-v1.lz but contains trailing data that the decompressor must ignore. good-1-v1-trailing-2.lz is like good-1-v1.lz but contains trailing data whose first three bytes match the .lz magic bytes. With lzip >= 1.20 this file results in an error unless one uses the command line option --loose-trailing. lzip 1.3 to 1.19 decode this file successfully by default. XZ Utils uses the old behavior because it allows lzma_code() to stop at the first byte of the trailing data as long as the first byte isn't 0x4C (L in US-ASCII); otherwise the first 1-3 bytes that equal to the magic bytes are consumed and lost in lzma_code(), and this is visible in xz too: $ ( xz -dc ; cat ) < good-1-v1-trailing-2.lz Hello World! Trailing garbage $ ( xz -dc --single-stream ; cat ) < good-1-v1-trailing-2.lz Hello World! LZITrailing garbage good-2-v0-v1.lz contains two members of which the first is version 0 and the second version 1. lzip versions 1.3 to 1.17 (inclusive) can decompress this. good-2-v1-v0.lz contains two members of which the first is version 1 and the second version 0. lzip versions 1.3 to 1.17 (inclusive) can decompress this. good-2-v1-v1.lz contains two version 1 members. lzip versions 1.3 and newer can decompress this. 4.2. Unsupported Files unsupported-1-v234.lz is like good-1-v1.lz except the version field has been set to 234 (0xEA) which, as of writing, isn't defined or supported by any .lz implementation. 4.3. Bad Files bad-1-v1-magic-1.lz is like good-1-v1.lz but the first magic byte is wrong. bad-1-v1-magic-2.lz is like good-1-v1.lz but the last (fourth) magic byte is wrong. bad-1-v1-dict-1.lz has too low value in the dictionary size field. bad-1-v1-dict-2.lz has too high value in the dictionary size field. bad-1-v1-crc32.lz has wrong CRC32 value. bad-1-v0-uncomp-size.lz is version 0 format with incorrect value in the uncompressed size field. bad-1-v1-uncomp-size.lz is version 1 format with incorrect value in the uncompressed size field. bad-1-v1-member-size.lz has incorrect value in the member size field. bad-1-v1-trailing-magic.lz has the four .lz magic bytes as trailing data. This should be detected as a truncated file and thus result in an error. That is, the last four bytes of the file should not be ignored as trailing garbage. lzip >= 1.18 matches this behavior while older versions ignore the last four bytes and don't indicate an error.