xz/doc/liblzma-advanced.txt

Advanced features of liblzma
----------------------------

0. Introduction

    Most developers need only the basic features of liblzma. These
    features allow single-threaded encoding and decoding of .lzma files
    in streamed mode.

    In some cases developers want more. The .lzma file format is
    designed to allow multi-threaded encoding and decoding and limited
    random-access reading. These features are possible in non-streamed
    mode and limitedly also in streamed mode.

    To take advange of these features, the application needs a custom
    .lzma file format handler. liblzma provides a set of tools to ease
    this task, but it's still quite a bit of work to get a good custom
    .lzma handler done.


1. Where to begin

    Start by reading the .lzma file format specification. Understanding
    the basics of the .lzma file structure is required to implement a
    custom .lzma file handler and to understand the rest of this document.


2. The basic components

2.1. Stream Header and tail

    Stream Header begins the .lzma Stream and Stream tail ends it. Stream
    Header is defined in the file format specification, but Stream tail
    isn't (thus I write "tail" with a lower-case letter). Stream tail is
    simply the Stream Flags and the Footer Magic Bytes fields together.
    It was done this way in liblzma, because the Block coders take care
    of the rest of the stuff in the Stream Footer.

    For now, the size of Stream Header is fixed to 11 bytes. The header
    <lzma/stream_flags.h> defines LZMA_STREAM_HEADER_SIZE, which you
    should use instead of a hardcoded number. Similarly, Stream tail
    is fixed to 3 bytes, and there is a constant LZMA_STREAM_TAIL_SIZE.

    It is possible, that a future version of the .lzma format will have
    variable-sized Stream Header and tail. As of writing, this seems so
    unlikely though, that it was considered simplest to just use a
    constant instead of providing a functions to get and store the sizes
    of the Stream Header and tail.


2.x. Stream tail

    For now, the size of Stream tail is fixed to 3 bytes. The header
    <lzma/stream_flags.h> defines LZMA_STREAM_TAIL_SIZE, which you
    should use instead of a hardcoded number.


3. Keeping track of size information

    The lzma_info_* functions found from <lzma/info.h> should ease the
    task of keeping track of sizes of the Blocks and also the Stream
    as a whole. Using these functions is strongly recommended, because
    there are surprisingly many situations where an error can occur,
    and these functions check for possible errors every time some new
    information becomes available.

    If you find lzma_info_* functions lacking something that you would
    find useful, please contact the author.


3.1. Start offset of the Stream

    If you are storing the .lzma Stream inside anothe file format, or
    for some other reason are placing the .lzma Stream to somewhere
    else than to the beginning of the file, you should tell the starting
    offset of the Stream using lzma_info_start_offset_set().

    The start offset of the Stream is used for two distinct purporses.
    First, knowing the start offset of the Stream allows
    lzma_info_alignment_get() to correctly calculate the alignment of
    every Block. This information is given to the Block encoder, which
    will calculate the size of Header Padding so that Compressed Data
    is alignment at an optimal offset.

    Another use for start offset of the Stream is in random-access
    reading. If you set the start offset of the Stream, lzma_info_locate()
    will be able to calculate the offset relative to the beginning of the
    file containing the Stream (instead of offset relative to the
    beginning of the Stream).


3.2. Size of Stream Header

    While the size of Stream Header is constant (11 bytes) in the current
    version of the .lzma file format, this may change in future.


3.3. Size of Header Metadata Block

    This information is needed when doing random-access reading, and
    to verify the value of this field stored in Footer Metadata Block.


3.4. Total Size of the Data Blocks


3.5. Uncompressed Size of Data Blocks


3.6. Index




x. Alignment

    There are a few slightly different types of alignment issues when
    working with .lzma files.

    The .lzma format doesn't strictly require any kind of alignment.
    However, if the encoder carefully optimizes the alignment in all
    situations, it can improve compression ratio, speed of the encoder
    and decoder, and slightly help if the files get damaged and need
    recovery.

    Alignment has the most significant effect compression ratio FIXME


x.1. Compression ratio

    Some filters take advantage of the alignment of the input data.
    To get the best compression ratio, make sure that you feed these
    filters correctly aligned data.

    Some filters (e.g. LZMA) don't necessarily mind too much if the
    input doesn't match the preferred alignment. With these filters
    the penalty in compression ratio depends on the specific type of
    data being compressed.

    Other filters (e.g. PowerPC executable filter) won't work at all
    with data that is improperly aligned. While the data can still
    be de-filtered back to its original form, the benefit of the
    filtering (better compression ratio) is completely lost, because
    these filters expect certain patterns at properly aligned offsets.
    The compression ratio may even worse with incorrectly aligned input
    than without the filter.


x.1.1. Inter-filter alignment

    When there are multiple filters chained, checking the alignment can
    be useful not only with the input of the first filter and output of
    the last filter, but also between the filters.

    Inter-filter alignment important especially with the Subblock filter.


x.1.2. Further compression with external tools

    This is relatively rare situation in practice, but still worth
    understanding.

    Let's say that there are several SPARC executables, which are each
    filtered to separate .lzma files using only the SPARC filter. If
    Uncompressed Size is written to the Block Header, the size of Block
    Header may vary between the .lzma files. If no Padding is used in
    the Block Header to correct the alignment, the starting offset of
    the Compressed Data field will be differently aligned in different
    .lzma files.

    All these .lzma files are archived into a single .tar archive. Due
    to nature of the .tar format, every file is aligned inside the
    archive to an offset that is a multiple of 512 bytes.

    The .tar archive is compressed into a new .lzma file using the LZMA
    filter with options, that prefer input alignment of four bytes. Now
    if the independent .lzma files don't have the same alignment of
    the Compressed Data fields, the LZMA filter will be unable to take
    advantage of the input alignment between the files in the .tar
    archive, which reduces compression ratio.

    Thus, even if you have only single Block per file, it can be good for
    compression ratio to align the Compressed Data to optimal offset.


x.2. Speed

    Most modern computers are faster when multi-byte data is located
    at aligned offsets in RAM. Proper alignment of the Compressed Data
    fields can slightly increase the speed of some filters.


x.3. Recovery

    Aligning every Block Header to start at an offset with big enough
    alignment may ease or at least speed up recovery of broken files.


y. Typical usage cases

y.x. Parsing the Stream backwards

    You may need to parse the Stream backwards if you need to get
    information such as the sizes of the Stream, Index, or Extra.
    The basic procedure to do this follows.

    Locate the end of the Stream. If the Stream is stored as is in a
    standalone .lzma file, simply seek to the end of the file and start
    reading backwards using appropriate buffer size. The file format
    specification allows arbitrary amount of Footer Padding (zero or more
    NUL bytes), which you skip before trying to decode the Stream tail.

    Once you have located the end of the Stream (a non-NULL byte), make
    sure you have at least the last LZMA_STREAM_TAIL_SIZE bytes of the
    Stream in a buffer. If there isn't enough bytes left from the file,
    the file is too small to contain a valid Stream. Decode the Stream
    tail using lzma_stream_tail_decoder(). Store the offset of the first
    byte of the Stream tail; you will need it later.

    You may now want to do some internal verifications e.g. if the Check
    type is supported by the liblzma build you are using.

    Decode the Backward Size field with lzma_vli_reverse_decode(). The
    field is at maximum of LZMA_VLI_BYTES_MAX bytes long. Check that
    Backward Size is not zero. Store the offset of the first byte of
    the Backward Size; you will need it later.

    Now you know the Total Size of the last Block of the Stream. It's the
    value of Backward Size plus the size of the Backward Size field. Note
    that you cannot use lzma_vli_size() to calculate the size since there
    might be padding; you need to use the real observed size of the
    Backward Size field.

    At this point, the operation continues differently for Single-Block
    and Multi-Block Streams.


y.x.1. Single-Block Stream

    There might be Uncompressed Size field present in the Stream Footer.
    You cannot know it for sure unless you have already parsed the Block
    Header earlier. For security reasons, you probably want to try to
    decode the Uncompressed Size field, but you must not indicate any
    error if decoding fails. Later you can give the decoded Uncompressed
    Size to Block decoder if Uncopmressed Size isn't otherwise known;
    this prevents it from producing too much output in case of (possibly
    intentionally) corrupt file.

    Calculate the the start offset of the Stream:

        backward_offset - backward_size - LZMA_STREAM_HEADER_SIZE

    backward_offset is the offset of the first byte of the Backward Size
    field. Remember to check for integer overflows, which can occur with
    invalid input files.

    Seek to the beginning of the Stream. Decode the Stream Header using
    lzma_stream_header_decoder(). Verify that the decoded Stream Flags
    match the values found from Stream tail. You can use the
    lzma_stream_flags_is_equal() macro for this.

    Decode the Block Header. Verify that it isn't a Metadata Block, since
    Single-Block Streams cannot have Metadata. If Uncompressed Size is
    present in the Block Header, the value you tried to decode from the
    Stream Footer must be ignored, since Uncompressed Size wasn't actually
    present there. If Block Header doesn't have Uncompressed Size, and
    decoding the Uncompressed Size field from the Stream Footer failed,
    the file is corrupt.

    If you were only looking for the Uncompressed Size of the Stream,
    you now got that information, and you can stop processing the Stream.

    To decode the Block, the same instructions apply as described in
    FIXME. However, because you have some extra known information decoded
    from the Stream Footer, you should give this information to the Block
    decoder so that it can verify it while decoding:
      - If Uncompressed Size is not present in the Block Header, set
        lzma_options_block.uncompressed_size to the value you decoded
        from the Stream Footer.
      - Always set lzma_options_block.total_size to backward_size +
        size_of_backward_size (you calculated this sum earlier already).


y.x.2. Multi-Block Stream

    Calculate the start offset of the Footer Metadata Block:

        backward_offset - backward_size

    backward_offset is the offset of the first byte of the Backward Size
    field. Remember to check for integer overflows, which can occur with
    broken input files.

    Decode the Block Header. Verify that it is a Metadata Block. Set
    lzma_options_block.total_size to backward_size + size_of_backward_size
    (you calculated this sum earlier already). Then decode the Footer
    Metadata Block.

    Store the decoded Footer Metadata to lzma_info structure using
    lzma_info_set_metadata(). Set also the offset of the Backward Size
    field using lzma_info_size_set(). Then you can get the start offset
    of the Stream using lzma_info_size_get(). Note that any of these steps
    may fail so don't omit error checking.

    Seek to the beginning of the Stream. Decode the Stream Header using
    lzma_stream_header_decoder(). Verify that the decoded Stream Flags
    match the values found from Stream tail. You can use the
    lzma_stream_flags_is_equal() macro for this.

    If you were only looking for the Uncompressed Size of the Stream,
    it's possible that you already have it now. If Uncompressed Size (or
    whatever information you were looking for) isn't available yet,
    continue by decoding also the Header Metadata Block. (If some
    information is missing, the Header Metadata Block has to be present.)

    Decoding the Data Blocks goes the same way as described in FIXME.


y.x.3. Variations

    If you know the offset of the beginning of the Stream, you may want
    to parse the Stream Header before parsing the Stream tail.