liblzma: Fix a buffer overflow in lzma_index_append()

If lzma_index_decoder() was used to decode an Index that contained no
Records, the resulting lzma_index had an invalid internal "prealloc"
value. If lzma_index_append() was called on this lzma_index, too
little memory would be allocated and a buffer overflow would occur.

While this combination of the API functions is meant to work, in the
real-world apps this call sequence is rare or might not exist at all.

This bug is older than xz 5.0.0, so all stable releases are affected.

Reported-by: GitHub user christos-spearbit

src/liblzma/common/index.c

@@ -433,6 +433,26 @@ lzma_index_prealloc(lzma_index *i, lzma_vli records)
 	if (records > PREALLOC_MAX)
 		records = PREALLOC_MAX;
 
+	// If index_decoder.c calls us with records == 0, it's decoding
+	// an Index that has no Records. In that case the decoder won't call
+	// lzma_index_append() at all, and i->prealloc isn't used during
+	// the Index decoding either.
+	//
+	// Normally the first lzma_index_append() call from the Index decoder
+	// would reset i->prealloc to INDEX_GROUP_SIZE. With no Records,
+	// lzma_index_append() isn't called and the resetting of prealloc
+	// won't occur either. Thus, if records == 0, use the default value
+	// INDEX_GROUP_SIZE instead.
+	//
+	// NOTE: lzma_index_append() assumes i->prealloc > 0. liblzma <= 5.8.2
+	// didn't have this check and could set i->prealloc = 0, which would
+	// result in a buffer overflow if the application called
+	// lzma_index_append() after decoding an empty Index. Appending
+	// Records after decoding an Index is a rare thing to do, but
+	// it is supposed to work.
+	if (records == 0)
+		records = INDEX_GROUP_SIZE;
+
 	i->prealloc = (size_t)(records);
 	return;
 }
@@ -685,6 +705,7 @@ lzma_index_append(lzma_index *i, const lzma_allocator *allocator,
 		++g->last;
 	} else {
 		// We need to allocate a new group.
+		assert(i->prealloc > 0);
 		g = lzma_alloc(sizeof(index_group)
 				+ i->prealloc * sizeof(index_record),
 				allocator);