History log of /haiku/src/kits/package/hpkg/PackageFileHeapWriter.cpp
Revision Date Author Comments
# dfbf1c8a 02-Aug-2019 Augustin Cavalier <waddlesplash@gmail.com>

Package Kit: Avoid discarding potential error values.

Spotted by Clang and the [[nodiscard]] patch.


# 4a1d8b1a 16-Jul-2019 Adrien Destugues <pulkomandy@pulkomandy.tk>

PVS V575: strange values passed to well known functions

Change-Id: Id6008d93777f860d0b341bcd012fe6f86c99cf35
Reviewed-on: https://review.haiku-os.org/c/1610
Reviewed-by: Stephan Aßmus <superstippi@gmx.de>


# bb9112df 10-Oct-2014 Michael Lotz <mmlr@mlotz.ch>

Cleanup: Remove doubled semicolon and a trailing whitespace.


# d98a3e02 09-Oct-2014 Michael Lotz <mmlr@mlotz.ch>

package kit: Fix corruption of last package chunk on update.

PackageFileHeapWriter::_UnwriteLastPartialChunk() used ReadData() to
extract the last partial chunk into the pending buffer. This indirectly
calls PackageFileHeapWriter::ReadAndDecompressChunk(), which assumes
data past the last full chunk to come from the pending data buffer.

Since the pending data buffer is not filled in at that point, the call
to ReadAndDecompressChunk() simply did nothing, leaving the object with
a correctly sized but completely nulled pending data buffer. The last
partial chunk of a package would therefore always get corrupted when
updating a package.

Fixes #11306 that provided a reduced test case that happened to corrupt
the only chunk of a package, nulling the .PackageInfo and therefore
making the error more obvious as subsequent parsing of the info failed.


# 8f5130ed 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

package kit: Actually add support for B_HPKG_COMPRESSION_NONE

Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.


# e527b796 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.


# b3263ad3 30-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.


# 6a89a36a 24-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.


# 84297881 23-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Remove BPackageKit::[...]::AbstractDataWriter and friends

* FDDataWriter and ZlibDataWriter weren't used anymore.
* AbstractDataWriter was implemented only by PackageFileHeapWriter,
which was only used by WriterImplBase.
* Add a PackageFileHeapWriter::AddDataThrows() which has semantics
equivalent to the previously inherited WriteDataThrows().


# 32832cbe 23-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Remove BPackageKit::BDataOutput

Use BDataIO instead.


# e2f30519 24-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: user big endian for chunk sizes


# 4ffb9234 23-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapWriter::RemoveDataRanges(): Fix empty heap check


# 88adb4a8 19-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapWriter: Add error message

... in case compression fails unexpectedly.


# b05413ed 18-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapWriter: Fix update case

* Pull _UnwriteLastPartialChunk() out of Reinit() for reuse.
* _UnwriteLastPartialChunk(): fPendingDataSize wasn't set.
* _PushChunks(): Some simplifications for clarity.
* ChunkBuffer/RemoveDataRanges(): Use data reading and decompression
methods provided by our base class instead of duplicating the
implementation.
* RemoveDataRanges():
- _FlushPendingData() before starting, so we don't ignore the pending
data and _UnwriteLastPartialChunk() when done, so a partial chunk
is read back into the pending data buffer.
- fUncompressedHeapSize wasn't reset before the main processing loop,
thus resulting in an erroneous size later on.


# 96af1d28 18-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapWriter: Small style fix


# 796343ed 18-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

Allow specifying the compression level for package creation

* Introduce BPackageWriterParameters which comprises all parameters
for package creation, currently flags and compression level. Such an
object can be passed to BPackageWriter::Init() and is passed on to
PackageWriterImpl and WriterImplBase.
* PackageFileHeapWriter: Add compressionLevel property and pass the
value on to ZlibCompressor.
* package add/create: Add options -0 ... -9 to specify the compression
level to be used.


# 1f633814 17-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.


# bb9112df023feb0398638dad9074db49a5c1a10f 10-Oct-2014 Michael Lotz <mmlr@mlotz.ch>

Cleanup: Remove doubled semicolon and a trailing whitespace.


# d98a3e026c2c8e81509ab5bd163d766f0389a6f5 09-Oct-2014 Michael Lotz <mmlr@mlotz.ch>

package kit: Fix corruption of last package chunk on update.

PackageFileHeapWriter::_UnwriteLastPartialChunk() used ReadData() to
extract the last partial chunk into the pending buffer. This indirectly
calls PackageFileHeapWriter::ReadAndDecompressChunk(), which assumes
data past the last full chunk to come from the pending data buffer.

Since the pending data buffer is not filled in at that point, the call
to ReadAndDecompressChunk() simply did nothing, leaving the object with
a correctly sized but completely nulled pending data buffer. The last
partial chunk of a package would therefore always get corrupted when
updating a package.

Fixes #11306 that provided a reduced test case that happened to corrupt
the only chunk of a package, nulling the .PackageInfo and therefore
making the error more obvious as subsequent parsing of the info failed.


# 8f5130edfa454cc7ee21e4c3aef2a151f7c3c567 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

package kit: Actually add support for B_HPKG_COMPRESSION_NONE

Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.


# e527b796319f21ca025f68e1964df140daa6de35 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.


# b3263ad3e1662cc419849a2d2cf539f9636613dc 30-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.


# 6a89a36aa0ce6a53f9758ba767863ac6b4c6156c 24-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.


# 84297881bbcc26aa41292a57130d61ac1247577c 23-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Remove BPackageKit::[...]::AbstractDataWriter and friends

* FDDataWriter and ZlibDataWriter weren't used anymore.
* AbstractDataWriter was implemented only by PackageFileHeapWriter,
which was only used by WriterImplBase.
* Add a PackageFileHeapWriter::AddDataThrows() which has semantics
equivalent to the previously inherited WriteDataThrows().


# 32832cbe47f991cc6d2b29824903181d8baaaa63 23-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Remove BPackageKit::BDataOutput

Use BDataIO instead.


# e2f30519ab5b526d1e5b603f847c4fc3d18af111 24-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: user big endian for chunk sizes


# 4ffb9234137d9a173489d3158cf4180d07bf0ac3 23-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapWriter::RemoveDataRanges(): Fix empty heap check


# 88adb4a8e7d20cd9b2a1b8471c3b0391323a9324 19-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapWriter: Add error message

... in case compression fails unexpectedly.


# b05413ed784e0a3822a2549dda19c799e232fa9e 18-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapWriter: Fix update case

* Pull _UnwriteLastPartialChunk() out of Reinit() for reuse.
* _UnwriteLastPartialChunk(): fPendingDataSize wasn't set.
* _PushChunks(): Some simplifications for clarity.
* ChunkBuffer/RemoveDataRanges(): Use data reading and decompression
methods provided by our base class instead of duplicating the
implementation.
* RemoveDataRanges():
- _FlushPendingData() before starting, so we don't ignore the pending
data and _UnwriteLastPartialChunk() when done, so a partial chunk
is read back into the pending data buffer.
- fUncompressedHeapSize wasn't reset before the main processing loop,
thus resulting in an erroneous size later on.


# 96af1d283c9e144a5c0596ab1200dbc5abfccf9e 18-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapWriter: Small style fix


# 796343ed73b1707ed97e91e22b72b8ca81bcdbad 18-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

Allow specifying the compression level for package creation

* Introduce BPackageWriterParameters which comprises all parameters
for package creation, currently flags and compression level. Such an
object can be passed to BPackageWriter::Init() and is passed on to
PackageWriterImpl and WriterImplBase.
* PackageFileHeapWriter: Add compressionLevel property and pass the
value on to ZlibCompressor.
* package add/create: Add options -0 ... -9 to specify the compression
level to be used.


# 1f633814fa347b0bd822f683af731615df600347 17-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.