History log of /haiku/src/kits/package/hpkg/PackageFileHeapAccessorBase.cpp
Revision Date Author Comments
# 0ecd7516 28-Feb-2023 Augustin Cavalier <waddlesplash@gmail.com>

Package Kit: Use an object_cache in kernel mode for decompression buffers.

The kernel heap only uses object caches for objects up to size 8192.
Larger allocations have to go through the raw allocator. That can
get pretty expensive.

Adding instrumentation around the malloc/free calls in this function
showed that on my machine, some 596ms during boot were spent on
*malloc/free alone*, all else aside. After this change, we are at
around 110ms, or a >5x improvement. Running an fgrep -R on /system/
after boot increased the cumulative time in memory functions to over
5 seconds, while after this change it is "only" 1170ms.

Honestly, it seems like the object depots should be able to be faster
than that, even if this function is called thousands of times. But that
is a problem for a different investigation.

It would be even faster for every consumer of this data in
packagefs just allocated one set of buffers up front, or at least
for a single "read session", but plumbing that all the way
through the myriad abstractions of the Package Kit will
not be easy, and is left for another time, as well.


# 8f5130ed 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

package kit: Actually add support for B_HPKG_COMPRESSION_NONE

Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.


# e527b796 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.


# cdfeba5a 08-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

BPackageWriter: Add Recompress()

Allows rewriting an existing package file with a different compression.


# b3263ad3 30-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.


# 6a89a36a 24-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.


# 32832cbe 23-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Remove BPackageKit::BDataOutput

Use BDataIO instead.


# d2d1af83 18-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile


# 4c235c74 18-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Revert "Fix more issues detected by gcc4 compiler:"

This reverts commit 19f3bae0716c4eb7dd2a5716778560580be68071.


# 19f3bae0 11-Feb-2014 Adrien Destugues <pulkomandy@pulkomandy.tk>

Fix more issues detected by gcc4 compiler:

More ssize_t/status_t mixup.


# 9af2105d 10-Feb-2014 Adrien Destugues <pulkomandy@pulkomandy.tk>

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.


# e2f30519 24-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: user big endian for chunk sizes


# c3bd329f 20-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapAccessorBase: Fix error output


# 520a7a76 18-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapAccessorBase: small refactoring

* Pull method DecompressChunkData() out of ReadAndDecompressChunkData()
for potential reuse.
* Also fix/improve some error output strings.


# 1f633814 17-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.


# 8f5130edfa454cc7ee21e4c3aef2a151f7c3c567 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

package kit: Actually add support for B_HPKG_COMPRESSION_NONE

Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.


# e527b796319f21ca025f68e1964df140daa6de35 12-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.


# cdfeba5a1e49d0993e7ab6853c77392bbc10d3c4 08-Jul-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

BPackageWriter: Add Recompress()

Allows rewriting an existing package file with a different compression.


# b3263ad3e1662cc419849a2d2cf539f9636613dc 30-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.


# 6a89a36aa0ce6a53f9758ba767863ac6b4c6156c 24-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.


# 32832cbe47f991cc6d2b29824903181d8baaaa63 23-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Remove BPackageKit::BDataOutput

Use BDataIO instead.


# d2d1af830b3f144937ea5e4c3f0e5b74c4c3d737 18-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile


# 4c235c7497ee3bc08a0bb53ae6776fe14c1f4462 18-Jun-2014 Ingo Weinhold <ingo_weinhold@gmx.de>

Revert "Fix more issues detected by gcc4 compiler:"

This reverts commit 19f3bae0716c4eb7dd2a5716778560580be68071.


# 19f3bae0716c4eb7dd2a5716778560580be68071 11-Feb-2014 Adrien Destugues <pulkomandy@pulkomandy.tk>

Fix more issues detected by gcc4 compiler:

More ssize_t/status_t mixup.


# 9af2105d36856be60528c361edf14a263f242338 10-Feb-2014 Adrien Destugues <pulkomandy@pulkomandy.tk>

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.


# e2f30519ab5b526d1e5b603f847c4fc3d18af111 24-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: user big endian for chunk sizes


# c3bd329fa27d34588a31ec8c61f2a78e41b9c968 20-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapAccessorBase: Fix error output


# 520a7a76c7c6528f654a4e9a6d0d291df63e7532 18-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

PackageFileHeapAccessorBase: small refactoring

* Pull method DecompressChunkData() out of ReadAndDecompressChunkData()
for potential reuse.
* Also fix/improve some error output strings.


# 1f633814fa347b0bd822f683af731615df600347 17-May-2013 Ingo Weinhold <ingo_weinhold@gmx.de>

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.