PackageReaderImpl: Delay reading sections until ParseContent()

package kit: Actually add support for B_HPKG_COMPRESSION_NONE

Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.

Remove BPackageKit::BDataOutput

Use BDataIO instead.

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.

ReaderImplBase: Verify that the attribute type matches

... the one expected for the respective attribute. Before it was
possible that e.g. a uint was read and then interpreted as a
const char*, if a string was expected for that attribute.

Package/repository file format: Add a minor version header field

* Add minor_version to hpkg_header and hpkg_repo_header and make
heap_compression uint16.
* If the minor version of a package/repository file is greater than the
current one unknown attributes are ignored without error. This allows
introducing new harmless attributes without making the resulting files
unreadable for older package kit versions.

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.

package: Suppress version mismatch errors where V1 is supported

* Add flags parameter to Init() of BPackageReader and friends.
* Introduce flag B_HPKG_READER_DONT_PRINT_VERSION_MISMATCH_MESSAGE and
don't print a version mismatch error when given.
* package extract/list: Use the new flag.

ReaderImplBase: Fix gcc 4 build

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

package kit: some fixes for multi-version support

* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.

Add BLowLevelPackageContentHandler section hooks

* Add hooks HandleSectionStart() and HandleSectionEnd(). They are
invoked to bracket package file section, so the handler can
discriminate which section the attributes belong to.
HandleSectionStart() features a return parameter _handleSection, which
allows to handler to pick which sections it wants to handle.
* "package dump" does now print the section names.

Add BLowLevelPackageContentHandler section hooks

* Add hooks HandleSectionStart() and HandleSectionEnd(). They are
invoked to bracket package file section, so the handler can
discriminate which section the attributes belong to.
HandleSectionStart() features a return parameter _handleSection, which
allows to handler to pick which sections it wants to handle.
* "package dump" does now print the section names.

* read package attributes before reading the TOC, as that allows to
take action in the content-handler that depend on the values of
package attributes

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40490 a95241bf-73f2-0310-859d-f6bbb57e9c96

Pulled reusable stuff from PackageReaderImpl into new class ReaderImplBase.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40485 a95241bf-73f2-0310-859d-f6bbb57e9c96

Cleanup following up on r40466:
* remove commented methods no longer being used


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40471 a95241bf-73f2-0310-859d-f6bbb57e9c96

Largish adjustments to PackagerReaderImpl and package attribute
handling:
* package attributes are now compatible with the low level attribute
handling of other HPKG attributes (such that 'package dump' now shows
package attributes, too)
* dropped type names from hpkg format, the attributes were identified
by IDs already and this simplifies the code considerably. Type names
are now handled in BLowLevelPackageHandler only.
* instead of rolling their own mechanism, high-level package attributes
handling is now implemented via a corresonding set of
AttributeHandler-subclasses
* adjusted package writer to only write package attributes that are
needed (empty ones are left out)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40466 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented repository writing:
* fleshed out RepositoryWriterImpl
* renamed BRepositoryHeader to BRepositoryInfo (in accordance with
BPackageInfo)
* adjusted BRepositoryInfo to be able to parse itself from a
driver_settings file
* added package_repo binary (only 'create' works as of yet)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40405 a95241bf-73f2-0310-859d-f6bbb57e9c96

* implemented checking of licenses, such that it is no longer possible
to create a package that refers to a license that is not a system
license or contained in the package
* added package-info flags (currently only approve_license and
system_package)
* adjusted package-reader/writer and pkgman accordingly

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40393 a95241bf-73f2-0310-859d-f6bbb57e9c96

Refactoring step towards implementation of RepositoryWriter:
* pulled commonly useful parts out of PackageWriterImpl into
WriterImplBase
* moved CachedStringTable and related methods into a separate class,
StringCache, in order to support having more than one string cache
per package file
* made package attribute section use a string cache, too, as that's
going to be very useful for repositories
* instead of writing package attributes directly, we now collect
corresponding PackageAttributes and write those later
* adjusted package reader accordingly


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40376 a95241bf-73f2-0310-859d-f6bbb57e9c96

* some rearrangements and renames with respect to public/private
types and defines of HPKG
* added header, magic and version for haiku repository format

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40370 a95241bf-73f2-0310-859d-f6bbb57e9c96

* cage the somewhat ugly duality between TOC and package attributes
section access into the SectionInfo structure, of which
PackageReaderImpl now keeps two instances and a pointer to the
current of these two


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40359 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented reading of package info attributes from hpkg file
* added kernel-compatible datatypes for reading package info attribute
values (PackageInfoAttributeValue.h) - these will be used at a later
stage by the package-fs to transport those attributes to userland
when asked to do so (by ioctl)
* implemented parsing of package info attributes in PackageReaderImpl
* added support for compressed package attribute section to
PackageReaderImpl
* completed the writing of package info attributes in PackageWriterImpl
and fixed a couple of bugs exposed by parsing
* adjusted 'package list' to show the package info attributes as they
are found


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40354 a95241bf-73f2-0310-859d-f6bbb57e9c96

Refactored hpkg implementation to provide some separation between
public and private API (still far from ideal, but a start):
* moved several HPKG-classes into the public namespace BPackageKit::HPKG
* added fImpl-wrappers around PackageReader and PackageWriter to hide
most of the gory details
* adjusted 'package'-binary and packagefs accordingly


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40320 a95241bf-73f2-0310-859d-f6bbb57e9c96

PackageReaderImpl: Delay reading sections until ParseContent()

package kit: Actually add support for B_HPKG_COMPRESSION_NONE

Until now we always declared in the HPKG header that the package file is
zlib compressed. For uncompressed files we would just store all
individual chunks uncompressed. Now we handle completely uncompressed
files slightly differently: We don't write the redundant chunk size
table anymore. The size savings are minor, but it makes the uncompressed
format read-streamable which may be handy.

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.

Move package kit Zlib* classes to support kit

Also move to B* namespace and no longer expose the zlib dependency in
the headers.

Remove BPackageKit::BDataOutput

Use BDataIO instead.

Revert "Move ZlibDecompressor to libshared"

This reverts commit 9af2105d36856be60528c361edf14a263f242338.

Conflicts:
src/kits/package/Jamfile

Move ZlibDecompressor to libshared

* This will be used to implement compressed http streams
* Remove the custom BDataOutput class, and use BDataIO instead, for
easier integration with existing code.

ReaderImplBase: Verify that the attribute type matches

... the one expected for the respective attribute. Before it was
possible that e.g. a uint was read and then interpreted as a
const char*, if a string was expected for that attribute.

Package/repository file format: Add a minor version header field

* Add minor_version to hpkg_header and hpkg_repo_header and make
heap_compression uint16.
* If the minor version of a package/repository file is greater than the
current one unknown attributes are ignored without error. This allows
introducing new harmless attributes without making the resulting files
unreadable for older package kit versions.

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.

package: Suppress version mismatch errors where V1 is supported

* Add flags parameter to Init() of BPackageReader and friends.
* Introduce flag B_HPKG_READER_DONT_PRINT_VERSION_MISMATCH_MESSAGE and
don't print a version mismatch error when given.
* package extract/list: Use the new flag.

ReaderImplBase: Fix gcc 4 build

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

package kit: some fixes for multi-version support

* Use enums/constants/functions instead of preprocessor macros.
* Missing include in PackageInfoAttributeValue.h.
* PackageReaderImpl::Init(): Check version before header size and
return B_MISMATCHED_VALUES instead of B_BAD_DATA, if the version
doesn't match. This allows callers to determine the condition and
try a reader for a different version. A more flexible interface for
that case would be nice, but since we want to support the old package
version only temporarily, the current solution should be good enough.

Add BLowLevelPackageContentHandler section hooks

* Add hooks HandleSectionStart() and HandleSectionEnd(). They are
invoked to bracket package file section, so the handler can
discriminate which section the attributes belong to.
HandleSectionStart() features a return parameter _handleSection, which
allows to handler to pick which sections it wants to handle.
* "package dump" does now print the section names.

Add BLowLevelPackageContentHandler section hooks

* Add hooks HandleSectionStart() and HandleSectionEnd(). They are
invoked to bracket package file section, so the handler can
discriminate which section the attributes belong to.
HandleSectionStart() features a return parameter _handleSection, which
allows to handler to pick which sections it wants to handle.
* "package dump" does now print the section names.

* read package attributes before reading the TOC, as that allows to
take action in the content-handler that depend on the values of
package attributes

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40490 a95241bf-73f2-0310-859d-f6bbb57e9c96

Pulled reusable stuff from PackageReaderImpl into new class ReaderImplBase.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40485 a95241bf-73f2-0310-859d-f6bbb57e9c96

Cleanup following up on r40466:
* remove commented methods no longer being used


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40471 a95241bf-73f2-0310-859d-f6bbb57e9c96

Largish adjustments to PackagerReaderImpl and package attribute
handling:
* package attributes are now compatible with the low level attribute
handling of other HPKG attributes (such that 'package dump' now shows
package attributes, too)
* dropped type names from hpkg format, the attributes were identified
by IDs already and this simplifies the code considerably. Type names
are now handled in BLowLevelPackageHandler only.
* instead of rolling their own mechanism, high-level package attributes
handling is now implemented via a corresonding set of
AttributeHandler-subclasses
* adjusted package writer to only write package attributes that are
needed (empty ones are left out)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40466 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented repository writing:
* fleshed out RepositoryWriterImpl
* renamed BRepositoryHeader to BRepositoryInfo (in accordance with
BPackageInfo)
* adjusted BRepositoryInfo to be able to parse itself from a
driver_settings file
* added package_repo binary (only 'create' works as of yet)


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40405 a95241bf-73f2-0310-859d-f6bbb57e9c96

* implemented checking of licenses, such that it is no longer possible
to create a package that refers to a license that is not a system
license or contained in the package
* added package-info flags (currently only approve_license and
system_package)
* adjusted package-reader/writer and pkgman accordingly

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40393 a95241bf-73f2-0310-859d-f6bbb57e9c96

Refactoring step towards implementation of RepositoryWriter:
* pulled commonly useful parts out of PackageWriterImpl into
WriterImplBase
* moved CachedStringTable and related methods into a separate class,
StringCache, in order to support having more than one string cache
per package file
* made package attribute section use a string cache, too, as that's
going to be very useful for repositories
* instead of writing package attributes directly, we now collect
corresponding PackageAttributes and write those later
* adjusted package reader accordingly


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40376 a95241bf-73f2-0310-859d-f6bbb57e9c96

* some rearrangements and renames with respect to public/private
types and defines of HPKG
* added header, magic and version for haiku repository format

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40370 a95241bf-73f2-0310-859d-f6bbb57e9c96

* cage the somewhat ugly duality between TOC and package attributes
section access into the SectionInfo structure, of which
PackageReaderImpl now keeps two instances and a pointer to the
current of these two


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40359 a95241bf-73f2-0310-859d-f6bbb57e9c96

Implemented reading of package info attributes from hpkg file
* added kernel-compatible datatypes for reading package info attribute
values (PackageInfoAttributeValue.h) - these will be used at a later
stage by the package-fs to transport those attributes to userland
when asked to do so (by ioctl)
* implemented parsing of package info attributes in PackageReaderImpl
* added support for compressed package attribute section to
PackageReaderImpl
* completed the writing of package info attributes in PackageWriterImpl
and fixed a couple of bugs exposed by parsing
* adjusted 'package list' to show the package info attributes as they
are found


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40354 a95241bf-73f2-0310-859d-f6bbb57e9c96

Refactored hpkg implementation to provide some separation between
public and private API (still far from ideal, but a start):
* moved several HPKG-classes into the public namespace BPackageKit::HPKG
* added fImpl-wrappers around PackageReader and PackageWriter to hide
most of the gory details
* adjusted 'package'-binary and packagefs accordingly


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40320 a95241bf-73f2-0310-859d-f6bbb57e9c96