kit/package: Error on short file read

* It's safe to assume that if the file is shorter than
the provided header, things will go poorly.
* Avoids a random vauge ReadBuffer error.
* This doesn't fix #15230, but makes the issue clearer.

Change-Id: I3471e6de384a0c9be94049ad891c01be980f7846
Reviewed-on: https://review.haiku-os.org/c/1679
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

ReaderImplBase::Init(): Make file size check optional

If the file doesn't support GetSize(), skip the header total size file
size check.

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.

package kit: ReaderImplBase: Remove superfluous scratch buffer

package kit: ReaderImplBase: Add public attrs section accessor

Separate notification from deletion in repository-handlers.

* Add NotifyDone() to all repository-attribute handlers and invoke that
to notify any listeners.
* Unify deletion to a single implementation of Delete() in the base
class. Before, the root handler for a repository didn't do that, but
just triggered the notification.

hpkg format: Generalize user/global settings file attributes

* For all identifiers: Rename global settings file to global writable
file. We want to use the respective attribute also for other writable
files, not only settings files.
* User settings file/global writable file info/attribute: Add
isDirectory property/child attribute. This allows declaring global/
user settings directories associated with the package.

hpkg format: Add attributes for declaring users and groups

ReaderImplBase: Verify that the attribute type matches

... the one expected for the respective attribute. Before it was
possible that e.g. a uint was read and then interpreted as a
const char*, if a string was expected for that attribute.

hpkg format: Add attributes for declaring settings files

Global and user settings files can be declared. For global ones an
update policy can be specified. If not specified, the settings file is
not included in the package, but created by the program (or user) later.
If an update type is specified, it defines what to do with the settings
file when updating the package to a newer version.

User settings files are never included in the package; they are always
created by the program or the user. If the package contains a template/
default settings file, it can be declared, but for informative purposes
only.

Package/repository file format: Add a minor version header field

* Add minor_version to hpkg_header and hpkg_repo_header and make
heap_compression uint16.
* If the minor version of a package/repository file is greater than the
current one unknown attributes are ignored without error. This allows
introducing new harmless attributes without making the resulting files
unreadable for older package kit versions.

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.

package: Suppress version mismatch errors where V1 is supported

* Add flags parameter to Init() of BPackageReader and friends.
* Introduce flag B_HPKG_READER_DONT_PRINT_VERSION_MISMATCH_MESSAGE and
don't print a version mismatch error when given.
* package extract/list: Use the new flag.

ReaderImplBase: Fix gcc 4 build

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

Add BLowLevelPackageContentHandler section hooks

* Add hooks HandleSectionStart() and HandleSectionEnd(). They are
invoked to bracket package file section, so the handler can
discriminate which section the attributes belong to.
HandleSectionStart() features a return parameter _handleSection, which
allows to handler to pick which sections it wants to handle.
* "package dump" does now print the section names.

BLowLevelPackageContentHandler interface extension

HandleAttributeDone(): Add parentToken parameter.

Add BLowLevelPackageContentHandler section hooks

* Add hooks HandleSectionStart() and HandleSectionEnd(). They are
invoked to bracket package file section, so the handler can
discriminate which section the attributes belong to.
HandleSectionStart() features a return parameter _handleSection, which
allows to handler to pick which sections it wants to handle.
* "package dump" does now print the section names.

BLowLevelPackageContentHandler interface extension

HandleAttributeDone(): Add parentToken parameter.

Pulled reusable stuff from PackageReaderImpl into new class ReaderImplBase.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40485 a95241bf-73f2-0310-859d-f6bbb57e9c96

ReaderImplBase::Init(): Make file size check optional

If the file doesn't support GetSize(), skip the header total size file
size check.

Switch package file accessor classes to use BPositionIO

* PackageFileHeap{Reader,Writer} as well as Package{Reader,Writer} and
their implementation and super classes do now internally use a
BPositionIO instead of a FD to access the package file. This provides
more flexibility needed for features to come.
* BPackageReader has already grown a new Init() version with a
BPositionIO* parameter.

Switch package kit to BZlibCompressionAlgorithm

... and remove the Zlib{Compressor,Decompressor} API.

package kit: ReaderImplBase: Remove superfluous scratch buffer

package kit: ReaderImplBase: Add public attrs section accessor

Separate notification from deletion in repository-handlers.

* Add NotifyDone() to all repository-attribute handlers and invoke that
to notify any listeners.
* Unify deletion to a single implementation of Delete() in the base
class. Before, the root handler for a repository didn't do that, but
just triggered the notification.

hpkg format: Generalize user/global settings file attributes

* For all identifiers: Rename global settings file to global writable
file. We want to use the respective attribute also for other writable
files, not only settings files.
* User settings file/global writable file info/attribute: Add
isDirectory property/child attribute. This allows declaring global/
user settings directories associated with the package.

hpkg format: Add attributes for declaring users and groups

ReaderImplBase: Verify that the attribute type matches

... the one expected for the respective attribute. Before it was
possible that e.g. a uint was read and then interpreted as a
const char*, if a string was expected for that attribute.

hpkg format: Add attributes for declaring settings files

Global and user settings files can be declared. For global ones an
update policy can be specified. If not specified, the settings file is
not included in the package, but created by the program (or user) later.
If an update type is specified, it defines what to do with the settings
file when updating the package to a newer version.

User settings files are never included in the package; they are always
created by the program or the user. If the package contains a template/
default settings file, it can be declared, but for informative purposes
only.

Package/repository file format: Add a minor version header field

* Add minor_version to hpkg_header and hpkg_repo_header and make
heap_compression uint16.
* If the minor version of a package/repository file is greater than the
current one unknown attributes are ignored without error. This allows
introducing new harmless attributes without making the resulting files
unreadable for older package kit versions.

packagefs: Add caching for the package file heap reader

* ReaderImplBase:
- Add virtual CreateCachedHeapReader() which can create a cached
reader based on the given heap reader.
- Rename HeapReader() to RawHeapReader() and add HeapReader() for the
cached heap reader.
- Add DetachHeapReader() to allow a clients to remove the heap
reader(s) after deleting the ReaderImplBase object.
* packagefs:
- Add CachedDataReader class, which wraps a given
BAbstractBufferedDataReader and provides caching for it using a
VMCache. The implementation is based on the IOCache implementation.
- Use CachedDataReader to wrap the heap reader. For file data that
means they are cached twice -- in the heap reader cache and in the
file cache -- but due to the heap reader using a VMCache as well,
the pages will be recycled automatically anyway. For attribute data
the cache should be very helpful, since they weren't cached at all
before.

package: Suppress version mismatch errors where V1 is supported

* Add flags parameter to Init() of BPackageReader and friends.
* Introduce flag B_HPKG_READER_DONT_PRINT_VERSION_MISMATCH_MESSAGE and
don't print a version mismatch error when given.
* package extract/list: Use the new flag.

ReaderImplBase: Fix gcc 4 build

hpkg format: compress the whole heap

Instead of handling compression for individual file/attribute data we
do now compress the whole heap where they are stored. This
significantly improves compression ratios. We still divide the
uncompressed data into 64 KiB chunks and use a chunk offset array for
the compressed chunks to allow for quick random access without too much
overhead. The tradeoff is a limited possible compression ratio -- i.e.
we won't be as good as tar.gz (though surprisingly with my test
archives we did better than zip).

The other package file sections (package attributes and TOC) are no
longer compressed individually. Their uncompressed data are simply
pushed onto the heap where the usual compression strategy applies. To
simplify things the repository format has been changed in the same
manner although it doesn't otherwise use the heap, since it only stores
meta data.

Due to the data compression having been exposed in public and private
API, this change touches a lot of package kit using code, including
packagefs and the boot loader packagefs support. The latter two haven't
been tested yet. Moreover packagefs needs a new kind of cache so we
avoid re-reading the same heap chunk for two different data items it
contains.

Add BLowLevelPackageContentHandler section hooks

* Add hooks HandleSectionStart() and HandleSectionEnd(). They are
invoked to bracket package file section, so the handler can
discriminate which section the attributes belong to.
HandleSectionStart() features a return parameter _handleSection, which
allows to handler to pick which sections it wants to handle.
* "package dump" does now print the section names.

BLowLevelPackageContentHandler interface extension

HandleAttributeDone(): Add parentToken parameter.

Add BLowLevelPackageContentHandler section hooks

* Add hooks HandleSectionStart() and HandleSectionEnd(). They are
invoked to bracket package file section, so the handler can
discriminate which section the attributes belong to.
HandleSectionStart() features a return parameter _handleSection, which
allows to handler to pick which sections it wants to handle.
* "package dump" does now print the section names.

BLowLevelPackageContentHandler interface extension

HandleAttributeDone(): Add parentToken parameter.

Pulled reusable stuff from PackageReaderImpl into new class ReaderImplBase.

git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@40485 a95241bf-73f2-0310-859d-f6bbb57e9c96