Break translators out of haiku.hpkg into a separate haiku_datatranslators.hpkg.

Translators and media-plugins are the main source of dependencies in haiku.hpkg,
and thus the main source of packages being pulled into chroots, especially
HaikuPorter chroots. (FFmpeg pulls in a rather large array of sub-
dependencies, itself.) So, here we break all the translators into their
own sub-package.

For now, haiku.hpkg is declared to depend on haiku_datatranslators,
so that users will not suddenly update and have no translators.
In the future, this will be dropped.

Note that this is only done for the primary arch at present.
Secondary architecture translators remain in the main secondary package
for now.

Change-Id: Id0b352f34f7110b79ec7787792bf3ae0edab4054
Reviewed-on: https://review.haiku-os.org/c/haiku/+/4477
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

Overhaul "haiku" PackageInfo.

* Merge all except the x86_gcc2 one into a single "generic" one.
It is now exactly the same on all architectures, save a single
ifdef for x86_64, which needs a different compat version, as there
are still some packages built that far back and never rebuilt since!

* Group the requires according to category: first any arch packages,
then commands, then the GCC syslibs, then packages always depended
upon if they are available, and finally packages only depended upon
in "regular" builds.

* Put all packages inside HAIKU_BUILD_FEATURE_* ifdef tests,
and order them alphabetically within their category.

If I have done this all correctly, at least x86_64 should get an
identically generated haiku .PackageInfo as before this commit, save
the reordering changes. Others may differ based on package availability,
as they were missing the requisite #ifdef sections.

build/jam: Inline HOST_ADD_BUILD_COMPATIBILITY_LIB_DIR where possible.

The "exec" tool can only handle one command with environs set at
the beginning of the line, so now we set the ADD_BUILD_COMPAT...
in this format. This also seems to be a general performance
improvement to builds using real shells, too.

Change-Id: If4b3117651b5475039d5e8116cd3de398582290a

build: Include extra information in the package version.

Previously only the "effective revision", i.e. the hrev by itself,
was used. Now we include the "commits ahead" count and the "dirty" flag,
if they're there, using _ instead of - of course.

Change-Id: I5238d01926c2cca242bffbdef30bff606a173a06
Reviewed-on: https://review.haiku-os.org/596
Reviewed-by: waddlesplash <waddlesplash@gmail.com>
(cherry picked from commit e7c5d994267fb7d49e3dd50c2854e9b37c9ffca1)
Reviewed-on: https://review.haiku-os.org/597

x86_64/x86_gcc2: don't depend on libgcc_s, libstdc++, libsupc++.

PackageRules: add secondaryArch to the packageInfo grist.

this fixes the generation of multiple secondary architecture package infos.

PackageRules: use valid macro names in defines.

haiku-x86_64: make some dependencies feature dependent.

Fix #11458, pt.1: invoking jam from source dir does not work.

* Apply change suggested by Ingo: replace LocalDepends with Depends.

Create packages_build folders per build type.

* The content of the preprocessed package-info files and the package
contents depend on the build type, so we use a different folder for
each build type.

Whitespace cleanup in root and 'build' folder.

* Removed trailing whitespace and empty lines at top of files.

Replace HAIKU_{BOOTSTRAP,MINIMUM}_BUILD with HAIKU_BUILD_TYPE.

* Instead of separate variables, HAIKU_BUILD_TYPE is set to one of
'bootstrap', 'minimum' or 'regular'.
* Adjust uses of HAIKU_BOOTSTRAP_BUILD accordingly.

Move creating the Haiku repo info to jam

* PreprocessPackageInfo rule: Pull out new rule
PreprocessPackageOrRepositoryInfo which does the sed substituation and
optionally the filtering through the C preprocessor.
* HaikuRepository rule: Generate the repository info file (from the
given template). No longer do that in the build_haiku_repository
script.

Add DetermineEffectiveHaikuRevision rule

Simplifies PreprocessPackageInfo a bit.

Add support for explicitly preventing rebuilding packages

Due to depending on dynamically built files, package files will always
be rebuilt when they are needed (e.g. when an image is built). The build
variable HAIKU_DONT_REBUILD_PACKAGES can be defined to prevent
rebuilding existing package files (even ones that are out-of-date).

Main target for this change is buildbot. We want to make sure that the
packages in the repository it builds are exactly identical to the ones
in the images, which may not be the case when the packages are rebuilt
(due to different timestamps of contained files). The respective build
order should be:

1. Build repository.
2. Without cleaning the generated directory, build different image types
with HAIKU_DONT_REBUILD_PACKAGES defined.

Update requires entries for the packages we build

* Add build tool update_package_requires. Given a package info file and
a repository cache file, it updates the minimum versions of the
requires entries of the package info file according to what is
provided by the repository.
* PreprocessPackageInfo rule: Use update_package_requires (with the
HaikuPorts repository file).

Encode the hrev revision into the package versions

Don't code the Haiku version into all the package infos

PreprocessPackageInfo: add substitution

... HAIKU_SECONDARY_PACKAGING_ARCH_SUFFIX. It's "_<arch>" for a
secondary architecture and "" for the primary one.

Filter package infos through sed

%HAIKU_PACKAGING_ARCH% and %HAIKU_SECONDARY_PACKAGING_ARCH% in the
package infos are replaced before filtering them through the C
preprocessor.

PreprocessPackageInfo: define macro for secondary architecture

More work towards hybrid support

* All packaging architecture dependent variables do now have a
respective suffix and are set up for each configured packaging
architecture, save for the kernel and boot loader variables, which
are still only set up for the primary architecture.
For convenience TARGET_PACKAGING_ARCH, TARGET_ARCH, TARGET_LIBSUPC++,
and TARGET_LIBSTDC++ are set to the respective values for the primary
packaging architecture by default.
* Introduce a set of MultiArch* rules to help with building targets for
multiple packaging architectures. Generally the respective targets are
(additionally) gristed with the packaging architecture. For libraries
the additional grist is usually omitted for the primary architecture
(e.g. libroot.so and <x86>libroot.so for x86_gcc2/x86 hybrid), so that
Jamfiles for targets built only for the primary architecture don't
need to be changed.
* Add multi-arch build support for all targets needed for the stage 1
cross devel package as well as for libbe (untested).

Move package info pre-processing to jam build system

Also, pre-process all package infos (not only the generic ones) and
define the macro HAIKU_BOOTSTRAP_BUILD, if building a package for a
bootstrap image.

Remove the old "CVS" package rules

... and all their invocations.

BuildHaikuPackage rule: Locate extract script

Missed that when adding the script. Therefore it would be created in the
current directory and when building multiple packages concurrently the
script would be overwritten.

FHaikuPackageGrist: Avoid potential package name clashes

Adjust handling of package infos.

* add HAIKU_PACKAGING_ARCH, which is set to the target packaging
architecture
* introduce support for generic package infos, which are package infos
that are the same for all architectures, except for the declaration
of the package architecture itself
* move package info files underneath architecture-specific or generic
folder

Support extracting archives to the packages we build

* BuildHaikuPackage rule: Create the script that contains the extraction
commands.
* build_haiku_package: Add extractFile() function (stripped down version
from build_haiku_image).

In build_haiku_image the functionality was mainly used to extract the
optional packages, which is no longer done. We still need it e.g. for
the Wifi firmware packages that want to be extracted.

BuildHaikuPackage rule: Add grist to package info file

... to avoids clashes with equally named targets.

Add HAIKU_PACKAGE_COMPRESSION_LEVEL build variable

It allows to control the compression level used for package creation
and update. The default (9) is *very* slow, so developers may want to
use a smaller level during the regular development process to keep
turn-around times low.

Complete MIME related support for image/package creation

* Update MimeSet rule to use the MIME DB the build system creates.
* Add CreateAppMimeDBEntries rule and call it from Link for targets that
might be applications that need to be registered with the MIME DB. For
the target the rule is invoked with it creates a directory into which
the entries for the types to be registered are written. The directory
is associated with the target via the HAIKU_MIME_DB_ENTRIES variable.
* AddFilesToContainer: If a target is added that has MIME DB entries,
also add those to the container.
* build_haiku_package: Call mimeset for the package contents.

Replace various boolean rule parameters by flags

Instead of the boolean alwaysUpdate, isCDPackage, and stripDebugSymbols
parameters use a uniform flags parameter which can contain any of the
respectively supported flags (alwaysUpdate, cdPackage,
stripDebugSymbols, !stripDebugSymbols).

Add build profile action update-packages

Equivalent to "jam <list of all hpkgs> && jam @... <list of all hpkgs>",
i.e. it makes sure all hpkgs that are supposed to be in the image are
rebuilt respectively downloaded and copied to the image. It doesn't
remove old packages nor the activation files -- that still has to be
done manually.

ExtractArchiveToContainer: Add update support

Add an alwaysUpdate parameter like the ExtractArchiveTo{Image,Package}
rules have and do the handling in the base rule.

Support for package "update" build profile action

* Add global HAIKU_PACKAGES_UPDATE_ONLY build system variable, which is
set for the "update" and "update-all" build profile actions.
* Change the HAIKU_INCLUDE_IN_CONTAINER_VAR variable on packages to
HAIKU_INCLUDE_IN_PACKAGES for all packages. The variable is also
set for the "update" and "update-all" build profile actions.
* Introduce HAIKU_CONTAINER_INHERIT_UPDATE_VARIABLE on container
variable. If set, the contained variable will be set on the container
when one or more files in it are updated. It is set on packages so an
update of a file in a package causes the package to be updated in the
image.
* Introduce HAIKU_CONTAINER_ALWAYS_CREATE_DIRECTORIES on container
variable. If set, directories will be created also in update mode. It
is set on packages.

Fix image rules wrt system/ relative entries

* Introduce on-container jam variable HAIKU_CONTAINER_SYSTEM_DIR_TOKENS
containing the directory tokens relative to the container root to
refer to system.
* Use the variable in *ToContainer rules that need to put something in
system.
* AddFilesToContainer: Fix overlooked reference to AddFilesToHaikuImage.
* AddBootModuleSymlinksToContainer: Use relative symlinks. This avoids
special-casing in the boot loader for packagefs.

Package rule: introduce current package notion

The HaikuPackage rule now sets the variable
HAIKU_CURRENTLY_BUILT_HAIKU_PACKAGE to the given package and all
*ToPackage rules use that instead of a parameter. This saves passing the
package in each of those rule invocations.

AddDirectoryTo{HaikuImage,...} refactoring

Move common code from AddDirectoryToHaikuImage/AddDirectoryToPackage to
AddDirectoryToContainer. AddDirectoryToPackage was incorrect, using an
incorrect search directory for the attribute files.

Build a system Haiku package

* Create rules (build/jam/PackageRules) and a script,
build_haiku_package, to build hpkg files.
* Move all rule invocations that copy files and created symlinks and
directories in the "system" directory from HaikuImage to
HaikuPackages, which creates a package "haiku.hpkg".
* build_haiku_image: Comment out adding the copyrights info to
AboutSystem for the moment.

The Copy rule respects an already set SEARCH path now.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@16262 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged changes from branch build_system_redesign at revision 14573.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@14574 a95241bf-73f2-0310-859d-f6bbb57e9c96

Fix #11458, pt.1: invoking jam from source dir does not work.

* Apply change suggested by Ingo: replace LocalDepends with Depends.

Create packages_build folders per build type.

* The content of the preprocessed package-info files and the package
contents depend on the build type, so we use a different folder for
each build type.

Whitespace cleanup in root and 'build' folder.

* Removed trailing whitespace and empty lines at top of files.

Replace HAIKU_{BOOTSTRAP,MINIMUM}_BUILD with HAIKU_BUILD_TYPE.

* Instead of separate variables, HAIKU_BUILD_TYPE is set to one of
'bootstrap', 'minimum' or 'regular'.
* Adjust uses of HAIKU_BOOTSTRAP_BUILD accordingly.

Move creating the Haiku repo info to jam

* PreprocessPackageInfo rule: Pull out new rule
PreprocessPackageOrRepositoryInfo which does the sed substituation and
optionally the filtering through the C preprocessor.
* HaikuRepository rule: Generate the repository info file (from the
given template). No longer do that in the build_haiku_repository
script.

Add DetermineEffectiveHaikuRevision rule

Simplifies PreprocessPackageInfo a bit.

Add support for explicitly preventing rebuilding packages

Due to depending on dynamically built files, package files will always
be rebuilt when they are needed (e.g. when an image is built). The build
variable HAIKU_DONT_REBUILD_PACKAGES can be defined to prevent
rebuilding existing package files (even ones that are out-of-date).

Main target for this change is buildbot. We want to make sure that the
packages in the repository it builds are exactly identical to the ones
in the images, which may not be the case when the packages are rebuilt
(due to different timestamps of contained files). The respective build
order should be:

1. Build repository.
2. Without cleaning the generated directory, build different image types
with HAIKU_DONT_REBUILD_PACKAGES defined.

Update requires entries for the packages we build

* Add build tool update_package_requires. Given a package info file and
a repository cache file, it updates the minimum versions of the
requires entries of the package info file according to what is
provided by the repository.
* PreprocessPackageInfo rule: Use update_package_requires (with the
HaikuPorts repository file).

Encode the hrev revision into the package versions

Don't code the Haiku version into all the package infos

PreprocessPackageInfo: add substitution

... HAIKU_SECONDARY_PACKAGING_ARCH_SUFFIX. It's "_<arch>" for a
secondary architecture and "" for the primary one.

Filter package infos through sed

%HAIKU_PACKAGING_ARCH% and %HAIKU_SECONDARY_PACKAGING_ARCH% in the
package infos are replaced before filtering them through the C
preprocessor.

PreprocessPackageInfo: define macro for secondary architecture

More work towards hybrid support

* All packaging architecture dependent variables do now have a
respective suffix and are set up for each configured packaging
architecture, save for the kernel and boot loader variables, which
are still only set up for the primary architecture.
For convenience TARGET_PACKAGING_ARCH, TARGET_ARCH, TARGET_LIBSUPC++,
and TARGET_LIBSTDC++ are set to the respective values for the primary
packaging architecture by default.
* Introduce a set of MultiArch* rules to help with building targets for
multiple packaging architectures. Generally the respective targets are
(additionally) gristed with the packaging architecture. For libraries
the additional grist is usually omitted for the primary architecture
(e.g. libroot.so and <x86>libroot.so for x86_gcc2/x86 hybrid), so that
Jamfiles for targets built only for the primary architecture don't
need to be changed.
* Add multi-arch build support for all targets needed for the stage 1
cross devel package as well as for libbe (untested).

Move package info pre-processing to jam build system

Also, pre-process all package infos (not only the generic ones) and
define the macro HAIKU_BOOTSTRAP_BUILD, if building a package for a
bootstrap image.

Remove the old "CVS" package rules

... and all their invocations.

BuildHaikuPackage rule: Locate extract script

Missed that when adding the script. Therefore it would be created in the
current directory and when building multiple packages concurrently the
script would be overwritten.

FHaikuPackageGrist: Avoid potential package name clashes

Adjust handling of package infos.

* add HAIKU_PACKAGING_ARCH, which is set to the target packaging
architecture
* introduce support for generic package infos, which are package infos
that are the same for all architectures, except for the declaration
of the package architecture itself
* move package info files underneath architecture-specific or generic
folder

Support extracting archives to the packages we build

* BuildHaikuPackage rule: Create the script that contains the extraction
commands.
* build_haiku_package: Add extractFile() function (stripped down version
from build_haiku_image).

In build_haiku_image the functionality was mainly used to extract the
optional packages, which is no longer done. We still need it e.g. for
the Wifi firmware packages that want to be extracted.

BuildHaikuPackage rule: Add grist to package info file

... to avoids clashes with equally named targets.

Add HAIKU_PACKAGE_COMPRESSION_LEVEL build variable

It allows to control the compression level used for package creation
and update. The default (9) is *very* slow, so developers may want to
use a smaller level during the regular development process to keep
turn-around times low.

Complete MIME related support for image/package creation

* Update MimeSet rule to use the MIME DB the build system creates.
* Add CreateAppMimeDBEntries rule and call it from Link for targets that
might be applications that need to be registered with the MIME DB. For
the target the rule is invoked with it creates a directory into which
the entries for the types to be registered are written. The directory
is associated with the target via the HAIKU_MIME_DB_ENTRIES variable.
* AddFilesToContainer: If a target is added that has MIME DB entries,
also add those to the container.
* build_haiku_package: Call mimeset for the package contents.

Replace various boolean rule parameters by flags

Instead of the boolean alwaysUpdate, isCDPackage, and stripDebugSymbols
parameters use a uniform flags parameter which can contain any of the
respectively supported flags (alwaysUpdate, cdPackage,
stripDebugSymbols, !stripDebugSymbols).

Add build profile action update-packages

Equivalent to "jam <list of all hpkgs> && jam @... <list of all hpkgs>",
i.e. it makes sure all hpkgs that are supposed to be in the image are
rebuilt respectively downloaded and copied to the image. It doesn't
remove old packages nor the activation files -- that still has to be
done manually.

ExtractArchiveToContainer: Add update support

Add an alwaysUpdate parameter like the ExtractArchiveTo{Image,Package}
rules have and do the handling in the base rule.

Support for package "update" build profile action

* Add global HAIKU_PACKAGES_UPDATE_ONLY build system variable, which is
set for the "update" and "update-all" build profile actions.
* Change the HAIKU_INCLUDE_IN_CONTAINER_VAR variable on packages to
HAIKU_INCLUDE_IN_PACKAGES for all packages. The variable is also
set for the "update" and "update-all" build profile actions.
* Introduce HAIKU_CONTAINER_INHERIT_UPDATE_VARIABLE on container
variable. If set, the contained variable will be set on the container
when one or more files in it are updated. It is set on packages so an
update of a file in a package causes the package to be updated in the
image.
* Introduce HAIKU_CONTAINER_ALWAYS_CREATE_DIRECTORIES on container
variable. If set, directories will be created also in update mode. It
is set on packages.

Fix image rules wrt system/ relative entries

* Introduce on-container jam variable HAIKU_CONTAINER_SYSTEM_DIR_TOKENS
containing the directory tokens relative to the container root to
refer to system.
* Use the variable in *ToContainer rules that need to put something in
system.
* AddFilesToContainer: Fix overlooked reference to AddFilesToHaikuImage.
* AddBootModuleSymlinksToContainer: Use relative symlinks. This avoids
special-casing in the boot loader for packagefs.

Package rule: introduce current package notion

The HaikuPackage rule now sets the variable
HAIKU_CURRENTLY_BUILT_HAIKU_PACKAGE to the given package and all
*ToPackage rules use that instead of a parameter. This saves passing the
package in each of those rule invocations.

AddDirectoryTo{HaikuImage,...} refactoring

Move common code from AddDirectoryToHaikuImage/AddDirectoryToPackage to
AddDirectoryToContainer. AddDirectoryToPackage was incorrect, using an
incorrect search directory for the attribute files.

Build a system Haiku package

* Create rules (build/jam/PackageRules) and a script,
build_haiku_package, to build hpkg files.
* Move all rule invocations that copy files and created symlinks and
directories in the "system" directory from HaikuImage to
HaikuPackages, which creates a package "haiku.hpkg".
* build_haiku_image: Comment out adding the copyrights info to
AboutSystem for the moment.

The Copy rule respects an already set SEARCH path now.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@16262 a95241bf-73f2-0310-859d-f6bbb57e9c96

Merged changes from branch build_system_redesign at revision 14573.


git-svn-id: file:///srv/svn/repos/haiku/haiku/trunk@14574 a95241bf-73f2-0310-859d-f6bbb57e9c96