Move MIPS N64 compat out of {base,debug}32 into {base,debug}64

martin requested separation by ABI and it doesn't make much sense
to have 64-bit binaries in a set called 32.

Add new sets: base32, debug32, manhtml

- base32 contains (when MKCOMPAT=yes) shared libraries for 32-bit
compatibility, previously included in base

- debug32 contains (when MKCOMPAT=yes) debug symbols and static libraries
containing debug symbols for 32-bit compatiblity, previously included
in debug

- manhtml contains (when MKHTML=yes) the HTML files previously included
in 'man', which are of limited utility without third-party software.

The motivation for this change is to be able to easily exclude sets
from CD-ROM images that go over the size limit without xz compression
(which many NetBSD platforms struggle to extract at acceptable speeds).

Fix previous, use sed to fix up the set.${set} line in non-METAFILE mode

When building in priviledged mode, remove the entry for the output file
from etc/mtree/set.* - it may exist and contain stale data.

More conservative umask

Set the file creation mask consistently because the files these scripts
create can become part of an iso image.

sets: enhance MAKEVERBOSE support

Add more messages for various sets operations.
Add ${_MKMSG_CREATE} to bsd.own.mk.

Enhance maketars to be a bit more verbose even with -q,
to make it more obvious why certain operations may be slow.
May help debugging PR 47562.

maketars: add comments for skipped mtree keywords

Reorder skipkeys alphabetically and add comment explaining
the purpose and why sha256 is not part of the variable.

Deprecate extsrc/, EXTSRCSRCDIR, MKEXTSRC

Remove support for building extsrc/:
1. Makefile: remove do-extsrc target.
2. build.sh: remove options -y and -Y extsrcdir.
3. distrib/sets: remove support for extsrc in various tools
including the options -L ext and -y, and the extsrc sets.
4. doc/BUILDING.mdoc: remove docs for extsrc/, EXTSRCSRCDIR, MKEXTSRC (etc)
5. bsd.own.mk and various Makefiles: remove support for extsrc/,
EXTSRCSRCDIR, MKEXTSRC.

As proposed on tech-kern and tech-userlevel on 2022-01-07
and followed up on 2022-08-21.

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

Patch the build script and toolchain to allow passing through
HOST_CPPFLAGS, HOST_CXXFLAGS, but not LIBRARY_PATH because it breaks
the build. These variables are necessary to support building NetBSD
from a GNU Guix or NixOS host, where /usr/include, /lib, and all
but /bin/sh do not exist. In many cases, support for HOST_CPPFLAGS
was incomplete. From Ryan Sundberg

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

branches: 1.89.2;
Prepare for .tar.xz sets

Revert previous - will pass the additional args via the environment.

Allow for ${COMPRESS_PROGRAM} to hold the name of a compressor program
plus additional arguments (like -9).

branches: 1.86.10; 1.86.12;
This fixes a bug where mtree would use names & IDs from the host system
instead of from the build.

branches: 1.85.2; 1.85.4;
Obey MKREPRO_TIMESTAMP when creating sets (create all the files in the
sets with that date).

Simplify slightly

Instead of guessing wether to preserve permissions on install operations
from the METALOG variable, default them to yes and add a new argument -U to
switch preservation off.

Do not try to preserve permissions when copying to the -i installdir
in the METALOG case (it makes no sense at all).

Add a new arg to override the value of the "setlistdir" variable (that is:
where the mtree spec files are created)

We always have a valid mtree file, so always specify -M

merge d flag.

- use dest instead of DESTDIR
- factor out pax usage, and make it use only the first word from the sets

Create full sets using mtree from the flists (not just filenames).

- add xdebug set
- document debug better
- fix previous debug fallout.

Add a new "debug" set that gets built when ${MKDEBUG} is set. On evbarm:
-rw-r--r-- 1 root src 35806068 Jan 16 10:47 base.tgz
-rw-r--r-- 1 root src 57374069 Jan 16 10:48 comp.tgz
-rw-r--r-- 1 root src 101286193 Jan 16 10:49 debug.tgz
...
This allows always building the debugging code and libraries but not
installing it (although now it is placed in the default sets if ${MKDEBUG}
is set

branches: 1.74.6; 1.74.12;
add new option to build/use tools/pigz for compressing sets. for now
USE_PIGZGZIP defaults to "no". (it depends upon local <pthread.h> and -lz.)

you can set it to "yes" on most modern platforms just fine.

sh(1) style. No functional changes.

Fix syntax error on OSX 10.5.
While here, simplify handling with OPTARG using IFS.

developped with and 'go for it' dsl@

Exit with a sensible error message if makeflist output is empty.
This should be better than an incomprehensible error message from
mtree.

Support "extsrc", externally added programs and libraries. Users can write
their own reach-overs, cross-build, install, and get set files just like base
and X11 / X.org. (These sets are not included as TNF releases.)

If there are errors in creating a ${set}.tgz tar file, then don't
just press on regardless.

In various places that manipulate mtree specs, use "mtree -CS" to sort.
Remove some now-redundant standalone sort commands.

Consistently use gzip with the -n flag so dates are not embedded.

Not tested on all platforms.

Explicitly sort entries on preparing set files from METALOG.
METALOG could have different order due to install(1) race
on parallel builds, and mtree(8) doesn't sort files.
Should fix inconsistent shared sets among builds as seen in
/pub/NetBSD-daily/netbsd-5/200904010000Z/shared/ and
/pub/NetBSD-daily/netbsd-5/200904010002Z/shared/ dirs.
Okay'ed by snj@.

Clean up set files to make them consistent using mtree. Patch from lukem.

branches: 1.64.6; 1.64.8;
Add '-q' to maketars and makesrctars, to quieten them.
Improve MAKEVERBOSE support.

Cleanup .tgz.tmp files on exit.

branches: 1.62.6; 1.62.8;
Add the tests.tgz set

This adds a new tests.tgz set to releases which includes all the tests
for the system. It is important to note that this set does not rely on
comp.tgz: a user of the system can run the tests without having the
development tools installed, which can be useful in a production machine.

branches: 1.61.10;
No space in "function()", "command;" or "$(command)".
Requested by christos.

When printing messages to stderr, always use
echo >&2 "${prog}: ..."

Approved by christos

Remove the code that used "${MTREE} -C ..." to sanitise the plist. We
can now assume that it's already been sanitised before maketars was
invoked.

Reviewed by agc

Ensure that (almost) all shell variables in distrib/sets/* scripts are
used with curly braces and quotes, as in "${var}".

Also ensure that command substitution is quoted, as in "$(command)", and
convert `command` to $(command).

Reviewed by agc

Add shell variables defined in sets.subr for almost all commands used
by scripts in distrib/sets/*. This is intended to be useful when cross
building. Only trivial commands like echo, cat, and rm are excluded
from this treatment.

While I am here, make ${MTREE} and ${DB} follow the pattern used by most
other such variables.

Reviewed by agc.

Move default values for several shell variables into sets.subr.

Reviewed by agc

Set default vars so blank and unset both end up w. a reasonable default

Make all the sets scripts work even when not run from the directory that
they reside in.
Change any uses of dirname to use shell variable substitution instead.

Write to a temporary file and mv to real name to avoid getting a
truncated file if/when interrupted.

Clearly the error message was supposed to print $metalog, not
$METALOG.

When not using a metalog, only copy the generated flist.<set> over
/etc/mtree/set.<set> if they are different. This should allow
installsets to operate from a pre-populated read-only DESTDIR.
Based on fix from Juergen Hannken-Illjes.

Create ${DESTDIR}/etc/mtree/set.<setname> setfiles for each given <setname>.
Add `-S' to exit after creating the setfiles.
Send errors to stderr instead of stdout.

Also remove "xetc" from the default list of sets copied with -i installdir

rename all the <bsd.own.mk> derived variables to uppercase, and export
them to the environment.

Significant overhaul of how the sets are used:

* Add an optional third field to the sets file which is a list of
comma separated keywords that control if the line is printed.
Currently supported keywords
kerberos4 ${MKKERBEROS4} != no
kerberos ${MKKERBEROS} != no
lint ${MKLINT} != no
obsolete ${obsolete} != 0.
In this case, non obsolete files are not printed.
(This will allow future support for builds with variables such as
MKHESIOD and MKYP set to "no".)

* Use sh(1)'s getopts where appropriate, and otherwise cleanup the
various scripts.

* Move defaults for sets.subr from sets.defaults into sets.subr.
Move replicated code for determining stuff such as shlibs type
from various scripts into sets.subr.

* Merge the obsolete.*, krb.*, krb4.* and lint.* into the appropriate
main lists with the relevant third field keyword(s).

Redo creation of sets/sums so this can be parallelized if using -j.
Suggested in PR#20115 and PR#20116

Factor duplicated code for setting variables such as machine_cpu
and machine_arch to their defaults into sets.defaults, and source
it at the top of each script.

Also, to be consistent with variable naming, s/arch/machine_arch/.

sets.defaults introduces two new variables, krb and krb4, which
will affect whether Kerberos- or Kerberos IV-only files are put
into the set lists.

Report correct error code (in particular when mtree objects to METALOG file).

* use ${MKTEMP} -d instead of home-grown tempdir creation code
* use $(...) instead of `...`
* fix a recent modification to follow the same coding style as the
rest of the file... (mmm, consistency)

Since the METALOG is updated by every install, there is no need to
remake the sets if the set files are newer than METALOG.
Useful for running DISTRIBUTION_DONE= nbmake in src/etc in order to
rebuild with new kernels and/or new ramdisk.

if the mtree parse of METALOG fails, exit.

Temporarily revert last. It now screws over symlinks from the destdir,
which is the inverse problem. It is fixable by piping one pax -w into
a pax -r instead of using a pax -rw, but unfortunately that won't work
yet because I've hit a pax issue. I'll fix that and then fix this right.

do pax -L so that symlinks aren't blown away.
Beginning of work to fix PR install/8090 from Luke Mewburn

Return the correct exit status in the trap handler so errors get caught.
Also make sure the status is less than 256.

writable, not writeable.

improve description of -t

improve description

fix header to match usage

Rework METALOG manipulation:
- add '-D ${DESTDIR}' to INSTPRIV, so install(8) removes the leading
${DESTDIR} from the metalogged path
- provide ${METADB.add} variable (for "${CAT} -l >> ${METALOG}"), to make
it easier to replace manual metalog manipulation in the future.
- with manual metalog additions, don't add the leading ${DESTDIR} in the path
- in maketars, use "mtree -C ..." instead of
"mtree -D ... | sed -e 's,\(.*\) \(\..*\),\2 \1,";

Benefits:
- maketars "Parsing METALOG" step speeds up from 29 seconds to 1.2 seconds
on a P3-600.
(This also benefits "make installworld" at the top level.)
- ${DESTDIR}/METALOG is easier to read without the leading "${DESTDIR}"
on all the pathnames, and it's smaller as well.

Invoke ${PAX} with '-O' so that various errors don't hang the make
or the command with an interactive question. (This also prevents
weird problems if a directory element of '-f path' is missing and
'make -j N' is being used).

Always call maketars (and subsequently pax) with -N ${NETBSDSRCDIR}/etc,
so that we always use the user & group databases from there rather than
the host systems.

maketars:
- add "-N etcdir", to be used for pax & mtree -N. defaults to ${DESTDIR}/etc

Makefile:
- if UNPRIVED, call maketars with "-N ${NETBSDSRCDIR}/etc", so that
"make UNPRIVED= DESTDIR=/foo installsets" can work with an empty
${DESTDIR}/etc. (i.e, after a make build without install-etc-files)

consistently use $dest instead of $DESTDIR

Add '-i installdir', to copy the given sets from $DESTDIR to installdir
instead of creating tar files. If the default sets are used, "etc" is
removed from the list, because it's highly unlikely that it's wanted
in that case.

Tested against a DESTDIR created with "make UNPRIVED= DESTDIR=.... build".

use ${make} instead of ${MAKE} in the shell scripts, to prevent weird
lossage with the modified ${MAKE} getting passed to child make's

explicitly add " -f `dirname $0`/Makefile " to ${MAKE}, so that
these scripts do not need to be run from this directory in order
to function correctly

set MAKE="${MAKE:-make} -j 1" to prevent problems where the output
of make -j N>1 would result in junk in various shell variables

branches: 1.24.2;
keep separate temp files for "full" flists

- clean up SDIR (temporary directory) creation
- improve munging of NetBSD.dist and METALOG into metalog, using mtree to
"cleanup" / "merge" entries, and don't sort the result.
relies upon mtree to not change the order (see below)
- use (newly added) join.awk to output lines in metalog that are listed
in flist.${set}. join.awk doesn't require either file to be sorted
(unlike join(1)), which is required because we want to retain the
order of the metalog, because certain entries in the metalog (such
as hard links) make assumptions about the permissions of earlier entries.
this should fix [toolchain/16207]

Solve problem with with XFree86 3.3.6/4.x set "maketars" differently:
We don't want e.g. a seperate "xbase.tgz" and "xbase3.tgz" but one
combined archive.

Fix problem with building X11 release "tar" archives after support for
XFree86 4.x sets has been added. Problem noted by Richard Earnshaw in
private e-mail.

Instead of using mktemp -d (which isn't available on many systems), use mkdir
in a loop to create a temporary directory. Add a few chacks for the sanity
of /tmp first, so as to avoid making the loop infinite.

- use numeric rather than symbolic trap names; it's more portable
- read NetBSD.dist from ${DESTDIR}/etc/mtree instead of /usr/src/etc/mtree
- run mtree with "-N ${DESTDIR}/etc" instead of using the host user databases

Fixes to allow unprivileged building of the binary tar files to
contain the correct permissions.

Makefile:
- set SETSENV to the list of variables we wish to pass to each shell
script and use that, rather than duplicating the effort below
- call maketars with "-M ${METALOG}" if UNPRIVED is set.

getdirs.awk:
- takes a list of filenames and outputs the list along with all the
directories used by those filenames

maketars:
- support "-M metalog"
- merge etc/mtree/NetBSD.dist with the "-M metalog" for use in the tgz build
- for a given set, find the union between the set list and the metalog
generated above and use that as the list of files to ${PAX} up as the
tgz file.

Wait with exiting on error until all the install sets have been
attempted to be created.

This way, in "discover missing files" mode, you'll get the benefit
of getting all the missing files listed before this script bombs
out with a non-zero exit status, something which can be of benefit
on slow machines.

Use the logic in bsd.own.mk to determine machine/machine_arch.

Change pipeline in sets building so proper error detection for each stage
can be detected. This way sets stop building at the exact file that doesn't
exist rather than continuing.

Add the xmisc set.

branches: 1.13.10;
Use "$MAKE" instead of "make" for cross environment.

Use $out correctly, pointed out in pr 6526 by Dave Huang.

There should be no need for the "set -x" now, it clutters output.

The newly churned maketars was apparently not tested in its normal
mode with no options and no parameters.

You have to be careful about the difference between $@ and $*.
In particular, "if [ -n "$@" ]" does not do what you think it will.

* replace '-arch arch' with '-m machine' (as per make(1)'s $MACHINE). e.g, i386
* add '-a arch' - $MACHINE_ARCH. e.g, m68k, powerpc
* replace long -options with short -options (with backward compat)
* ``knf''ify scripts

Change ".tar.gz" tarball suffix to ".tgz", per release(7).

add -both arg for when one puts xsrc and src builds in same DESTDIR

When deciding which md set lists to use, prefer $MACHINE over
uname -m

Add xcontrib to xlists.

Add a new option '-x11' for the xsets.

branches: 1.3.2;
Feed makeflist to sh, instead of relying on x-bits; from Andrew
Wheadon in PR 3116.

More portable versions of makeflist and maketars that can be used to
cross-make a distribution. Also avoid duplicating makeflist logic.

branches: 1.1.1;
Initial revision

Add new sets: base32, debug32, manhtml

- base32 contains (when MKCOMPAT=yes) shared libraries for 32-bit
compatibility, previously included in base

- debug32 contains (when MKCOMPAT=yes) debug symbols and static libraries
containing debug symbols for 32-bit compatiblity, previously included
in debug

- manhtml contains (when MKHTML=yes) the HTML files previously included
in 'man', which are of limited utility without third-party software.

The motivation for this change is to be able to easily exclude sets
from CD-ROM images that go over the size limit without xz compression
(which many NetBSD platforms struggle to extract at acceptable speeds).

Fix previous, use sed to fix up the set.${set} line in non-METAFILE mode

When building in priviledged mode, remove the entry for the output file
from etc/mtree/set.* - it may exist and contain stale data.

More conservative umask

Set the file creation mask consistently because the files these scripts
create can become part of an iso image.

sets: enhance MAKEVERBOSE support

Add more messages for various sets operations.
Add ${_MKMSG_CREATE} to bsd.own.mk.

Enhance maketars to be a bit more verbose even with -q,
to make it more obvious why certain operations may be slow.
May help debugging PR 47562.

maketars: add comments for skipped mtree keywords

Reorder skipkeys alphabetically and add comment explaining
the purpose and why sha256 is not part of the variable.

Deprecate extsrc/, EXTSRCSRCDIR, MKEXTSRC

Remove support for building extsrc/:
1. Makefile: remove do-extsrc target.
2. build.sh: remove options -y and -Y extsrcdir.
3. distrib/sets: remove support for extsrc in various tools
including the options -L ext and -y, and the extsrc sets.
4. doc/BUILDING.mdoc: remove docs for extsrc/, EXTSRCSRCDIR, MKEXTSRC (etc)
5. bsd.own.mk and various Makefiles: remove support for extsrc/,
EXTSRCSRCDIR, MKEXTSRC.

As proposed on tech-kern and tech-userlevel on 2022-01-07
and followed up on 2022-08-21.

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

Patch the build script and toolchain to allow passing through
HOST_CPPFLAGS, HOST_CXXFLAGS, but not LIBRARY_PATH because it breaks
the build. These variables are necessary to support building NetBSD
from a GNU Guix or NixOS host, where /usr/include, /lib, and all
but /bin/sh do not exist. In many cases, support for HOST_CPPFLAGS
was incomplete. From Ryan Sundberg

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

branches: 1.89.2;
Prepare for .tar.xz sets

Revert previous - will pass the additional args via the environment.

Allow for ${COMPRESS_PROGRAM} to hold the name of a compressor program
plus additional arguments (like -9).

branches: 1.86.10; 1.86.12;
This fixes a bug where mtree would use names & IDs from the host system
instead of from the build.

branches: 1.85.2; 1.85.4;
Obey MKREPRO_TIMESTAMP when creating sets (create all the files in the
sets with that date).

Simplify slightly

Instead of guessing wether to preserve permissions on install operations
from the METALOG variable, default them to yes and add a new argument -U to
switch preservation off.

Do not try to preserve permissions when copying to the -i installdir
in the METALOG case (it makes no sense at all).

Add a new arg to override the value of the "setlistdir" variable (that is:
where the mtree spec files are created)

We always have a valid mtree file, so always specify -M

merge d flag.

- use dest instead of DESTDIR
- factor out pax usage, and make it use only the first word from the sets

Create full sets using mtree from the flists (not just filenames).

- add xdebug set
- document debug better
- fix previous debug fallout.

Add a new "debug" set that gets built when ${MKDEBUG} is set. On evbarm:
-rw-r--r-- 1 root src 35806068 Jan 16 10:47 base.tgz
-rw-r--r-- 1 root src 57374069 Jan 16 10:48 comp.tgz
-rw-r--r-- 1 root src 101286193 Jan 16 10:49 debug.tgz
...
This allows always building the debugging code and libraries but not
installing it (although now it is placed in the default sets if ${MKDEBUG}
is set

branches: 1.74.6; 1.74.12;
add new option to build/use tools/pigz for compressing sets. for now
USE_PIGZGZIP defaults to "no". (it depends upon local <pthread.h> and -lz.)

you can set it to "yes" on most modern platforms just fine.

sh(1) style. No functional changes.

Fix syntax error on OSX 10.5.
While here, simplify handling with OPTARG using IFS.

developped with and 'go for it' dsl@

Exit with a sensible error message if makeflist output is empty.
This should be better than an incomprehensible error message from
mtree.

Support "extsrc", externally added programs and libraries. Users can write
their own reach-overs, cross-build, install, and get set files just like base
and X11 / X.org. (These sets are not included as TNF releases.)

If there are errors in creating a ${set}.tgz tar file, then don't
just press on regardless.

In various places that manipulate mtree specs, use "mtree -CS" to sort.
Remove some now-redundant standalone sort commands.

Consistently use gzip with the -n flag so dates are not embedded.

Not tested on all platforms.

Explicitly sort entries on preparing set files from METALOG.
METALOG could have different order due to install(1) race
on parallel builds, and mtree(8) doesn't sort files.
Should fix inconsistent shared sets among builds as seen in
/pub/NetBSD-daily/netbsd-5/200904010000Z/shared/ and
/pub/NetBSD-daily/netbsd-5/200904010002Z/shared/ dirs.
Okay'ed by snj@.

Clean up set files to make them consistent using mtree. Patch from lukem.

branches: 1.64.6; 1.64.8;
Add '-q' to maketars and makesrctars, to quieten them.
Improve MAKEVERBOSE support.

Cleanup .tgz.tmp files on exit.

branches: 1.62.6; 1.62.8;
Add the tests.tgz set

This adds a new tests.tgz set to releases which includes all the tests
for the system. It is important to note that this set does not rely on
comp.tgz: a user of the system can run the tests without having the
development tools installed, which can be useful in a production machine.

branches: 1.61.10;
No space in "function()", "command;" or "$(command)".
Requested by christos.

When printing messages to stderr, always use
echo >&2 "${prog}: ..."

Approved by christos

Remove the code that used "${MTREE} -C ..." to sanitise the plist. We
can now assume that it's already been sanitised before maketars was
invoked.

Reviewed by agc

Ensure that (almost) all shell variables in distrib/sets/* scripts are
used with curly braces and quotes, as in "${var}".

Also ensure that command substitution is quoted, as in "$(command)", and
convert `command` to $(command).

Reviewed by agc

Add shell variables defined in sets.subr for almost all commands used
by scripts in distrib/sets/*. This is intended to be useful when cross
building. Only trivial commands like echo, cat, and rm are excluded
from this treatment.

While I am here, make ${MTREE} and ${DB} follow the pattern used by most
other such variables.

Reviewed by agc.

Move default values for several shell variables into sets.subr.

Reviewed by agc

Set default vars so blank and unset both end up w. a reasonable default

Make all the sets scripts work even when not run from the directory that
they reside in.
Change any uses of dirname to use shell variable substitution instead.

Write to a temporary file and mv to real name to avoid getting a
truncated file if/when interrupted.

Clearly the error message was supposed to print $metalog, not
$METALOG.

When not using a metalog, only copy the generated flist.<set> over
/etc/mtree/set.<set> if they are different. This should allow
installsets to operate from a pre-populated read-only DESTDIR.
Based on fix from Juergen Hannken-Illjes.

Create ${DESTDIR}/etc/mtree/set.<setname> setfiles for each given <setname>.
Add `-S' to exit after creating the setfiles.
Send errors to stderr instead of stdout.

Also remove "xetc" from the default list of sets copied with -i installdir

rename all the <bsd.own.mk> derived variables to uppercase, and export
them to the environment.

Significant overhaul of how the sets are used:

* Add an optional third field to the sets file which is a list of
comma separated keywords that control if the line is printed.
Currently supported keywords
kerberos4 ${MKKERBEROS4} != no
kerberos ${MKKERBEROS} != no
lint ${MKLINT} != no
obsolete ${obsolete} != 0.
In this case, non obsolete files are not printed.
(This will allow future support for builds with variables such as
MKHESIOD and MKYP set to "no".)

* Use sh(1)'s getopts where appropriate, and otherwise cleanup the
various scripts.

* Move defaults for sets.subr from sets.defaults into sets.subr.
Move replicated code for determining stuff such as shlibs type
from various scripts into sets.subr.

* Merge the obsolete.*, krb.*, krb4.* and lint.* into the appropriate
main lists with the relevant third field keyword(s).

Redo creation of sets/sums so this can be parallelized if using -j.
Suggested in PR#20115 and PR#20116

Factor duplicated code for setting variables such as machine_cpu
and machine_arch to their defaults into sets.defaults, and source
it at the top of each script.

Also, to be consistent with variable naming, s/arch/machine_arch/.

sets.defaults introduces two new variables, krb and krb4, which
will affect whether Kerberos- or Kerberos IV-only files are put
into the set lists.

Report correct error code (in particular when mtree objects to METALOG file).

* use ${MKTEMP} -d instead of home-grown tempdir creation code
* use $(...) instead of `...`
* fix a recent modification to follow the same coding style as the
rest of the file... (mmm, consistency)

Since the METALOG is updated by every install, there is no need to
remake the sets if the set files are newer than METALOG.
Useful for running DISTRIBUTION_DONE= nbmake in src/etc in order to
rebuild with new kernels and/or new ramdisk.

if the mtree parse of METALOG fails, exit.

Temporarily revert last. It now screws over symlinks from the destdir,
which is the inverse problem. It is fixable by piping one pax -w into
a pax -r instead of using a pax -rw, but unfortunately that won't work
yet because I've hit a pax issue. I'll fix that and then fix this right.

do pax -L so that symlinks aren't blown away.
Beginning of work to fix PR install/8090 from Luke Mewburn

Return the correct exit status in the trap handler so errors get caught.
Also make sure the status is less than 256.

writable, not writeable.

improve description of -t

improve description

fix header to match usage

Rework METALOG manipulation:
- add '-D ${DESTDIR}' to INSTPRIV, so install(8) removes the leading
${DESTDIR} from the metalogged path
- provide ${METADB.add} variable (for "${CAT} -l >> ${METALOG}"), to make
it easier to replace manual metalog manipulation in the future.
- with manual metalog additions, don't add the leading ${DESTDIR} in the path
- in maketars, use "mtree -C ..." instead of
"mtree -D ... | sed -e 's,\(.*\) \(\..*\),\2 \1,";

Benefits:
- maketars "Parsing METALOG" step speeds up from 29 seconds to 1.2 seconds
on a P3-600.
(This also benefits "make installworld" at the top level.)
- ${DESTDIR}/METALOG is easier to read without the leading "${DESTDIR}"
on all the pathnames, and it's smaller as well.

Invoke ${PAX} with '-O' so that various errors don't hang the make
or the command with an interactive question. (This also prevents
weird problems if a directory element of '-f path' is missing and
'make -j N' is being used).

Always call maketars (and subsequently pax) with -N ${NETBSDSRCDIR}/etc,
so that we always use the user & group databases from there rather than
the host systems.

maketars:
- add "-N etcdir", to be used for pax & mtree -N. defaults to ${DESTDIR}/etc

Makefile:
- if UNPRIVED, call maketars with "-N ${NETBSDSRCDIR}/etc", so that
"make UNPRIVED= DESTDIR=/foo installsets" can work with an empty
${DESTDIR}/etc. (i.e, after a make build without install-etc-files)

consistently use $dest instead of $DESTDIR

Add '-i installdir', to copy the given sets from $DESTDIR to installdir
instead of creating tar files. If the default sets are used, "etc" is
removed from the list, because it's highly unlikely that it's wanted
in that case.

Tested against a DESTDIR created with "make UNPRIVED= DESTDIR=.... build".

use ${make} instead of ${MAKE} in the shell scripts, to prevent weird
lossage with the modified ${MAKE} getting passed to child make's

explicitly add " -f `dirname $0`/Makefile " to ${MAKE}, so that
these scripts do not need to be run from this directory in order
to function correctly

set MAKE="${MAKE:-make} -j 1" to prevent problems where the output
of make -j N>1 would result in junk in various shell variables

branches: 1.24.2;
keep separate temp files for "full" flists

- clean up SDIR (temporary directory) creation
- improve munging of NetBSD.dist and METALOG into metalog, using mtree to
"cleanup" / "merge" entries, and don't sort the result.
relies upon mtree to not change the order (see below)
- use (newly added) join.awk to output lines in metalog that are listed
in flist.${set}. join.awk doesn't require either file to be sorted
(unlike join(1)), which is required because we want to retain the
order of the metalog, because certain entries in the metalog (such
as hard links) make assumptions about the permissions of earlier entries.
this should fix [toolchain/16207]

Solve problem with with XFree86 3.3.6/4.x set "maketars" differently:
We don't want e.g. a seperate "xbase.tgz" and "xbase3.tgz" but one
combined archive.

Fix problem with building X11 release "tar" archives after support for
XFree86 4.x sets has been added. Problem noted by Richard Earnshaw in
private e-mail.

Instead of using mktemp -d (which isn't available on many systems), use mkdir
in a loop to create a temporary directory. Add a few chacks for the sanity
of /tmp first, so as to avoid making the loop infinite.

- use numeric rather than symbolic trap names; it's more portable
- read NetBSD.dist from ${DESTDIR}/etc/mtree instead of /usr/src/etc/mtree
- run mtree with "-N ${DESTDIR}/etc" instead of using the host user databases

Fixes to allow unprivileged building of the binary tar files to
contain the correct permissions.

Makefile:
- set SETSENV to the list of variables we wish to pass to each shell
script and use that, rather than duplicating the effort below
- call maketars with "-M ${METALOG}" if UNPRIVED is set.

getdirs.awk:
- takes a list of filenames and outputs the list along with all the
directories used by those filenames

maketars:
- support "-M metalog"
- merge etc/mtree/NetBSD.dist with the "-M metalog" for use in the tgz build
- for a given set, find the union between the set list and the metalog
generated above and use that as the list of files to ${PAX} up as the
tgz file.

Wait with exiting on error until all the install sets have been
attempted to be created.

This way, in "discover missing files" mode, you'll get the benefit
of getting all the missing files listed before this script bombs
out with a non-zero exit status, something which can be of benefit
on slow machines.

Use the logic in bsd.own.mk to determine machine/machine_arch.

Change pipeline in sets building so proper error detection for each stage
can be detected. This way sets stop building at the exact file that doesn't
exist rather than continuing.

Add the xmisc set.

branches: 1.13.10;
Use "$MAKE" instead of "make" for cross environment.

Use $out correctly, pointed out in pr 6526 by Dave Huang.

There should be no need for the "set -x" now, it clutters output.

The newly churned maketars was apparently not tested in its normal
mode with no options and no parameters.

You have to be careful about the difference between $@ and $*.
In particular, "if [ -n "$@" ]" does not do what you think it will.

* replace '-arch arch' with '-m machine' (as per make(1)'s $MACHINE). e.g, i386
* add '-a arch' - $MACHINE_ARCH. e.g, m68k, powerpc
* replace long -options with short -options (with backward compat)
* ``knf''ify scripts

Change ".tar.gz" tarball suffix to ".tgz", per release(7).

add -both arg for when one puts xsrc and src builds in same DESTDIR

When deciding which md set lists to use, prefer $MACHINE over
uname -m

Add xcontrib to xlists.

Add a new option '-x11' for the xsets.

branches: 1.3.2;
Feed makeflist to sh, instead of relying on x-bits; from Andrew
Wheadon in PR 3116.

More portable versions of makeflist and maketars that can be used to
cross-make a distribution. Also avoid duplicating makeflist logic.

branches: 1.1.1;
Initial revision

Fix previous, use sed to fix up the set.${set} line in non-METAFILE mode

When building in priviledged mode, remove the entry for the output file
from etc/mtree/set.* - it may exist and contain stale data.

More conservative umask

Set the file creation mask consistently because the files these scripts
create can become part of an iso image.

sets: enhance MAKEVERBOSE support

Add more messages for various sets operations.
Add ${_MKMSG_CREATE} to bsd.own.mk.

Enhance maketars to be a bit more verbose even with -q,
to make it more obvious why certain operations may be slow.
May help debugging PR 47562.

maketars: add comments for skipped mtree keywords

Reorder skipkeys alphabetically and add comment explaining
the purpose and why sha256 is not part of the variable.

Deprecate extsrc/, EXTSRCSRCDIR, MKEXTSRC

Remove support for building extsrc/:
1. Makefile: remove do-extsrc target.
2. build.sh: remove options -y and -Y extsrcdir.
3. distrib/sets: remove support for extsrc in various tools
including the options -L ext and -y, and the extsrc sets.
4. doc/BUILDING.mdoc: remove docs for extsrc/, EXTSRCSRCDIR, MKEXTSRC (etc)
5. bsd.own.mk and various Makefiles: remove support for extsrc/,
EXTSRCSRCDIR, MKEXTSRC.

As proposed on tech-kern and tech-userlevel on 2022-01-07
and followed up on 2022-08-21.

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

Patch the build script and toolchain to allow passing through
HOST_CPPFLAGS, HOST_CXXFLAGS, but not LIBRARY_PATH because it breaks
the build. These variables are necessary to support building NetBSD
from a GNU Guix or NixOS host, where /usr/include, /lib, and all
but /bin/sh do not exist. In many cases, support for HOST_CPPFLAGS
was incomplete. From Ryan Sundberg

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

branches: 1.89.2;
Prepare for .tar.xz sets

Revert previous - will pass the additional args via the environment.

Allow for ${COMPRESS_PROGRAM} to hold the name of a compressor program
plus additional arguments (like -9).

branches: 1.86.10; 1.86.12;
This fixes a bug where mtree would use names & IDs from the host system
instead of from the build.

branches: 1.85.2; 1.85.4;
Obey MKREPRO_TIMESTAMP when creating sets (create all the files in the
sets with that date).

Simplify slightly

Instead of guessing wether to preserve permissions on install operations
from the METALOG variable, default them to yes and add a new argument -U to
switch preservation off.

Do not try to preserve permissions when copying to the -i installdir
in the METALOG case (it makes no sense at all).

Add a new arg to override the value of the "setlistdir" variable (that is:
where the mtree spec files are created)

We always have a valid mtree file, so always specify -M

merge d flag.

- use dest instead of DESTDIR
- factor out pax usage, and make it use only the first word from the sets

Create full sets using mtree from the flists (not just filenames).

- add xdebug set
- document debug better
- fix previous debug fallout.

Add a new "debug" set that gets built when ${MKDEBUG} is set. On evbarm:
-rw-r--r-- 1 root src 35806068 Jan 16 10:47 base.tgz
-rw-r--r-- 1 root src 57374069 Jan 16 10:48 comp.tgz
-rw-r--r-- 1 root src 101286193 Jan 16 10:49 debug.tgz
...
This allows always building the debugging code and libraries but not
installing it (although now it is placed in the default sets if ${MKDEBUG}
is set

branches: 1.74.6; 1.74.12;
add new option to build/use tools/pigz for compressing sets. for now
USE_PIGZGZIP defaults to "no". (it depends upon local <pthread.h> and -lz.)

you can set it to "yes" on most modern platforms just fine.

sh(1) style. No functional changes.

Fix syntax error on OSX 10.5.
While here, simplify handling with OPTARG using IFS.

developped with and 'go for it' dsl@

Exit with a sensible error message if makeflist output is empty.
This should be better than an incomprehensible error message from
mtree.

Support "extsrc", externally added programs and libraries. Users can write
their own reach-overs, cross-build, install, and get set files just like base
and X11 / X.org. (These sets are not included as TNF releases.)

If there are errors in creating a ${set}.tgz tar file, then don't
just press on regardless.

In various places that manipulate mtree specs, use "mtree -CS" to sort.
Remove some now-redundant standalone sort commands.

Consistently use gzip with the -n flag so dates are not embedded.

Not tested on all platforms.

Explicitly sort entries on preparing set files from METALOG.
METALOG could have different order due to install(1) race
on parallel builds, and mtree(8) doesn't sort files.
Should fix inconsistent shared sets among builds as seen in
/pub/NetBSD-daily/netbsd-5/200904010000Z/shared/ and
/pub/NetBSD-daily/netbsd-5/200904010002Z/shared/ dirs.
Okay'ed by snj@.

Clean up set files to make them consistent using mtree. Patch from lukem.

branches: 1.64.6; 1.64.8;
Add '-q' to maketars and makesrctars, to quieten them.
Improve MAKEVERBOSE support.

Cleanup .tgz.tmp files on exit.

branches: 1.62.6; 1.62.8;
Add the tests.tgz set

This adds a new tests.tgz set to releases which includes all the tests
for the system. It is important to note that this set does not rely on
comp.tgz: a user of the system can run the tests without having the
development tools installed, which can be useful in a production machine.

branches: 1.61.10;
No space in "function()", "command;" or "$(command)".
Requested by christos.

When printing messages to stderr, always use
echo >&2 "${prog}: ..."

Approved by christos

Remove the code that used "${MTREE} -C ..." to sanitise the plist. We
can now assume that it's already been sanitised before maketars was
invoked.

Reviewed by agc

Ensure that (almost) all shell variables in distrib/sets/* scripts are
used with curly braces and quotes, as in "${var}".

Also ensure that command substitution is quoted, as in "$(command)", and
convert `command` to $(command).

Reviewed by agc

Add shell variables defined in sets.subr for almost all commands used
by scripts in distrib/sets/*. This is intended to be useful when cross
building. Only trivial commands like echo, cat, and rm are excluded
from this treatment.

While I am here, make ${MTREE} and ${DB} follow the pattern used by most
other such variables.

Reviewed by agc.

Move default values for several shell variables into sets.subr.

Reviewed by agc

Set default vars so blank and unset both end up w. a reasonable default

Make all the sets scripts work even when not run from the directory that
they reside in.
Change any uses of dirname to use shell variable substitution instead.

Write to a temporary file and mv to real name to avoid getting a
truncated file if/when interrupted.

Clearly the error message was supposed to print $metalog, not
$METALOG.

When not using a metalog, only copy the generated flist.<set> over
/etc/mtree/set.<set> if they are different. This should allow
installsets to operate from a pre-populated read-only DESTDIR.
Based on fix from Juergen Hannken-Illjes.

Create ${DESTDIR}/etc/mtree/set.<setname> setfiles for each given <setname>.
Add `-S' to exit after creating the setfiles.
Send errors to stderr instead of stdout.

Also remove "xetc" from the default list of sets copied with -i installdir

rename all the <bsd.own.mk> derived variables to uppercase, and export
them to the environment.

Significant overhaul of how the sets are used:

* Add an optional third field to the sets file which is a list of
comma separated keywords that control if the line is printed.
Currently supported keywords
kerberos4 ${MKKERBEROS4} != no
kerberos ${MKKERBEROS} != no
lint ${MKLINT} != no
obsolete ${obsolete} != 0.
In this case, non obsolete files are not printed.
(This will allow future support for builds with variables such as
MKHESIOD and MKYP set to "no".)

* Use sh(1)'s getopts where appropriate, and otherwise cleanup the
various scripts.

* Move defaults for sets.subr from sets.defaults into sets.subr.
Move replicated code for determining stuff such as shlibs type
from various scripts into sets.subr.

* Merge the obsolete.*, krb.*, krb4.* and lint.* into the appropriate
main lists with the relevant third field keyword(s).

Redo creation of sets/sums so this can be parallelized if using -j.
Suggested in PR#20115 and PR#20116

Factor duplicated code for setting variables such as machine_cpu
and machine_arch to their defaults into sets.defaults, and source
it at the top of each script.

Also, to be consistent with variable naming, s/arch/machine_arch/.

sets.defaults introduces two new variables, krb and krb4, which
will affect whether Kerberos- or Kerberos IV-only files are put
into the set lists.

Report correct error code (in particular when mtree objects to METALOG file).

* use ${MKTEMP} -d instead of home-grown tempdir creation code
* use $(...) instead of `...`
* fix a recent modification to follow the same coding style as the
rest of the file... (mmm, consistency)

Since the METALOG is updated by every install, there is no need to
remake the sets if the set files are newer than METALOG.
Useful for running DISTRIBUTION_DONE= nbmake in src/etc in order to
rebuild with new kernels and/or new ramdisk.

if the mtree parse of METALOG fails, exit.

Temporarily revert last. It now screws over symlinks from the destdir,
which is the inverse problem. It is fixable by piping one pax -w into
a pax -r instead of using a pax -rw, but unfortunately that won't work
yet because I've hit a pax issue. I'll fix that and then fix this right.

do pax -L so that symlinks aren't blown away.
Beginning of work to fix PR install/8090 from Luke Mewburn

Return the correct exit status in the trap handler so errors get caught.
Also make sure the status is less than 256.

writable, not writeable.

improve description of -t

improve description

fix header to match usage

Rework METALOG manipulation:
- add '-D ${DESTDIR}' to INSTPRIV, so install(8) removes the leading
${DESTDIR} from the metalogged path
- provide ${METADB.add} variable (for "${CAT} -l >> ${METALOG}"), to make
it easier to replace manual metalog manipulation in the future.
- with manual metalog additions, don't add the leading ${DESTDIR} in the path
- in maketars, use "mtree -C ..." instead of
"mtree -D ... | sed -e 's,\(.*\) \(\..*\),\2 \1,";

Benefits:
- maketars "Parsing METALOG" step speeds up from 29 seconds to 1.2 seconds
on a P3-600.
(This also benefits "make installworld" at the top level.)
- ${DESTDIR}/METALOG is easier to read without the leading "${DESTDIR}"
on all the pathnames, and it's smaller as well.

Invoke ${PAX} with '-O' so that various errors don't hang the make
or the command with an interactive question. (This also prevents
weird problems if a directory element of '-f path' is missing and
'make -j N' is being used).

Always call maketars (and subsequently pax) with -N ${NETBSDSRCDIR}/etc,
so that we always use the user & group databases from there rather than
the host systems.

maketars:
- add "-N etcdir", to be used for pax & mtree -N. defaults to ${DESTDIR}/etc

Makefile:
- if UNPRIVED, call maketars with "-N ${NETBSDSRCDIR}/etc", so that
"make UNPRIVED= DESTDIR=/foo installsets" can work with an empty
${DESTDIR}/etc. (i.e, after a make build without install-etc-files)

consistently use $dest instead of $DESTDIR

Add '-i installdir', to copy the given sets from $DESTDIR to installdir
instead of creating tar files. If the default sets are used, "etc" is
removed from the list, because it's highly unlikely that it's wanted
in that case.

Tested against a DESTDIR created with "make UNPRIVED= DESTDIR=.... build".

use ${make} instead of ${MAKE} in the shell scripts, to prevent weird
lossage with the modified ${MAKE} getting passed to child make's

explicitly add " -f `dirname $0`/Makefile " to ${MAKE}, so that
these scripts do not need to be run from this directory in order
to function correctly

set MAKE="${MAKE:-make} -j 1" to prevent problems where the output
of make -j N>1 would result in junk in various shell variables

branches: 1.24.2;
keep separate temp files for "full" flists

- clean up SDIR (temporary directory) creation
- improve munging of NetBSD.dist and METALOG into metalog, using mtree to
"cleanup" / "merge" entries, and don't sort the result.
relies upon mtree to not change the order (see below)
- use (newly added) join.awk to output lines in metalog that are listed
in flist.${set}. join.awk doesn't require either file to be sorted
(unlike join(1)), which is required because we want to retain the
order of the metalog, because certain entries in the metalog (such
as hard links) make assumptions about the permissions of earlier entries.
this should fix [toolchain/16207]

Solve problem with with XFree86 3.3.6/4.x set "maketars" differently:
We don't want e.g. a seperate "xbase.tgz" and "xbase3.tgz" but one
combined archive.

Fix problem with building X11 release "tar" archives after support for
XFree86 4.x sets has been added. Problem noted by Richard Earnshaw in
private e-mail.

Instead of using mktemp -d (which isn't available on many systems), use mkdir
in a loop to create a temporary directory. Add a few chacks for the sanity
of /tmp first, so as to avoid making the loop infinite.

- use numeric rather than symbolic trap names; it's more portable
- read NetBSD.dist from ${DESTDIR}/etc/mtree instead of /usr/src/etc/mtree
- run mtree with "-N ${DESTDIR}/etc" instead of using the host user databases

Fixes to allow unprivileged building of the binary tar files to
contain the correct permissions.

Makefile:
- set SETSENV to the list of variables we wish to pass to each shell
script and use that, rather than duplicating the effort below
- call maketars with "-M ${METALOG}" if UNPRIVED is set.

getdirs.awk:
- takes a list of filenames and outputs the list along with all the
directories used by those filenames

maketars:
- support "-M metalog"
- merge etc/mtree/NetBSD.dist with the "-M metalog" for use in the tgz build
- for a given set, find the union between the set list and the metalog
generated above and use that as the list of files to ${PAX} up as the
tgz file.

Wait with exiting on error until all the install sets have been
attempted to be created.

This way, in "discover missing files" mode, you'll get the benefit
of getting all the missing files listed before this script bombs
out with a non-zero exit status, something which can be of benefit
on slow machines.

Use the logic in bsd.own.mk to determine machine/machine_arch.

Change pipeline in sets building so proper error detection for each stage
can be detected. This way sets stop building at the exact file that doesn't
exist rather than continuing.

Add the xmisc set.

branches: 1.13.10;
Use "$MAKE" instead of "make" for cross environment.

Use $out correctly, pointed out in pr 6526 by Dave Huang.

There should be no need for the "set -x" now, it clutters output.

The newly churned maketars was apparently not tested in its normal
mode with no options and no parameters.

You have to be careful about the difference between $@ and $*.
In particular, "if [ -n "$@" ]" does not do what you think it will.

* replace '-arch arch' with '-m machine' (as per make(1)'s $MACHINE). e.g, i386
* add '-a arch' - $MACHINE_ARCH. e.g, m68k, powerpc
* replace long -options with short -options (with backward compat)
* ``knf''ify scripts

Change ".tar.gz" tarball suffix to ".tgz", per release(7).

add -both arg for when one puts xsrc and src builds in same DESTDIR

When deciding which md set lists to use, prefer $MACHINE over
uname -m

Add xcontrib to xlists.

Add a new option '-x11' for the xsets.

branches: 1.3.2;
Feed makeflist to sh, instead of relying on x-bits; from Andrew
Wheadon in PR 3116.

More portable versions of makeflist and maketars that can be used to
cross-make a distribution. Also avoid duplicating makeflist logic.

branches: 1.1.1;
Initial revision

When building in priviledged mode, remove the entry for the output file
from etc/mtree/set.* - it may exist and contain stale data.

More conservative umask

Set the file creation mask consistently because the files these scripts
create can become part of an iso image.

sets: enhance MAKEVERBOSE support

Add more messages for various sets operations.
Add ${_MKMSG_CREATE} to bsd.own.mk.

Enhance maketars to be a bit more verbose even with -q,
to make it more obvious why certain operations may be slow.
May help debugging PR 47562.

maketars: add comments for skipped mtree keywords

Reorder skipkeys alphabetically and add comment explaining
the purpose and why sha256 is not part of the variable.

Deprecate extsrc/, EXTSRCSRCDIR, MKEXTSRC

Remove support for building extsrc/:
1. Makefile: remove do-extsrc target.
2. build.sh: remove options -y and -Y extsrcdir.
3. distrib/sets: remove support for extsrc in various tools
including the options -L ext and -y, and the extsrc sets.
4. doc/BUILDING.mdoc: remove docs for extsrc/, EXTSRCSRCDIR, MKEXTSRC (etc)
5. bsd.own.mk and various Makefiles: remove support for extsrc/,
EXTSRCSRCDIR, MKEXTSRC.

As proposed on tech-kern and tech-userlevel on 2022-01-07
and followed up on 2022-08-21.

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

Patch the build script and toolchain to allow passing through
HOST_CPPFLAGS, HOST_CXXFLAGS, but not LIBRARY_PATH because it breaks
the build. These variables are necessary to support building NetBSD
from a GNU Guix or NixOS host, where /usr/include, /lib, and all
but /bin/sh do not exist. In many cases, support for HOST_CPPFLAGS
was incomplete. From Ryan Sundberg

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

branches: 1.89.2;
Prepare for .tar.xz sets

Revert previous - will pass the additional args via the environment.

Allow for ${COMPRESS_PROGRAM} to hold the name of a compressor program
plus additional arguments (like -9).

branches: 1.86.10; 1.86.12;
This fixes a bug where mtree would use names & IDs from the host system
instead of from the build.

branches: 1.85.2; 1.85.4;
Obey MKREPRO_TIMESTAMP when creating sets (create all the files in the
sets with that date).

Simplify slightly

Instead of guessing wether to preserve permissions on install operations
from the METALOG variable, default them to yes and add a new argument -U to
switch preservation off.

Do not try to preserve permissions when copying to the -i installdir
in the METALOG case (it makes no sense at all).

Add a new arg to override the value of the "setlistdir" variable (that is:
where the mtree spec files are created)

We always have a valid mtree file, so always specify -M

merge d flag.

- use dest instead of DESTDIR
- factor out pax usage, and make it use only the first word from the sets

Create full sets using mtree from the flists (not just filenames).

- add xdebug set
- document debug better
- fix previous debug fallout.

Add a new "debug" set that gets built when ${MKDEBUG} is set. On evbarm:
-rw-r--r-- 1 root src 35806068 Jan 16 10:47 base.tgz
-rw-r--r-- 1 root src 57374069 Jan 16 10:48 comp.tgz
-rw-r--r-- 1 root src 101286193 Jan 16 10:49 debug.tgz
...
This allows always building the debugging code and libraries but not
installing it (although now it is placed in the default sets if ${MKDEBUG}
is set

branches: 1.74.6; 1.74.12;
add new option to build/use tools/pigz for compressing sets. for now
USE_PIGZGZIP defaults to "no". (it depends upon local <pthread.h> and -lz.)

you can set it to "yes" on most modern platforms just fine.

sh(1) style. No functional changes.

Fix syntax error on OSX 10.5.
While here, simplify handling with OPTARG using IFS.

developped with and 'go for it' dsl@

Exit with a sensible error message if makeflist output is empty.
This should be better than an incomprehensible error message from
mtree.

Support "extsrc", externally added programs and libraries. Users can write
their own reach-overs, cross-build, install, and get set files just like base
and X11 / X.org. (These sets are not included as TNF releases.)

If there are errors in creating a ${set}.tgz tar file, then don't
just press on regardless.

In various places that manipulate mtree specs, use "mtree -CS" to sort.
Remove some now-redundant standalone sort commands.

Consistently use gzip with the -n flag so dates are not embedded.

Not tested on all platforms.

Explicitly sort entries on preparing set files from METALOG.
METALOG could have different order due to install(1) race
on parallel builds, and mtree(8) doesn't sort files.
Should fix inconsistent shared sets among builds as seen in
/pub/NetBSD-daily/netbsd-5/200904010000Z/shared/ and
/pub/NetBSD-daily/netbsd-5/200904010002Z/shared/ dirs.
Okay'ed by snj@.

Clean up set files to make them consistent using mtree. Patch from lukem.

branches: 1.64.6; 1.64.8;
Add '-q' to maketars and makesrctars, to quieten them.
Improve MAKEVERBOSE support.

Cleanup .tgz.tmp files on exit.

branches: 1.62.6; 1.62.8;
Add the tests.tgz set

This adds a new tests.tgz set to releases which includes all the tests
for the system. It is important to note that this set does not rely on
comp.tgz: a user of the system can run the tests without having the
development tools installed, which can be useful in a production machine.

branches: 1.61.10;
No space in "function()", "command;" or "$(command)".
Requested by christos.

When printing messages to stderr, always use
echo >&2 "${prog}: ..."

Approved by christos

Remove the code that used "${MTREE} -C ..." to sanitise the plist. We
can now assume that it's already been sanitised before maketars was
invoked.

Reviewed by agc

Ensure that (almost) all shell variables in distrib/sets/* scripts are
used with curly braces and quotes, as in "${var}".

Also ensure that command substitution is quoted, as in "$(command)", and
convert `command` to $(command).

Reviewed by agc

Add shell variables defined in sets.subr for almost all commands used
by scripts in distrib/sets/*. This is intended to be useful when cross
building. Only trivial commands like echo, cat, and rm are excluded
from this treatment.

While I am here, make ${MTREE} and ${DB} follow the pattern used by most
other such variables.

Reviewed by agc.

Move default values for several shell variables into sets.subr.

Reviewed by agc

Set default vars so blank and unset both end up w. a reasonable default

Make all the sets scripts work even when not run from the directory that
they reside in.
Change any uses of dirname to use shell variable substitution instead.

Write to a temporary file and mv to real name to avoid getting a
truncated file if/when interrupted.

Clearly the error message was supposed to print $metalog, not
$METALOG.

When not using a metalog, only copy the generated flist.<set> over
/etc/mtree/set.<set> if they are different. This should allow
installsets to operate from a pre-populated read-only DESTDIR.
Based on fix from Juergen Hannken-Illjes.

Create ${DESTDIR}/etc/mtree/set.<setname> setfiles for each given <setname>.
Add `-S' to exit after creating the setfiles.
Send errors to stderr instead of stdout.

Also remove "xetc" from the default list of sets copied with -i installdir

rename all the <bsd.own.mk> derived variables to uppercase, and export
them to the environment.

Significant overhaul of how the sets are used:

* Add an optional third field to the sets file which is a list of
comma separated keywords that control if the line is printed.
Currently supported keywords
kerberos4 ${MKKERBEROS4} != no
kerberos ${MKKERBEROS} != no
lint ${MKLINT} != no
obsolete ${obsolete} != 0.
In this case, non obsolete files are not printed.
(This will allow future support for builds with variables such as
MKHESIOD and MKYP set to "no".)

* Use sh(1)'s getopts where appropriate, and otherwise cleanup the
various scripts.

* Move defaults for sets.subr from sets.defaults into sets.subr.
Move replicated code for determining stuff such as shlibs type
from various scripts into sets.subr.

* Merge the obsolete.*, krb.*, krb4.* and lint.* into the appropriate
main lists with the relevant third field keyword(s).

Redo creation of sets/sums so this can be parallelized if using -j.
Suggested in PR#20115 and PR#20116

Factor duplicated code for setting variables such as machine_cpu
and machine_arch to their defaults into sets.defaults, and source
it at the top of each script.

Also, to be consistent with variable naming, s/arch/machine_arch/.

sets.defaults introduces two new variables, krb and krb4, which
will affect whether Kerberos- or Kerberos IV-only files are put
into the set lists.

Report correct error code (in particular when mtree objects to METALOG file).

* use ${MKTEMP} -d instead of home-grown tempdir creation code
* use $(...) instead of `...`
* fix a recent modification to follow the same coding style as the
rest of the file... (mmm, consistency)

Since the METALOG is updated by every install, there is no need to
remake the sets if the set files are newer than METALOG.
Useful for running DISTRIBUTION_DONE= nbmake in src/etc in order to
rebuild with new kernels and/or new ramdisk.

if the mtree parse of METALOG fails, exit.

Temporarily revert last. It now screws over symlinks from the destdir,
which is the inverse problem. It is fixable by piping one pax -w into
a pax -r instead of using a pax -rw, but unfortunately that won't work
yet because I've hit a pax issue. I'll fix that and then fix this right.

do pax -L so that symlinks aren't blown away.
Beginning of work to fix PR install/8090 from Luke Mewburn

Return the correct exit status in the trap handler so errors get caught.
Also make sure the status is less than 256.

writable, not writeable.

improve description of -t

improve description

fix header to match usage

Rework METALOG manipulation:
- add '-D ${DESTDIR}' to INSTPRIV, so install(8) removes the leading
${DESTDIR} from the metalogged path
- provide ${METADB.add} variable (for "${CAT} -l >> ${METALOG}"), to make
it easier to replace manual metalog manipulation in the future.
- with manual metalog additions, don't add the leading ${DESTDIR} in the path
- in maketars, use "mtree -C ..." instead of
"mtree -D ... | sed -e 's,\(.*\) \(\..*\),\2 \1,";

Benefits:
- maketars "Parsing METALOG" step speeds up from 29 seconds to 1.2 seconds
on a P3-600.
(This also benefits "make installworld" at the top level.)
- ${DESTDIR}/METALOG is easier to read without the leading "${DESTDIR}"
on all the pathnames, and it's smaller as well.

Invoke ${PAX} with '-O' so that various errors don't hang the make
or the command with an interactive question. (This also prevents
weird problems if a directory element of '-f path' is missing and
'make -j N' is being used).

Always call maketars (and subsequently pax) with -N ${NETBSDSRCDIR}/etc,
so that we always use the user & group databases from there rather than
the host systems.

maketars:
- add "-N etcdir", to be used for pax & mtree -N. defaults to ${DESTDIR}/etc

Makefile:
- if UNPRIVED, call maketars with "-N ${NETBSDSRCDIR}/etc", so that
"make UNPRIVED= DESTDIR=/foo installsets" can work with an empty
${DESTDIR}/etc. (i.e, after a make build without install-etc-files)

consistently use $dest instead of $DESTDIR

Add '-i installdir', to copy the given sets from $DESTDIR to installdir
instead of creating tar files. If the default sets are used, "etc" is
removed from the list, because it's highly unlikely that it's wanted
in that case.

Tested against a DESTDIR created with "make UNPRIVED= DESTDIR=.... build".

use ${make} instead of ${MAKE} in the shell scripts, to prevent weird
lossage with the modified ${MAKE} getting passed to child make's

explicitly add " -f `dirname $0`/Makefile " to ${MAKE}, so that
these scripts do not need to be run from this directory in order
to function correctly

set MAKE="${MAKE:-make} -j 1" to prevent problems where the output
of make -j N>1 would result in junk in various shell variables

branches: 1.24.2;
keep separate temp files for "full" flists

- clean up SDIR (temporary directory) creation
- improve munging of NetBSD.dist and METALOG into metalog, using mtree to
"cleanup" / "merge" entries, and don't sort the result.
relies upon mtree to not change the order (see below)
- use (newly added) join.awk to output lines in metalog that are listed
in flist.${set}. join.awk doesn't require either file to be sorted
(unlike join(1)), which is required because we want to retain the
order of the metalog, because certain entries in the metalog (such
as hard links) make assumptions about the permissions of earlier entries.
this should fix [toolchain/16207]

Solve problem with with XFree86 3.3.6/4.x set "maketars" differently:
We don't want e.g. a seperate "xbase.tgz" and "xbase3.tgz" but one
combined archive.

Fix problem with building X11 release "tar" archives after support for
XFree86 4.x sets has been added. Problem noted by Richard Earnshaw in
private e-mail.

Instead of using mktemp -d (which isn't available on many systems), use mkdir
in a loop to create a temporary directory. Add a few chacks for the sanity
of /tmp first, so as to avoid making the loop infinite.

- use numeric rather than symbolic trap names; it's more portable
- read NetBSD.dist from ${DESTDIR}/etc/mtree instead of /usr/src/etc/mtree
- run mtree with "-N ${DESTDIR}/etc" instead of using the host user databases

Fixes to allow unprivileged building of the binary tar files to
contain the correct permissions.

Makefile:
- set SETSENV to the list of variables we wish to pass to each shell
script and use that, rather than duplicating the effort below
- call maketars with "-M ${METALOG}" if UNPRIVED is set.

getdirs.awk:
- takes a list of filenames and outputs the list along with all the
directories used by those filenames

maketars:
- support "-M metalog"
- merge etc/mtree/NetBSD.dist with the "-M metalog" for use in the tgz build
- for a given set, find the union between the set list and the metalog
generated above and use that as the list of files to ${PAX} up as the
tgz file.

Wait with exiting on error until all the install sets have been
attempted to be created.

This way, in "discover missing files" mode, you'll get the benefit
of getting all the missing files listed before this script bombs
out with a non-zero exit status, something which can be of benefit
on slow machines.

Use the logic in bsd.own.mk to determine machine/machine_arch.

Change pipeline in sets building so proper error detection for each stage
can be detected. This way sets stop building at the exact file that doesn't
exist rather than continuing.

Add the xmisc set.

branches: 1.13.10;
Use "$MAKE" instead of "make" for cross environment.

Use $out correctly, pointed out in pr 6526 by Dave Huang.

There should be no need for the "set -x" now, it clutters output.

The newly churned maketars was apparently not tested in its normal
mode with no options and no parameters.

You have to be careful about the difference between $@ and $*.
In particular, "if [ -n "$@" ]" does not do what you think it will.

* replace '-arch arch' with '-m machine' (as per make(1)'s $MACHINE). e.g, i386
* add '-a arch' - $MACHINE_ARCH. e.g, m68k, powerpc
* replace long -options with short -options (with backward compat)
* ``knf''ify scripts

Change ".tar.gz" tarball suffix to ".tgz", per release(7).

add -both arg for when one puts xsrc and src builds in same DESTDIR

When deciding which md set lists to use, prefer $MACHINE over
uname -m

Add xcontrib to xlists.

Add a new option '-x11' for the xsets.

branches: 1.3.2;
Feed makeflist to sh, instead of relying on x-bits; from Andrew
Wheadon in PR 3116.

More portable versions of makeflist and maketars that can be used to
cross-make a distribution. Also avoid duplicating makeflist logic.

branches: 1.1.1;
Initial revision

More conservative umask

Set the file creation mask consistently because the files these scripts
create can become part of an iso image.

sets: enhance MAKEVERBOSE support

Add more messages for various sets operations.
Add ${_MKMSG_CREATE} to bsd.own.mk.

Enhance maketars to be a bit more verbose even with -q,
to make it more obvious why certain operations may be slow.
May help debugging PR 47562.

maketars: add comments for skipped mtree keywords

Reorder skipkeys alphabetically and add comment explaining
the purpose and why sha256 is not part of the variable.

Deprecate extsrc/, EXTSRCSRCDIR, MKEXTSRC

Remove support for building extsrc/:
1. Makefile: remove do-extsrc target.
2. build.sh: remove options -y and -Y extsrcdir.
3. distrib/sets: remove support for extsrc in various tools
including the options -L ext and -y, and the extsrc sets.
4. doc/BUILDING.mdoc: remove docs for extsrc/, EXTSRCSRCDIR, MKEXTSRC (etc)
5. bsd.own.mk and various Makefiles: remove support for extsrc/,
EXTSRCSRCDIR, MKEXTSRC.

As proposed on tech-kern and tech-userlevel on 2022-01-07
and followed up on 2022-08-21.

Move DRM-driver firmware from base to its own set, gpufw

This set is only installed on amd64,i386,evbarm.
This set is installed on minimal installs and on install media, in
case someone needs it for basic driver functionality.

Comments:
Switched to a single MK tunable for it - that is probably unneeded.

An upcoming DRM update will include even fatter firmware, and we'd
like to minimize the impact of it.

Patch the build script and toolchain to allow passing through
HOST_CPPFLAGS, HOST_CXXFLAGS, but not LIBRARY_PATH because it breaks
the build. These variables are necessary to support building NetBSD
from a GNU Guix or NixOS host, where /usr/include, /lib, and all
but /bin/sh do not exist. In many cases, support for HOST_CPPFLAGS
was incomplete. From Ryan Sundberg

Separate devicetree .dts -> .dtb building from kernel builds. They are now
part of a separate set, "dtb.tgz", and only built when MKDTB=yes. This
defaults to yes for earmv[67]* and aarch64, and no everywhere else.

Split out /rescue to its own set and adapt installers/images to add it.

This is meant to make updates safer: if something goes wrong with updating
base, we still have the old, standalone /rescue to recover from.

branches: 1.89.2;
Prepare for .tar.xz sets

Revert previous - will pass the additional args via the environment.

Allow for ${COMPRESS_PROGRAM} to hold the name of a compressor program
plus additional arguments (like -9).

branches: 1.86.10; 1.86.12;
This fixes a bug where mtree would use names & IDs from the host system
instead of from the build.

branches: 1.85.2; 1.85.4;
Obey MKREPRO_TIMESTAMP when creating sets (create all the files in the
sets with that date).

Simplify slightly

Instead of guessing wether to preserve permissions on install operations
from the METALOG variable, default them to yes and add a new argument -U to
switch preservation off.

Do not try to preserve permissions when copying to the -i installdir
in the METALOG case (it makes no sense at all).

Add a new arg to override the value of the "setlistdir" variable (that is:
where the mtree spec files are created)

We always have a valid mtree file, so always specify -M

merge d flag.

- use dest instead of DESTDIR
- factor out pax usage, and make it use only the first word from the sets

Create full sets using mtree from the flists (not just filenames).

- add xdebug set
- document debug better
- fix previous debug fallout.

Add a new "debug" set that gets built when ${MKDEBUG} is set. On evbarm:
-rw-r--r-- 1 root src 35806068 Jan 16 10:47 base.tgz
-rw-r--r-- 1 root src 57374069 Jan 16 10:48 comp.tgz
-rw-r--r-- 1 root src 101286193 Jan 16 10:49 debug.tgz
...
This allows always building the debugging code and libraries but not
installing it (although now it is placed in the default sets if ${MKDEBUG}
is set

branches: 1.74.6; 1.74.12;
add new option to build/use tools/pigz for compressing sets. for now
USE_PIGZGZIP defaults to "no". (it depends upon local <pthread.h> and -lz.)

you can set it to "yes" on most modern platforms just fine.

sh(1) style. No functional changes.

Fix syntax error on OSX 10.5.
While here, simplify handling with OPTARG using IFS.

developped with and 'go for it' dsl@