Introduce NOIEEE80211 option, by which IEEE 802.11 support is dropped from
ifconfig(8) in crunched binaries. This saves ~9KB for, e.g., m68k.

1. Remove all the special handling of variables (-d -p -P -s -S) that
were dealing with DBG (-d) LDSTATIC/NOPIE (-p), and the rest with
disabling/enabling sanitizers.
2. Use emalloc/estrdup for all the allocators instead of only some cases.
3. Add -V varspec which passes variables on the command line (as DBG
and LDSTATIC used to be passed before) instead of appending them
to the on-the-fly Makefile using -v varspec.
4. Change the distrib and rescue Makefiles to use -V instead of the removed
flags.

The motivation of this is to make variable handling consistent, less magical,
and remove the need for changing crunchgen each time we want to add disabling
an option by default.

(as proposed in tech-toolchain)

branches: 1.25.26;
Pass DBG to crunchgen so it can get into the created mk file.

branches: 1.24.18; 1.24.24;
Use ${TOOL_AWK} instead of ${AWK} or plain "awk" in make commands.
Pass AWK=${TOOL_AWK:Q} to shell scripts that use awk.

branches: 1.23.20;
Stop make wingeing when the crunchgen binary can't be located.

We need the absolute path of the crunchgen binary in order to make the
generated makefile depend on the binary.
Needed for USE_TOOLS=no builds

Fix breakage from yesterday!
Don't pass crunchgen the name of its own binary as an extra argument.
(make must have silently done this when the extra argument was .PHONY)

Make ${CRUNCHBIN}.mk depend on ${TOOL_CRUNCHGEN:[-1]} instead of .PHONY.
Recreating the makefile when the crunchgen program has changed is probably
adequate, and it allows a lot of later optimisations to still apply.
In particular:
- the final link for ramdiskbin
- the create of the embedded filesystem
- the inclusion of the ramdisk into the kernel
- compressing the kernel
Unless something else actually requires one of the above.

Make the crunchgen generated makefile regen on each run

Use
${MAKE} ${CRUNCHENV} ...
instead of
${CRUNCHENV} ${MAKE} ...
so that CRUNCHENV overrides any user-provided command-line make(1) variables.

This fixes /rescue build problems for people who pass settings such
as MKKERBEROS=yes in via the make(1) command-line.

Only support SMALLPROG_INET6 if ${USE_INET6} != "no".
Part of PR 28994.

Rename a large chunk of the make(1) variables which refer to a
program/tool from "FOO" to "TOOL_FOO". The new variables are:
TOOL_ASN1_COMPILE TOOL_CAP_MKDB TOOL_CAT TOOL_CKSUM TOOL_COMPILE_ET
TOOL_CONFIG TOOL_CRUNCHGEN TOOL_CTAGS TOOL_DB TOOL_EQN TOOL_FGEN
TOOL_GENCAT TOOL_GROFF TOOL_HEXDUMP TOOL_INDXBIB TOOL_INSTALLBOOT
TOOL_INSTALL_INFO TOOL_M4 TOOL_MAKEFS TOOL_MAKEINFO TOOL_MAKEWHATIS
TOOL_MDSETIMAGE TOOL_MENUC TOOL_MKCSMAPPER TOOL_MKESDB
TOOL_MKLOCALE TOOL_MKMAGIC TOOL_MKTEMP TOOL_MSGC TOOL_MTREE
TOOL_PAX TOOL_PIC TOOL_PREPMKBOOTIMAGE TOOL_PWD_MKDB TOOL_REFER
TOOL_ROFF_ASCII TOOL_ROFF_DVI TOOL_ROFF_HTML TOOL_ROFF_PS
TOOL_ROFF_RAW TOOL_RPCGEN TOOL_SOELIM TOOL_SUNLABEL TOOL_TBL
TOOL_UUDECODE TOOL_VGRIND TOOL_ZIC

For each, provide default in <bsd.sys.mk> of the form:
TOOL_FOO?= foo
and for the ${USETOOLS}=="yes" case in <bsd.own.mk>, provide override:
TOOL_FOO= ${TOOLDIR}/bin/${_TOOL_PREFIX}foo

Document all of these in bsd.README.

This cleans up a chunk of potential (and actual) namespace collision
within our build infrastructure, as well as improves consistency in
the share/mk documentation and provision of appropriate defaults for
each of these variables.

Don't force "make -j 1" any more, since David Laight appears to have
fixed the parallel-make issues in crunchgen generated Makefiles.

Avoid building the xxx.mk file (etc) multiple times in a parralel build.
Change name of target build from xxx.mk from 'all' to xxx.crunched to avoid
picking up baggage from bsd.own.mk and bsd.prog.mk.
Fixes toolchain/21563 (with new crunchgen binary)

Force ramdiskbin.mk to be run to update the crunched binary.
Explicitly check times on built targets so to stop make rebuilding
the later targets.
nbmake in src/distrib/i386/floppies/ramdisk-big (etc) will now
pick up changes to source files that make the crunched binary.

add multiple-include protection.
clean up some comments.

Rework how SMALLPROG was being used and how INET6 support was enabled.
There's two variables added to Makefile.crunch:

SMALLPROG If 1, add SMALLPROG=1 to CRUNCHENV. [default: 1]
This can be used by various Makefiles to
determine if a `small' version of the program
is required.

SMALLPROG_INET6 If 1 and SMALLPROG == 1, add SMALLPROG_INET6=1
to CRUNCHENV. [default: 0]
This can be used by various Makefiles to
determine if the `small' version has INET6
support enabled.

Only enable SMALLPROG_INET6 for programs on the ramdisks that have INET6 in
the appropriate kernel, and cleanup INET6 setting.
(Note: this means that sysinst won't get INET6 support if the kernel
it's running on doesn't have it).

This change saves a fair bit of space on various install images
(including i386 ramdisk-{ps2,small,tiny}) that didn't have INET6 in
the kernel but were unconditionally getting INET6 support in their tools.

- build crunchgen(1)-created Makefiles with -j 1, as they're not
parallel build safe at this time.
- invoke crunchgen with -q, as the verbose output looks ugly with "make -j N"
and in build logs.

branches: 1.9.2;
support optional CRUNCHENV - environment variables to pass to
crunchgen(1) and the make -f ${CRUNCHBIN}.mk.
defaults to SMALLPROG=1

replace _SRC_TOP_ with NETBSDSRCDIR, since the latter is the "public" version

move default PARSELISTENV and related goop into separate Makefile.parselist,
and add DISTRIBDIR to PARSELISTENV.

Add support for optional ${CRUNCHGEN_FLAGS}.
Per discussion with Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>

add missing backslash. noted by Havard Eidnes

parselist.awk
- improve comments
- provide a general purpose variable replacement; replace contents of ${FOO}
with ENVIRON["FOO"].
- deprecate @MACHINE@ and @MACHINE_ARCH@ in favour of using ${MACHINE}
and ${MACHINE_ARCH} instead
- add errx() function; like err(), but doesn't print the line number
- invoke this script with various variables defined via PARSELISTENV

*/Makefile*
- use PARSELISTENV instead of POPULATEENV, and add variables like DESTDIR
- other minor tweaks

*/list*
- replace @MACHINE@ with ${MACHINE}

- improve error checking
- add support for passing variables to parselist.awk that are used to replace
text in each line of the lists. currently supported strings:
text variable
---- --------
@MACHINE_ARCH@ MACHINE_ARCH
@MACHINE@ MACHINE

use CLEANFILES to remove cruft

distrib/common - shared files for building distribution media, initially
comprising of:

Makefile.crunch
build a crunchgen(1)ed binary from the provided lists

Makefile.image
build a tree from the provided lists, and
build an ffs file system image from that tree using makefs(8),
without requiring root privileges

parselist.awk
parse list files generating different output:
crunchgen config
mtree specfile
sh commands to populate a tree

1. Remove all the special handling of variables (-d -p -P -s -S) that
were dealing with DBG (-d) LDSTATIC/NOPIE (-p), and the rest with
disabling/enabling sanitizers.
2. Use emalloc/estrdup for all the allocators instead of only some cases.
3. Add -V varspec which passes variables on the command line (as DBG
and LDSTATIC used to be passed before) instead of appending them
to the on-the-fly Makefile using -v varspec.
4. Change the distrib and rescue Makefiles to use -V instead of the removed
flags.

The motivation of this is to make variable handling consistent, less magical,
and remove the need for changing crunchgen each time we want to add disabling
an option by default.

(as proposed in tech-toolchain)

Pass DBG to crunchgen so it can get into the created mk file.

branches: 1.24.18; 1.24.24;
Use ${TOOL_AWK} instead of ${AWK} or plain "awk" in make commands.
Pass AWK=${TOOL_AWK:Q} to shell scripts that use awk.

branches: 1.23.20;
Stop make wingeing when the crunchgen binary can't be located.

We need the absolute path of the crunchgen binary in order to make the
generated makefile depend on the binary.
Needed for USE_TOOLS=no builds

Fix breakage from yesterday!
Don't pass crunchgen the name of its own binary as an extra argument.
(make must have silently done this when the extra argument was .PHONY)

Make ${CRUNCHBIN}.mk depend on ${TOOL_CRUNCHGEN:[-1]} instead of .PHONY.
Recreating the makefile when the crunchgen program has changed is probably
adequate, and it allows a lot of later optimisations to still apply.
In particular:
- the final link for ramdiskbin
- the create of the embedded filesystem
- the inclusion of the ramdisk into the kernel
- compressing the kernel
Unless something else actually requires one of the above.

Make the crunchgen generated makefile regen on each run

Use
${MAKE} ${CRUNCHENV} ...
instead of
${CRUNCHENV} ${MAKE} ...
so that CRUNCHENV overrides any user-provided command-line make(1) variables.

This fixes /rescue build problems for people who pass settings such
as MKKERBEROS=yes in via the make(1) command-line.

Only support SMALLPROG_INET6 if ${USE_INET6} != "no".
Part of PR 28994.

Rename a large chunk of the make(1) variables which refer to a
program/tool from "FOO" to "TOOL_FOO". The new variables are:
TOOL_ASN1_COMPILE TOOL_CAP_MKDB TOOL_CAT TOOL_CKSUM TOOL_COMPILE_ET
TOOL_CONFIG TOOL_CRUNCHGEN TOOL_CTAGS TOOL_DB TOOL_EQN TOOL_FGEN
TOOL_GENCAT TOOL_GROFF TOOL_HEXDUMP TOOL_INDXBIB TOOL_INSTALLBOOT
TOOL_INSTALL_INFO TOOL_M4 TOOL_MAKEFS TOOL_MAKEINFO TOOL_MAKEWHATIS
TOOL_MDSETIMAGE TOOL_MENUC TOOL_MKCSMAPPER TOOL_MKESDB
TOOL_MKLOCALE TOOL_MKMAGIC TOOL_MKTEMP TOOL_MSGC TOOL_MTREE
TOOL_PAX TOOL_PIC TOOL_PREPMKBOOTIMAGE TOOL_PWD_MKDB TOOL_REFER
TOOL_ROFF_ASCII TOOL_ROFF_DVI TOOL_ROFF_HTML TOOL_ROFF_PS
TOOL_ROFF_RAW TOOL_RPCGEN TOOL_SOELIM TOOL_SUNLABEL TOOL_TBL
TOOL_UUDECODE TOOL_VGRIND TOOL_ZIC

For each, provide default in <bsd.sys.mk> of the form:
TOOL_FOO?= foo
and for the ${USETOOLS}=="yes" case in <bsd.own.mk>, provide override:
TOOL_FOO= ${TOOLDIR}/bin/${_TOOL_PREFIX}foo

Document all of these in bsd.README.

This cleans up a chunk of potential (and actual) namespace collision
within our build infrastructure, as well as improves consistency in
the share/mk documentation and provision of appropriate defaults for
each of these variables.

Don't force "make -j 1" any more, since David Laight appears to have
fixed the parallel-make issues in crunchgen generated Makefiles.

Avoid building the xxx.mk file (etc) multiple times in a parralel build.
Change name of target build from xxx.mk from 'all' to xxx.crunched to avoid
picking up baggage from bsd.own.mk and bsd.prog.mk.
Fixes toolchain/21563 (with new crunchgen binary)

Force ramdiskbin.mk to be run to update the crunched binary.
Explicitly check times on built targets so to stop make rebuilding
the later targets.
nbmake in src/distrib/i386/floppies/ramdisk-big (etc) will now
pick up changes to source files that make the crunched binary.

add multiple-include protection.
clean up some comments.

Rework how SMALLPROG was being used and how INET6 support was enabled.
There's two variables added to Makefile.crunch:

SMALLPROG If 1, add SMALLPROG=1 to CRUNCHENV. [default: 1]
This can be used by various Makefiles to
determine if a `small' version of the program
is required.

SMALLPROG_INET6 If 1 and SMALLPROG == 1, add SMALLPROG_INET6=1
to CRUNCHENV. [default: 0]
This can be used by various Makefiles to
determine if the `small' version has INET6
support enabled.

Only enable SMALLPROG_INET6 for programs on the ramdisks that have INET6 in
the appropriate kernel, and cleanup INET6 setting.
(Note: this means that sysinst won't get INET6 support if the kernel
it's running on doesn't have it).

This change saves a fair bit of space on various install images
(including i386 ramdisk-{ps2,small,tiny}) that didn't have INET6 in
the kernel but were unconditionally getting INET6 support in their tools.

- build crunchgen(1)-created Makefiles with -j 1, as they're not
parallel build safe at this time.
- invoke crunchgen with -q, as the verbose output looks ugly with "make -j N"
and in build logs.

branches: 1.9.2;
support optional CRUNCHENV - environment variables to pass to
crunchgen(1) and the make -f ${CRUNCHBIN}.mk.
defaults to SMALLPROG=1

replace _SRC_TOP_ with NETBSDSRCDIR, since the latter is the "public" version

move default PARSELISTENV and related goop into separate Makefile.parselist,
and add DISTRIBDIR to PARSELISTENV.

Add support for optional ${CRUNCHGEN_FLAGS}.
Per discussion with Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>

add missing backslash. noted by Havard Eidnes

parselist.awk
- improve comments
- provide a general purpose variable replacement; replace contents of ${FOO}
with ENVIRON["FOO"].
- deprecate @MACHINE@ and @MACHINE_ARCH@ in favour of using ${MACHINE}
and ${MACHINE_ARCH} instead
- add errx() function; like err(), but doesn't print the line number
- invoke this script with various variables defined via PARSELISTENV

*/Makefile*
- use PARSELISTENV instead of POPULATEENV, and add variables like DESTDIR
- other minor tweaks

*/list*
- replace @MACHINE@ with ${MACHINE}

- improve error checking
- add support for passing variables to parselist.awk that are used to replace
text in each line of the lists. currently supported strings:
text variable
---- --------
@MACHINE_ARCH@ MACHINE_ARCH
@MACHINE@ MACHINE

use CLEANFILES to remove cruft

distrib/common - shared files for building distribution media, initially
comprising of:

Makefile.crunch
build a crunchgen(1)ed binary from the provided lists

Makefile.image
build a tree from the provided lists, and
build an ffs file system image from that tree using makefs(8),
without requiring root privileges

parselist.awk
parse list files generating different output:
crunchgen config
mtree specfile
sh commands to populate a tree

Pass DBG to crunchgen so it can get into the created mk file.

branches: 1.24.18; 1.24.24;
Use ${TOOL_AWK} instead of ${AWK} or plain "awk" in make commands.
Pass AWK=${TOOL_AWK:Q} to shell scripts that use awk.

branches: 1.23.20;
Stop make wingeing when the crunchgen binary can't be located.

We need the absolute path of the crunchgen binary in order to make the
generated makefile depend on the binary.
Needed for USE_TOOLS=no builds

Fix breakage from yesterday!
Don't pass crunchgen the name of its own binary as an extra argument.
(make must have silently done this when the extra argument was .PHONY)

Make ${CRUNCHBIN}.mk depend on ${TOOL_CRUNCHGEN:[-1]} instead of .PHONY.
Recreating the makefile when the crunchgen program has changed is probably
adequate, and it allows a lot of later optimisations to still apply.
In particular:
- the final link for ramdiskbin
- the create of the embedded filesystem
- the inclusion of the ramdisk into the kernel
- compressing the kernel
Unless something else actually requires one of the above.

Make the crunchgen generated makefile regen on each run

Use
${MAKE} ${CRUNCHENV} ...
instead of
${CRUNCHENV} ${MAKE} ...
so that CRUNCHENV overrides any user-provided command-line make(1) variables.

This fixes /rescue build problems for people who pass settings such
as MKKERBEROS=yes in via the make(1) command-line.

Only support SMALLPROG_INET6 if ${USE_INET6} != "no".
Part of PR 28994.

Rename a large chunk of the make(1) variables which refer to a
program/tool from "FOO" to "TOOL_FOO". The new variables are:
TOOL_ASN1_COMPILE TOOL_CAP_MKDB TOOL_CAT TOOL_CKSUM TOOL_COMPILE_ET
TOOL_CONFIG TOOL_CRUNCHGEN TOOL_CTAGS TOOL_DB TOOL_EQN TOOL_FGEN
TOOL_GENCAT TOOL_GROFF TOOL_HEXDUMP TOOL_INDXBIB TOOL_INSTALLBOOT
TOOL_INSTALL_INFO TOOL_M4 TOOL_MAKEFS TOOL_MAKEINFO TOOL_MAKEWHATIS
TOOL_MDSETIMAGE TOOL_MENUC TOOL_MKCSMAPPER TOOL_MKESDB
TOOL_MKLOCALE TOOL_MKMAGIC TOOL_MKTEMP TOOL_MSGC TOOL_MTREE
TOOL_PAX TOOL_PIC TOOL_PREPMKBOOTIMAGE TOOL_PWD_MKDB TOOL_REFER
TOOL_ROFF_ASCII TOOL_ROFF_DVI TOOL_ROFF_HTML TOOL_ROFF_PS
TOOL_ROFF_RAW TOOL_RPCGEN TOOL_SOELIM TOOL_SUNLABEL TOOL_TBL
TOOL_UUDECODE TOOL_VGRIND TOOL_ZIC

For each, provide default in <bsd.sys.mk> of the form:
TOOL_FOO?= foo
and for the ${USETOOLS}=="yes" case in <bsd.own.mk>, provide override:
TOOL_FOO= ${TOOLDIR}/bin/${_TOOL_PREFIX}foo

Document all of these in bsd.README.

This cleans up a chunk of potential (and actual) namespace collision
within our build infrastructure, as well as improves consistency in
the share/mk documentation and provision of appropriate defaults for
each of these variables.

Don't force "make -j 1" any more, since David Laight appears to have
fixed the parallel-make issues in crunchgen generated Makefiles.

Avoid building the xxx.mk file (etc) multiple times in a parralel build.
Change name of target build from xxx.mk from 'all' to xxx.crunched to avoid
picking up baggage from bsd.own.mk and bsd.prog.mk.
Fixes toolchain/21563 (with new crunchgen binary)

Force ramdiskbin.mk to be run to update the crunched binary.
Explicitly check times on built targets so to stop make rebuilding
the later targets.
nbmake in src/distrib/i386/floppies/ramdisk-big (etc) will now
pick up changes to source files that make the crunched binary.

add multiple-include protection.
clean up some comments.

Rework how SMALLPROG was being used and how INET6 support was enabled.
There's two variables added to Makefile.crunch:

SMALLPROG If 1, add SMALLPROG=1 to CRUNCHENV. [default: 1]
This can be used by various Makefiles to
determine if a `small' version of the program
is required.

SMALLPROG_INET6 If 1 and SMALLPROG == 1, add SMALLPROG_INET6=1
to CRUNCHENV. [default: 0]
This can be used by various Makefiles to
determine if the `small' version has INET6
support enabled.

Only enable SMALLPROG_INET6 for programs on the ramdisks that have INET6 in
the appropriate kernel, and cleanup INET6 setting.
(Note: this means that sysinst won't get INET6 support if the kernel
it's running on doesn't have it).

This change saves a fair bit of space on various install images
(including i386 ramdisk-{ps2,small,tiny}) that didn't have INET6 in
the kernel but were unconditionally getting INET6 support in their tools.

- build crunchgen(1)-created Makefiles with -j 1, as they're not
parallel build safe at this time.
- invoke crunchgen with -q, as the verbose output looks ugly with "make -j N"
and in build logs.

branches: 1.9.2;
support optional CRUNCHENV - environment variables to pass to
crunchgen(1) and the make -f ${CRUNCHBIN}.mk.
defaults to SMALLPROG=1

replace _SRC_TOP_ with NETBSDSRCDIR, since the latter is the "public" version

move default PARSELISTENV and related goop into separate Makefile.parselist,
and add DISTRIBDIR to PARSELISTENV.

Add support for optional ${CRUNCHGEN_FLAGS}.
Per discussion with Izumi Tsutsui <tsutsui@ceres.dti.ne.jp>

add missing backslash. noted by Havard Eidnes

parselist.awk
- improve comments
- provide a general purpose variable replacement; replace contents of ${FOO}
with ENVIRON["FOO"].
- deprecate @MACHINE@ and @MACHINE_ARCH@ in favour of using ${MACHINE}
and ${MACHINE_ARCH} instead
- add errx() function; like err(), but doesn't print the line number
- invoke this script with various variables defined via PARSELISTENV

*/Makefile*
- use PARSELISTENV instead of POPULATEENV, and add variables like DESTDIR
- other minor tweaks

*/list*
- replace @MACHINE@ with ${MACHINE}

- improve error checking
- add support for passing variables to parselist.awk that are used to replace
text in each line of the lists. currently supported strings:
text variable
---- --------
@MACHINE_ARCH@ MACHINE_ARCH
@MACHINE@ MACHINE

use CLEANFILES to remove cruft

distrib/common - shared files for building distribution media, initially
comprising of:

Makefile.crunch
build a crunchgen(1)ed binary from the provided lists

Makefile.image
build a tree from the provided lists, and
build an ffs file system image from that tree using makefs(8),
without requiring root privileges

parselist.awk
parse list files generating different output:
crunchgen config
mtree specfile
sh commands to populate a tree

Pull up revision 1.10 (requested by lukem):
- build crunchgen(1)-created Makefiles with -j 1, as they're not
parallel build safe at this time.
- invoke crunchgen with -q, as the verbose output looks ugly with "make -j N"
and in build logs.

Sync with HEAD.

Rebase to HEAD as of a few days ago.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")