crunchgen: fix "keep" for an ELF world, break it out

"keep" currently adds a leading underscore, which hasn't been useful or
accurate since a.out days. Preserve the symbol name as it's given
rather than mangle it to match ELF-style symbol names.

This was partially fixed back in
6cd35234a092d ("Assume ELF-style symbol names now.") for crunchgen, but
the keeplist wasn't changed to match it.

While we're here, break it out to bsd.crunchgen.mk for later use in
bsdbox.

Reviewed by: adrian, imp
Differential Revision: https://reviews.freebsd.org/D42499

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

rescue: Restore 'make depend' call to fix WITH_META_MODE after r334008.

The rescue/crunchgen build avoids linking binaries for the objects it is
building by doing 'make foo.o bar.o' rather than 'make all'. This breaks the
implicit 'beforebuild: depend' dependency which ensured that all source files
were generated and up-to-date before building the object files. This
manifested as a WITH_META_MODE build problem for bin/sh in the rescue build
with syntax.{c,h} not properly being regenerated or recognized as changed in
the dependency graph.

Sponsored by: Dell EMC
MFC after: 1 week
Reported by: many

No need to run 'make depend' with FAST_DEPEND logic in-tree.

Sponsored by: Dell EMC Isilon

AUTO_OBJ: No need to tree-walk with 'make obj' here.

Sponsored by: Dell EMC Isilon

Pass down any PROG-specific buildopts for forwarded targets.

This currently has no impact.

Sponsored by: Dell EMC Isilon

Add support for CRUNCH_LIBS_ for specifying a lib only for 1 prog.

Sponsored by: Dell EMC Isilon

AUTO_OBJ can work for crunchgen build-tools.

Sponsored by: Dell EMC Isilon

Disable MK_AUTO_OBJ as a make argument like r325078.

This avoids needlessly trying to enable it opportunistically.

Sponsored by: Dell EMC Isilon

Quote ${MAKE} when passing in env in case it contains spaces.

Downstream we are wrapping MAKE with a limits(1) call which
interferes with these non-quoted cases.

Sponsored by: Dell EMC Isilon

META_MODE: Fix build-tools still sometimes rebuilding during target build.

In a cross-build, the build-tools are native host binaries. We do not
want to rebuild them when building for the target. Bmake previously
did not support checking .NOMETA on an existing target, so .NOMETA_CMP
was used here. However, .NOMETA_CMP still triggers meta mode conditions
if the number of commands or the command changes. In r312467 the paths
to build ncurses files were modified and thus triggered meta mode to
rebuild the build tools (make_keys, make_hash) in ncurses during the
target build. Bmake 20160604 committed in r301462 changed .NOMETA to
also skip meta mode logic for an existing .meta file as well, thus it
is now the proper fix here.

I explored moving the build-tools output to WORLDTMP/tools with
relatively good success, but have concerns that doing so would be
problematic for downstream vendors who use LOCAL_TOOL_DIRS and
expect the tools to be in current OBJDIR for the target. It also
adds more complexity into finding the tools during target build
and handling of where they are for rescue/rescue and
mkcsmapper_static/mkesdb_static which should really not be connected in
build-tools anyway.

MFC after: 2 weeks
Reported by: many
Sponsored by: Dell EMC Isilon

DIRDEPS_BUILD: Include crunched object meta files for gendirdeps.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

META_MODE/DIRDEPS_BUILD: Fix various issues with crunch builds.

- DIRDEPS_BUILD: Fix crunchgen builds losing their library dependencies
on a nop-rebuild.
- META_MODE: Fix not rebuilding various crunch.mk targets if their .meta
files warrant a rebuild. They were lacking .meta files previously.
This adds .NOMETA to the crunch objects being used since they are
already built. Bmake was forcing a rebuild on them since their
.meta files were not in the expected place; there is no reason to
rebuild them.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

META_MODE: Don't expect .meta files for side-effect generated files.

This is similar to r301285.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

WITH_META_MODE: Fix rescue rebuilding build-tools.

This is the same issue as r297997.

Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division

DIRDEPS_BUILD: Fix and hookup rescue/rescue to the build.

Sponsored by: EMC / Isilon Storage Division

WITH_AUTO_OBJ: Fix crunchgen builds.

Passing MAKEOBJDIRPREFIX to the main prog build (rescue) would confuse
WITH_AUTO_OBJ and cause it to create a recursed object directory that
then broke the actual prog build. This is normally not a problem since
we do not call 'make -f prog.mk obj' before building anything in it.

Crunchgen(1) also assumes that if -o is not passed then if an object
directory does not already exist then it should build in the source
directories. The normal buildworld process will have already ran 'make
obj' in each of the component directories so this is not a problem.
With WITH_AUTO_OBJ though this is not the case. So we must tell
crunchgen(1) that MK_AUTO_OBJ=yes will create the directory and to not
require it be present before generating its Makefile.

Sponsored by: EMC / Isilon Storage Division

Avoid redundant 'make objs' tree-walk when building the main prog.

The main prog has a dependency on the submake targets to ensure they are
built. From bsd.crunchgen.mk though we already have our own dependency
on 'make objs' so there is no need for another one. Crunchgen(1) is
doing the right thing here so it is not modified.

This also prevents the CC fix tainting the submake environment with
META_MODE and causing rebuilds. The CC passed is is only intended for
the main prog itself.

Sponsored by: EMC / Isilon Storage Division

Pass CC/CXX/CFLAGS/CXXFLAGS/LDFLAGS to the main crunch exec build.

This fixes --sysroot and other CFLAGS/LDFLAGS not being respected
in the crunchgen build since it is not including bsd.sys.mk and
other files. For example, this fixes building rescue itself without
--sysroot and other CFLAGS.

Sponsored by: EMC / Isilon Storage Division

WITH_META_MODE: Only expect a .meta file for the main target.

Since multiple files are generated from one build command, only
the first to run will actually generate a .meta file. This fix
prevents 'required but missing' rebuilds on each target.

Sponsored by: EMC / Isilon Storage Division

DIRDEPS_BUILD: Don't show finished stats, which confuses crunchgen(1).

It would show 'make error:' lines for each auto object directory created.

Sponsored by: EMC / Isilon Storage Division

DIRDEPS_BUILD: Fix some things building in MAKELEVEL0.

For PROGS this was recursing twice since MAKELEVEL0 is for 'dirdeps'
which then really builds in a sub-make.

Sponsored by: EMC / Isilon Storage Division

There seems to be no reason to duplicate CANONICALOBJDIR logic from bsd.obj.mk,
which is included for crunchgen builds.

No change in build output occurred with this change.

Sponsored by: EMC / Isilon Storage Division

Add bsd.crunchgen.mk to bsd.README.

MFC after: 1 week

Fix regression from r289734 that caused crunchgen "subdirs" to not be
properly recursed.

The .for loop was defining a ${__dir} variable that was being set at a
different evaluation time than the target itself, so every 'cd ${__dir}'
became the last value that was in ${__dir}. This resulted in 'make obj'
not properly being ran in the tree that would leave .depend files
scattered around when 'make all' was ran in rescue/.

To fix this, define a CRUNCH_SRCDIR_* for every prog if it does not
already have one and then use that variable in every relevant place.
This allows simplifying some logic as well.

Reported by: emaste
X-MFC-With: r289734
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division

Fix incorrect defined() usage from style clean up in r289735.

Submitted by: ngie
MFC after: 2 weeks
X-MFC-With: r289735
Sponsored by: EMC / Isilon Storage Division

Fix style. Namely use {} rather than ().

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Rewrite crunchgen target handling for progs so that it can be parallelized.

This covers 'clean', 'cleandepend', 'cleandir', 'obj', 'objlink' and
'build-tools'.

This uses the same method as bsd.subdir.mk.

MFC after: 2 weeks
X-MFC-With: r289731
Sponsored by: EMC / Isilon Storage Division

Clean up some bsd.crunchgen.mk issues.

- Remove handling of 'make -P' since that is for fmake only.
- Add '+' where appropriate for sub-make calls.
- Pass MK_TESTS=no to all of the sub-makes to prevent recursing into test
directories for targets such as 'obj', 'clean', 'depend', etc.

Sponsored by: EMC / Isilon Storage Division

Use ${CRUNCHGEN} rather than bare word.

Reviewed by: brooks

NO_DEBUG_FILES -> MK_DEBUG_FILES=no in last remaining place.

Add a new knob WITH_DEBUG_FILES to control the building of standalone
debug files for userland programs and libraries. The "-g" debug flag
is automatically applied when WITH_DEBUG_FILES is set.

The debug files are now named ${prog}.debug and ${shlib}.debug for
consistency with other systems and documentation. In addition they are
installed under /usr/lib/debug, to simplify the process of installing
them if needed after a crash. Users of bsd.{prog,lib}.mk outside of the
base system place the standalone debug files in a .debug subdirectory.
GDB automatically searches both of these directories for standalone
debug files.

Thanks to everyone who contributed changes, review, and testing during
development.

Add support for bmake. This includes:
1. Don't do upgrade_checks when using bmake. As long as we have WITH_BMAKE,
there's a bootstrap complication in ths respect. Avoid it. Make the
necessary changes to have upgrade_checks work wth bmake anyway.
2. Remove the use of -E. It's not needed in our build because we use ?= for
the respective variables, which means that we'll take the environment
value (if any) anyway.
3. Properly declare phony targets as phony as bmake is a lot smarter (and
thus agressive) about build avoidance.
4. Make sure CLEANFILES is complete and use it on .NOPATH. bmake is a lot
smarter about build avoidance and should not find files we generate in
the source tree. We should not have files in the repository we want to
generate, but this is an easier way to cross this hurdle.
5. Have behavior under bmake the same as it is under make with respect to
halting when sub-commands fail. Add "set -e" to compound commands so
that bmake is informed when sub-commands fail.
6. Make sure crunchgen uses the same make as the rest of the build. This
is important when the make utility isn't called make (but bmake for
example).
7. While here, add support for using MAKEOBJDIR to set the object tree
location. It's the second alternative bmake looks for when determining
the actual object directory (= .OBJDIR).

Submitted by: Simon Gerraty <sjg@juniper.net>
Submitted by: John Van Horne <jvanhorne@juniper.net>

Sync FreeBSD's bmake branch with Juniper's internal bmake branch.

Requested by: Simon Gerraty <sjg@juniper.net>

Ensure crunchen uses the same make binary as the rest of the build.

Submitted by: Simon Gerraty <sjg@juniper.net>

Allow crunchgen binary link generation to be disabled.

If CRUNCH_GENERATE_LINKS is set to "no", then no links will be
generated.

This defaults to "yes" so things like release crunch building
still works.

Re-enable generating links.

Break out the rules which generate crunchgen'ed binaries into a separate
.mk file so they can be reused.

Introduce a new option, CRUNCH_BUILDTOOLS, which lists the binaries that
require tools built in the local architecture. sh and csh both require this.
It was previously hardcoded in rescue/rescue/Makefile .

Introduce a new option, CRUNCH_SHLIBS, which lists the shared libraries
to link against. These override the static libraries listed in CRUNCH_LIBS.
Some build environments may wish to use a handful of shared libraries
(eg libc.so) so other small, dynamic binaries can be run in the environment.

Remove the now-shared code from rescue/rescue/Makefile and introduce the
CRUNCH_BUILDTOOLS option for the above shells.