MFC r322951:

Respect MK_TCSH with build-tools and native-xtools

This helps reduce the WORLDTMP footprint slightly.

Based on a patch I submitted 5 years ago to GNATS.

PR: 174051
Relnotes: yes (anyone who cross-builds with MK_TCSH=yes will run into
build failures if the host doesn't have tcsh(1))
Reminded by: Fabian Keil <fk@fabiankeil.de>

MFC r263778:

Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
all the SUBDIR entries in parallel, instead of serially. Apply this
option to a selected number of Makefiles, which can greatly speed up the
build on multi-core machines, when using make -j.

This can be extended to more Makefiles later on, whenever they are
verified to work correctly with parallel building.

I tested this on a 24-core machine, with make -j48 buildworld (N = 6):

before stddev after stddev
======= ====== ======= ======
real time 1741.1 16.5 959.8 2.7
user time 12468.7 16.4 14393.0 16.8
sys time 1825.0 54.8 2110.6 22.8

(user+sys)/real 8.2 17.1

E.g. the build was approximately 45% faster in real time. On machines
with less cores, or with lower -j settings, the speedup will not be as
impressive. But at least you can now almost max out a machine with
buildworld!

Submitted by: jilles

MFC r263833:

Enable parallel building for gnu/usr.bin and usr.bin/clang too.

Sync sh(1) in stable/10 to head.

This is a MFC of all the commits listed below.

My original goal of this change was to only merge the move of the tests
from tools/regression/bin/ into the new layout (which include tests for
sh(1) and other tools as well). However, doing so is tricky due to the
ongoing work in sh(1) and, especially, the many changes to its tests
since stable/10 was first branched.

Merging everything is the simplest way to achieve this goal and, as a
bonus point, we get various fixes and miscellaneous improvements into
the branch.

Per jilles' suggestion, I'm avoiding the merge of a couple of changes
(r256850 and r257506) that required depending kernel changes. I'm also
avoiding very recent changes that have not had a long enough time to be
validated in current.

This is "make tinderbox" clean.

r256735 sh: Remove one syscall when waiting for a foreground job.
r257399 sh: Allow trapping SIGINT/SIGQUIT after ignore because of '&'.
r257504 sh: Reorder union node to reduce its size on 64-bit platforms.
r257920 sh: Add a test case for would-be assignments that are not due to quoting.
r257929 sh: Properly quote alias output from command -v.
r258489 sh: Add tests for the </dev/null implicit in a background command.
r258533 sh: Add more tests for the </dev/null implicit in a background command.
r258535 sh: Make <&0 disable the </dev/null implicit in a background command.
r258776 sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro.
r259047 sh: Split set -x output into a separate function.
r259210 Migrate tools/regression/bin/ tests to the new layout.
r259844 sh: Remove an unused variable.
r259846 sh: Initialize OPTIND=1 even if it came from the environment.
r259874 sh: Simplify code related to PPID variable.
r259946 sh: Don't check input for non-whitespace if history is disabled.
r260246 sh(1): Discourage use of -e.
r260506 Run the sh(1) and test(1) tests as unprivileged.
r260586 Mark the bin/pax tests as requiring perl.
r260634 Use TAP_TESTS_PERL to register the legacy_test in bin/pax.
r260635 Replace hand-crafted Kyuafiles with automatic generation.
r260654 sh: Remove SIGWINCH handler and just check for resize before every read.
r261121 sh: Add test for nested alias.
r261125 sh: Solve the alias recursion problem in a less hackish way.
r261141 sh: Do not depend on parse/execute split in new alias test.
r261160 sh: Add tests for alias names after another alias.
r261192 sh: Allow aliases to force alias substitution on the following word.
r262533 sh: Make expari() static.
r262565 sh: Do not corrupt internal representation if LINENO inner expansion fails.
r262697 sh: Simplify expari().

Reviewed by: jilles

MFC r263778:

Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
all the SUBDIR entries in parallel, instead of serially. Apply this
option to a selected number of Makefiles, which can greatly speed up the
build on multi-core machines, when using make -j.

This can be extended to more Makefiles later on, whenever they are
verified to work correctly with parallel building.

I tested this on a 24-core machine, with make -j48 buildworld (N = 6):

before stddev after stddev
======= ====== ======= ======
real time 1741.1 16.5 959.8 2.7
user time 12468.7 16.4 14393.0 16.8
sys time 1825.0 54.8 2110.6 22.8

(user+sys)/real 8.2 17.1

E.g. the build was approximately 45% faster in real time. On machines
with less cores, or with lower -j settings, the speedup will not be as
impressive. But at least you can now almost max out a machine with
buildworld!

Submitted by: jilles

MFC r263833:

Enable parallel building for gnu/usr.bin and usr.bin/clang too.

Sync sh(1) in stable/10 to head.

This is a MFC of all the commits listed below.

My original goal of this change was to only merge the move of the tests
from tools/regression/bin/ into the new layout (which include tests for
sh(1) and other tools as well). However, doing so is tricky due to the
ongoing work in sh(1) and, especially, the many changes to its tests
since stable/10 was first branched.

Merging everything is the simplest way to achieve this goal and, as a
bonus point, we get various fixes and miscellaneous improvements into
the branch.

Per jilles' suggestion, I'm avoiding the merge of a couple of changes
(r256850 and r257506) that required depending kernel changes. I'm also
avoiding very recent changes that have not had a long enough time to be
validated in current.

This is "make tinderbox" clean.

r256735 sh: Remove one syscall when waiting for a foreground job.
r257399 sh: Allow trapping SIGINT/SIGQUIT after ignore because of '&'.
r257504 sh: Reorder union node to reduce its size on 64-bit platforms.
r257920 sh: Add a test case for would-be assignments that are not due to quoting.
r257929 sh: Properly quote alias output from command -v.
r258489 sh: Add tests for the </dev/null implicit in a background command.
r258533 sh: Add more tests for the </dev/null implicit in a background command.
r258535 sh: Make <&0 disable the </dev/null implicit in a background command.
r258776 sh: Prefer memcpy() to strcpy() in most cases. Remove the scopy macro.
r259047 sh: Split set -x output into a separate function.
r259210 Migrate tools/regression/bin/ tests to the new layout.
r259844 sh: Remove an unused variable.
r259846 sh: Initialize OPTIND=1 even if it came from the environment.
r259874 sh: Simplify code related to PPID variable.
r259946 sh: Don't check input for non-whitespace if history is disabled.
r260246 sh(1): Discourage use of -e.
r260506 Run the sh(1) and test(1) tests as unprivileged.
r260586 Mark the bin/pax tests as requiring perl.
r260634 Use TAP_TESTS_PERL to register the legacy_test in bin/pax.
r260635 Replace hand-crafted Kyuafiles with automatic generation.
r260654 sh: Remove SIGWINCH handler and just check for resize before every read.
r261121 sh: Add test for nested alias.
r261125 sh: Solve the alias recursion problem in a less hackish way.
r261141 sh: Do not depend on parse/execute split in new alias test.
r261160 sh: Add tests for alias names after another alias.
r261192 sh: Allow aliases to force alias substitution on the following word.
r262533 sh: Make expari() static.
r262565 sh: Do not corrupt internal representation if LINENO inner expansion fails.
r262697 sh: Simplify expari().

Reviewed by: jilles