Copy stable/11@r303970 to releng/11.0 as part of the 11.0-RELEASE
cycle.

Prune svn:mergeinfo from the new branch, and rename it to RC1.

Update __FreeBSD_version.

Use the quarterly branch for the default FreeBSD.conf pkg(8) repo and
the dvd1.iso packages population.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r303274:

Don't disable binutils/elftoolchain bootstrapping with external compiler.

Approved by: re (kib)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

WITH_SYSTEM_COMPILER: Avoid using GCC4.8+ logic on native GCC4.2 platforms.

As the XXX notes, these should really be checking MK_GNUCXX since there is
already a version check in share/mk/src.opts.mk to disable it. Fixing that
here is more complex though. This could also be using X_COMPILER_FEATURES
but uses X_COMPILER_VERSION to keep in sync with the src.opts.mk logic.

Tested by: andreast
Sponsored by: EMC / Isilon Storage Division
Approved by: re (gjb)

Fix native powerpc64 build of lib32 with in-tree GCC.

- This was broken by r300350 and r300885.
- Add some comments around the external GCC logic since it is spread out
and in need of some cleanup.
- The problem was that X_COMPILER_TYPE is always defined from CC->XCC's
default, so if /usr/bin/cc is GCC (as it is on native powerpc64) then
X_COMPILER_TYPE was getting GCC and triggering the external logic in
Makefile.libcompat. It was intended to always provide -isystem with
GCC since --sysroot is used into the lib32 sysroot which won't modify
the header path without the -isystem. The use of the libc++/std=c++11
override was only intended to be used for external compilers though
(more accurately GCC 4.8+ but that's a separate assumption to
cleanup). Apply the same logic from Makefile.inc1 to Makefile.libcompat
to only add the libc++ override when needed for external compilers.

Pointyhat to: bdrewery
Tested with: native ppc64 (swills), universe, ppc64 xtoolchain,
amd64 xtoolchain, sparc64 cross-build of ppc64 (host GCC 4.2)
Reported by: andreast, swills
Approved by: re (gjb)
Sponsored by: EMC / Isilon Storage Division

Use -L to specify compat32 library paths instead of -Y

-Y is an uncommon linker option that is rather similar to -L. In
discussion with Peter it seems early amd64 development might have
required the -Y-specific behaviour, but it is no longer necessary.

Switch to -L which is more widely supported and much more commonly
used, to make it easier to link the FreeBSD base system with linkers
other than ld.bfd.

Submitted by: Rafael Ávila de Espíndola
Differential Revision: https://reviews.freebsd.org/D6681

Libcompat: Swap CXX/CFLAGS.

This is the same as done for the native build in r300770 to ensure that
the libc++ build reads from SYSROOT/usr/include/c++/v1 before reading
from SYSROOT/usr/include.

GCC External: Revert r300886, r300904, r300917, r300918

The fix in r300873 is mostly enough. A fix for lib32 will be
committed.separately.

Libcompat: Set build tools in environment rather than make overrides.

This allows the CXX hack in r300917 for external GCC to work for
the lib32 build. It is also the same pattern as the native
build uses by adding the tools into CROSSENV for external
toolchain, rather than make overrides.

Sponsored by: EMC / Isilon Storage Division

Move external GCC compiler hacks to bsd.sys.mk.

This allows respecting -nostdinc, -nostdinc++ and -nostdlib before
making the decision to add in -isystem, etc. The -isystem flags
are problematic for building lib/libc++ and lib/libcxxrt which wants
to only use its own headers.

More information the need of these flags can be found at
https://gcc.gnu.org/ml/gcc/2016-03/msg00219.html

This also reverts r300873.

Sponsored by: EMC / Isilon Storage Division

Libcompat: Only pass -isystem =/usr/include for external GCC.

This is the same as the main build logic. GCC with a cross-compiler
requires using -isystem to =/usr/include to get the search order
correct.

Reported by: dim, asomers
Sponsored by: EMC / Isilon Storage Division

Set CPP from XCPP for the libcompat build.

Submitted by: Mark Millard <markmi@dsl-only.net>

META_MODE: Don't rebuild build-tools targets during normal build.

This avoids 'build command changed' due to CFLAGS/CC changes during the
normal build. Without this the build-tools targets end up rebuilding
for the *target* rather than keeping the native versions built in
build-tools.

Sponsored by: EMC / Isilon Storage Division

Remove leftover _LDSCRIPTROOT missed in r297270.

Sponsored by: EMC / Isilon Storage Division

LIB32+WITHOUT_TOOLCHAIN: Fix build by staging includes.

This is the same fix as r297281 for the normal build.

Sponsored by: EMC / Isilon Storage Division

Remove the old depend (mkdep) code and make FAST_DEPEND the one true way.

Reviewed by: emaste, hselasky (partial), brooks (brief)
Discussed on: arch@
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5742

Fix libcompat not handling some external toolchain flags.

- Use libc++ with GCC.
- Use CROSS_BINUTILS_PREFIX with -B (r280980 addressed this mostly already)

Sponsored by: EMC / Isilon Storage Division

Build libcompat (lib32) with a --sysroot pointing into its stage directory.

This overrides the cross-compiler's default sysroot to use the WORLD32's
sysroot for building the lib32 libraries. Previously the cross-compiler
would default the sysroot to the 64bit WORLDTMP and -B/-L/-isystem flags
were used to build using the lib32 files. This leads to multiple issues
discussed later. Some extra headers are now needed to be staged since the
64bit WORLDTMP is not referenced at all for headers. The 64bit WORLDTMP
is still used via PATH for build tools. Overriding the default
target/arch is retained in the CC/CXX overrides.

This allows reverting the LDSCRIPT rewriting in installworld from r296921 and
r235122, thus allowing read-only objdirs to work for installing again.

This removes the need for _LDSCRIPTROOT.

This allows progressing the change to always use --sysroot for the build
rather than only relying on the cross-compiler's default sysroot. The
work for that is in D3970 and needed to resolve WITHOUT_CROSS_COMPILER
not using a --sysroot [1].

PR: 196193 [1]
Sponsored by: EMC / Isilon Storage Division

LIBRARIES_ONLY should only be defined during install32.

r245561 added it to prevent extra files from being installed during
the install32 phase (to prevent duplicates in the meta log with -DNO_ROOT).
The flag should not be passed during build32 though since it may
prevent staging of includes during the 'make includes' phase on
library directories.

Sponsored by: EMC / Isilon Storage Division

Explicit 'make obj' for library dirs in build32 is not needed.

In r138291 some directories were explicitly made to run 'make obj',
but each target in 'make libraries' already runs 'make obj' for
each library since r161580.

Sponsored by: EMC / Isilon Storage Division

Use the newly minted Makefile.libcompat to implement libsoft libraries
for the armv6 ABI switch. This also make WITH_LIBSOFT functional on
the arm platform. As a transition thing, this seems to work even
without switching the ABI (we basically build the same libraries
twice when MK_LIBSOFT=yes until the ABI cut over next
month). MK_LIBSOFT remains default no.

Follow-up r296709: Fix build32 not properly building all libraries.

Pointyhat to: bdrewery
Reported by: antoine

Move Makefile.lib32 to Makefile.libcompat and generalize it.

This is in preparation for LIBSOFT.

This file only supports *1* LIBCOMPAT value currently and must be capitalized.
In Makefile.libcompat given LIBCOMPAT=FOO there can be values set for
LIBFOOCFLAGS, LIBFOOCPUFLAGS, LIBFOOWMAKEENV, LIBFOOWMAKEFLAGS, LIBFOOCPUFLAGS,
and LIBFOODTRACE. These will have the standard cross-build values appended
onto them.

This could be extended to support multiple libcompat libraries in the future
once there is a need.

Reviewed by: imp
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5612

Factor out lib32 generation to its own file. This is prep for a
similar Makefile.libsoft which will do the same for armv6 soft fp API
libraries in prep for pulling the trigger on moving to armv6 hard
float. Once there's two files, I'll work with bdrewery@ to merge the
two files as they are mostly the same. The high rate of churn for
Makefile* makes it quite difficult to make progress out of tree.

Differential Review: https://reviews.freebsd.org/D5566

Don't ever create object directories here with MK_AUTO_OBJ.

Sponsored by: EMC / Isilon Storage Division

Upgrade our copies of clang, llvm, lldb and compiler-rt to 3.8.0
release.

Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.

Release notes for llvm and clang will soon be available here:
<http://llvm.org/releases/3.8.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.8.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Roman Divacky, Davide Italiano and Antoine Brodin
for their help.

Relnotes: yes

Conditionalize building makewhatis(1) for upgrades that need it.

Remove building of the legacy makewhatis(1) since it was only needed for
6.0 upgrades.

On my 2.2 GHz system libsqlite3 takes 60-100 seconds to build, which due to
its serialized nature can hold up the build waiting on it to finish in
bootstrap-tools.

makewhatis(1) was only required to be a build tool to support upgrades
from 6.0 which was properly removed in r208324 but later reverted due to
installworld using it. The installworld issue was fixed in r275622
when it was added to ITOOLS. A BOOTSTRAPPING check was missed when
makewhatis(1) was replaced with mandoc in r283777.

Sponsored by: EMC / Isilon Storage Division

Don't add duplicates from LOCAL_LIB_DIRS.

This causes parallel failures.

Sponsored by: EMC / Isilon Storage Division

Fix over/under-linking in contrib/ofed.

Sponsored by: EMC / Isilon Storage Division

NO_ROOT: Create the METALOG directory on installworld/distributeworld.

When not using NO_ROOT the DESTDIR is implicitly created by the
distrib-dirs call. However with NO_ROOT it is trying to write
to the METALOG right away before running distrib-dirs which
may fail.

Sponsored by: EMC / Isilon Storage Division

Add dd as a bootstrap tool, for status= support

In r295757 I added status=none to buildworld's use of dd, but FreeBSD
versions before April 2014 lack support.

Convert casperd(8) daemon to the libcasper.
After calling the cap_init(3) function Casper will fork from it's original
process, using pdfork(2). Forking from a process has a lot of advantages:
1. We have the same cwd as the original process.
2. The same uid, gid and groups.
3. The same MAC labels.
4. The same descriptor table.
5. The same routing table.
6. The same umask.
7. The same cpuset(1).
From now services are also in form of libraries.
We also removed libcapsicum at all and converts existing program using Casper
to new architecture.

Discussed with: pjd, jonathan, ed, drysdale@google.com, emaste
Partially reviewed by: drysdale@google.com, bdrewery
Approved by: pjd (mentor)
Differential Revision: https://reviews.freebsd.org/D4277

FAST_DEPEND: Skip 'make depend' for buildworld and kernel since it is auto-ran now.

Sponsored by: EMC / Isilon Storage Division

Don't descend into test directories for distribution->installconfig.

Sponsored by: EMC / Isilon Storage Division

Add libpe for elfcopy(1) PE/COFF support

Sponsored by: The FreeBSD Foundation

Simplify running the FreeBSD test suite

Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.

`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.

Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.

Document `make check` and `make checkworld` in build(7).

Other minor changes:

- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
`make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
not found

The MFC is [partly] contingent on other build related changes being MFCed.

Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

Deduplicate distrib-dirs logic from r289086 in distribution.

This does it correctly this time, rather than the incorrect version in
r295167.

Sponsored by: EMC / Isilon Storage Division

Revert r295167 at bdrewery's request

$ svn merge -c -295167 .

JHB reports Navdeep reports that it breaks distribution and etcupdate.

Approved by: bdrewery

Deduplicate distrib-dirs logic from r289086 in distribution.

Sponsored by: EMC / Isilon Storage Division

Adjust install .WAITs for lib/ and etc/ to allow parallelization more.

Only 'installworld' needs to be protected and only when not using
-DNO_ROOT, which implies not installing to / and not needing the
lib dependency protections.

Sponsored by: EMC / Isilon Storage Division

Correct a comment.

Submitted by: jhb

Chase r292622: Update path to ioctl.c for incremental build hack.

Sponsored by: EMC / Isilon Storage Division

Fix upgrading from OSVERSION 1000002-1000032 after r288829.

r288829 states that lex requires the latest m4, but was not always building it.
Move lex to the same logic as m4 since they are closely tied now.

MFC after: 3 days
Sponsored by: EMC / Isilon Storage Division
Reported by: Slawa Olhovchenkov <slw@zxy.spb.ru>

Provide knob NO_INSTALLEXTRAKERNELS. If defined, extra kernels in KERNCONF
won't be installed, only the first one would.

Merge ^/projects/release-install-debug:

- Rework MANIFEST generation and parsing via bsdinstall(8).
- Allow selecting debugging distribution sets during install.
- Rework bsdinstall(8) to fetch remote debug distribution sets
when they are not available on the local install medium.
- Allow selecting additional non-GENERIC kernels during install.
At present, GENERIC is still required, and installed by default.

Tested with: head@r293203
Sponsored by: The FreeBSD Foundation

Fixup native-xtools target for poudriere cross build jails after svn
r291955 by excluding all debug files.

Reported by: swills

Reviewed by: emaste

Always expose LOCALBASE, not just when CROSS_TOOLCHAIN is defined

Instead of using which(1) to look for doxygen, look for it in <LOCALBASE>/bin .
$PATH gets mangled by make buildenv, etc so it's better to just be explicit
about the path if someone uses that for instance.

Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

Follow-up r290423: Don't use CSH for buildenv shell.

It does not properly import PATH; the PATH is reset by included profile
files on startup which breaks the biggest feature of buildenv (using
sysrooted cc from WORLDTMP)

Spotted by: smh, kib
Sponsored by: EMC / Isilon Storage Division

Split KNOWN_ARCHES so we have one per line to simplify merging future
architectures.

Start support for the RISC-V 64-bit architecture developed by UC Berkeley.

RISC-V is a new ISA designed to support computer research and education, and
is now become a standard open architecture for industry implementations.

This is a minimal set of changes required to run 'make kernel-toolchain'
using external (GNU) toolchain.

The FreeBSD/RISC-V project home: https://wiki.freebsd.org/riscv.

Reviewed by: andrew, bdrewery, emaste, imp
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5
Differential Revision: https://reviews.freebsd.org/D4445

Fix some makeman issues.

- Don't bother looking up REVISION/BRANCH/etc from release/, or the
CPUTYPE check, as these are not used for makeman and wastes time. The also
invokes auto.obj.mk after I reverted auto.obj.mk ignoring -V in r291312.
- Don't modify CC or PATH when WITH_CCACHE_BUILD or WITH_META_MODE is enabled
as it leads to bsd.compiler.mk errors.

Sponsored by: EMC / Isilon Storage Division

Fix spelling of internal hack.

Reported by: ngie

bsd.subdir.mk: Only recurse on called targets, rather than dependencies.

This is to fix 'make all' causing it to recurse on both 'all' and 'buildconfig'
due to 'buildconfig' being in ALL_SUBDIR_TARGETS and being a dependency of
'all'.

This now adds all of the '*includes', '*files' targets as subdir targets,
allowing them to recurse.

This also removes the need for some 'realinstall' hacks in bsd.subdir.mk since
it no longer recurses; only 'install' will recurse and call the proper
'beforeinstall', 'realinstall', and 'afterinstall' in each sub-directory.

This fixes 'make includes' and 'make files' to not be a rerolled ${MAKE}
sub-shell but to rather just recurse on 'inclues' and 'files'. This avoids
various issues such as the one fixed in r289462. As such revert Makefile.inc1
back to using 'includes' which avoids an extra tree walk and parallelizes
the includes phases better.

Makefile.inc1 includes a guard so that 'make all' will not use SUBDIR_PARALLEL,
added in r289438. This is so users do not get a probably broken build if they
run 'make all' from the top-level. Before the change in this commit, the
workaround for 'make everything' was 'par-all' which would depend on 'all' and
cause a proper parallel recursion. Now that will not work so a new
_PARALLEL_SUBUDIR_OK is used to allow it.

This is still part of an effort to combine bsd.(files|incs|confs).mk and move
some of its logic out of bsd.subdir.mk, as attempted in r289282 and reverted in
r289331. This commit fixes the problems found there which was mostly double
recursing during 'includes' which would recurse on itself and 'buildincludes'
and 'installincludes', all in parallel. The logic is still in bsd.subdir.mk
for now.

I've been cautious about this commit but have experienced no breakage on the
tree except for the 'par-all' case which was already a hack. If something foo
is depending on something bar that should recurse, it is very likely that the
foo target is being recursed on already meaning that bar will still effectively
recurse once sub-directories call foo.

Discussed on: arch@
MFC after: never
Sponsored by: EMC / Isilon Storage Division

Add NO_INSTALLKERNEL to undo the assumption that the first KERNCONF will be
installed as "kernel". This is relevant for packaging of the kernel when
not wanting a default "kernel.txz".

Submitted by: Russell Cattelan <cattelan@thebarn.com>
MFC after: 2 weeks
Obtained from: OneFS
Sponsored by: EMC / Isilon Storage Division

Allow storing package(world|kernel) tarballs into a different location at PACKAGEDIR.

Submitted by: Russell Cattelan <cattelan@thebarn.com>
Discussed with: gjb
MFC after: 2 weeks
Obtained from: OneFS
Sponsored by: EMC / Isilon Storage Division

Fix errors being ignored in many phases of the build since the bmake integration.

Say it with me, "I will not chain commands with && in Makefiles"

This was originally fixed and explained quite well by bde@ in r36074. The
initial bmake integration caused 'set -e' to stop being used which lead to
r252419. Later 'set -e' expectations were fixed with bmake in r254980.

Because of the && here, errors would be ignored when building in parallel and
a dependency failed. Such as bootstrap-tools since it builds everything in
parallel. If any tool failed in obj/depend/all, it would just ignore the error
and continue to build. This later would result in cascaded errors that only
confused the real issue. This could also cause commands after the failed
command to still execute, leading to more confusion.

This should be fine if the command is in a sub-shell such as: (cmd1 && cmd2)

This reverts r252419.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Add support to libkvm for reading vmcores from other architectures.
- Add a kvaddr_type to represent kernel virtual addresses instead of
unsigned long.
- Add a struct kvm_nlist which is a stripped down version of struct nlist
that uses kvaddr_t for n_value.
- Add a kvm_native() routine that returns true if an open kvm descriptor
is for a native kernel and memory image.
- Add a kvm_open2() function similar to kvm_openfiles(). It drops the
unused 'swapfile' argument and adds a new function pointer argument for
a symbol resolving function. Native kernels still use _fdnlist() from
libc to resolve symbols if a resolver function is not supplied, but cross
kernels require a resolver.
- Add a kvm_nlist2() function similar to kvm_nlist() except that it uses
struct kvm_nlist instead of struct nlist.
- Add a kvm_read2() function similar to kvm_read() except that it uses
kvaddr_t instead of unsigned long for the kernel virtual address.
- Add a new kvm_arch switch of routines needed by a vmcore backend.
Each backend is responsible for implementing kvm_read2() for a given
vmcore format.
- Use libelf to read headers from ELF kernels and cores (except for
powerpc cores).
- Add internal helper routines for the common page offset hash table used
by the minidump backends.
- Port all of the existing kvm backends to implement a kvm_arch switch and
to be cross-friendly by using private constants instead of ones that
vary by platform (e.g. PAGE_SIZE). Static assertions are present when
a given backend is compiled natively to ensure the private constants
match the real ones.
- Enable all of the existing vmcore backends on all platforms. This means
that libkvm on any platform should be able to perform KVA translation
and read data from a vmcore of any platform.

Tested on: amd64, i386, sparc64 (marius)
Differential Revision: https://reviews.freebsd.org/D3341

Remove unneeded libmd from bootstrap-tools (reverting r246784).

The bootstrap-tools are supposed to be host tools, which in most cases, use
host headers and libraries. As such, directly including the src tree's headers
for libmd here causes the need to link libmd in since it will be built with
the new symbols (which /usr/lib/libmd.so) won't have unless it is new enough.
During the target build in buildworld the target headers are staged into
WORLDTMP and used via --sysroot, allowing the target xinstall to be built with
the new/target libmd.

The .PATH here was also not doing anything since xinstall does not use libmd
source files.

Sponsored by: EMC / Isilon Storage Division
MFC after: 2 weeks

Avoid setting schg in the objtree for lib32 build.

Reported by: kib
Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week

Fix 'make cleanworld' to respect TARGET/TARGET_ARCH for cross-build objtrees.

This simplifies the logic to always try removing the objdir if it exists
and to fallback on a 'cleandir' if no objdir exists. The reasoning for
this is to avoid rm -rf src/* (r126024)

Sponsored by: EMC / Isilon Storage Division
MFC after: 2 weeks

Move 'make cleandir' from Makefile to Makefile.inc1.

This leads the way for fixing cross-build cleanup, and eventually replacing
'cleandir' with it during the build.

Sponsored by: EMC / Isilon Storage Division
MFC after: 2 weeks

Add built-in ccache build support via WITH_CCACHE_BUILD option.

ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build. This is explained in
more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times. When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'. Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports. This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead. ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss. For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct. Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it. Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings. GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual. Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization. There is also a memcached backend available [5]. Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout. A
possible working layout is as follows:
Source: /some/prefix/src1
Source: /some/prefix/src2
Source: /some/prefix/src3
Objdir: /some/prefix/obj
Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative. Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files and ccache objects.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
x buildworld-before
+ buildworld-ccache-empty
* buildworld-ccache-full
% buildworld-ccache-full-fastdep
# buildworld-fastdep
+-------------------------------------------------------------------------------+
|% * # +|
|% * # +|
|% * # xxx +|
| |A |
| A|
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 4519 4525.04 4520.73 4521.59 3.1104823
Difference at 95.0% confidence
758.027 +/- 43.4565
20.1412% +/- 1.15466%
(Student's t, pooled s = 19.1726)
* 3 1823.08 1827.2 1825.62 1825.3 2.0785572
Difference at 95.0% confidence
-1938.26 +/- 43.298
-51.5007% +/- 1.15045%
(Student's t, pooled s = 19.1026)
% 3 1266.96 1279.37 1270.47 1272.2667 6.3971113
Difference at 95.0% confidence
-2491.3 +/- 44.3704
-66.1952% +/- 1.17895%
(Student's t, pooled s = 19.5758)
# 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)

buildkernel:
x buildkernel-before
+ buildkernel-ccache-empty
* buildkernel-ccache-empty-fastdep
% buildkernel-ccache-full
# buildkernel-ccache-full-fastdep
@ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|# @ % * |
|# @ % * x + |
|# @ % * xx ++|
| MA |
| MA|
| A |
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 727.97 731.91 728.06 729.31333 2.2492295
Difference at 95.0% confidence
156.88 +/- 4.17129
27.4058% +/- 0.728695%
(Student's t, pooled s = 1.84034)
* 3 527.1 528.29 528.08 527.82333 0.63516402
Difference at 95.0% confidence
-44.61 +/- 2.33254
-7.79305% +/- 0.407478%
(Student's t, pooled s = 1.02909)
% 3 400.4 401.05 400.62 400.69 0.3306055
Difference at 95.0% confidence
-171.743 +/- 2.16453
-30.0023% +/- 0.378128%
(Student's t, pooled s = 0.954969)
# 3 201.94 203.34 202.28 202.52 0.73020545
Difference at 95.0% confidence
-369.913 +/- 2.40293
-64.6212% +/- 0.419774%
(Student's t, pooled s = 1.06015)
@ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR: 182944 [4]
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes

Add a hack to workaround ZSH as BUILDENV_SHELL breaking CPUTYPE.

ZSH considers CPUTYPE a magic variable that will be the output of 'uname -m'
even if already set in environment when starting up.

The CPUTYPE?= check in Makefile.inc1 and supporting overriding CPUTYPE
manually in the buildenv shell make automatic workarounds too tricky
here. ZSH should really respect variables set in the environment before
trashing them.

X-MFC-With: r290423
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Allow 'make buildenv' to work anywhere in the src tree.

Sponsored by: EMC / Isilon Storage Division

Allow 'make buildenv' to default to the caller's shell by using SHELL.

Also pass BUILDENV=1 into the sub-shell to allow modifying PS1 in .profile such
as:
if [ -n "${BUILDENV}" ]; then
PS1="(buildenv) ${PS1}"
fi

SHELL defaults to 'sh' in share/mk/sys.mk, but is typically passed down by
the shell invoking make as well. Rather than forcing all 'buildenv' users
to use plain /bin/sh, let them use their favorite shell.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
Discussed with: imp

Fix CC being wrong during install* targets.

Move CROSS_TOOLS stuff to top of file (before bsd.compiler.mk) so that
decisions made by bsd.compiler.mk can properly affect the defaults in
src.opts.mk. Move that to after bsd.compiler.mk. Add a comment about
why we include bsd.compiler.mk here despite the fact that src.opts.mk
currently does too. Also remove bsd.arch.inc.mk that's been OBE.

Differential Revision: https://reviews.freebsd.org/D4087

Rename libohash to libopenbsd.

libopenbsd is an internal library which
to bring in compatibility stuff from OpenBSD.
This will allow us to bring in more
OpenBSD utilities into the FreeBSD base system.

We similarly use libnetbsd for bringing in stuff from NetBSD.

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D4078

Add a note about DEPFLAGS which is currently a hack around not properly passing
CXXFLAGS to sub-makes.

The bad passing also causes bsd.dep.mk's logic to selectively pull only some
flags from C[XX]FLAGS to not apply which can be seen with '-L' being passed to
mkdep when using an external compiler.

native-xtools: Replace common path with NXBDESTDIR.

Also combine some mkdir calls.

Sponsored by: EMC / Isilon Storage Division

native-xtools: Fix build with WITH_DEBUG_FILES.

Sponsored by: EMC / Isilon Storage Division

Parallelize build-tools.

MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division

Warn that NOCLEAN is deprecated - use NO_CLEAN instead

NO_CLEAN has been the correct spelling for over a decade.

Let SUBDIR_OVERRIDE with 'make buildworld' be more useful.

Now it can be used to effectively "build in a subdir". It will use the
'cross-tools', 'libraries', and 'includes' phases of 'buildworld' to properly
setup a WORLDTMP to use. Then it will build 'everything' only in the
listed SUBDIR_OVERRIDE directories. It is still required to list custom
library directories in LOCAL_LIB_DIRS if SUBDIR_OVERRIDE is something
that contains libraries outside of the normal area (such as
SUBDIR_OVERRIDE=contrib/ofed needing LOCAL_LIB_DIRS=contrib/ofed/usr.lib)

Without these changes, SUBDIR_OVERRIDE with buildworld was broken or hit
obscure failures due to missing libraries, includes, or cross compiler.

SUBDIR_OVERRIDE with 'make <target that is not buildworld>' will continue to
work as it did before although its usefulness is questionable.

With a fully populated WORLDTMP, building with a SUBDIR_OVERRIDE with
-DNO_CLEAN only takes a few minutes to start building the target
directories. This is still much better than building unneeded things via
'everything' when testing small subset changes. A BUILDFAST or
SKIPWORLDTMP might make sense for this as well.

- Add in '_worldtmp' as we still need to create WORLDTMP as later targets,
such as '_libraries' and '_includes' use it. This probably was avoiding
calling '_worldtmp' to not remove WORLDTMP for debugging purposes, but
-DNO_CLEAN can be used for that.

- '_legacy' must be included since '_build-tools' uses -legacy.
The SUBDIR_OVERRIDE change came in r95509, while -legacy being part
of build-tools came in r113136.

- 'bootstrap-tools' is still skipped as this feature is not for
upgrades.

- Fix buildworld combined with SUBDIR_OVERRIDE not installing all includes.

The original change for SUBDIR_OVERRIDE in r95509 kept '_includes'
and '_libraries' as building everything possible as the SUBDIR_OVERRIDE
could need anything from them. However in r96462 the real 'includes'
target was changed from manual sub-makes to just recursing 'includes'
on SUBDIR, thus not all includes have been installed into WORLDTMP since then
when combined with 'buildworld'.

This is not done unless calling 'make buildworld' as it would be
unexpected to have it go into all directories when doing 'make
SUBDIR_OVERRIDE=mydir includes'.

- Also need to build the cross-compiler so it is used with --sysroot.
If this is burdensome then telling the build to use the local compiler
as an external compiler (thus using a proper --sysroot to WORLDTMP) is
possible by setting CC=/usr/bin/cc, CXX=/usr/bin/c++, etc.

- Don't build the lib32 distribution with SUBDIR_OVERRIDE in buildworld
since it won't contain anything related to SUBDIR_OVERRIDE. Testing
of the lib32 build can be done with 'make build32'.

- Document these changes in build.7

Sponsored by: EMC / Isilon Storage Division
MFC after: 2 weeks

Fix my change in r289435 causing 'etc' to be added to SUBDIR when using
SUBDIR_OVERRIDE.

MFC after: 2 weeks
X-MFC-With: r289435
Sponsored by: EMC / Isilon Storage Division

Add some missing '+', .MAKE, and .PHONY modifiers.

Some of these targets were lacking both .MAKE and a '+'. Others were just
inconsistent.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Handle lib32 files during delete-old* when MK_LIB32=no.

Extend OptionalObsoleteFiles.inc to delete all lib32 files when MK_LIB32 is
set to no on a system that previously had lib32 libraries installed.

Also, to prevent "make delete-old-dirs" from always deleting lib32 directories
after an installworld, move the lib32 subtree to its own mtree file that only
gets applied when MK_LIB32=yes.

Test: Ran "make delete-old" and "make delete-old-libs" on a system that never
had MK_LIB32 enabled, and on a system where MK_LIB32 was enabled and later
disabled. Did this both on amd64 and powerpc64.

Test: Ran "make tinderbox" without errors.

Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D3923

Remove unneeded MK_CTF=no when MK_CDDL=no.

This has been handled since r228158 made MK_CTF dependent on MK_CDDL
in share/mk/bsd.opts.mk.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Fix wrong PATH being set for world 'includes' stage after r289438.

The 'includes' target is currently a pseudo target in bsd.subdir.mk that
does 'cd ${.CURDIR} && ${MAKE} buildincludes && ${MAKE} installincludes',
versus all over targets that just recurse.

In Makefile.inc1 the older duplicated bsd.subdir.mk logic for calling
'includes' was being executed in each subdir directly, meaning 'cd lib && make
includes' became 'cd lib && make buildincludes && make installincludes'. Now
that the bsd.subdir.mk logic is used it is calling 'make buildincludes && make
installincludes' from the top-level which pulls in the PATH=<default path>
from /Makefile.

The sub-make logic for 'includes' in bsd.subdir.mk was attempted to be removed
in r289282 but turned out to be wrong. I have a working version now but
it is not yet ready for commit. So for now in Makefile.inc1 split out
'includes' to 'buildincludes' and 'installincludes' which will avoid the
problem.

MFC after: 2 weeks
X-MFC-With: r289438
Sponsored by: EMC / Isilon Storage Division

For 'buildenvvars' show any .exported variables as well to cover recent
exporting of OSRELDATE and VERSION. These already do export to 'buildenv'
fine.

Sponsored by: EMC / Isilon Storage Division

Always export VERSION to the environment to avoid looking it up again in
sub-makes.

Some of the world phases that used plain '${MAKE} -f Makefile.inc1' were not
passing this variable along which caused them to look it up again. By
using bmake's .export we can remove it from all of the other environment
lines.

Add a comment about the usage for VERSION for ctfmerge.

Sponsored by: EMC / Isilon Storage Division

Export OSRELDATE so sub-makes don't look it up again.

We pass BOOTSTRAPPING=${OSRELDATE} to some of the sub-makes. Rather than
chase every ${MAKE} invokation, just export it as bmake lets us.

Sponsored by: EMC / Isilon Storage Division

Rework the world subdir build targets to use the standard SUBDIR_PARALLEL mechanism.

Back in r30113, the 'par-*' targets were added to parallelize portions of
the build in a very similar fashion as the SUBDIR_PARALLEL feature used in
r263778. Calling a target without 'par-' (for 'parallel') resulted in the
standard bsd.subdir.mk handling without parallelization. Given we have
SUBDIR_PARALLEL now there is no reason to duplicate the handling here.

In build logs this will result in the ${dir}.${target}__D targets now showing
as the normal ${target}_subdir_${dir} targets.

I audited all of the uses of Makefile.inc1 and Makefile's targets that use
bsd.subdir.mk and found that all but 'all' and 'install' were fine to use
as always parallel.
- For 'install' (from installworld -j) the ordering of lib/ and libexec/
before the rest of the system (described in r289433), and etc/ being last
(described in r289435), is all that matters. So now a .WAIT is added in
the proper places when invoking any 'install*' target. A parallel
installworld does work and took 46% of the time a non-parallel
install would take on my system with -j15 to ZFS.
- For 'all' I left the default handling for this to not run in parallel. A
'par-all' target is still used by the 'everything' stage of buildworld
to continue building in parallel as it already has been. This works
because most of the dependencies are handled by the early bootstrap
phases as well as 'libraries' and 'includes' phases. This lets
all of the SUBDIR build in parallel fine, such as bin/ and lib/. This
will not work if the user invokes 'all' though as we have dependencies
spread all over the system with no way to depend between them (except
for the dirdeps feature in the META_MODE build). Calling 'make all'
from the top-level is still useful at least when using SUBDIR_OVERRIDE.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Fix adding manpages installed by LOCAL_DIRS to whatis file.

The ordering of 'etc' in the install has a long history dating back to the
first time it was realized it needed to be "last" in r4486. That commit
still left it before LOCAL_DIRS though. By having it before LOCAL_DIRS
any manpages they install were not being added to the whatis database in the
install image. They would likely show up in the file after a periodic
rebuild of the file though.

Currently the whatis file is built by an 'afterinstall' hook in etc/Makefile
that calls share/man's 'makedb' target.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Remove lockf as an ITOOL.

It was added in r152006 to handle serializing access of info/dir when
installing INFO files. We no longer support INFO files since r276551
though.

Sponsored by: EMC / Isilon Storage Division

Correct a bitrotted comment about installworld order requirements.

The case of make(1) using a new /bin/sh issue was fixed in r173219 when ITOOLS
was introduced.

There are still issues with mid-install errors leaving a system unusable that
are currently non-trivial to solve. The safest ordering requires installing
rtld, libc and libthr (in that order) before anything else. We don't do that
now though. Much improvement is needed here still.

Discussed with: kip and kan (rtld/library ordering)
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Remove .MAKE from targets that do more than just run sub-makes, such as
calling rm or mtree.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Fix delete-old and check-old-files not removing old debug symbols.

This was handled for libraries in r256842 but for some reason was missed
for files (bsd.prog.mk).

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes

Fix buildworld with clean objdir after r289351.

Sponsored by: EMC / Isilon Storage Division

Consider top-level targets to be .PHONY as bmake won't build them
otherwise if a file with the same name is found in the directory.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Add temporary workaround for .MAKE being applied to _worldtmp, since r251750.

This was causing files to be removed from the objdir when -n was used.
_worldtmp makes no sub-make calls.

A more comprehensive solution is coming involving fine-grained '+' where
appropriate.

Sponsored by: EMC / Isilon Storage Division
MFC after: 1 week

Create /usr/tests *.debug file directory hierarchy

Reviewed by: bdrewery, ngie
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3896

Fix installing config files as non root

Reported by: adrian

Change make distribution so that it now call installconfig in all dirs along
with the current behaviour of calling "distribution" in the etc target.

This allows mergemaster/etcupdate to still work when some configuration will be
moved to be handled in the same directories their source code lives in.

The latest version of lex requires the latest m4 to build, add a dependency
when running the build-tools stage.

The requirement is due to the -P flag used when running m4 from usr.bin/lex
Makefile to generate skel.c. With the old m4 that fails and the failure is
ignored, resulting in an empty(-ish) skel.c, which leads to later build
failures when the misconfigured new lex tool is run.

This enables building -current (and 10-stable after MFC) on a stable-8
system again.

MFC after: 3 days

Final step of eliminating the "games" distribution: Merge src/games
(or what's left of it, at least) into src/usr.bin.

This change will not be MFCed.

Discussed at: EuroBSDCon 2014
Committed from: EuroBSDCon 2015

remove unused sgsmsg utility (originally imported from opensolaris)

MFC after: 25 days

META_MODE: Make it easier to keep the bootstrap-tools args in sync.

Sponsored by: EMC / Isilon Storage Division

Remove EOL whitespace from Makefile.inc1

Move DTrace tools to cross-tools stage

Bootstrap tools exist for backwards compatibility support. DTrace tools
tools are also needed for cross builds, so belong in cross-tools.

Reviewed by: imp (earlier), markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2144

Let makeman run 'make showconfig' without hitting the aarch64 error.

Fix grammer in an error message

PR: 202310
Submitted by: Chris Petrik <chris@bsdjunk.com>
Approved by: will

Unconditionally build CTF tools in the bootstrap-tools phase of the build.

Stale CTF tools are a frequent source of DTrace issues, and they compile
quickly enough that the increase in build time is negligible.

Reviewed by: emaste, imp
Differential Revision: https://reviews.freebsd.org/D3670

The ${BUILDKERNELS:[2..-1]} appears to produce a non zero result for
a one word variable, which is quite unexpected from documentation.
So, to avoid double installation of a single kernel, protect the extra
kernels loop with ${BUILDKERNELS:[#]} > 1 conditional.

Sponsored by: Netflix
Sponsored by: Nginx, Inc.

Not only build with buildworld, but also install with installworld all
alternative kernels.

Sponsored by: Netflix
Sponsored by: Nginx, Inc.

When building multiple kernels use [2..-1] to extract !INSTALLKERNEL
from BUILDKERNELS list. This is more strict, since INSTALLKERNEL by
definition is the first word of BUILDKERNELS list. The previous
code failed if INSTALLKERNEL is a substring of additional kernel name.

Reviewed by: gjb
Sponsored by: Netflix
Sponsored by: Nginx, Inc.

Remove now unused LIBPRIVATEDIR

Sponsored by: Gandi.net

Sparc64 is the odd-man out, so form the if that way rather than
listing everybody else.

Separate ELFTOOLCHAIN_BOOTSTRAP from BINUTILS_BOOTSTRAP

For most cases they are equivalent, but BINUTILS_BOOTSTRAP is a
BROKEN_OPTION on arm64 as the in-tree GNU binutils do not support it,
so we need a separate internal flag for ELF Tool Chain.

Reviewed by: andrew, brooks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3381

Spell binaries in the customary way.

Submitted by: jhb@

Roll WITHOUT_ELFTOOLCHAIN_TOOLS into WITHOUT_TOOLCHAIN

The option was added only to ease the transition from GNU Binutils to
ELF Tool Chain tools, and that process is now complete (for the viable
replacements). Noting the removal in UPDATING is sufficient as we have
not shipped a release with the option.

Reviewed by: brooks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3240

Fix the fixing of the build I broke. rescue/rescue has the right
target, but rescue doesn't.

Pointy hat: imp@

Fix build.

Document build-tools better. Add rescue back because it builds /bin/sh
which has a build-tools target (see commit for how build-tools and
cross-tools differ).

Why on earth have we been building rescue as a build tool for the past
12 years? Nothing downstream in the build uses it. Eliminate it as a
build tool.

Reviewed by: emaste@ ("just delete it")

Crunchgen needs to be bootstrapped to pick up the STRIP->STRIPBIN
changes to prevent the 'rescue: not found' errors from happening.
Bump FreeBSD_version to 1100078 since there's been no version bumps
since this change was made. Only people that installed since r284356
really need to do this bootstrapping, but since crunchgen needs to
bootstrap for other reasons, bumping the number was the simplest.

Fix comment describing legacy target and wrap to 80 columns

Build libelf and libdwarf in the legacy stage

They need to be built and installed (including headers) prior to the
DTrace CTF tools.

Reviewed by: imp (as part of a larger change)
Sponsored by: The FreeBSD Foundation

cmp and cp are used by the kerberos install, so need to be imclided in
ITOOLS. They are tiny enough that I'm not making conditional: the
minuscule savings in disk space isn't worth the obfuscation of
Makefile.inc1.

First cut of DTrace for AArch64.

Reviewed by: andrew, emaste
Sponsored by: ARM Limited
Differential Revision: https://reviews.freebsd.org/D2738

Make all shared library a relative symlink

This makes sysroot usable for cross building, it also removes the need for
_SHLIBDIRPREFIX (keeps its definition since picobsd uses it and I have no time
to test it)

Differential Revision: https://reviews.freebsd.org/D2920
Submitted by: imp, adrian
Tested by: adrian

Only build gperf as a bootstrap tool if gcc is going to be built

Remove temporary changes that crept in r284417 when it should not have

Reported by: antoine, jbeich

Revert r284417 it is not necessary anymore

Revert r283938 it is not needed anymore and broken

gzip(1) cannot be used as bootstrap tools as it depends on syscalls that only
exists on HEAD

mandoc's makewhatis requires gnuzip to be in the path when deadling with
with compressed manpages.

This should fix spam during installworld: "exec: No such file or directory"

Reported by: many

Remove libmandoc from bootstrap tools

Only push libohash once in bootstrap tools

Switch to mandoc's version of makewhatis(8), whatis(1), apropos(1) utilities.

This change among other things improve search capabilities over the manpages
allowing fine grain query.

A new build option WITHOUT_MANDOCDB has been added to keep the ancient version
of the database and the tools. The plan is to entirely remove this option before
11.0-RELEASE.

Differential Revision: https://reviews.freebsd.org/D2603

Do not make libsqlite3 dependency chain only defined for kerberos, soon more
options would need libsqlite3 as well thus depends on this definitions

Fix build WITHOUT_CDDL by unconditionnaly adding libproc and librtld_db
on supported arches to the prebuild libs

_Really_, _Really_ fix buildworld by moving the conditionals down, fixing some
typos, and fixing the dependency when MK_LIBCPLUSPLUS != no

- `:D` operator evaluation is immediate, i.e. like .if defined(..). So r283159
was in effect a no-op commit.
- Fix dependency in MK_LIBCPLUSPLUS case in two ways:
-- lib/libc++ was the wrong dependency. It should have been libcxxrt.
-- lib/libc++ was missing __L, so again it was depending on the directory, not
the relevant .PHONY target.

Tested with: make tinderbox (amd64, arm, sparc64) and JFLAG=-j16
In collaboration with: bdrewery, imp, peter
BIG pointyhat to: ngie (for trying to commit things at 6am while staying up all
night working on other tasks)

Make kerberos use the same sqlite libraries as other sqlite consumer.

This reduces the number of copy of sqlite we have to just one and easier
tracking version of sqlite

Differential Revision: https://reviews.freebsd.org/D2443
Reviewed by: imp, stas, bjk

Fix breakage I didn't fully solve in r283151 by depending on the .PHONY
target, not the directory when building libctf for libproc

Reported by: many, Jenkins
Submitted by: rodrigc

Unbreak the arm64/sparc64 tinderbox by only compiling lib/libproc and
lib/librtld_db on architectures where they're supported

Reported by: bz, Jenkins
Pointyhat to: bapt

cddl/lib/libctf isn't always a requirement for lib/libproc; its use is dependent
on MK_CTF != "no". Use the other divined value instead of reinventing the wheel

Add _more_ missing dependencies for lib/libproc to further squash build races

Verified via `cd lib/libproc; make -VLIBADD`

Pointyhat to: ngie

Articulate all dependencies for lib/libproc to squash build races after r283139
on !arm64 and !sparc64

Pointyhat to: bapt
Sponsored by: EMC / Isilon Storage Division

Only build sys/boot/usb/tools if MK_USB != no

Sponsored by: EMC / Isilon Storage Division

Fix buildworld by adding libproc and librtld_db to the _prebuild_libs
Those are needed to build libdtrace

Update crunch bootstrapping test for recent fixes

- r277259 crunchide: Correct 64-bit section header offset
- r281674 crunchide: always include both 32- and 64-bit ELF support

With built-in cross-size support we also no longer need a special case
for cross-build crunchide.

Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2576

Remove redundant csu subdir logic

The appropriate subdirectories are handled by lib/csu/Makefile. There's
no need to duplicate this logic in Makefile.inc1 and lib/Makefile.

Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2523

Unbreak build following rev. 282726

(Makefile.inc1): add dependency of xinstall on libmd to
avoid failure of parallel bootstrap.

(lib/libmd/*.h): do not redefine symbols if already
defined as macros (libcrypt uses the same sources internally,
redefining symbols with a prefix of its own).

Fixes build failures caused by previous change to libmd.

Reported by: ian
Pointy hat to: thomas

Replace groff's soelim by soeliminate(1) renamed soelim(1)

Setting PATH to anything is useless as a bare command. Its only relevant
if its set in the environement of each command seperately.

Move the PATH setting to the NXBMAKE variable so its picked up to find
the one-off gperf build for the native-xtools target.

Pointed Out by: ngie

Fix native-xtools breakage when building a gcc enabled target on a clang
enabled host. Build a one-off gperf and put it in the PATH for the rest
of the target so the ONE call to gperf by the gcc build picks it up and
DTRT.

Reviewed by: imp

Serialize all of _kerberos5_bootstrap_tools to avoid build failures involving
make bootstrap-tools

On the plus side, this also greatly reduces complexity

MFC after: 1 week
Pointyhat to: ngie
Reported by: Willem Jan Withagen <wjw@digiware.nl>

Bump crunch BOOTSTRAPPING for ELF header offset fix in r277557

RELEASEDIR was removed in FreeBSD 9.x, at the same time /boot/loader
stopped using kgzip in the release process. We no longer need to build
kgzip as a cross tool, and tests for RELEASEDIR are obsolete, so
remove both.

Differential Revision: https://reviews.freebsd.org/D2313

Defeat race with MK_KERBEROS == yes introduced with bootstrap-tools
parallelization work done in r279197

- kerberos5/lib/libroken requires kerberos5/tools/make-roken to build
- kerberos5/tools/asn1_compile, kerberos5/tools/slc, and usr.bin/compile_et
require kerberos5/lib/libroken and kerberos5/lib/libvers

This race is incredibly evident when cross-building sparc64 on
ref10-amd64.freebsd.org

MFC after: 1 week
Pointyhat to: ngie

People are still getting burned by the byacc upgraded, switch to
always doing byacc until someone figures out the more nuanced version
to switch off of.

Only use the CROSS_BINUTILS_PREFIX tool if it exists.

Reviewed by: emaste
Sponsored by: The FreeBSD Foundation

Ensure the cross assembler, linker and objcopy are used for the build32
stage, just like for the regular world stage.

Reviewed by: rodrigc, imp, bapt, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D2187

Add --sysroot to the compiler flags for clang or gcc external toolchains

Before this fix, --sysroot was only added to the compiler flags if a clang
external toolchain was used.

Reviewed by: imp

Add some more explanation to the different phases of the build.

Force MK_INCLUDES for the legacy stage

As legacy executes "make installincludes" we don't want it to be
disabled by a src.conf setting.

Reviewed by: imp
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2143

Start to import support for the AArch64 architecture from ARM. This change
only adds support for kernel-toolchain, however it is expected further
changes to add kernel and userland support will be committed as they are
reviewed.

As our copy of binutils is too old the devel/aarch64-binutils port needs
to be installed to pull in a linker.

To build either TARGET needs to be set to arm64, or TARGET_ARCH set to
aarch64. The latter is set so uname -p will return aarch64 as existing
third party software expects this.

Differential Revision: https://reviews.freebsd.org/D2005
Relnotes: Yes
Sponsored by: The FreeBSD Foundation

When cross-building with an external toolchain we still need a target strip

It is used by at least crunchide(1).

Add LIB_CXX so that C++ libraries will use CXX to link.

This fixes C++ libraries not implicitly linking in libc++. This is
generally not an issue because the final linking with the compiled binary
will involve CXX via PROG_CXX or other means. It is however
inconsistent with libraries implicitly linking in libc and problematic
for trying to build libraries with '-z defs' to ensure all direct
dependencies are linked in.

libatf-c++ is currently the only consumer of this new feature.

Differential Revision: https://reviews.freebsd.org/D2039
Reviewed by: imp
Discussed with: bapt
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Provide automatic cross-binutils path if no BINUTILS_BOOTSTRAP

The in-tree binutils does not support arm64, so will not work for the
forthcoming FreeBSD arm64 port. BROKEN_OPTIONS will include
BINUTILS_BOOTSTRAP, so provide a default CROSS_BINUTILS_PREFIX for this
case.

Reviewed by: imp
Sponsored by: The FreeBSD Foundation

It appears that xlint isn't used in the build process, so it certainly
doesn't need to be a build tool.

Support CROSS_BINUTILS_PREFIX with in-tree compiler

Reviewed by: bapt, imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D1974

Fill in missing dependencies for dtrace related tools so the bootstrap-tools
compiles properly on older hosts

Pointyhat to: me
X-MFC with: r279197
MFC after: 2 weeks

Parallelize building bootstrap-tools

Differential Revision: https://reviews.freebsd.org/D1901
MFC after: 2 weeks
Reviewed by: ian
No serious objections from: imp

Fix bootstrap on systems with old yacc with small MAXTABLE.

Restore the ability to use clang as an external compiler. This was
inadvertently removed when support for external GCC was added.

Deprecate XFLAGS in favour of the newer XCFLAGS/XCXXFLAGS.

Tested with: make universe, make CROSS_COMPILER_PREFIX=/usr/bin/ buildworld
Reviewed by: imp, bapt

release: use xz via pipe when compressing the tarballs.

libarchive(3) doesn't support the new liblzma API yet, but this change
allows us to enable multi-threaded xz compression.
``make release'' should now finish in half the time on a machine with
several cores and fast disks (our typical build server).

This behaviour only applies when building a release and it doesn't
affect buildworld/installworld. To disable threaded xz compression,
set XZ_THREADS=1.

Reviewed by: gjb
Tested by: gjb

Fix a typo when invoking tar(1) to create the debug distributions.

Step 1 of eliminating the "games" distribution: Move binaries to /usr/bin;
update paths; and include everything in the "base" distribution.

The "games" distribution being optional made sense when there were more
games and we had small disks; but the "games-like" games were moved into
the ports tree a dozen years ago and the remaining "utility-like" games
occupy less than 0.001% of my laptop's small hard drive. Meanwhile every
new user is confronted by the question "do you want games installed" when
they they try to install FreeBSD.

The next steps will be:

2. Removing punch card (bcd, ppt), phase-of-moon (pom), clock (grdc), and
caesar cipher (caesar, rot13) utilities. I intend to keep fortune, factor,
morse, number, primes, and random, since there is evidence that those are
still being used.

3. Merging src/games into src/usr.bin.

This change will not be MFCed.

Reviewed by: jmg
Discussed at: EuroBSDCon
Approved by: gjb (release-affecting changes)

Remove remnant from texinfo

Exclude 'tests' from DEBUG_DISTRIBUTIONS.

Sponsored by: The FreeBSD Foundation

Merge xz 5.2.0.

This brings support for multi-threaded compression. This brings close
N times faster compression where N is the number of CPU cores.
Because of this, liblzma now depends on libthr.

Soon libarchive will be modified to use the new lzma API.

Thanks to antoine@ for the exp-run.

Differential Revision: https://reviews.freebsd.org/D1786
Reviewed by: bapt

Conditionalize building radius support into libpam, ppp, etc via
MK_RADIUS_SUPPORT

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Restore addr2line to cross tools

Addr2line is not required for the build, and a per-arch binary is no
longer required with the switch to the ELF Tool Chain. However, building
these tools during the cross tools stage can be useful for developers
who cross build HEAD from stable/10, and adds very little to the build
time.

Reviewed by: ian, imp
Differential Revision: https://reviews.freebsd.org/D1583

Fix bootstrap on systems with old libdwarf and WITHOUT_CDDL

ELF Tool Chain tools need libelf and libdwarf.

Submitted by: jmallett (earlier version)
Reviewed by: jmallett
Sponsored by: The FreeBSD Foundation

Remove addr2line from cross elftoolchain tools list

It is not required, and there is no reason to install it just because it
came with the binutils cross tools.

Sponsored by: The FreeBSD Foundation

Remove GNU texinfo from base along with all info pages.

To be able to info pages consider installing texinfo from ports print/texinfo or
via pkg: pkg install texinfo

Differential Revision: https://reviews.freebsd.org/D1409
Reviewed by: emaste, imp (previous version)
Relnotes: yes

Use BINUTILS_BOOTSTRAP knob for binutils only

Previously it also disabled building elftoolchain bootstrap tools such as
strip(1).

Differential Revision: https://reviews.freebsd.org/D1398
Reviewed by: imp
Sponsored by: The FreeBSD Foundation

Update the text for building texinfo with build-tools to reflect the fact
that make install is being called as well

MFC after: 1 week
X-MFC with: r276052

Build selective portions of gnu/usr.bin/texinfo as part of build-tools to
ensure that building on a host without makeinfo (i.e. a host where
make delete-old -DWITHOUT_INFO was run), then building with MK_INFO == yes
doesn't manifest in build errors when building info pages

This manifested itself like the following when I was build testing an MFC
change on stable/10:

makeinfo --no-split -I /usr/src/gnu/lib/libregex/doc -I /usr/src/gnu/lib/libregex/doc regex.texi -o regex.info
makeinfo: not found
*** [regex.info] Error code 127

make[6]: stopped in /usr/src/gnu/lib/libregex/doc
1 error

Tested on a head VM without makeinfo installed and by building with MK_INFO=yes

MFC after: 1 week

Fix accidental MK_DEBUG_FILES=no addition to NXBMAKE in r275909

X-MFC with: r275909
Sponsored by: EMC / Isilon Storage Division

Don't build full clang toolchain or clang extras in stages 1-3 of buildworld

MFC after: 1 week
Reviewed by: dim (as part of a "larger" diff)
Phabric: D1336
Sponsored by: EMC / Isilon Storage Division

Fix building/installing tests when TESTSBASE != /usr/tests

The work in r258233 hardcoded the assumption that tests was the last component
of the tests tree by pushing tests as an explicit prefix for the paths in
BSD.tests.dist and /usr was the prefix for all tests, per BSD.usr.dist and all
of the mtree calls used in Makefile.inc1. This assumption breaks if/when one
provides a custom TESTSBASE "prefix", e.g. TESTSBASE=/mytests .

One thing that r258233 did properly though was remove "/usr/tests" creation
from BSD.usr.dist -- that should have not been there in the first place. That
was an "oops" on my part for the work that was originally committed in r241823

MFC after: 2 weeks
Phabric: D1301
Reviewed by: imp
Sponsored by: EMC / Isilon Storage Division

Fix sporadic build failures due to race when running make installworld
when strip gets replaced at install time by adding it to ITOOLS for the
default usr.bin/xinstall STRIP_CMD

This will fix the failure noted in this Jenkins build step:
https://jenkins.freebsd.org/job/Build-UFS-image/688/

This will also fix the issue reported by alfred@ dealing with installing on
targets that differ from build hosts (e.g. installing on i386/i386 when built
on amd64/amd64)

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Add an UPDATING entry and warning about the change in r274807 to help users
transition to the new behavior.

Discussed with: jmallett
Sponsored by: DARPA, AFRL

Add makewhatis to ITOOLS if MK_MAN != no

This will fix installation with differing host targets in installworld, so
one can build i386/i386 on an amd64 host, then install to an i386/i386 target

Reported by: alfred
Phabric: D1280
MFC after: 1 week

Build gperf with xdev as part of _xb-bootstrap-tools

This will unbreak the build when "env MK_CXX=no make delete-old" has been run
on the build host post-r272849

Tested with the following commands:

/bin/sh
export __MAKE_CONF=/dev/null SRCCONF=/dev/null
export MK_CLANG=no MK_CLANG_BOOTSTRAP=no MK_GCC=yes MK_GCC_BOOTSTRAP=yes
export MK_GNUCXX=yes TARGET=armv6 TARGET_ARCH=arm make xdev
make toolchain
make xdev-build
sudo -E make xdev-install
/usr/armv6-freebsd/usr/bin/cc -dumpmachine | grep arm

X-MFC with: r272849
Reported by: Dan Raymond <draymond@foxvalley.net>, gjb

Build infrastructure for elftoolchain tools

Set WITH_ELFTOOLCHAIN_TOOLS in src.conf to use the elftoolchain version
of the following tools:

* addr2line
* elfcopy (strip / mcs)
* nm
* size
* strings

Reviewed by: bapt (earlier version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D1224

Slightly alter the handling of LOCAL_LIB_DIRS to skip addition of
directories in LOCAL_LIB_DIRS if they are subdirectories of directories
listed in LOCAL_DIRS. This allows a hierarchy like:

foo
foo/lib
foo/usr.bin
foo/usr.sbin

to be supported with LOCAL_DIRS=foo LOCAL_DIRS=foo/lib.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Add a guard against attempting to invoke the buildenv target with -j# as
that silently exits rather than doing something useful.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Add mv, echo and hostname to the native-xtools target. Missed these
when profiling.

MFC after: 2 weeks

Flush out many more static binaries that can be used in a jail for building
pkgs via poudriere. This removes a significant amount of emulation speeding
up run times.

Phabric: D1172
Reviewed by: imp
MFC after: 2 weeks

Add more static binaries to the native-xtools target. These help speed
up the cross builds significantly.

adding:
bin/sh
bin/csh
sbin/md5
usr.bin/bzip2
usr.bin/fetch
usr.bin/gzip
usr.bin/tar
usr.bin/unzip
usr.bin/xz

Phabric: D1146
Reviewed by: imp bdrewery
MFC after: 2 weeks

Fix build race in Makefile.inc1 after r274226 by adding lib/libm__L dependency
for gnu/lib/libdialog

Sponsored by: EMC / Isilon Storage Division

For really fast machines, an edge-case may exist where dpv(3) may be
built before contrib dependency, dialog(3). Add dialog(3) to the list
of _prebuild_libs to ensure that this does not happen.

Tested on: 11.0-CURRENT amd64 @ r274205
Thanks to: kargl, Larry Rosenman <ler@lerctr.org>, ngie, markj
Recommended by: ngie
Reviewed by: ngie, markj
MFC after: 21 days
X-MFC-to: stable/10 stable/9
X-MFC-with: 274116 274120 274121 274123 274144 274146 274192 274203

SUBDIR_DEPENDS__ in lib/Makefile is not working out so well for me.
Add to using _prebuild_libs in (top-level) Makefile.inc1.
NB: Unbreak build yet again (we'll get this right eventually)

Reviewed by: markj, ngie
Thanks to: ian, markj, ngie, Nikolai Lifanov <lifanov@mail.lifanov.com>
MFC after: 21 days
X-MFC-to: stable/10 stable/9
X-MFC-with: 274116 274120 274121 274123 274144 274146 274192

Rename XFLAGS to XCFLAGS and XXFLAGS to XCXXFLAGS

This is less confusing names and actually more reflexting what they are intended
to.

Discussed with: brooks

My previous commit exposed an issue as it fixed a different
issue. lib/atf isn't a prereq_lib, since it isn't required for other
libraries to build. Remove it. The old kludge of always building it
had effectively been retired. Since we don't want to build the
libraries with the tests when we're bootstrapping, invent
MK_TESTS_SUPPORT which normally defaults to the current MK_TESTS
value, except when explicitly defined. Make lib/atf depend on it being
yes. When building the libraries set MK_TESTS to no, and
MK_TESTS_SUPPORT to the current value of MK_TESTS so that later stages
of the build work correctly. This should fix (and does for me)
people's issues with parallel builds racing between lib/atf and
libexec/atf. Since lib/atf is built during the libraries phase, the
race disappears.

Generate both userland and kernel option settings for showconfig.

PR: 191920

You aren't allowed to test WITH_xxx or WITHOUT_xxx here, so remove it.
Even if you were allowed to test for it, the test makes no sense as it
always results in adding -DWITH_ATF unless WITH_ATF was already
defined. But if MK_ATF != no, then we know it was defined. This, in
turn, caused tools/build/options/makemake always think WITH_ATF is the
default, which removed control of that from sys.conf.mk.

To get the intent of the deleted comment, another mechanism is
required, assuming that the intent of that comment is desirable.

Make the external toolchain support grows to the knowleged of XXFLAGS for C++ dedicated flags
and DEPFLAGS for mkdep flags
Pass the path to the libc++ headers in both, enforce the gnu++11 standard in the XXFLAGS
to satisfy libc++ requirements pass the libc++ objectdir as a location where to find
libraries so it can find libstdc++.so and libstdc++.A

Reviewed by: imp

Add size(1) to the cross build toolchain

Fix typo

Add OBJCOPY to the list of external tools

Fix build when XCC is defined and X_COMPILER_TYPE is not

Add CROSS_TOOLCHAIN macro select pre seeded external toolchain configuration files
The goal is to provide pre seeded toolchain configurations withing the ports tree
to allow the use of an external toolchain in a simple way:

make CROSS_TOOLCHAIN=powerpc64-gcc TARGET=powerpc TARGET_ARCH=powerpc64 buildworld

This will look for the external toolchain definition in /usr/local/share/mk/powerpc64-gcc.mk
While here add the notion of X_COMPILER_TYPE to the external toolchain framework to allow
to deal with differences between gcc and clang in regards of cross building

Fix dependency errors when linking libproc.

Reported by: Oliver Hartmann
X-MFC-With: r272488

Add dependencies to various libraries to libzfs and libzpool.

Submitted by: sef

Fix the TARGET_ABI value clang uses. It shpuld be gnueabi on all ARM
soft-float architectures, and gnueabihf for hard-float.

Remove MK_ARM_EABI, the armeb issues have been fixed. The code to support
the oabi is still in the tree, but it is expected this will be removed
as developers work on surrounding code.

With this commit the ARM EABI is the only supported supported ABI by
FreeBSD on ARMa 32-bit processors.

X-MFC after: never
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D876

Have distrib-dirs, distribution, hier, hierarchy, redistribute, and reinstall
depend on the .MAKE special target

This will allow users to do something like the following to print out the
results of the running the simulated make target with bmake, like some of the
other top-level make targets in Makefile.inc1:

% make -f Makefile.inc1 -n distribution TARGET=i386 TARGET_ARCH=i386

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Root the lib32 object tree under the overall object tree.

This enables a common root directory for all object files for a given tree,
which eases sharing a common MAKEOBJDIRPREFIX, and cleaning up of object trees.

In particular, one can simply (from the source directory) rm -rf /usr/obj$(pwd)
to destroy all object files for it. Or to copy/sync files, etc.

Reviewed by: bdrewery
CR: https://reviews.freebsd.org/D796
MFC after: 1 month
Sponsored by: Spectra Logic

Revert r267233 for now. PIE support needs to be reworked.

1. 50+% of NO_PIE use is fixed by adding -fPIC to INTERNALLIB and other
build-only utility libraries.
2. Another 40% is fixed by generating _pic.a variants of various libraries.
3. Some of the NO_PIE use is a bit absurd as it is disabling PIE (and ASLR)
where it never would work anyhow, such as csu or loader. This suggests
there may be better ways of adding support to the tree. Many of these
cases can be fixed such that -fPIE will work but there is really no
reason to have it in those cases.
4. Some of the uses are working around hacks done to some Makefiles that are
really building libraries but have been using bsd.prog.mk because the code
is cleaner. Had they been using bsd.lib.mk then NO_PIE would not have
been needed.

We likely do want to enable PIE by default (opt-out) for non-tree consumers
(such as ports). For in-tree though we probably want to only enable PIE
(opt-in) for common attack targets such as remote service daemons and setuid
utilities. This is also a great performance compromise since ASLR is expected
to reduce performance. As such it does not make sense to enable it in all
utilities such as ls(1) that have little benefit to having it enabled.

Reported by: kib

Create the native-xtools target. This target creates only the cross
building toolchain for the host computer. This toolchain produces
TARGET_ARCH and assumes the rest of the system contains libraries for
the target. It is intended to be used in a "qemu-user jail" where all
the binaries would otherwise be the target architecture's to build
ports. However, emulation of the compilers is too slow, so we build
native binaries for that. Rather than use the xdev produced binaries,
with all their weird links and paths, these binaries use the native
paths. They will not work unless installed into the qemu-user jail.

Differential Revision: https://phabric.freebsd.org/D518
Reviewed by: sbruno@

Fix atmconfig compilation when MK_ATM == yes and MK_BSNMP == no

Makefile.inc1:
Always compile gensnmptree with bootstrap-tools when MK_BSNMP != no
instead of depending on a potentially stale tool installed on the build host

sbin/atm/atmconfig/Makefile:
- Always remove oid.h to avoid cluttering up the build/src tree.
- Consolidate all of the RESCUE/MK_BSNMP != no logic under one
conditional to improve readability
- Remove unnecessary ${.OBJDIR} prefixing for oid.h and use ${.TARGET} instead
of spelling out oid.h
- Add a missing DPADD for ${LIBCRYPTO} when compiled MK_BSNMP == yes and
MK_OPENSSL == yes and not compiling for /rescue/rescue

sbin/atm/atmconfig/main.c:
Change #ifndef RESCUE to #ifdef WITH_BSNMP in main.c to make it
clear that we're compiling bsnmp support into atmconfig

Approved by: jmmv (mentor)
Phabric: D579
PR: 143830
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Run dtrace in 32-bit mode when compiling 32-bit libraries.

MFC after: 3 days

m4 now requires libohash, ensure it gets built when bootstrapping.

10 has a new flex (2.5.37) and the config.h for unbound has been updated to
take this into account. Alas it breaks source upgrade from any version of
9 because flex is not built as a bootstrap-tools (it would be for older
versions).

That means "libunbound/configlexer.c" is built with the old flex but using
config.h for the new one. Build is thus broken going from 9.* to 10.

Make flex a bootstrap-tools entry if host is less than 1000033 to take into
account the flex update in 10.

Tested on both 9.2-RC3 and 9.3 by myself and dim@. Running buildworld in
head but as both 10 and 11 has the new flex, it will not matter.

Reviewed by: imp
Approved by: des, imp
MFC after: 1 week
Phabric: D554

Merge from CheriBSD:

Make the sysinit tool a build tool rather than building in with
/usr/bin/cc and running it from OBJDIR. (It will be moved to usr.bin
once a manpage is written and a few style cleanups are done.)

Split the makefile bits for Hans' kernel shim layer into their own
includable kshim.mk.

Move USB support into a .mk file so loaders can include it.

Deprecate the use of XDEV and XDEV_ARCH and just use TARGET/TARGET_ARCH
for the xdev build target, which is awesome and totally works.

Reapply svn R268377 with correct name of libsupc++ here as this does
resolve one dependancy race when building the xdev target.

the xdev target builds for amd64, i386, mips, mips64 and armv6 with this commit,
must be built as root, must be built from /usr/src, must not have a /usr/obj and
places the xdev tools in /usr/$TARGET_ARCH-freebsd

the xdev target still leaves some assorted files strewn about your /usr/src when
this is done and needs to be investigated further.

Phabric: https://phabric.freebsd.org/D385
Submitted by: bsdimp

Run mtree for BSD.tests.dist during make xdev-install, if the tests are
enabled (which they are in the default configuration). Otherwise, it
will fail because ${XDDESTDIR}/usr/include/atf-c does not exist.

MFC after: 3 days

Make MK_GNUCXX mean "build the libstdc++ and libsupc++ libraries" and
nothing more. Force it to be "no" when MK_CXX is "no" to simplify
usage. It no longer also means "build g++" since we no longer have a
platform where that's interesting now that pc98 no longer needs clang
and gcc, but not g++. pc98 now just uses clang after boot2 changes.

Separate out the links creation from the other targets. This was
supposed to have been done for the original commit, but somebody
forgot.

Pointy-hat-to: imp@

Typo means that this couldn't have possibly fixed anything, so revert
r268377.

xdev builds libsupc++ and libstdc++ in a slightly strange way. This
cause a race to be exposed between the two. Compensate for this race
by serializing the build/install of libstdc++ before libsupc++.

Remove ia64 from the list of known architectures and add an entry to
UPDATING. This is the first step towards the removal of ia64 from
head. A buildworld for ia64 will now yield:

% make buildworld
make[1]: "/usr/src/Makefile.inc1" line 151: Unknown target ia64:ia64.

While here, trim the ia64-specific additions from ObsoleteFiles.inc

Discussed at: BSDcan

Merge ^/projects/release-debugdist into ^/head:
r262491, r262493, r262516, r267345, r267397:

r262491:
Add DEBUG_DISTRIBUTIONS, and set it to include base and
EXTRA_DISTRIBUTIONS, excluding 'doc', since the documentation
distribution does not have corresponding debug information.

Use DEBUG_DISTRIBUTIONS in the 'distributeworld installworld'
and 'packageworld' targets, to reduce the number of occurances
of excluding distributions that do not have .debug files.

r262493:
In release/Makefile, explicitly set WITHOUT_DEBUG_FILES=1
for dvdrom and cdrom targets. (Later reverted.)

Exclude the *.debug.txz distributions from dvdrom and
cdrom images, but include them for ftp distribution.

r262516:
Rename ${dist}.debug.txz to ${dist}-dbg.txz to prevent the
following output:
eval: ${base....}: Bad substitution
eval: ${doc....}: Bad substitution
eval: ${games....}: Bad substitution
eval: ${lib32....}: Bad substitution

This also follows other naming conventions seen in the
wild.

r267345:
Explicitly set MK_DEBUG_FILES=no, which overrides the
WITH_DEBUG_FILES=1 and WITHOUT_DEBUG_FILES=1 collisions
previously experienced.

This change allows us to create the {base,kernel}_debug.txz
distributions without accidentally installing the *.debug
files on the medium itself.

r267397:
Remove evaluations of MK_DEBUG_FILES where not needed.
If DEBUG_DISTRIBUTIONS is empty, which is true if
MK_DEBUG_FILES evaluates to 'no' above, the loop does
nothing.

MFC after: 1 month
Tested on: head@r267801
Reviewed by: brooks [1], emaste, imp [1]
[1] earlier version
Sponsored by: The FreeBSD Foundation

Rename the WITHOUT_VT_SUPPORT knob to WITHOUT_VT

The _SUPPORT knobs have a consistent meaning which differs from the
behaviour controlled by this knob. As the knob is opt-out and has not
appeared in a release the impact should be low.

Suggested by: imp, wblock
MFC after: 1 week

- Add a LOCAL_ITOOLS to allow adding additional tools required for the
installworld and distributeworld targets

PR: 179562
Submitted by: Garrett Cooper <yaneurabeya@gmail.com>
MFC after: 1 week

Build vt(4) fonts during buildworld

vtfontcvt(8) is now built during buildworld, so can be used as a
bootstrap tool to create vt(4) fonts from source .hex or .bdf font
files, rather than having uuencoded binary fonts in the tree.

Sponsored by: The FreeBSD Foundation

Put the test suite in its own tests.txz distribution file.

Force all the contents of /usr/tests to go into a separate distribution
file so that users of binary releases can easily choose to not install it.

To make this possible, we need two fixes:
- bsd.subdir.mk needs to properly honor NO_SUBDIR in all cases so that we
do not recurse into 'tests' subdirectories when we needn't. Otherwise,
we end up with some Kyuafiles in base.txz.
- etc/Makefile needs to skip installing tests in its 'distribute' target
so that a Kyuafile doesn't leak into base.txz.

Approved by: gjb

In preparation for ASLR [1] support add WITH_PIE to support building with -fPIE.

This is currently an opt-in build flag. Once ASLR support is ready and stable
it should changed to opt-out and be enabled by default along with ASLR.

Each application Makefile uses opt-out to ensure that ASLR will be enabled by
default in new directories when the system is compiled with PIE/ASLR. [2]

Mark known build failures as NO_PIE for now.

The only known runtime failure was rtld.

[1] http://www.bsdcan.org/2014/schedule/events/452.en.html
Submitted by: Shawn Webb <lattera@gmail.com>
Discussed between: des@ and Shawn Webb [2]

When libelf and libdwarf were updated, we didn't bump the minimal
version needed for CTF tools, so sometimes we'd use the host's CTF
tools that didn't work. Be sure to bootstrap in that case.

Fix build: Build libavl as prebuild-lib.

X-MFC-With: 266520

Eliminate EARLY_BUILD flag. It is redundant and means MK_CLANG_FULL=no
and MK_LLDB=no, so set those explicitly (now that we can do
that). Simplify tests for these variables as well, since we know they
will always be defined regardless of the phase of the build.

Migrate NO_WARN to MK_WARN. Support legacy NO_WARN usage. Remove a
check for EARLY_BUILD because it isn't necessary (MK_WARN=no will
always be defined for that).

Document src.opts.mk changes and the decoupling of /etc/src.conf from
anything but the source tree.

Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.

Add appropriate quoting to allow building with a KERNCONFDIR containing
spaces.

PR: kern/162736
MFC after: 1 week

Turns out this .if evaluated not in the context of the makedtb target
sometimes due to Makefile expansion rules. Make the test for things
being NULL elsewhere...

Allow FDT_DTS_FILE to be a list, either in the makedtb target, or in a
kernel config file. If you also want to have a static DTB compiled
into your kernel, however, it cannot be a list. We have no mechanism
in the kernel for picking one, so that doesn't make sense and will
result in a compile-time error.

Add dependency for libm to libstdc++. This fixes high -j value builds
when not building with clang.

Submitted by: ian@

make_dtb.sh is designed to be used in a kernel build environment where
MACHINE is defined to the target's value, not the host's
value. However, in Makefile.inc1, it is still defined to be the host's
value. Make the makedtb target work by expanding TARGET in the
existance test, and passing MACHINE=$TARGET in the call to make_dtb.sh

Eliminate NO_INCS. It is used one place, and MK_TOOLCHAIN=no will
handle the job just as well. Opt for simplicity rather than one more
option.

Spell NO_PROFILE= as MK_PROFILE=no.

Remove NO_CTF, exccept as an undocumented compatibility
option. Convert all other uses to MK_CTF=no. Set MK_CTF=no rather than
the indirect WITHOUT_CDDL in filemon regression. It is expected that
NO_CTF will be removed in FreeBSD 12 entirely.

Separate out enabling building clang and/or gcc for the system and
building clang and/or gcc as the bootstrap compiler. Normally, the
default compiler is used. WITH_CLANG_BOOTSTRAP and/or
WITH_GCC_BOOTSTRAP will enable building these compilers as part
bootstrap phase. WITH/WITHOUT_CLANG_IS_CC controls which compiler is
used by default for the bootstrap phase, as well as which compiler is
installed as cc. buildworld now successfully completes building the
cross compiler with WITHOUT_CLANG=t and WITHOUT_GCC=t and produces a
built system with neither of these included.

Similarlly, MK_BINUTILS_BOOTSTRAP controls whether binutils is built
during this phase.

WITHOUT_CROSS_COMPILER will now force MK_BINUTILS_BOOTSTRAP=no,
MK_CLANG_BOOTSTRAP=no and MK_GCC_BOOTSTRAP=no.

BOOTSTRAP_COMPILER was considered, but rejected, since pc98 needs both
clang and gcc to bootstrap still. It should be revisisted in the
future if this requirement goes away. Values should be gcc, clang or
none. It could also be a list.

The odd interaction with Xfoo cross/external tools needs work, but
is beyond the scope of this change as well.

Up the minimum system to build FreeBSD current to 8.0-RELEASE. The
issues with vendors that needed 7.x support have been resolved. Many
vendors are still using 8.x build platforms, however, so bumping this
up to 9.0 will have to wait until that is resolved. Actual support for
building from 8.x still relies on those vendors fixing bugs that are
present as most developers have moved onto 9.x or newer platforms.

Reviewed by: marcel@

Determine whether to build clang and its bootstrap tools the same
way. This allows a clang bootstrap to happen, even when WITHOUT_CLANG
is defined. This is a minimal version of a more extensive change which
can be MFC'd more easily. However, we have to also test to see if
we're building clang as not cc, since the bootstrap for that needs
these cross tools and it is easier to build them in just one place.

MFC after: 1 week

Prevent XDTP from being a relative path
XDTP is used as the default SYSROOT for clang and thus should be an absolute path.

PR: arm/188249
Submitted by: Edgar Martinez <wink15987@gmail.com>
Reviewed by: imp

XDDESTDIR should not be set by the user, instead the user might want to tweak
XDTP and let the standard DESTDIR decide where to install.

Discussed with: imp

Some xdev fixes:

- if TARGET_ARCH is not defined and XDEV_ARCH is defined then early define
TARGET_ARCH to the valud of XDEV_ARCH: This allow the xdev-build target
to be able to correctly chose the compiler it needs to build
- Allow overwriting XDTP to allow a user to not chose where the xdev env will
live in
- Fix build for gcc only xdev (like ia64) by providing the proper -B to the
toolchain and not relying on gcc being installed already in base
- Fix TOOLS_PREFIX so the generated toolchain has the right default sysroot when
installed intead of getting the DESTDIR one
- Fix supporting DESTDIR
- Also overwrite CXX (needed for cross building c++ libraries with clang) and
CPP (needed to cross build some libraries when gcc is the target default
compiler but gcc is not installed on the building host)

Discussed with: imp

It is possible that MK_GNUCXX is "yes" but MK_CXX is "no" so make sure
MK_CXX isn't "no" before building these libraries.

Ensure make xdev respect which compiler is CC
and which compilers should be built according to standard build options

There's no need to guess at the COMPILER_TYPE to pass it down. We
guess wrong for buildkernel when CC=gcc49, say. Eliminate all the
guessing. COMPILER_TYPE propigates properly on its own, if specified,
and we guess it correctly otherwise lower in the build. Also, fix
conditionals for armv6hf when using an external compiler chain. They
were broken before, but unused. Also, prefer checking the compiler
type over CLANG_IS_CC since the latter is only supposed to be used to
determine what symlinks to install (more fixes to follow).

Move stray targets out of the xdev section.

Kill NO_TESTS and use MK_TESTS=no when we want to skip them.

Allow the build system to safely set MK_FOO to avoid the ambiguity
when both WITH_FOO and WITHOUT_FOO are set. Use this where
possible. Only disallow setting of MK_FOO on the command line. This
was preferable to inventing a new mechanism or fixing the undef bug
(bin/183762) which precludes users from turning off anything we turn
off for parts of the build with WITHOUT_FOO prior to this.

When building g++, we need to build libsupc earlier to avoid a race
with libproc. Not sure why people didn't see this before now, but I
get it often for higher (20-30) -j builds, but never for -j1 builds.

Only set XDDESTDIR if it wasn't already set to allow people to
override it.

Add a new ARM TARGET_ARCH, armv6hf. This is considered experimental.

This targets the existing ARMv6 and ARMv7 SoCs that contain a VFP unit.
This is an optional coprocessors may not be present in all devices, however
it appears to be in all current SoCs we support.

armv6hf targets the VFP variant of the ARM EABI and our copy of gcc is too
old to support this. Because of this there are a number of WITH/WITHOUT
options that are unsupported and must be left as the default value. The
options and their required value are:
* WITH_ARM_EABI
* WITHOUT_GCC
* WITHOUT_GNUCXX

In addition, without an external toolchain, the following need to be left
as their default:
* WITH_CLANG
* WITH_CLANG_IS_CC

As there is a different method of passing float and double values to
functions the ABI is incompatible with existing armv6 binaries. To use
this a full rebuild of world is required. Because no floating point values
are passed into the kernel an armv6 kernel with VFP enabled will work with
an armv6hf userland and vice versa.

Fix kern/187712: config(8) does not respect KERNCONFDIR.
The impact of this bug is that you cannot build a kernel if both of the
following are true:
1) The kernel config file is in a non-default location
2) The kernel config file uses the "include" statement from config(5).

usr.sbin/config/main.c
usr.sbin/config/config.8
usr.sbin/config/config.h
usr.sbin/config/lang.l
Added a "-I path" option to config(8). By analogy to cc(1), it adds
an extra path in which the "include" statement will search for
files.

Makefile.inc1
Pass "-I ${KERNCONFDIR}" to config(8).

PR: kern/187712
Reviewed by: will, imp (previous version)
MFC after: 3 weeks
Sponsored by: Spectra Logic Corporation

With the more generous footprints today, it makes little sense to use
UFS1 by default any more. Switch to UFS2.

Remove IPX support.

IPX was a network transport protocol in Novell's NetWare network operating
system from late 80s and then 90s. The NetWare itself switched to TCP/IP
as default transport in 1998. Later, in this century the Novell Open
Enterprise Server became successor of Novell NetWare. The last release
that claimed to still support IPX was OES 2 in 2007. Routing equipment
vendors (e.g. Cisco) discontinued support for IPX in 2011.

Thus, IPX won't be supported in FreeBSD 11.0-RELEASE.

Use ${MAKE} so that we always use the same version/implementation
of make.

Integrate device-tree upstream files into the build process:
(1) Invoke cpp to bring in files via #include (although the old
/include/ stuff is supported still).
(2) bring in files from either vendor tree or freebsd-custom files
when building.
(3) move all dts* files from sys/boot/fdt/dts to
sys/boot/fdt/dts/${MACHINE} as appropriate.
(4) encode all the magic to do the build in sys/tools/fdt/make_dtb.sh
so that the different places in the tree use the exact same logic.
(5) switch back to gpl dtc by default. the bsdl one in the tree has
significant issues not easily addressed by those unfamiliar with
the code.

Merge from CheriBSD:
commit 1b41f6de7ca09e04fdc6f66bc478ea6c981a41b9
Author: Brooks Davis <brooks@one-eyed-alien.net>
Date: Mon Jan 27 22:59:02 2014 +0000

Now that mtree is always nmtree use it as mtree

Tested on: ref9-amd64
X-MFC after: never
Sponsored by: DARPA, AFRL

Fix the build so -DNO_TESTS is passed in various phases that don't
require tests in order to build or install. Crucially, don't try to
install tests during the lib32 install phase. This commit supersedes
r261081, which fixed the lib32 install phase problem, but didn't fix
other phases.

Submitted by: Garrett Cooper
Reviewed by: sjg
MFC after: 13 days

Replace the old unix_seqpacket and unix_seqpacket_exercise tests, which
were a little broken and not automatable, with unix_seqpacket_test.
It's coverage is a superset of the old tests and it uses ATF. It
includes test cases for bugs kern/185813 and kern/185812.

PR: kern/185812
PR: kern/185813
Sponsored by: Spectra Logic
MFC after: 2 weeks

Generate /var/db/services.db during 'make distribution' so that it is
present during new installs. Update etcupdate and mergemaster to
ignore the generated file.

Tested by: gjb (release build)
MFC after: 1 month

Remove aicasm as a build dependency. It made sense when the ahc and ahd
drivers and their firmware were under active development, but those days
have passed. The firmware now exists in pre-compiled form, no longer
dependent on it's sources or on aicasm. If you wish to rebuild the
firmware from source, the glue still exists under the 'make firmware'
target in sys/modules/aic7xxx.

This also fixes the problem introduced with r257777 et al with building
kernels the old fashioned way in sys/$arch/compile/$CONFIG when the
ahc/ahd drivers were included.

Please welcome casperd daemon. It (and its services) will be responsible for
giving access to functionality that is not available in capability mode
sandbox. The functionality can be precisely restricted.

Start with the following services:
- system.dns - provides API compatible to:
- gethostbyname(3),
- gethostbyname2(3),
- gethostbyaddr(3),
- getaddrinfo(3),
- getnameinfo(3),
- system.grp - provides getgrent(3)-compatible API,
- system.pwd - provides getpwent(3)-compatible API,
- system.random - allows to obtain entropy from /dev/random,
- system.sysctl - provides sysctlbyname(3-compatible API.

Sponsored by: The FreeBSD Foundation

Move my simple logging API to a separate library. It is now already used
by hastctl(8), hastd(8) and auditdistd(8) and will soon be also used
by casperd(8) and its services. There is no documentation and pjdlog.h
header file is not installed in /usr/include/ to keep it private.
Unfortunately we don't have /lib/private/ at this point, only
/usr/lib/private/, so the library is installed in /lib/.

Sponsored by: The FreeBSD Foundation

Attempt to move the POSIX iconv* symbols out of runtime linker space.
FreeBSD systems usually implemented this as a third party module and
our implementation hasn't played as nicely with the old way as it could
have.

To that end:
* Rename the iconv* symbols in libc.so.7 to have a __bsd_ prefix.
* Provide .symver compatability with existing 10.x+ binaries that
referenced the iconv symbols. All existing binaries should work.
* Like on Linux/glibc systems, add a libc_nonshared.a to the ldscript
at /usr/lib/libc.so.
* Move the "iconv*" wrapper symbols to libc_nonshared.a

This should solve the runtime ambiguity about which symbols resolve
to where. If you compile against the iconv in libc, your runtime
dependencies will be unambiguous.

Old 9.x libraries and binaries will always resolve against their
libiconv.so.3 like they did on 9.x. They won't resolve against libc.

Old 10.x binaries will be satisified by the .symver helpers.

This should allow ports to selectively compile against the libiconv
port if needed and it should behave without ambiguity now.

Discussed with: kib

Move all atf directories to the tests mtree.

This is to ensure that test-related directories don't get needlessly
created (and later deleted) when MK_TESTS=no.

Problem found by jhb@.

Approved by: rpaulo (mentor)

Strip the -pN patch level from the VERSION string which gets encoded into
CTF data. Otherwise FreeBSD Update builds think every kernel file has
changed every time there's a security advisory, since the FreeBSD Update
build code isn't smart enough to look inside CTF data to ignore those
changes.

Pointy hat to: cperciva
MFC after: 1 day, or before the next BETA

This change builds kernel tools based on the same assumption as building
the kernel itself: If building for the same architecture as the build host,
the kernel build assumes that the host toolchain is capable of building the
kernel. If it's not, "make kernel-toolchain" will bootstrap a new set of
tools that will work.

With this change the same assumptions are made for building kernel tools,
and the existing host toolchain is used to do the build (notably, the build
doesn't link the tools with the legacy libraries, which may not even exist).
If ever for some reason the host toolchain isn't capable of building the
kernel tools, then doing a "make kernel-toolchain" will bootstrap newer
tools to get the job done.

So when built as part of buildworld or kernel-toolchain, the kernel tools
are built using the XMAKE (via BMAKE) commands and environment. When built
as part of building just the kernel on a same-target host, the tools are
built using the new KTMAKE commands and environment. What doesn't jump
out at you in the diffs is that the difference between BMAKE and KTMAKE
is that BMAKE contains this magic line which changes how the build is done
because it changes what files get included for .include <bsd.prog.mk> and
other standard includes:

MAKEFLAGS="-m ${.CURDIR}/tools/build/mk ${.MAKEFLAGS}"

and KTMAKE doesn't, and contains this instead:

TOOLS_PREFIX=${WORLDTMP}

Hopefully this brings the "how to build aicasm with the right toolchain"
saga to a conclusion that works in all usage scenarios that have
historically been supported.

Subsume the functionality of MK_ATF into MK_TESTS.

There is no reason to keep the two knobs separate: if tests are
enabled, the ATF libraries are required; and if tests are disabled,
the ATF libraries are not necessary. Keeping the two just serves
to complicate the build.

Reviewed by: freebsd-testing
Approved by: rpaulo (mentor)

Add libatf-c++ to the prebuild libs.

Some tests may require C++ so we must ensure this library exists as part
of the bootstrap process or else they will fail to build. Do this by
just depending on lib/atf as part of the bootstrap libraries instead of
using lib/atf/libatf-c.

Submitted by: Garrett Cooper <yaneurabeya at gmail dot com>
Approved by: rpaulo (mentor)

Fix buildworld when WITH_TESTS is enabled.

The addition of the TESTS knob and its enabling of the build of tests in
lib/libcrypt/tests/ broke the build. The reason is that we cannot descend
into tests/ subdirectories until all prerequisites have been built, which
in the case of tests may be "a lot of things" (libatf-c in this case).

Ensure that we do not walk tests/ directories during the bootstrapping of
the libraries as part of buildworld.

Reviewed by: freebsd-testing
Approved by: rpaulo (mentor)

Build kernel tools along with other tools during world build, as well as
during kernel build (if they didn't get done with world). This will make
-DMODULES_WITH_WORLD work, and it ensures the kernel tools are built
as part of 'make kernel-toolchain'.

Instead of modeling the kernel-tools build after the bootstrap tools, build
kernel tools the way cross-tools get built. This seems to result in the
tool getting installed in the right place. It also seems more correct in
retrospect, because if a tool emitted code or binary data as part of
building the kernel, it should do so in target-specific ways (endianess,
architecture, whatever). That issue is moot for aicasm, our only current
tool, but it still seems to be more correct in principle.

Do not build aicasm with the cross-tools/kernel-toolchain, instead add a
proper kernel-tools step/target modeled after the world build-tools stuff.

This is a re-do of r257730 which was backed out in r257734, but this time
it's one byte smaller... a leftover trailing backslash resulted in a .for
loop with no rules, so no compiler stuff got built and later steps built
with the wrong toolset.

Revert r257730:
Make head/ buildable again, instead of spewing garbage like:
/src/gnu/lib/csu/../../../contrib/gcc/config/rs6000/crtsavres.asm:280:
Error: no such instruction: `lwz 28,-16(11)'

Do not build aicasm with the cross-tools/kernel-toolchain, instead add a
proper kernel-tools step/target modeled after the world build-tools stuff.

Rework the aicasm build machinery so that it gets built along with toolchain
components instead of with the kernel and/or modules. This ensures that it
gets built with the host compiler, not the compiler in obj/... used to build
the target components (which may be a cross-compiler outputting code for a
different architecture and using header files with types and options set up
for the wrong architecture).

Reviewed by: imp

In r257079, SRCDIR is pointed to ${.CURDIR} when not set. However,
Makefile.inc1 is being called in sub-make's where make(1) would,
by default, implicitly chdir(2) to ${.OBJDIR} before executing any
targets. This would make some targets, like delete-old, when trying
to derive various variables introduced by change r256921 using
``make -f Makefile.inc1'' that also rely on SRCDIR to fail.

This changeset adds an explicit cd ${.CURDIR} before these unwrapped
make calls, making them in line with the other ones that are already
being wrapped with the explicit chdir's.

Tested by: gjb
MFC after: 5 days

Fix build. Both clang and gcc are required on pc98.

X-MFC with: r256915

Set up the /usr/tests hierarchy.

Populate /usr/tests with the only test programs that currently live
in the tree (those in lib/libcrypt/tests/) and add all the build
machinery to accompany this change.

In particular:

- Add a WITHOUT_TESTS variable that users can define to request that
no tests be put in /usr/tests.
- Add a top-level Kyuafile for /usr/tests and a way to create similar
Kyuafiles in top-level subdirectories.
- Add a BSD.tests.dist file to define the directory layout of
/usr/tests.

Submitted by: Julio Merino jmmv google.com
Reviewed by: sjg
MFC after: 2 weeks

Fix build host pollution by avoiding calling 'uname -srp' to
determine values for 'VERSION'.

Looked at by: cperciva

Revert r256921 to prevent error output when in the wrong directory.
This should have been reverted with the stable/10/Makefile.inc1
revert, but apparently my commit did not go through.

Discussed with: cperciva (originally)

Thou shalt not leak build host state into the system being compiled.

The VERSION variable is encoded into the SUNW_ctf sections of the kernel
and every kernel module when dtrace is enabled; starting with 9.2-RELEASE
(when dtrace was turned on in GENERIC) this means that different host kernels
will result in very different kernel binaries being generated. This tripped
up freebsd-update builds after the build boxes were updated from 9.x to 10.x.

MFC after: 3 days (stable/9)
X-MFC after: 0 days (stable/10)
Security: Rendered two members of so@ temporarily insane

Stop conflating WITHOUT_CLANG with WITHOUT_CLANG_IS_CC. This allows
bootstrapping a copy of clang without building clang for the base system
which is useful for nanobsd and similar setups. It's still probably
wrong to conflate what is installed as /usr/bin/cc with the selection
of a bootstrap compiler under WITH*_CLANG_IS_CC, but that's for another
day.

MFC after: 1 week
Sponsored by: DARPA/AFRL

Fix 'make delete-old-libs' and 'make check-libs' to delete .debug
files created by WITH_DEBUG_FILES. Also cleanup .symbols files from
the period between r244236 when .symbols were supported and r251512
when they were renamed to .debug.

Only propose to delete a .debug file if the corresponding library
itself was deleted already.

Reported by: des
Reviewed by: emaste (earlier version)
Approved by: bapt
MFC after: 3 days

Allow 'make xdev' to work when DESTDIR is set.

Submitted by: Patrick Kelsey <kelsey@ieee.org>

Bump OS versions in the toolchain triples to 11.0, and bump the
__FreeBSD_cc_version predefined macros in clang and gcc.

Approved by: re (gjb)

Revert-and-redo r255955: the sort -r should be added to delete-old-dirs.

Approved by: re (gjb)

Reverse directories order when doing 'make delete-old-dir'. This
ensures subdirectories gets removed before their parents when doing
make delete-old.

Approved by: re (gjb)
MFC after: 2 weeks

Remove BIND.

Approved by: re (gjb)

Don't parse NO_ROOT metadata for extra kernels if NO_ROOT isn't defined.

Approved by: re (gjb)
MFC after: 1 week

Remove the armv6eb architecture as it is unused, and almost certainly
broken. None of our kernels can boot armv6eb. The little-endian kernels do
not have the required code to be able to switch endian when running a
big-endian executable.

Approved by: re (gjb)

Fix 'make installcheck' to check for missing UID/GID as well, broken
since r249893, by adding a separate _installcheck_world and
_installcheck_kernel so the destination targets can be more explicit
on which they are needed for.

installcheck will call both, while installworld only calls
_installcheck_world and installkernel only calls _installcheck_kernel

While here, mark the internal targets as starting with _.

Reported by: des
Reviewed by: des
Pointyhat to: bdrewery
Approved by: re (delphij)

Add unbound to the list of UIDs / GIDs to check fore before installing.

Approved by: re (blanket)

Clean up the OpenSSH build. It is now possible to build most components
as static binaries, if desired. The one exception is sshd, which runs
into trouble due to libpam.a's includion of pam_ssh.

Make OpenSSH use LDNS if available. This allows it to verify signed
SSHFP records.

Approved by: re (blanket)

Create a private library directory (LIBPRIVATEDIR) for libraries which
we don't want to expose but which can't or shouldn't be static.

To mark a library as private, define PRIVATELIB in its Makefile. It
will be installed in LIBPRIVATEDIR, which is normally /usr/lib/private
(or /usr/lib32/private for 32-bit libraries on 64-bit platforms).

To indicate that a program or library depends on a private library,
define USEPRIVATELIB in its Makefile. The correct version of
LIBPRIVATEDIR will be added to its run-time library search path.

Approved by: re (blanket)

- Fix LOCAL_MTREE so it properly handles multiple files and quotes
its value into submakes

PR: conf/179466
Submitted by: Garrett Cooper <yaneurabeya@gmail.com>
Approved by: bapt
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Update nvi-1.79 to 2.1.1-4334a8297f

This is the gsoc-2011 project to clean up and backport multibyte support
from other nvi forks in a form we can use.

USE_WIDECHAR is on unless building for the rescue crunchgen. This should
allow editing in the native locale encoding.

USE_ICONV depends on make.conf having 'WITH_ICONV=YES' for now. This
adds the ability to do things like edit a KOI8-R file while having $LANG
set to (say) en_US.UTF-8. iconv is used to transcode the characters for
display.

Other points:
* It uses gencat and catopen/etc instead of homegrown msg catalog stuff.
* A lot of stuff has been trimmed out, eg: the perl and tcl bindings which
we could never use in base anyway.
* It uses ncursesw when in widechar mode. This could be interesting.

GSoC info: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/zy/1
Repo at: https://github.com/lichray/nvi2

Obtained from: Zhihao Yuan <lichray@gmail.com>

Generate debug data release dist sets

If WITH_DEBUG_FILES is set the standalone debug data will be excluded
from each ${dist}.txz and placed in a ${dist}.debug.txz.

Submitted by: gjb
Reviewed by: brooks

Sprinkle some .MAKE magic

Use && rather than ; when success of previous job matters.

Simplify the hierarchy target's logic. [0]

Run hierarchy with WORLDTMP in the path so it works when it is invoked
directly. Such use is nearly alwasy wrong but appears to be common.[1]

PR: conf/178209 [0], conf/178547 [1]
Submitted by: Garrett Cooper <yaneurabeya@gmail.com> [0]
MFC after: 5 days

Flag recursive make targets with .MAKE
so that job token pipe is passed to them.
To avoid surprising anyone, only add .MAKE to ${TGTS} when -n
has not been specified (at least for Makefile).

Reviewed by: obrien

Be more agressive about bootstrapping ctfmerge and ctfconvert so
builds from existing releases have a chance of working properly.

Sponsored by: DARPA, AFRL
MFC after: 3 days

Add a new knob WITH_DEBUG_FILES to control the building of standalone
debug files for userland programs and libraries. The "-g" debug flag
is automatically applied when WITH_DEBUG_FILES is set.

The debug files are now named ${prog}.debug and ${shlib}.debug for
consistency with other systems and documentation. In addition they are
installed under /usr/lib/debug, to simplify the process of installing
them if needed after a crash. Users of bsd.{prog,lib}.mk outside of the
base system place the standalone debug files in a .debug subdirectory.
GDB automatically searches both of these directories for standalone
debug files.

Thanks to everyone who contributed changes, review, and testing during
development.

release: Allow empty extra distributions.

For example, WITHOUT_SHAREDOCS= in src.conf creates an empty doc
distribution.

Submitted by: Kurt Lidl
Tested by: Kurt Lidl
Discussed with: gjb
MFC after: 1 week

Restore (at least temporarily) SHARED=symlinks in the build includes
stage of buildworld. This was accidentally included in r251140.

Reported by: jhb

Always define INSTALL_DDIR and define it such that it contains no extra
/ characters rather than removing them later on. This should fix
release builds.

PR: conf/178963
Reviewed by: gjb, hrs

r245757 introduced warning output if update method is set to CVS_UPDATE
or SUP_UPDATE.

CVS exporter for head/ is turned off for nearly one month now.

It is finally time to swing the ax at these update methods.

Reviewed by: eadler
MFC after: 1 month

Update comments to reflect use of svn/svnup to keep the src/ tree up
to date.

MFC after: 3 days

Temporarily revert r251058 - it breaks documented use of makeoptions
including the tinderbox.

http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-i386-i386.full
make: don't know how to make modules-all. Stop
make: stopped in /obj/i386.i386/src/sys/PAE
*** Error code 2

During buildkernel, print a banner before building modules.

Remove backwards compat layer which was added in r71419 (in 2001).

This change is not intended for MFC.

Reviewed by: peter

Fix building on slightly older -current and stable systems after
the switch to bmake. The rescue bits are built via crunchgen,
which didn't respect the MAKE environment variable until r237574
(i.e. ~11 months ago). This resulted in a failure due to bmake's
internal -J flag being passed around and not being understood by
the standard (i.e. host's) make.
Note that the failure is conditional upon having the jobServer
feature enabled within bmake.

Connect flex 2.5.37 to the build and bump __FreeBSD_version.

Restore the ability to build on systems with 32-bit compat when
the system compiler is not clang. clang and gcc appear to differ
signficantly in their interpretation of -isystem and --sysroot. Further
work is likely required to support an external gcc.

Reported by: andreast, fidaj@ukr.net, sergey.dyatko@gmail.com

Fix distributekernel in the non NO_ROOT case.

PR: conf/178775
Submitted by: Garrett Cooper <yaneurabeya@gmail.com>

Implement NO_ROOT support for distributekernel and packagekernel.

Add support for an external cross compiler. The cross compiler is
specified by passing the XCC, XCXX, and XCPP variables (corresponding to
CC, CXX, and CPP) to buildworld/buildkernel. The compiler must be clang
or be configured to target the appropriate architecture.

To speed build times, if XCC is an absolute path or
WITHOUT_CROSS_COMPILER is defined then no cross compiler will be built
during the cross-tools stage.

Limited documentation of this feature can currently be found at:

https://wiki.freebsd.org/ExternalToolchain

This functionality should be considered experimental and is subject to
change without notice.

Sponsored by: DARPA, AFRL
Discussed with: imp, sjg

Add compile_et to kerberos bootstrap tools.
This makes it possible to build a regular world on a host without kerberos.

PR: conf/178421
Submitted by: Garrett Cooper
MFC after: 1 month

Optimize SUBDIR_OVERRIDE such that SUBDIR isn't automatically defined if
SUBDIR_OVERRIDE is defined.

PR: conf/174071
Submitted by: Garrett Cooper <yaneurabeya@gmail.com>
Sponsored by: EMC / Isilon Storage Division

A transition period of more than two years is more than enough:
Remove the compatibility code added in 2011-02-10.

This change is not intended for MFC

Reviewed by: imp

Fix installworld with DB_FROM_SRC after r249807

Approved by: bapt
Reported by: Tom Everett

Fix installkernel requiring users/groups defined in CHECK_UIDS
and CHECK_GIDS to exist since r152680. This is only needed for
installworld. The documented procedure of running mergemaster -p
to check for missing users is only needed for installworld, not
for installkernel. This fixes auditdistd incorrectly being
required for installkernel.

PR: misc/174405
Approved by: bapt

This should have been committed in r249666.

MFC after: 1 month

Upgrade our copy of llvm/clang to trunk r178860, in preparation of the
upcoming 3.3 release (branching and freezing expected in a few weeks).

Preliminary release notes can be found at the usual location:
<http://llvm.org/docs/ReleaseNotes.html>

An MFC is planned once the actual 3.3 release is finished.

Fix xdev-install when installing to a location other than /

Fix xdev. Clang and libc were not building correctly on older worlds, for
example, on 9.1.
* To fix clang add an _xb-bootstrap-tools target that mirrors the existing
bootstrap-tools target in the full world.
* For libc have the compiler use the newly installed includes, and, while
here, tell the compiler about the xdev library path as some other
libraries will link against the installed libraries.

Fix a typo, 'xdev-build' should be spelled with a 'd' at the end.

Approved by: cognet

Merge libzfs_core branch:
includes MFV 238590, 238592, 247580

MFV 238590, 238592:
In the first zfs ioctl restructuring phase, the libzfs_core library was
introduced. It is a new thin library that wraps around kernel ioctl's.
The idea is to provide a forward-compatible way of dealing with new
features. Arguments are passed in nvlists and not random zfs_cmd fields,
new-style ioctls are logged to pool history using a new method of
history logging.

http://blog.delphix.com/matt/2012/01/17/the-future-of-libzfs/

MFV 247580 [1]:
To address issues of several deadlocks and race conditions the locking
code around dsl_dataset was rewritten and the interface to synctasks
was changed.

User-Visible Changes:
"zfs snapshot" can create more arbitrary snapshots at once (atomically)
"zfs destroy" destroys multiple snapshots at once
"zfs recv" has improved performance

Backward Compatibility:
I have extended the compatibility layer to support full backward
compatibility by remapping or rewriting the responsible ioctl arguments.
Old utilities are fully supported by the new kernel module.

Forward Compatibility:
New utilities work with old kernels with the following restrictions:
- creating, destroying, holding and releasing of multiple snapshots
at once is not supported, this includes recursive (-r) commands

Illumos ZFS issues:
2882 implement libzfs_core
2900 "zfs snapshot" should be able to create multiple,
arbitrary snapshots at once
3464 zfs synctask code needs restructuring

References:
https://www.illumos.org/issues/2882
https://www.illumos.org/issues/2900
https://www.illumos.org/issues/3464 [1]

MFC after: 1 month
Sponsored by: Hybrid Logic Inc. [1]

In all situations where we need to bootstrap yacc, we also need to
bootstrap liby. This was not readily apparent, because a normal 'make
buildworld' or 'make toolchain' builds liby before building anything
that uses yacc. However, 'make kernel-toolchain' does not build
headers or libraries, so it was not possible to build a kernel from
head on, say, stable/9 without first building the complete toolchain.

MFC after: 1 week

- Fix libmd dependency. It is needed in the bootstrap library list because
usr.bin/xinstall depends on it.
- Remove libutil from usr.bin/xinstall/Makefile. No symbol was actually used.

Reviewed by: brooks

Add nmtree to ITOOLS if it is installed on the host instead of keying off
the BOOTSTRAPPING variable. The previous test was wrong because
BOOTSTRAPPING is 0 in most cases.

Tested by: db

Fix xdev by using the install shell script as it knows about the -l
argument thile the local version install may not.

- Add CHECKSUM.* support in Makefile[1].
- Use ln -fs to create a symlink.
- Remove pkgadd for docports.
- Use WITHOUT_JADETEX=yes instead of WITH_JADETEX=no.
- Add {WORLD,KERNEL}_FLAGS to [BTWK]MAKE.
- Use makefs(8) and gpart(8) for sparc64 ISO image[2].
- Add publisher option to makefs(8)[2].

Based on work by: gjb[1]
Discussed with: marius, nwhitehorn[2]

Log the addition of login.conf.db, passwd, pwd.db, and spwd.db via cat -l.

Make cat a bootstrap tool to facilitate this.

Fix installworld when nmtree is not on the host by only adding it to
ITOOLS when we will be using the host copy.

Fix installkernel when -DNO_ROOT is set.

Fix installworld when the host does not need to bootstrap nmtree.

Reported by: lev, jmg

Fix update method (s/SUP/CVS) warning.

Submitted by: Ryan Stone
MFC after: 1 day

Strip any trailing slash off of the -D argument to install to support the
rather fragile destdir stripping in the metalog.

Introduce a new option -DNO_ROOT that allows install and distribution
targets to be run without root privilege.

Information about ownership, group, flags, and suid bits are stored in
the file specified by METALOG which defaults to ${DESTDIR}/METALOG.
This file can be used in conjunction with bsdtar or makefs to generate
archives or file system images with correct permissions.

The packageworld target has been altered to use this metadata allowing
non-root releases (subject to further changes in release/Makefile.)

Sponsored by: DARPA, AFRL
Reviewed by: ian, ray

Fix spurious leading / on dtc path in Makefile.inc1. Spotted by jhb.

Import new (BSDL) device tree compiler. Now built by default, so that it can't
be used on the host system (and not installed on the device, if required). The
GPL'd one is still available if there are any devices that need it (make
universe passes with it, including kernels that use fdt, but there may be some
out-of-tree ones). WITH_GPL_DTC can be used to select the old one, for now.

Probably won't be MFC'd, but we'll remove the GPL'd version in head after the
new one has had a lot more testing and ship it in 10.0.

Chase r245756 of make.conf(5):

- If update method is SUP_UPDATE or CVS, warn that those
update methods are deprecated.

- While here, remove bogus NO_WWWUPDATE.

MFC after: 3 days
X-MFC-with: r245756

Always bootstrap nmtree if it is not available on the host. It is fairly
small and it is quite annoying to get all the way through buildworld
only to discover that you forgot the DB_FROM_SRC (or soon NO_ROOT)
option.