Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC improved vt(4) font generation

r266851: Add VGAROM 8x8, 8x14 and 8x16 fonts for vt(4)

These are converted from syscons(4) cp437 fonts.

r267306: Add vgarom font source

These are in 'GNU Unifont' format, and are converted from syscons(4)
cp437 fonts.

r267400: Add thin versions of VGAROM 8x8 and 8x16 fonts for vt(4)

These are converted from syscons(4) cp437-thin-8x* fonts.

r267423: Build vt(4) fonts during buildworld

vtfontcvt(8) is now built during buildworld, so can be used as a
bootstrap tool to create vt(4) fonts from source .hex or .bdf font
files, rather than having uuencoded binary fonts in the tree.

r267578: Add glyphs from converted syscons iso* fonts

This consists of the unique glyphs from the following font files in
/usr/share/syscons/fonts:

iso*.fnt ISO-8859-1 West European
iso02*.fnt ISO-8859-2 Central European
iso04*.fnt ISO-8859-4 Baltic
iso05*.fnt ISO-8859-5 Cyrillic
iso07*.fnt ISO-8859-7 Greek
iso08*.fnt ISO-8859-8 Hebrew
iso09*.fnt ISO-8859-9 Turkish
iso15*.fnt ISO-8859-15 West European

r268022: Rename the WITHOUT_VT_SUPPORT knob to WITHOUT_VT

The _SUPPORT knobs have a consistent meaning which differs from the
behaviour controlled by this knob. As the knob is opt-out and has not
appeared in a release the impact should be low.

Approved by: re
Sponsored by: The FreeBSD Foundation

MFC r270027:

tmconfig compilation when MK_ATM == yes and MK_BSNMP == no

Makefile.inc1:
Always compile gensnmptree with bootstrap-tools when MK_BSNMP != no
instead of depending on a potentially stale tool installed on the build host

sbin/atm/atmconfig/Makefile:
- Always remove oid.h to avoid cluttering up the build/src tree.
- Consolidate all of the RESCUE/MK_BSNMP != no logic under one
conditional to improve readability
- Remove unnecessary ${.OBJDIR} prefixing for oid.h and use ${.TARGET} instead
of spelling out oid.h
- Add a missing DPADD for ${LIBCRYPTO} when compiled MK_BSNMP == yes and
MK_OPENSSL == yes and not compiling for /rescue/rescue

sbin/atm/atmconfig/main.c:
Change #ifndef RESCUE to #ifdef WITH_BSNMP in main.c to make it
clear that we're compiling bsnmp support into atmconfig

Revert r270079 which was MFC of r269688, build libohash while bootstrapping.

The changes to split the ohash code out of m4 into its own library haven't
been MFC'd, so this change was causing an error during bootstrap when
building stable-10.

Submitted by: Dai Sieh <dsieh@dsieh.com>
Pointy hat to: ian

MFC r260401

Remove aicasm as a build dependency. It made sense when the ahc and ahd
drivers and their firmware were under active development, but those days
have passed. The firmware now exists in pre-compiled form, no longer
dependent on it's sources or on aicasm. If you wish to rebuild the
firmware from source, the glue still exists under the 'make firmware'
target in sys/modules/aic7xxx.

This also fixes the problem introduced with r257777 et al with building
kernels the old fashioned way in sys/$arch/compile/$CONFIG when the
ahc/ahd drivers were included.

MFC r257637, r257730, r257734, r257777, r257825, r257838, r257873:

Changes to how the aicasm tool is built. This series of changes results
in the aicasm tool being built as part of the tools stages of world and
kernel builds.

Most of these changes will ultimately be undone when r260401 is MFC'd,
but it will leave in place the new kernel-build-tool machinery (KTMAKE
stuff) in case a new special kernel tool ever comes along.

MFC r266473,267331,267511:

Use an intermediate target to associate with _SUBDIR which is marked .MAKE
this allows make -n to do tree walks as expected without
doing anything else (as intended).
Use prefix _sub. to help avoid conflict with any real target.

Put the test suite in its own tests.txz distribution file.
Force all the contents of /usr/tests to go into a separate distribution
file so that users of binary releases can easily choose to not install

Create a mechanism for providing fine-grained build order dependencies
during SUBDIR_PARALLEL builds. This augments the coarse .WAIT mechanism,
which is still useful if you've got a situation such as "almost everything
depends on A and B".

MFC r269688: m4 now requires libohash, build it when bootstrapping.

MFC r269662:

10 has a new flex (2.5.37) and the config.h for unbound has been updated to
take this into account. Alas it breaks source upgrade from any version of
9 because flex is not built as a bootstrap-tools (it would be for older
versions).

That means "libunbound/configlexer.c" is built with the old flex but using
config.h for the new one. Build is thus broken going from 9.* to 10.

Make flex a bootstrap-tools entry if host is less than 1000033 to take into
account the flex update in 10.

Tested on both 9.2-RC3 and 9.3 by myself and dim@. Running buildworld in
head but as both 10 and 11 has the new flex, it will not matter.

Reviewed by: imp
Approved by: des, imp
MFC after: 1 week
Phabric: D554

MFC r269744
Run dtrace in 32-bit mode when compiling 32-bit libraries.

MFC r268957:

Run mtree for BSD.tests.dist during make xdev-install, if the tests are
enabled (which they are in the default configuration). Otherwise, it
will fail because ${XDDESTDIR}/usr/include/atf-c does not exist.

MFC r267599:

- Add a LOCAL_ITOOLS to allow adding additional tools required for the
installworld and distributeworld targets

PR: 179562

MFC r266520:

Explicitly link libzfs against libavl as it is done in OpenSolaris
(4543:12bb2876a62e). Without this, some third party applications
may break because the lack of AVL related symbols.

FreeBSD base system are not affected because the FreeBSD ZFS command
line tools were all linked against libavl and thus hide the underlying
issue.

PR: bin/183081

MFC 263949, 265063: Add library dependencies for a couple bootstrap libs
to fix build failures for high -jN values when building with gcc.

MFC 262614, 262625, 262626, 262627, 262682, 262714, 262725, 262736

Integrate device-tree upstream files into the build process:
(1) Invoke cpp to bring in files via #include (although the old
/include/ stuff is supported still).
(2) bring in files from either vendor tree or freebsd-custom files
when building.
(3) move all dts* files from sys/boot/fdt/dts to
sys/boot/fdt/dts/${MACHINE} as appropriate.
(4) encode all the magic to do the build in sys/tools/fdt/make_dtb.sh
so that the different places in the tree use the exact same logic.
(5) switch back to gpl dtc by default. the bsdl one in the tree has
significant issues not easily addressed by those unfamiliar with
the code.

Only try to build the static dtb when we're building a static dtb.

Use proper include path for dtc as well as cpp.

Fix syntax errors (missing ; other minor glitches) in existing dts files.

MFC rev. 265260:
Add appropriate quoting to allow building with a KERNCONFDIR containing
spaces.

PR: kern/162736

MFC: r263881, r263952, r263983, r264068, r264090, r264131

Some xdev fixes:

- if TARGET_ARCH is not defined and XDEV_ARCH is defined then early define
TARGET_ARCH to the valud of XDEV_ARCH: This allow the xdev-build target
to be able to correctly chose the compiler it needs to build
- Allow overwriting XDTP to allow a user to not chose where the xdev env will
live in
- Fix build for gcc only xdev (like ia64) by providing the proper -B to the
toolchain and not relying on gcc being installed already in base
- Fix TOOLS_PREFIX so the generated toolchain has the right default sysroot when
installed intead of getting the DESTDIR one
- Fix supporting DESTDIR
- Also overwrite CXX (needed for cross building c++ libraries with clang) and
CPP (needed to cross build some libraries when gcc is the target default
compiler but gcc is not installed on the building host)
- Prevent XDTP from being a relative path:
XDTP is used as the default SYSROOT for clang and thus should be an absolute path.
- Respect default CC

MFC r263429

Fix kern/187712: config(8) does not respect KERNCONFDIR.
The impact of this bug is that you cannot build a kernel if both of the
following are true:
1) The kernel config file is in a non-default location
2) The kernel config file uses the "include" statement from config(5).

usr.sbin/config/main.c
usr.sbin/config/config.8
usr.sbin/config/config.h
usr.sbin/config/lang.l
Added a "-I path" option to config(8). By analogy to cc(1), it adds
an extra path in which the "include" statement will search for
files.

Makefile.inc1
Pass "-I ${KERNCONFDIR}" to config(8).

Turn off warnings for building aicasm. Various versions of yacc exist,
which can result in different warnings-as-errors in this tool, but these
are all completely harmless.

This is a direct commit to stable/7, stable/8, stable/9 and stable/10,
since this kernel build tool has already been removed in head.

MFC r261132

Fix the build so -DNO_TESTS is passed in various phases that don't require
tests in order to build or install. Crucially, don't try to install tests
during the lib32 install phase. This commit supersedes r261081, which fixed
the lib32 install phase problem, but didn't fix other phases.

Submitted by: Garrett Cooper
Reviewed by: sjg

Plug the ATF tests into the build.

This is a MFC into stable/10 of:
- r257849 Add libatf-c++ to the prebuild libs.
- r257853 Build and install the atf tests.
- r258233 Move all atf directories to the tests mtree.
- r258285 Fix the build of some ATF tests.

This change is "make tinderbox" clean on ref10-amd64 with the default
settings of WITHOUT_TESTS. It is likely for the WITH_TESTS build to
still be broken because not all relevant changes have been merged yet.

Set up the /usr/tests hierarchy.

This is a MFC of the following into stable/10:
- r257097 Set up the /usr/tests hierarchy.
- r257098 Add missing WITHOUTTESTS file.
- r257100 Add a tests(7) manual page.
- r257105 Disable WITHTESTS= for now.
- r257848 Fix buildworld when WITHTESTS is enabled.
- r257850 Subsume the functionality of MKATF into MKTESTS.
- r257851 Handle the removal of the test suite when WITHOUTTESTS=yes.
- r257852 Install category Kyuafiles from their category directories.
- r258232 Install BSD.tests.mtree when MKTESTS is yes.

Note that building with WITH_TESTS is still broken at this point (and
hence why WITHOUT_TESTS is the set as the default). Subsequent pullups
will fix the remaining issues.

MFC 256640: Allow 'make xdev' to work when DESTDIR is set.

Hoist all the mergeinfo up to the root in preparation for enforcing merges
to the root only. All MFC's were rerecorded to the root.

Going forward, if an MFC includes mergeinfo, it will need to be made to
the root and committed from the root. Merges with --ignore-ancestry
or diff | patch can go anywhere.

The mergeinfo in HEAD is in a bad state from years of neglect and manual
tampering and this was branched into 10.x. This confuses the coalescing
code and prevents it from doing its job.

Approved by: re (gjb, implicit)

MFC r258086:
Strip the -pN patch level from the VERSION string which gets encoded into
CTF data. Otherwise FreeBSD Update builds think every kernel file has
changed every time there's a security advisory, since the FreeBSD Update
build code isn't smart enough to look inside CTF data to ignore those
changes.

Approved by: re (gjb)

MFC r258283 - move iconv* symbols out of libc.so.7 namespace with
compatibility symbols to aid transition for existing 10.x installs.

Approved by: re (kib)

MFC r257350:

In r257079, SRCDIR is pointed to ${.CURDIR} when not set. However,
Makefile.inc1 is being called in sub-make's where make(1) would,
by default, implicitly chdir(2) to ${.OBJDIR} before executing any
targets. This would make some targets, like delete-old, when trying
to derive various variables introduced by change r256921 using
``make -f Makefile.inc1'' that also rely on SRCDIR to fail.

This changeset adds an explicit cd ${.CURDIR} before these unwrapped
make calls, making them in line with the other ones that are already
being wrapped with the explicit chdir's.

Approved by: re (hrs)

MFC r257329 (nyan):
Fix build, both clang and gcc are required on pc98.

Approved by: re (kib)
Sponsored by: The FreeBSD Foundation

MFC r256915

Stop conflating WITHOUT_CLANG with WITHOUT_CLANG_IS_CC. This allows
bootstrapping a copy of clang without building clang for the base system

which is useful for nanobsd and similar setups. It's still probably
wrong to conflate what is installed as /usr/bin/cc with the selection
of a bootstrap compiler under WITH*_CLANG_IS_CC, but that's for another
day.

Sponsored by: DARPA/AFRL
Approved by: re (gjb)

MFC r257079:
Fix build host pollution by avoiding calling 'uname -srp' to
determine values for 'VERSION'.

Approved by: re (glebius)

MFC r256842:

Fix 'make delete-old-libs' and 'make check-libs' to delete .debug
files created by WITH_DEBUG_FILES. Also cleanup .symbols files from
the period between r244236 when .symbols were supported and r251512
when they were renamed to .debug.

Only propose to delete a .debug file if the corresponding library
itself was deleted already.

Approved by: bapt
Approved by: re (glebius)

Revert r256944, which does not quite get the src/ directory correct.

Discussed with: cperciva
Approved by: re (implicit)
Pointyhat: gjb

MFC r256921:
Thou shalt not leak build host state into the system being compiled.

Approved by: re (gjb)

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Revert-and-redo r255955: the sort -r should be added to delete-old-dirs.

Approved by: re (gjb)

Reverse directories order when doing 'make delete-old-dir'. This
ensures subdirectories gets removed before their parents when doing
make delete-old.

Approved by: re (gjb)
MFC after: 2 weeks

Remove BIND.

Approved by: re (gjb)

Don't parse NO_ROOT metadata for extra kernels if NO_ROOT isn't defined.

Approved by: re (gjb)
MFC after: 1 week

Remove the armv6eb architecture as it is unused, and almost certainly
broken. None of our kernels can boot armv6eb. The little-endian kernels do
not have the required code to be able to switch endian when running a
big-endian executable.

Approved by: re (gjb)

Fix 'make installcheck' to check for missing UID/GID as well, broken
since r249893, by adding a separate _installcheck_world and
_installcheck_kernel so the destination targets can be more explicit
on which they are needed for.

installcheck will call both, while installworld only calls
_installcheck_world and installkernel only calls _installcheck_kernel

While here, mark the internal targets as starting with _.

Reported by: des
Reviewed by: des
Pointyhat to: bdrewery
Approved by: re (delphij)

Add unbound to the list of UIDs / GIDs to check fore before installing.

Approved by: re (blanket)

Clean up the OpenSSH build. It is now possible to build most components
as static binaries, if desired. The one exception is sshd, which runs
into trouble due to libpam.a's includion of pam_ssh.

Make OpenSSH use LDNS if available. This allows it to verify signed
SSHFP records.

Approved by: re (blanket)

Create a private library directory (LIBPRIVATEDIR) for libraries which
we don't want to expose but which can't or shouldn't be static.

To mark a library as private, define PRIVATELIB in its Makefile. It
will be installed in LIBPRIVATEDIR, which is normally /usr/lib/private
(or /usr/lib32/private for 32-bit libraries on 64-bit platforms).

To indicate that a program or library depends on a private library,
define USEPRIVATELIB in its Makefile. The correct version of
LIBPRIVATEDIR will be added to its run-time library search path.

Approved by: re (blanket)

- Fix LOCAL_MTREE so it properly handles multiple files and quotes
its value into submakes

PR: conf/179466
Submitted by: Garrett Cooper <yaneurabeya@gmail.com>
Approved by: bapt
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Update nvi-1.79 to 2.1.1-4334a8297f

This is the gsoc-2011 project to clean up and backport multibyte support
from other nvi forks in a form we can use.

USE_WIDECHAR is on unless building for the rescue crunchgen. This should
allow editing in the native locale encoding.

USE_ICONV depends on make.conf having 'WITH_ICONV=YES' for now. This
adds the ability to do things like edit a KOI8-R file while having $LANG
set to (say) en_US.UTF-8. iconv is used to transcode the characters for
display.

Other points:
* It uses gencat and catopen/etc instead of homegrown msg catalog stuff.
* A lot of stuff has been trimmed out, eg: the perl and tcl bindings which
we could never use in base anyway.
* It uses ncursesw when in widechar mode. This could be interesting.

GSoC info: http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/zy/1
Repo at: https://github.com/lichray/nvi2

Obtained from: Zhihao Yuan <lichray@gmail.com>

Generate debug data release dist sets

If WITH_DEBUG_FILES is set the standalone debug data will be excluded
from each ${dist}.txz and placed in a ${dist}.debug.txz.

Submitted by: gjb
Reviewed by: brooks

Sprinkle some .MAKE magic

Use && rather than ; when success of previous job matters.

Simplify the hierarchy target's logic. [0]

Run hierarchy with WORLDTMP in the path so it works when it is invoked
directly. Such use is nearly alwasy wrong but appears to be common.[1]

PR: conf/178209 [0], conf/178547 [1]
Submitted by: Garrett Cooper <yaneurabeya@gmail.com> [0]
MFC after: 5 days

Flag recursive make targets with .MAKE
so that job token pipe is passed to them.
To avoid surprising anyone, only add .MAKE to ${TGTS} when -n
has not been specified (at least for Makefile).

Reviewed by: obrien

Be more agressive about bootstrapping ctfmerge and ctfconvert so
builds from existing releases have a chance of working properly.

Sponsored by: DARPA, AFRL
MFC after: 3 days

Add a new knob WITH_DEBUG_FILES to control the building of standalone
debug files for userland programs and libraries. The "-g" debug flag
is automatically applied when WITH_DEBUG_FILES is set.

The debug files are now named ${prog}.debug and ${shlib}.debug for
consistency with other systems and documentation. In addition they are
installed under /usr/lib/debug, to simplify the process of installing
them if needed after a crash. Users of bsd.{prog,lib}.mk outside of the
base system place the standalone debug files in a .debug subdirectory.
GDB automatically searches both of these directories for standalone
debug files.

Thanks to everyone who contributed changes, review, and testing during
development.

release: Allow empty extra distributions.

For example, WITHOUT_SHAREDOCS= in src.conf creates an empty doc
distribution.

Submitted by: Kurt Lidl
Tested by: Kurt Lidl
Discussed with: gjb
MFC after: 1 week

Restore (at least temporarily) SHARED=symlinks in the build includes
stage of buildworld. This was accidentally included in r251140.

Reported by: jhb

Always define INSTALL_DDIR and define it such that it contains no extra
/ characters rather than removing them later on. This should fix
release builds.

PR: conf/178963
Reviewed by: gjb, hrs

r245757 introduced warning output if update method is set to CVS_UPDATE
or SUP_UPDATE.

CVS exporter for head/ is turned off for nearly one month now.

It is finally time to swing the ax at these update methods.

Reviewed by: eadler
MFC after: 1 month

Update comments to reflect use of svn/svnup to keep the src/ tree up
to date.

MFC after: 3 days

Temporarily revert r251058 - it breaks documented use of makeoptions
including the tinderbox.

http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-i386-i386.full
make: don't know how to make modules-all. Stop
make: stopped in /obj/i386.i386/src/sys/PAE
*** Error code 2

During buildkernel, print a banner before building modules.

Remove backwards compat layer which was added in r71419 (in 2001).

This change is not intended for MFC.

Reviewed by: peter

Fix building on slightly older -current and stable systems after
the switch to bmake. The rescue bits are built via crunchgen,
which didn't respect the MAKE environment variable until r237574
(i.e. ~11 months ago). This resulted in a failure due to bmake's
internal -J flag being passed around and not being understood by
the standard (i.e. host's) make.
Note that the failure is conditional upon having the jobServer
feature enabled within bmake.

Connect flex 2.5.37 to the build and bump __FreeBSD_version.

Restore the ability to build on systems with 32-bit compat when
the system compiler is not clang. clang and gcc appear to differ
signficantly in their interpretation of -isystem and --sysroot. Further
work is likely required to support an external gcc.

Reported by: andreast, fidaj@ukr.net, sergey.dyatko@gmail.com

Fix distributekernel in the non NO_ROOT case.

PR: conf/178775
Submitted by: Garrett Cooper <yaneurabeya@gmail.com>

Implement NO_ROOT support for distributekernel and packagekernel.

Add support for an external cross compiler. The cross compiler is
specified by passing the XCC, XCXX, and XCPP variables (corresponding to
CC, CXX, and CPP) to buildworld/buildkernel. The compiler must be clang
or be configured to target the appropriate architecture.

To speed build times, if XCC is an absolute path or
WITHOUT_CROSS_COMPILER is defined then no cross compiler will be built
during the cross-tools stage.

Limited documentation of this feature can currently be found at:

https://wiki.freebsd.org/ExternalToolchain

This functionality should be considered experimental and is subject to
change without notice.

Sponsored by: DARPA, AFRL
Discussed with: imp, sjg

Add compile_et to kerberos bootstrap tools.
This makes it possible to build a regular world on a host without kerberos.

PR: conf/178421
Submitted by: Garrett Cooper
MFC after: 1 month

Optimize SUBDIR_OVERRIDE such that SUBDIR isn't automatically defined if
SUBDIR_OVERRIDE is defined.

PR: conf/174071
Submitted by: Garrett Cooper <yaneurabeya@gmail.com>
Sponsored by: EMC / Isilon Storage Division

A transition period of more than two years is more than enough:
Remove the compatibility code added in 2011-02-10.

This change is not intended for MFC

Reviewed by: imp

Fix installworld with DB_FROM_SRC after r249807

Approved by: bapt
Reported by: Tom Everett

Fix installkernel requiring users/groups defined in CHECK_UIDS
and CHECK_GIDS to exist since r152680. This is only needed for
installworld. The documented procedure of running mergemaster -p
to check for missing users is only needed for installworld, not
for installkernel. This fixes auditdistd incorrectly being
required for installkernel.

PR: misc/174405
Approved by: bapt

This should have been committed in r249666.

MFC after: 1 month

Upgrade our copy of llvm/clang to trunk r178860, in preparation of the
upcoming 3.3 release (branching and freezing expected in a few weeks).

Preliminary release notes can be found at the usual location:
<http://llvm.org/docs/ReleaseNotes.html>

An MFC is planned once the actual 3.3 release is finished.

Fix xdev-install when installing to a location other than /

Fix xdev. Clang and libc were not building correctly on older worlds, for
example, on 9.1.
* To fix clang add an _xb-bootstrap-tools target that mirrors the existing
bootstrap-tools target in the full world.
* For libc have the compiler use the newly installed includes, and, while
here, tell the compiler about the xdev library path as some other
libraries will link against the installed libraries.

Fix a typo, 'xdev-build' should be spelled with a 'd' at the end.

Approved by: cognet

Merge libzfs_core branch:
includes MFV 238590, 238592, 247580

MFV 238590, 238592:
In the first zfs ioctl restructuring phase, the libzfs_core library was
introduced. It is a new thin library that wraps around kernel ioctl's.
The idea is to provide a forward-compatible way of dealing with new
features. Arguments are passed in nvlists and not random zfs_cmd fields,
new-style ioctls are logged to pool history using a new method of
history logging.

http://blog.delphix.com/matt/2012/01/17/the-future-of-libzfs/

MFV 247580 [1]:
To address issues of several deadlocks and race conditions the locking
code around dsl_dataset was rewritten and the interface to synctasks
was changed.

User-Visible Changes:
"zfs snapshot" can create more arbitrary snapshots at once (atomically)
"zfs destroy" destroys multiple snapshots at once
"zfs recv" has improved performance

Backward Compatibility:
I have extended the compatibility layer to support full backward
compatibility by remapping or rewriting the responsible ioctl arguments.
Old utilities are fully supported by the new kernel module.

Forward Compatibility:
New utilities work with old kernels with the following restrictions:
- creating, destroying, holding and releasing of multiple snapshots
at once is not supported, this includes recursive (-r) commands

Illumos ZFS issues:
2882 implement libzfs_core
2900 "zfs snapshot" should be able to create multiple,
arbitrary snapshots at once
3464 zfs synctask code needs restructuring

References:
https://www.illumos.org/issues/2882
https://www.illumos.org/issues/2900
https://www.illumos.org/issues/3464 [1]

MFC after: 1 month
Sponsored by: Hybrid Logic Inc. [1]

In all situations where we need to bootstrap yacc, we also need to
bootstrap liby. This was not readily apparent, because a normal 'make
buildworld' or 'make toolchain' builds liby before building anything
that uses yacc. However, 'make kernel-toolchain' does not build
headers or libraries, so it was not possible to build a kernel from
head on, say, stable/9 without first building the complete toolchain.

MFC after: 1 week

- Fix libmd dependency. It is needed in the bootstrap library list because
usr.bin/xinstall depends on it.
- Remove libutil from usr.bin/xinstall/Makefile. No symbol was actually used.

Reviewed by: brooks

Add nmtree to ITOOLS if it is installed on the host instead of keying off
the BOOTSTRAPPING variable. The previous test was wrong because
BOOTSTRAPPING is 0 in most cases.

Tested by: db

Fix xdev by using the install shell script as it knows about the -l
argument thile the local version install may not.

- Add CHECKSUM.* support in Makefile[1].
- Use ln -fs to create a symlink.
- Remove pkgadd for docports.
- Use WITHOUT_JADETEX=yes instead of WITH_JADETEX=no.
- Add {WORLD,KERNEL}_FLAGS to [BTWK]MAKE.
- Use makefs(8) and gpart(8) for sparc64 ISO image[2].
- Add publisher option to makefs(8)[2].

Based on work by: gjb[1]
Discussed with: marius, nwhitehorn[2]

Log the addition of login.conf.db, passwd, pwd.db, and spwd.db via cat -l.

Make cat a bootstrap tool to facilitate this.

Fix installworld when nmtree is not on the host by only adding it to
ITOOLS when we will be using the host copy.

Fix installkernel when -DNO_ROOT is set.

Fix installworld when the host does not need to bootstrap nmtree.

Reported by: lev, jmg

Fix update method (s/SUP/CVS) warning.

Submitted by: Ryan Stone
MFC after: 1 day

Strip any trailing slash off of the -D argument to install to support the
rather fragile destdir stripping in the metalog.

Introduce a new option -DNO_ROOT that allows install and distribution
targets to be run without root privilege.

Information about ownership, group, flags, and suid bits are stored in
the file specified by METALOG which defaults to ${DESTDIR}/METALOG.
This file can be used in conjunction with bsdtar or makefs to generate
archives or file system images with correct permissions.

The packageworld target has been altered to use this metadata allowing
non-root releases (subject to further changes in release/Makefile.)

Sponsored by: DARPA, AFRL
Reviewed by: ian, ray

Fix spurious leading / on dtc path in Makefile.inc1. Spotted by jhb.

Import new (BSDL) device tree compiler. Now built by default, so that it can't
be used on the host system (and not installed on the device, if required). The
GPL'd one is still available if there are any devices that need it (make
universe passes with it, including kernels that use fdt, but there may be some
out-of-tree ones). WITH_GPL_DTC can be used to select the old one, for now.

Probably won't be MFC'd, but we'll remove the GPL'd version in head after the
new one has had a lot more testing and ship it in 10.0.

Chase r245756 of make.conf(5):

- If update method is SUP_UPDATE or CVS, warn that those
update methods are deprecated.

- While here, remove bogus NO_WWWUPDATE.

MFC after: 3 days
X-MFC-with: r245756

Always bootstrap nmtree if it is not available on the host. It is fairly
small and it is quite annoying to get all the way through buildworld
only to discover that you forgot the DB_FROM_SRC (or soon NO_ROOT)
option.

Remove a harmless (somewhat to my surprise) bogon that crept into r245440.

Rework the mtree portion of etc/Makefile's distrib-dirs target to run
mtree in a shell loop so there is only one mtree commandline. Move the
implementation of LOCAL_MTREE into etc/Makefile.

Sponsored by: DARPA, AFRL
Reviewed by: mtree :)

Correct the distrib-dirs target in the -DDB_FROM_SRC case.

Add a new LIBRARIES_ONLY make variable to disable the build and install
of files other than the actual libraries.

Use LIBRARIES_ONLY to supress the inclusion of files in the lib32
distribution that are duplicates of files in base.

Sponsored by: DARPA, AFRL
Reviewed by: emaste

Add an option DB_FROM_SRC to use src/etc's user/group databases when
installing. This allows things like running installworld for 10-CURRENT
on a 9.0-RELEASE system without adding extra users and groups to the
passwd and group files.

To prevent potentially risky uid/gid mismatches on systems with
non-standard local values, require that DESTDIR be set if DB_FROM_SRC is
set.

Sponsored by: DARPA, AFRL
Reviewed by: peter

Revert r245316. Systems with non-standard uids/gids are more prevalent
that I'd feared. Discussion is ongoing about the scope of a safer
solution.

Use the -N option to install and nmtree to eliminate the need for the
checks for missing users and groups.

Sponsored by: DARPA, AFRL

Use find -exec to install zoneinfo instead of requiring xargs to be an
install tool.

Suggested by: delphij

Add xargs to the set of install tools when zoneinfo is not disabled.
This fixes installworld which I had broken in r245265.

Reported by: Nikolai Lifanov <lifanov@mail.lifanov.com>

Add auditdistd to the pre-install required uid check list.

The build-tools step during build32 also needs the -legacy magic.

Approved by: marcel (mentor)

Unbreak amd64 cross-build where amd64 is the target. While clang
may be installed as cc and we don't need to build gcc as a
cross-tools, we still build gcc and thus need cc_tools built
as a build tool. Not doing this results in building gengenrtl with
the target compiler while we need to run it on the build machine.

Work around pc98 tinderbox failures in sys/boot/pc98, by making sure a
cross gcc gets built during the cross-tools stage.

MFC after: 1 week
X-MFC-With: 242706

Add ATF to the build. This is may be a bit rought around the egdes,
but committing it helps to get everyone on the same page and makes
sure we make progress.

Tinderbox breakages that are the result of this commit are entirely
the committer's fault -- in other words: buildworld testing on amd64
only.

Credits follow:

Submitted by: Garrett Cooper <yanegomi@gmail.com>
Sponsored by: Isilon Systems
Based on work by: keramida@
Thanks to: gnn@, mdf@, mlaier@, sjg@
Special thanks to: keramida@

Allow LOCAL_(DIRS,LIBS) to install files in directories not found in the
system mtree files via a LOCAL_MTREE variable which contains a list of
mtree files to be applyed along with the base mtree files to the tmp root
and DESTDIR.

Fix the showconfig target. bmake sends debug output to stderr.

Submitted by: Simon Garrety <sjg@juniper.net>

Fix 32-bit library builds after the removal of adding -E to make for AS,
CC, CXX and LD. This fix implements the intended as it should have been
implemented all along: by passing AS, CC, CXX and LD on the commandline
of the sub-make instead of in the environment of the sub-make.

Breakage pointed-out by: dim@

Add support for bmake. This includes:
1. Don't do upgrade_checks when using bmake. As long as we have WITH_BMAKE,
there's a bootstrap complication in ths respect. Avoid it. Make the
necessary changes to have upgrade_checks work wth bmake anyway.
2. Remove the use of -E. It's not needed in our build because we use ?= for
the respective variables, which means that we'll take the environment
value (if any) anyway.
3. Properly declare phony targets as phony as bmake is a lot smarter (and
thus agressive) about build avoidance.
4. Make sure CLEANFILES is complete and use it on .NOPATH. bmake is a lot
smarter about build avoidance and should not find files we generate in
the source tree. We should not have files in the repository we want to
generate, but this is an easier way to cross this hurdle.
5. Have behavior under bmake the same as it is under make with respect to
halting when sub-commands fail. Add "set -e" to compound commands so
that bmake is informed when sub-commands fail.
6. Make sure crunchgen uses the same make as the rest of the build. This
is important when the make utility isn't called make (but bmake for
example).
7. While here, add support for using MAKEOBJDIR to set the object tree
location. It's the second alternative bmake looks for when determining
the actual object directory (= .OBJDIR).

Submitted by: Simon Gerraty <sjg@juniper.net>
Submitted by: John Van Horne <jvanhorne@juniper.net>

Introduce a new make variable COMPILER_TYPE that specifies what
type of compiler is being used (currently clang or gcc). COMPILER_TYPE
is set in the new bsd.compiler.mk file based on the value of the CC
variable or, should it prove informative, by running ${CC} --version
and examining the output.

To avoid negative performance impacts in the default case and correct
value for COMPILER_TYPE type is determined and passed in the environment
of submake instances while building world.

Replace adhoc attempts at determining the compiler type by examining
CC or MK_CLANG_IS_CC with checks of COMPILER_TYPE. This eliminates
bootstrapping complications when first setting WITH_CLANG_IS_CC.

Sponsored by: DARPA, AFRL
Reviewed by: Yamaya Takashi <yamayan@kbh.biglobe.ne.jp>, imp, linimon
(with some modifications post review)
MFC after: 2 weeks

Not all Pmake derivatives silently handle empty shell output, so ensure there
is something for make(1) to consume. Bmake gives output such as:
"warning: Couldn't read shell's output for "/bin/sh -c true"
Note we parted from traditional Pmake behavior in r18864 / r18255.

Fix 'distributeworld' target when WITHOUT_GAMES is set.

Reviewed by: des
Approved by: des
MFC after: 3 days
X-MFC-To: 9-only

Merging of projects/armv6, part 3

r238211:
Support TARGET_ARCH=armv6 and TARGET_ARCH=armv6eb

This adds a new TARGET_ARCH for building on ARM
processors that support the ARMv6K multiprocessor
extensions. In particular, these processors have
better support for TLS and mutex operations.

This mostly touches a lot of Makefiles to extend
existing patterns for inferring CPUARCH from ARCH.
It also configures:
* GCC to default to arm1176jz-s
* GCC to predefine __FreeBSD_ARCH_armv6__
* gas to default to ARM_ARCH_V6K
* uname -p to return 'armv6'
* make so that MACHINE_ARCH defaults to 'armv6'
It also changes a number of headers to use
the compiler __ARM_ARCH_XXX__ macros to configure
processor-specific support routines.

Submitted by: Tim Kientzle <kientzle@freebsd.org>

Partial MFV (illumos-gate 13753:2aba784c276b)
2762 zpool command should have better support for feature flags

References:
https://www.illumos.org/issues/2762

MFC after: 2 weeks

Add "hier" as an alternate spelling of "hierarchy" to match hier(9).

Minor wording change. The previous commit message should have included:

Sponsored by: DARPA, AFRL

MFP4: 212854, 212854

Add a LOCAL_LIB_DIRS variable to complement the existing LOCAL_DIRS
and LOCAL_TOOL_DIRS variables. Directories in LOCAL_LIB_DIRS are
built at the end of the _generic_libs target.

Reviewed by: imp (212854)

During buildworld and buildkernel, define EARLY_BUILD in the earlier
stages (build-tools, cross-tools, etc) of the build, so we can detect in
bsd.*.mk whether to pass compiler-specific flags to ${CC}.

In particular, this commit will allow using WITH_CLANG_IS_CC when the
base compiler is still gcc, and when ${CC}, ${CXX} and ${CPP} are left
at their defaults. The early stages will then be built using gcc, and
no clang-specific flags will be passed to it. The later stages will be
built as usual.

The EARLY_BUILD define can also serve other uses, such as building the
world stage C++ executables with libc++ instead of libstdc++: during the
early build stages, we cannot assume libc++ is already available, so we
must still build with libstdc++ at that time.

MFC after: 1 week

Bump __FreeBSD_version for the byacc import, and update _bootstrap_tools.

Fix bootstrapping from FreeBSD 7.x: we also need to build ar.

By request, add back support from 7.0 and newer, since the changes for
that are so minimal.

Do a bit of house cleaning and remove the old, obsolete upgrade from
6.0 support and bump the minimum version to 8.0. These versions have
not been tested and are believed to be broken.

Introduce the ${SHLIB_LDSCRIPT} variable to have an ld(1) script
instead of a symlink for .so files.

Reviewed by: kib, kan (previous version), dim
Approved by: kib (mentor)
Silence from: -hackers@
MFC after: 1 week

Fix ordering issue. 'make xdev' can fail with -jN because it tries to run the
xdev-install step while xdev-build is still running.

Submitted by: Ian Lepore

o) Fix mips/mips -> mips for Makefile.inc1.
o) Rebuild src.conf.5.

Assume a big-endian default on MIPS and drop the "eb" suffix from MACHINE_ARCH.
This makes our naming scheme more closely match other systems and the
expectations of much third-party software. MIPS builds which are little-endian
should require and exhibit no changes. Big-endian TARGET_ARCHes must be
changed:
From: To:
mipseb mips
mipsn32eb mipsn32
mips64eb mips64

An entry has been added to UPDATING and some foot-shooting protection (complete
with warnings which should become errors in the near future) to the top-level
base system Makefile.

Build CTF tools as a part of toolchain for cross-compilation case

- Do not build libcom_err and compile_et when kerberos is disabled. They
depends on several heimdal libraries and not used by anything but kerberos
tools.

- Update FreeBSD Heimdal distribution to version 1.5.1. This also brings
several new kerberos related libraries and applications to FreeBSD:
o kgetcred(1) allows one to manually get a ticket for a particular service.
o kf(1) securily forwards ticket to another host through an authenticated
and encrypted stream.
o kcc(1) is an umbrella program around klist(1), kswitch(1), kgetcred(1)
and other user kerberos operations. klist and kswitch are just symlinks
to kcc(1) now.
o kswitch(1) allows you to easily switch between kerberos credentials if
you're running KCM.
o hxtool(1) is a certificate management tool to use with PKINIT.
o string2key(1) maps a password into key.
o kdigest(8) is a userland tool to access the KDC's digest interface.
o kimpersonate(8) creates a "fake" ticket for a service.

We also now install manpages for some lirbaries that were not installed
before, libheimntlm and libhx509.

- The new HEIMDAL version no longer supports Kerberos 4. All users are
recommended to switch to Kerberos 5.

- Weak ciphers are now disabled by default. To enable DES support (used
by telnet(8)), use "allow_weak_crypto" option in krb5.conf.

- libtelnet, pam_ksu and pam_krb5 are now compiled with error on warnings
disabled due to the function they use (krb5_get_err_text(3)) being
deprecated. I plan to work on this next.

- Heimdal's KDC now require sqlite to operate. We use the bundled version
and install it as libheimsqlite. If some other FreeBSD components will
require it in the future we can rename it to libbsdsqlite and use for these
components as well.

- This is not a latest Heimdal version, the new one was released while I was
working on the update. I will update it to 1.5.2 soon, as it fixes some
important bugs and security issues.

Make sure libgcc_s is finished building in _startup_libs before
building libcxxrt with high -j levels. The workaround in
libc++/Makefile isn't necessary once that race is solved.

Reviewed by: theravin

Fix a thinko in r232322, where gcc (and its tools) are not built during
the cross-tools stage, if CC=clang and WITH_CLANG_IS_CC is not set.

This causes no 'cc' to be installed in the temporary cross-tools tree,
making lint fall over later in the build, because it ignores ${CC} and
attempts to run 'cc' anyway.

To fix this, only skip building gcc during cross-tools, if WITHOUT_GCC
is set, or if WITH_CLANG_IS_CC is set.

Pointy hat to: dim
MFC after: 2 weeks

Add a WITH_CLANG_IS_CC option for src.conf(5), disabled by default, that
installs clang as /usr/bin/cc, /usr/bin/c++ and /usr/bin/cpp.

Note this does *not* disable building and installing gcc, which will
still be available as /usr/bin/gcc, /usr/bin/g++ and /usr/bin/gcpp. If
you want to disable gcc completely, you must use WITHOUT_GCC.

MFC after: 2 weeks

Allow specification of build shell for the buildenv target.

Submitted by: ian lepore

When the buildkernel stage 2.3 (build tools) runs, the PATH is still set
to the default from the top-level Makefile. Therefore, invocations of
lex and yacc (used during building of aicasm) will use the executables
in /usr/bin, not those optionally built during the previous buildworld
or kernel-toolchain. This makes kernel builds from older FreeBSD
releases more difficult than necessary.

Fix this by setting PATH to ${BPATH}:${PATH} in stage 2.3, so the
bootstrap tools directories are searched before the regular ones.

Silence from: svn-src-{all,head}
MFC after: 1 week

Restore functionality to pack several kernels into release. All
kernels specified by KERNCONF are built and packed into release.
The first one is packed into kernel.txz, all others to
kernel.CONFIG.txz.

The first one is installed on bootables in /boot.

Fix the broken non-cross compile build. Oops!

Another pointy hat to: adrian, for stirring up more trouble.

Allow extra directories to be added to the build-tools target.

Things such as "sh" require local tools to be built before
cross-compiling. This allows for extra software (that's
built via LOCAL_DIRS) to also have a build-tools target where
required.

Remove -mfancy-math from LIB32CPUFLAGS for amd64. It has been default
for our gcc since more than three years (see r181534, which is also in
stable/9 and stable/8). This flag used to be for the benefit of the old
in-kernel math emulator, which was removed more than eight years ago.

Pointed out by: arundel
MFC after: 1 week

- CTF knob is now implemented using common scheme: MK_CTF=yes/no is
defined based on WITH/WITHOUT_CTF settings, default is WITHOUT_CTF,
NO_CTF overrides WITH_CTF (used by Makefile.inc1)
- CTFCONVERT_CMD/NORMAL_CTFCONVERT are now defined to empty string
if make(1) can handle empty commands

Turn off profiled libs build by default.
Can be enabled back using WITH_PROFILE=yes in /etc/src.conf

Fix breakage after r227983; lib/libcxxrt still got built, because it was
not disabled in the usual way (by adding it to __DEFAULT_NO_OPTIONS in
share/mk/bsd.own.mk), and because the test for MK_LIBCPLUSPLUS in
Makefile.inc1 was incorrect.

Pointy hat to: dim

Provide proper error message when trying to build xdev, xdev-build or
xdev-install targets without either XDEV or XDEV_ARCH defined.

Import libc++ / libcxxrt into base. Not build by default yet (use
MK_LIBCPLUSPLUS=yes to enable). This is a work-in-progress. It works for
me, but is not guaranteed to work for anyone else and may eat your dog.

To build C++ using libc++, add -stdlib=libc++ to your CXX and LD flags.

Bug reports welcome, bug fixes even more welcome...

Approved by: dim (mentor)

Revert r227841 and part of r227798. We still build libpam in two passes,
but we use STATIC_CFLAGS instead of our own private .c.o rule.

MFC after: 3 weeks

Remove libpam from _prebuild_libs. This should unbreak the -jX build.

MFC after: 3 weeks

Simplify the libpam build by removing the shared modules' dependency
on the shared library. The modules are loaded by the library, so we
know it'll be there when we need it.

MFC after: 3 weeks

Rename the linker emulation name for powerpc and powerc64. This is needed that
we can also use the upstream binutils linker where we have to have a unique
name for the FreeBSD emulation.

Revert r227403 for now. Since the cross-tools stage purposefully
doesn't have ${WORLDTMP}/usr/bin in its PATH, if you build world with
CC=clang, tblgen tools from /usr/bin will be used instead of the ones
built under ${WORLDTMP}. This can lead to various errors, especially if
you upgrade from an older clang.

Note that building world with gcc would not experience these problems,
because it only uses the tblgen tools in the world stage, where PATH
does contain ${WORLDTMP}/usr/bin.

Pointy hat to: dim

Move building of clang's tblgen tools (and required libraries) from the
bootstrap-tools stage to the cross-tools stage. These tools are only
needed for generating llvm/clang include files, and are not necessary
for bootstrapping the build itself.

This shaves off some build time, because the required libraries are now
just built twice (during the cross-tools and world stages), instead of
three times.

Also, if you build world using WITHOUT_CLANG= in src.conf(5), no llvm or
clang code will be compiled at all anymore.

MFC after: 1 week

Make it possible to set CC and CXX (and optionally, AS and LD) in
make.conf(5), while allowing the build32 stage on 64-bit architectures
to still override them, so that stage can successfully build 32-bit
compatibility libraries.

Explanation:
1) The build32 stage sets environment variables CC, CXX, AS and LD for
its sub-make, to add 32-bit specific flags (-m32 and such).
2) The sub-make reads sys.mk, encounters CC?= and CXX?= assignments, so
does not alter them.
3) After some other stuff, sys.mk reads /etc/make.conf. When you have
"CC=xxx" and "CXX=yyy" statements in there, they will *override* the
build32-supplied CC/CXX values, nullifying the 32-bit specific flags.
4) Thus all objects get built as 64-bit anyway, and since LD is usually
not set in make.conf, it still has the 32-bit flags!
5) Now, whenever something is linked, you will get a "ld: Relocatable
linking with relocations from format elf64-x86-64-freebsd (foo.o) to
format elf32-i386-freebsd (bar.o) is not supported" error.

Fix this by adding "-ECC -ECXX -EAS -ELD" to the build32 sub-make
invocation, which forces those environment variables to always override
any assignment in makefiles. Thus making it possible to simply set:

CC=my-cc
CXX=my-c++

in your make.conf, or specify a path, even:

CC=/usr/local/bin/other-cc
CXX=/usr/local/bin/other-c++

Note this was never a problem on i386, since it has no build32 stage.

Silence from: current@
MFC after: 1 week

Attempt to fix build logic for gensnmptree.

There are two problems with the existing logic. It builds gensnmptree
on <700018, even if WITHOUT_BSNMP is set, but more importantly, we must
not forget to build gensnmptree on systems that have originally been
built without. This causes a buildworld on those systems to fail.

MFC after: 1 week

Upgrade our copy of llvm/clang to r142614, from upstream's release_30
branch. This brings us very close to the 3.0 release, which is expected
in a week or two.

MFC after: 1 week

Farewall, sysinstall! You served us well for many years, but 10.0 is one
digit beyond your time.

Various sysinstall dependencies (e.g. libftpio, libdisk, libodialog, etc.)
will be cleaned up in coming days. Some will take longer than others due to
a few other consumers (tzsetup and sade).

- Add missing interdependencies to kerberos libraries. Some of the
kerberos libraries were not linked properly (missing dependencies),
which causes 3rd party applications linking to fail when --as-needed
ld flag is used. I also added the --no-undefined ld(1) flag to make
sure that there're no missing dependencies.

MFC after: 3 days

- Merged awk upstream that includes a fix for a bug exposed by kmod_syms.mk.
- Provide a build aid for those who already have a buggy awk(1) installed.

Approved by: re (kib)

It's a bit odd, but "make update" in src/ can also update the ports/,
doc/, and now www/ trees, but only using the "cvsup" transport.

When "make update" is run using a tree's makefile, it can also use
"cvs" (except for www/) and "svn" (only src/).

Clean up documentation and code regarding "make update":

- Increase oddness by adding support for WWWSUPFILE and NO_WWWUPDATE to
Makefile.inc1 (analogous to PORTSSUPFILE/NO_PORTSUPDATE and
DOCSUPFILE/NO_DOCUPDATE; WWWSUPFILE already supported by www/Makefile).

- Document all trees that support CVS_UPDATE.

- Document all trees that support SUP_UPDATE.

- Document SVN_UPDATE.

- Document NO_WWWUPDATE.

- make.conf(5) mistakenly said that *SUPFILE* had defaults.

- Add an example entry for WWWSUPFILE.

Implement WITH{,OUT}_{GCC,BINUTILS} to provide finer-grained control
over building gcc and binutils. They default to true, unless
MK_TOOLCHAIN is no.

Reviewed by: ru@

Clean up a loose end from the conversion from gnu ar/ranlib to the BSD
one. Without this, we don't have ar or randlib in the tool path,
leading to much pain for some users. This pain is exposed by the
external toolchain enhancements that I'm working on.

Submitted by: John Hein (ages ago, and dropped on the floor by me: sorry)

Backout libinstall.a -> libpkg commit.

Discussed with: erwin, brooks, bapt

Disconnect sun4v architecture from the three.

Some files keep the SUN4V tags as a code reference, for the future,
if any rewamped sun4v support wants to be added again.

Reviewed by: marius
Tested by: sbruno
Approved by: re

+ DTrace as a bootstrap tool is only needed on certain older systems.
+ Be more consistent between BMAKE and TMAKE.
+ Add NO_CTF to crosstools as is done for bootstrap and build tools.

Reviewed by: marcel

Correct the kernel config name printed out during install.

PR: 156579
Submitted by: dhw

Note which of the built kernels is being installed.

PR: 156579
Submitted by: dhw

Remove libobjc and other Objective-C related components, as these are
extremely outdated, and not used by anything in the base system.

Silence from: current@

Complete WITHOUT_CXX support. It implies WITHOUT_GROFF and
WITHOUT_CLANG.

Don't build clang bootstrap/build-tools depending on this flag. We also
keep gperf, devd and libstdc++ around to prevent foot-shooting and to
make this a two-way street.

Make clang default on x86 and powerpc, but not on other architectures.
Make fdt default on arm and powerpc.

This now includes cross compiled targets, where before we tried to
make it host-based.

Also, move the lists of default yes and no options to a variable.

In general, only build tools should get this treatment in bsd.own.mk.
Also, the use of TARGET* in the bsd.*mk files is discouraged, but
necessary here due to the ordering of things in buildworld. We make
the native case work by testing MACHINE_ARCH after TARGET_ARCH.

Fix the delete-old/check-old targets to work with arbitrarily long
OLD_FILES/OLD_DIRS/OLD_LIBS lists.

If you specify enough WITHOUT_FOO flags, the argument list passed to the
shell will be too long. Using .for/.endfor make(1) "loop" will make the
parser of the Makefile explode. Hack around this with good old pipes.

No objections: netchild
Reported by: b.f.

- Merge in OFED 1.5.3 from projects/ofed/head

Improve the distributeworld target in Makefile.inc1 and update the release
infrastructure to use it. make distributeworld can now be used without
preparing its environment first and installs games into its distribution
using the regular make distribute logic instead of post-processing with
a script.

Also add two new targets, packageworld and packagekernel, that tar up the
results of distributeworld and distributekernel (also new), respectively.

Use a suitable DIRPRFX for each invocation of make in the build32 and
install32 targets so that the full path to each program or library is
displayed in the make output.

MFC after: 1 week

Commit two more files missed in r219089.

MFC after: 1 month

Add the BSD-licensed Citrus iconv to the base system with default off
setting. It can be built by setting the WITH_ICONV knob. While this
knob is unset, the library part, the binaries, the header file and
the metadata files will not be built or installed so it makes no impact
on the system if left turned off.

This work is based on the iconv implementation in NetBSD but a great
number of improvements and feature additions have been included:

- Some utilities have been added. There is a conversion table generator,
which can compare conversion tables to reference data generated by
GNU libiconv. This helps ensuring conversion compatibility.
- UTF-16 surrogate support and some endianness issues have been fixed.
- The rather chaotic Makefiles to build metadata have been refactored
and cleaned up, now it is easy to read and it is also easier to add
support for new encodings.
- A bunch of new encodings and encoding aliases have been added.
- Support for 1->2, 1->3 and 1->4 mappings, which is needed for
transliterating with flying accents as GNU does, like "u.
- Lots of warnings have been fixed, the major part of the code is
now WARNS=6 clean.
- New section 1 and section 5 manual pages have been added.
- Some GNU-specific calls have been implemented:
iconvlist(), iconvctl(), iconv_canonicalize(), iconv_open_into()
- Support for GNU's //IGNORE suffix has been added.
- The "-" argument for stdin is now recognized in iconv(1) as per POSIX.
- The Big5 conversion module has been fixed.
- The iconv.h header files is supposed to be compatible with the
GNU version, i.e. sources should build with base iconv.h and
GNU libiconv. It also includes a macro magic to deal with the
char ** and const char ** incompatibility.
- GNU compatibility: "" or "char" means the current local
encoding in use
- Various cleanups and style(9) fixes.

Approved by: delphij (mentor)
Obtained from: The NetBSD Project
Sponsored by: Google Summer of Code 2009

Flesh out WITHOUT_GROFF support to DTRT.

A full featured groff is required during buildworld, so build it always
and don't rely on it being present on the host system.

vgrind(1) is tightly coupled to a roff processor and will not be
built/installed when groff is disabled. Also much of the roff'ed
documentation under share/doc will not be built/installed when
WITHOUT_GROFF is defined.

Reviewed by: ru (partial)

Back out 218933 and 281934. The easy cases still worked, but some
hard cases broke in worse ways than the status quo. Back them out
until that can be resolved.

Select building CLANG based on ${TARGET_ARCH} and existence of
WITH{,OUT}_CLANG. We default to yes on some archs, no on others,
unless WITHOUT_CLANG or WITH_CLANG is defined respectively.

Upgrade our copy of llvm/clang to r126079, from upstream's trunk.

This contains many improvements, primarily better C++ support, an
integrated assembler for x86 and support for -pg.

Hmmm, specifying TARGET and TARGET_ARCH in the environment doesn't
seem to work when building xdev anymore (most likely my changes lately
moving the TARGET guessing stuff to Makefile from Makefile.inc1, but I
really don't grok why). Fix make xdev by putting them on the command
line. This will work either way while I try to figure it out.

CPUTYPE is now a required define for calling Makefile.inc1 diretly, so
make sure we define it for the xdev stuff.

Move xdev stuff to be last again in this file.

# xdev-build works now, but xdev-install appears to be broken though.

You are now *REQUIRED* to pass both TARGET and TARGET_ARCH to any
invocations of Makefile.inc1 (since that's supposed to be an internal
interface for world and related targets). Document this with a .error
message. For a transition period, support passing in just TARGET, but
give a .warning for that case: I plan on removing it in 9.0...

Move the architecture guessing from Makefile.inc1 to Makefile. We
need to do this because variables specified on the command line
override those specified in the Makefile. This is why we also moved
from TARGET to _TARGET in Makefile, and then set TARGET on the command
line when we fork a submake with Makefile.inc1.

This makes mips/mips work again, even without the workaround committed to
lib/libc/Makefile.

Retire TARGET_ABI.

Implement MACHINE_ARCH=mips64e[lb] to build N64 images. This replaces
MACHINE_ARCH=mipse[lb] TARGET_ABI=n64.

MACHINE_ARCH=mipsn32e[lb] has been added, but currently requires
WITHOUT_CDDL due to atomic issues in libzfs. I've not investigated
this much, but implemented this to preserve as much of the TARGET_ABI
functionality that I could. Since its presence doesn't affect the
working cases, I've kept it in for now.

Added mips64e[lb] to make universe, so more kernels build.

And I think this (finally) closes the curtain on the tbemd tree.

Do not lint code beyond necessity (with apologies to Wiliam of Ockham).

Don't lint externally maintained CDDL code, or relint the 32-bit libraries
in amd64 mode.

add / to the translation of TARGET_ARCH to TARGET

The fixes for mips/mips arm/arm compat hacks broke if TARGET or
TARGET_ARCH wasn't defined. Fix it for that case.

Submitted by: Garrett Cooper

Add legacy names for mips:mips and arm with TARGET_BIG_ENDIAN, for the moment.

Replace libgcc.a by libcompiler_rt.a.

libcompiler_rt.a is a BSD licensed C language runtime, which implements
many routines which are linked into binaries on architectures where
certain functionality is missing (e.g. 64 bits mul/div on i386).

Unfortunately, libcompiler_rt cannot replace libgcc entirely. Certain
features, such as an unwinder for exception handling, are missing.
That's why only libgcc.a is replaced for now, because this one does seem
to be complete.

Tested by: rene (amd64), nwhitehorn (powerpc), droso (i386 exprun)
and many others. Thanks!
Obtained from: user/ed/compiler-rt

Merge from tbemd:
o TARGET=mips --> little endian 32-bit mips build
o mipsel and mipseb TARGET_ARCH are both mips TARGETs
o Add some more architecture combinations

Check TARGET_ARCH as well as TARGET to determine if we are doing a cross
build. This is necessary to be able to cross-build 32-bit PowerPC from
a 64-bit PowerPC host.

Reviewed by: imp

Repair some build breakage introduced in r211725 and garbage collect some
code made obsolete in the same commit.

The order was correct before. I'd talked to Nathan about this before,
so this must have been mismerged into tbemd before I merged it back to
head. This unbreaks this on powerpc64

Submitted by: nathanw@

MFtbemd:

Prefer MACHNE_CPUARCH to MACHINE_ARCH in most contexts where you want
to test of all the CPUs of a given family conform.

Minor tweak from tbemd

Don't link drti.o with libelf_pic.a. This means that all software with
userland SDT probes must be linked with libelf explicitly.

Requested by: kib
Sponsored by: The FreeBSD Foundation

Add clang to the cross-tools build stage.

Submitted by: Dimitry Andric <dimitry at andric.com>

Add libelf to the prebuild libs.

This is needed for the DTrace instrumentation object.

Sponsored by: The FreeBSD Foundation

Revert r211436; it was a good idea, but not fully thought out.

Split kernel stage 3.2 (build everything) into 3.2 (build the kernel)
and 3.3 (build the modules). IMHO, this makes it a little easier to
track the progress of a kernel build using whereintheworld et al.

MFC after: 3 weeks

Relax FDT_DTS_FILE validation (and unbreak world build).

Pointed out by: kib

Fix FDT_DTS_FILE parsing to properly retrieve its value. This unbreaks the
'builddtb' target.

Make the processing more robust against non-existent kernel config files
(pointed out by imp@).

Connect powerpc64 to the build. It is not presently part of make universe,
which will be added soon.

Reviewed by: imp

Merge from tbemd:

use MACHINE_CPUARCH instead of MACHINE_CPU since the sources for the
CSU is based on MACHINE_CPUARCH

Reviewed by: arch@ (twice)

Merge from tbemd:

change the name of the object tree from ${TARGET} to
${TARGET}.${TARGET_ARCH} so we can do both big and little endian
builds in the same tree.

Reviewed by: arch@ (twice)

Merge from TBEMD:

Rather than having arch specific code in Makefile.inc1, generalize so
that we can control the settings of different options on a per
architecutre basis.

Reviewed by: arch@ (twice)

Use awk alone for extracting FDT_DTS_FILE instead of grep/cut, which is
simpler and more robust.

Submitted by: Andrzej Tobola

Connect FDT infrastructure to the build system.

Reviewed by: imp
Sponsored by: The FreeBSD Foundation

Hook clang into the build on i386/amd64/powerpc.

Approved by: ed (mentor)

Revert r208353. It did not work as I expected because WITHOUT_MAN is set
for bootstrap-tools target and WITHOUT_MAN_UTILS was implied by default.

Pointy hat: jkim

We do not have to build makewhatis as a bootstrap tool when it is disabled
by WITHOUT_MAN_UTILS.

Revert r208324. We still need it for installworld as $PATH is overriden.

Remove makewhatis(1) from bootstrap tools. We no longer support source
upgrades from versions prior to 6.0 on head.

Enable liblzma support in libarchive
Adjust dependencies for programs using libarchive
Add xz and linkage against liblzma to rescue system

Approved by: kientzle, delphij (mentor)
MFC after: 2 weeks

- Take libinstall.a out of pkg_install and make it a proper shared library.
- Rework the wrapper support to check libpkg version as well as pkg_install
version.
- Add libfetch to _prebuild_libs.
- There are no new features introduced.

Notes: the API is not stable, so basically, do not use libpkg in your
projects for now. Also there's no manpage for libpkg yet, because the API
will change drastically. I repeat, do not use libpkg for now.

Remove explicit setting of NO_CTF in WMAKEENV and in the make call for
the buildkernel. This way makeoptions WITH_CTF=yes not only works when
compiling the tradtitional way, but also when using buildkernel. This
does not enable the CTF part of the world, it still defaults to without
CTF info.

The cross/build-tools/bootstrap targets are not affected by this, they
still have and should keep the explicit NO_CTF.

Notified by: np

Add note about TARGET_ARCH

Expose MACHINE_CPU while building lib32 target.

Remove old documents when they are not compressed.
(NO_INFOCOMPRESS, NO_DOCCOMPRESS or NO_MANCOMPRESS)

MFC after: 1 month

Regen the list of prebuild libraries using tools/make_libdeps.sh.

Unbreak world:
- WITHOUT_OPENSSH (and WITH_KERBEROS)
- WITHOUT_KERBEROS and WITH_GSSAPI

PR: 137483
Submitted by: bf
MFC after: 2 weeks

Note: this breaks harder world WITHOUT_GSSAPI (and WITH_KERBEROS), but well

Remove libulog from the bootstrap again.

libulog now only provides functions that are used by various packages
from the ports tree, namely the libutempter ones. There is no reason to
link it into the crunch/fixit binaries anymore.

In 'make delete-old', use 'exec' to redirect an fd persistently.

That is, write 'exec 3<&0' instead of '3<&0'. Due to an sh(1) bug fixed in
r199953, the latter also persisted, provided that fd 3 was not open before.
With newer sh or fd 3 open, it would not delete orphaned catpages.

MFC after: 2 weeks

Disable K&R style function definitions for WARNS=6.

Unfortunately there are two slight problems with that:

- Yacc and lex might generate code that generates warnings because of
this. Require yacc and lex to be rebuilt during bootstrap. I'm not
incrementing __FreeBSD_version here, because I assume someone else
will do this eventually.

- When running `make buildkernel', it uses share/mk from the source
treeo to build aicasm. Because aicasm also depends on lex, this would
break. Lower WARNS to 5 for now. We should just increment it to 6
again somewhere in the very far future.

Add NO_KERNELOBJ flag, similar to NO_KERNEL{CONFIG,DEPEND,CLEAN},
which disables doing a make obj. Use it when you know it will work
only. KERNFAST now implies NO_KERNELOBJ, since you don't need to keep
doing obj when doing incremental kernel builds.

Convert pam_lastlog(8) to libulog.

The information used by the "Last login:"-line is obtained by using
ulog_setutxfile(3) to switch to the lastlog database. Login and logout
are performed using the utility functions ulog_login(3) and
ulog_logout(3).

This also means we must build libulog during bootstrap.

Approved by: des

Replace -iprefix with -isystem. We only need alternative header
files search path and thus -isystem is sufficient. -iprefix is
meant to do something entirely different.

Approved by: ed (mentor)
OKed by: ru, kan
Tested by: make universe

Now that the zoneinfo data is automatically updated when a new world
is installed, we should at least have the tzsetup tool available!

Suggested by: Andriy Gapon <avg@freebsd.org>
Noticed by: Ben Kaduk <minimarmot@gmail.com>
MFC after: 1 week

Move sed(1) from cross-tools to bootstrap-tools.

Chase dependency changes in libgssapi_krb5 module introruced by
revision r197995.

Fix comment about KERNFAST.

Second attempt at eliminating .text relocations in shared libraries
compiled with stack protector.

Use libssp_nonshared library to pull __stack_chk_fail_local symbol into
each library that needs it instead of pulling it from libc. GCC
generates local calls to this function which result in absolute
relocations put into position-independent code segment, making dynamic
loader do extra work every time given shared library is being relocated
and making affected text pages non-shareable.

Reviewed by: kib
Approved by: re (kib)

Back out previous revision until better tested fix is ready.

Approved by: re (impliciti, by approving previos check-in)

Eliminate .text relocations in shared libraries compiled with stack protector.

Use libssp_nonshared library to pull __stack_chk_fail_local symbol into
each library that needs it instead of pulling it from libc. GCC generates
local calls to this function which result in absolute relocations put into
position-independent code segment, making dynamic loader do extra work everys
time given shared library is being relocated and making affected text pages
non-shareable.

Reviewed by: kib
Approved by: re (kensmith)

Allow "make update" work with SVN, if you take the time to set up your
/etc/make.conf properly.

Only one SCM?
I do not think this is right.
Now I have fixed it.

Revert c190007 at the request of Ruslan. It was not correct because
ar had been moved from cross tools to bootstrap tools when bsd ar was
introduced.

Submitted by: ru@

Add usr.bin/ar to the list of cross tools. Before, when we had gnu
ar, it was built in the gnu/usr.bin/binutils tree. Now it isn't.

Submitted by: John Hein

Restore missing OSREL definition that accidetnally dropped from an
earlier version of this patch.

- Create the buildworld object directories with mtree instead of various
mkdir calls
- Remove the ugly workaroung from libc NLS, which was to create some of
these directories

- Reenable Native Language Support in libc. This feature was disabled due
to possible breakages in the catalog handling code. Since then, that
code has been replaced by the secure code from NetBSD but NLS in libc
remained turned off. Tests have shown that the feature is stable and
working so we can now turn it on again.

- Add several new catalog files:
- ca_ES.ISO8859-1
- de_DE.ISO8859-1
- el_GR.ISO8859-7 (by manolis@ and keramida@)
- es_ES.ISO8859-1 (kern/123179, by carvay@)
- fi_FI.ISO8859-1
- fr_FR.ISO8859-1 (kern/78756, by thierry@)
- hu_HU.ISO8859-2 (by gabor@)
- it_IT.ISO8859-15
- nl_NL.ISO8859-1 (corrections by rene@)
- no_NO.ISO8859-1
- mn_MN.UTF-8 (by ganbold@)
- sk_SK.ISO8859-2
- sv_SE.ISO8859-1
(The catalogs without explicit source has been obtained from NetBSD.)

Approved by: attilio

Don't put "install-info" to the list of install-tools if we're
installing with -DWITHOUT_INFO, otherwise one can experience a
failure trying to installworld on a system that is built with
-DWITHOUT_INFO (i.e., without /usr/bin/install-info).

Reported by: bland
MFC after: 3 days

Implement the xdev target. When you define XDEV=arch XDEV_ARCH=arch,
you can build the cross development tools and install them as
$XDEV-freebsd-xxx for each tool. This allows one to use autoconf to
find the tools for cross building scenarios.

Fix build when WITH_SSP is set explicitly.

Submitted by: Jeremie Le Hen

Implement an idea from Sam Leffler:
make KERNFAST=blah buildkernel
is now a short cut for
make KERNCONF=blah -DKERNFAST buildkernel

This change works for all kernel config files not named "1". I did
that to make sure that
make -DKERNFAST buildkernel
remains the same as
make -DKERNFAST KERNCONF=GENERIC buildkernel
rather than trying to build and configure "1". I've never seen a
kernel config file named "1," so I think this is a good compromise.

Add KERNFAST define. When defined, it skips all the config, depends
and clean steps. KERNFAST was selected to complement KERNCONF which
is typically used in these scenarios (especially with cross building).

Reviewed by: arch@

Add the kerberos5 libs to the install32 target.
(we've been building the all along, but never installing them)

Back out testing aid that was inadvertantly committed as part of r184205.

Noticed by: rdivacky

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

Rescue is "not yet" for MIPS.

If a CPUTYPE isn't specified, then don't use -march=k8 when compiling
32-bit compat libs on amd64 since -march=k8 may generate instructions
that are not implemented on Intel EM64T processors. Instead, use
a simpler set of default flags that should work on all amd64-capable
CPUs.

PR: amd64/113111
Submitted by: NIIMI Satoshi sa2c of sa2c.net
MFC after: 1 week

Various style fixes in the build32 rules.

Submitted by: bde (mostly)

Tweak the support for using ldd on 32-bit objects a bit further.
Specifically, build a 32-bit /usr/bin/ldd32 on amd64 which handles 32-bit
objects. Since it is a 32-bit binary, it can fork a child process which
can dlopen() a 32-bit shared library. The current 32-bit support in ldd
can't do this because it does the dlopen() from a 64-bit process. In order
to preserve an intuitive interface for users, the ldd binary automatically
execs /usr/bin/ldd32 for 32-bit objects. The end result is that ldd on
amd64 now transparently handles 32-bit shared libraries in addition to
32-bit binaries.

Submitted by: ps (indirectly)

Enable GCC stack protection (aka Propolice) for userland:
- It is opt-out for now so as to give it maximum testing, but it may be
turned opt-in for stable branches depending on the consensus. You
can turn it off with WITHOUT_SSP.
- WITHOUT_SSP was previously used to disable the build of GNU libssp.
It is harmless to steal the knob as SSP symbols have been provided
by libc for a long time, GNU libssp should not have been much used.
- SSP is disabled in a few corners such as system bootstrap programs
(sys/boot), process bootstrap code (rtld, csu) and SSP symbols themselves.
- It should be safe to use -fstack-protector-all to build world, however
libc will be automatically downgraded to -fstack-protector because it
breaks rtld otherwise.
- This option is unavailable on ia64.

Enable GCC stack protection (aka Propolice) for kernel:
- It is opt-out for now so as to give it maximum testing.
- Do not compile your kernel with -fstack-protector-all, it won't work.

Submitted by: Jeremie Le Hen <jeremie@le-hen.org>

Add MIPS to the list of known arches.

Add the DTrace build tools to the list of tools and pass NO_CTF to all
the stages of the build which don't require CTF conversion.

Fix conflicts after heimdal-1.1 import and add build infrastructure. Import
all non-style changes made by heimdal to our own libgssapi.

Bootstrap the build of btxld.

Remove options MK_LIBKSE and DEFAULT_THREAD_LIB now that we no longer
build libkse. This should fix WITHOUT_LIBTHR builds as a side effect.

Set AR=gnu-ar and RANLIB=gnu-ranlib on systems where we are forced
to use GNU tools. Remove the _WITH_GNUAR knob.

Prodded by: obrien

Sort.

Prodded by: obrien

Make again BSD ar(1) the default system ar(1), now properly handling
source upgrades by falling back to GNU ar(1) as necessary. Option
WITH_BSDAR is gone. Option _WITH_GNUAR to aid in upgrades is *not*
supposed to be set by the user.

Stop bootstrapping BSD ar(1) on the next __FreeBSD_version bump, as
there are no known bugs in it. Bump __FreeBSD_version to anticipate
this and to flag the switch to BSD ar(1), should it be needed for
something.

Input from: obrien, des, kaiw

Satisfy a particularly obstinate nit-picker :)

Give usr.bin/kdump/kdump_subr.c the same treatment as usr.bin/kdump/ioctl.c
and usr.bin/truss/ioctl.c. This is the correct way to address the problem
that arises when doing an incremental build after a header used by kdump
has been removed (cf. i4b disconnect a while ago)

Explained by: ru
MFC after: 2 weeks

Support source upgrades from at least 6.0-RELEASE.

Reviewed by: imp, obrien

Conditionally add mklocale to bootstrap-tools

Cleanup after last commit (remove undefined variables).

Remove 5.x and 6.x cruft - source upgrades to RELENG_8 from versions prior
to RELENG_7 are not supported.

Remove WITHOUT_FORTRAN from the XMAKE target, it doesn't exist anymore.

Reviewed by: imp, kan
Approved by: rwatson (mentor)
MFC after: 3 days

- Explicitly verify if all needed libs were found by ldd(1). Do so
through scanning its output as ldd(1) returns a non-zero status only
for really abnormal conditions such as an improper file format.
Now cp(1) won't get bogus "not" and "found" arguments if a lib
is missing. [1]

- Don't guess if an element of a complex pipeline is assigned to the main
shell or a sub-shell. Namely use stdio, not vars, to pass lists out from
loops. If using vars, there's the risk that a loop will run in a sub-shell
and the list won't make it to the main shell. It appears that braces and
parens give only limited control over the issue while stdio always works
as intended. Apply this solution to both $progs and $libs for consistency,
although I've failed to go without it only in the $libs part.

Requested by: emaste [1]

Decouple the install tools from the main system as much as possible.
I.e., not only copy them to a scratch dir, but also make them use saved
copies of libraries and locale files. That gives us several benefits:

1) ABI breakages should no longer affect installworld over the live system.

2) It becomes safe to run installworld while still running the old kernel.
However, it can be reasonable to save the old /rescue before that to be
able to run the old reboot(8), as the new binaries are rather likely to
fail with the old kernel. Anyhow, it's now possible to upgrade a system
in a single reboot _reliably_.

3) With a bit of hackery around rtld(8), it becomes possible to do destructive
cross-installs, e.g., i386->amd64 over the live system.

The only shared item left between the old and new systems is rtld(8),
which cannot be run from a saved copy easily because its full
pathname is stored in the respective field of each ELF executable.
(In theory, that field could be overridden, e.g., from the environment,
but this can lead to security issues.) That's why a destructive
cross-install isn't possible w/o hackery yet.

Fruitful ideas by: ru
Reviewed by: ru
Tested with: audit(4)

If a required install tool is missing, provide a more friendly error
message (instead of cp's usage info).

Tweak the handling of "WITHOUT_LIBPTHREAD". Also remove the accidental
treatment of 'LIBKSE' as an "old style" knob.

Submitted by: ru
Approved by: re(kensmith)

Repo copy libpthreads to libkse.
This introduces the WITHOUT_LIBKSE nob,
and changes WITHOUT_LIBPTHREADS to mean with neither threading libs.
Approved by: re(kensmith)

Revise the list of directories we manually create under
${WORLDTMP} and ${LIB32TMP}; some of them are no longer
needed, and some were never needed.

Approved by: re (kensmith)

Always install libpthread.* symlinks if at least one of
the threading libraries is built. This simplifies the
logic in makefiles that need to check if the pthreads
support is present. It also fixes a bug where we would
build a threading library that we shouldn't have built:
for example, building with WITHOUT_LIBTHR and the default
value of DEFAULT_THREADING_LIB (libthr) would mistakenly
build the libthr library, but not install it.

Approved by: re (kensmith)

Forced commit to say that the previous changeset was:

Submitted by: YAMAMOTO Shigeru <shigeru iij ad jp>
Approved by: re (rwatson, implicit)

Add sed(1) to cross tools. We do want newly built version during
buildworld.

Approved by: re (rwatson)

lex(1) no longer installs headers into GCC-internal dirs
(since src/usr.bin/lex/Makefile,v 1.20).

Revision 1.576 removed too much. I didn't notice because my /usr is on
ZFS, which doesn't support flags...

GCC will now properly link libc with all shared objects in order to record
their dependency on libc and its versioned symbols. For that to work,
libc needs to be built before any other shared library that might depend
on it. Add necessary glue to make that happen.

Don't use a subshell where it isn't needed.

Noticed by: John E Hein <jhein@timing.com>

Expose all of {check,delete}-old{,-dirs,-files,-libs}.

MFC after: 3 weeks

Remove stray comment which broke delete-old-dirs.

Partially revert the check-old / delete-old modifications to clarify the
logic.

Apply similar modifications to {check,delete}-old-dirs, which I had
overlooked.

Restore previous behaviour of aborting delete-old* if rm failed.

Pointed out by: netchild
MFC after: 3 weeks

Greatly speed up {check,delete}-old* by replacing make loops with sh loops.

MFC after: 3 weeks

Enable symbol versioning by default. Use WITHOUT_SYMVER to disable it.
Warning, after symbol versioning is enabled, going back is not easy
(use WITHOUT_SYMVER at your own risk).

Change the default thread library to libthr.

There most likely still needs to be a version bump for at least the
thread libraries. If necessary, this will happen later.

Link libzpool, ztest and zdb against libpthread.

Requested by: ru

Connect ZFS to the build.

Enable ncurses wide character support

Approved by: delphij (mentor)
Tested by: kris on pointyhat (early version), current@

Bump the bootstrapping requirements for gensnmptree(1) and remove
a note in UPDATING that tried to work around the build breakage.

Tested by: jhb
OK'ed by: njl

Resurrect one of the patches from attic and refine the
lib32 build somewhat. Specifically, instead of spamming
${CC} et al with -I${LIB32TMP}/usr/include which can be
harmful (as has been demonstrated by the ncursesw WIP),
use slightly different approach to achieve the same goal.
This also simplifies things a bit.

Prodded by: rafan

Clean the lib32 object directory with cleandir rather than rm -rf.

Discussed with: ru
MFC after: 2 weeks

Switch to new ncurses build glue

Approved by: delphij

There was a timeframe where crunchgen(1) was broken; force
an upgrade during the build so those who are affected can
build world again.

There doesn't seem to be a real reason for include/ to be installed
so early.

Add sun4v to the TARGET_ARCH when just TARGET is set (like pc98 is done).

This is required for 'make universe'.

buildworld fixes for sun4v
not sure why pmap.c is included as it is unchanged

Approved by: rwatson (mentor)
Reviewed by: jmg

Shuffle _*_libs assigments (no functional changes).

- Prebuild libgssapi only if building with Kerberos support.

- Remove libnetgraph from the list of prebuilt libraries as
no other library depends on it (snmp_netgraph.so does not
count as we don't build it in the "libraries" target).

- Restore libssh dependencies when compiling with Kerberos
support.

Previous revision wasn't enough for "make TARGET=<machine> build*"
to work. Some build-tools that are part of cross-tools (cc_tools)
also need to know the proper values of TARGET_ARCH and TARGET, so
tell them.

While experimenting with cross-building by specifying just TARGET,
I found one bug. Pass our idea of TARGET_ARCH and TARGET down to
XMAKE, the cross-tools make. Previously it worked because usually
TARGET_ARCH was specified on the initial make's command line.
This should also allow us to simplify the "universe" target, which
I'm currently testing.

Added timestamping to buildworld, similar to buildkernel.

- Rewrite logic to guess TARGET_ARCH/TARGET to be more readable,
but without changes in behavior.

- Add sanity checking for TARGET_ARCH/TARGET.

make "make TARGET=foo" work correctly. Before, it would fail to set
TARGET_ARCH correctly. Now it does, even for pc98. We should suggest
TARGET=foo in preference to TARGET_ARCH because the former is
unambiguous and the latter isn't, so update the docs.

This means that a long standing gripe I've had with this comes to a
close. I can build pc98 w/o specify both things. make TARGET=arm
works (rather than trying to build a arm:amd64 image and dying badly
in the attempt).

If you specify only TARGET_ARCH, then you get the old behavior.

# we can likely simplify the UNIVERSE target now to use this, but I'm not
# up for breaking that tonight :-).

# We should consider adding some kind of sanity check for TARGET_ARCH
# and TARGET.

Always make obj when building the libraries. This never hurts, and
helps some cross-architecture building tool installation patches that
I'm developing.

Remove alpha left-overs.

Remove sade from build-tools, the tinderbox problem should be fixed now.

Add the sade to build-tools list. This fixes a tinderbox.

Fix doxygen target.

Submitted by: "Alexander Mogilny" <sg@astral.ntu-kpi.kiev.ua>
Pointy hat to: netchild

Also need to quote values so they can be passed on correctly.

Create a new target 'buildenvvars'. This target reports the build
environment for cross building (the same one you'd get interactively
in make buildenv). This cannot be a simple

make -f Makefile.inc1 -V WMAKEENV

because in PATH is not set correctly unless one takes a trip through
the Makefile/Makefile.inc1 indirection, the logic of which is too
large to reproduce outside of Makefiles.

Connect the kernel subsystem API documentation generation framework to
the build infrastructure: "make doxygen"

Changes to the man pages will be made when some defaults are revisited.

Requested by: gnn

Send the pcvt(4) driver off to retirement.

Sort bootstrap-tools entries.

Rebuild gensnmptree(1) on 700014 systems too. While 700014 corresponds
to post January 26 systems where gensnmptree(1) code was already fixed,
there was a timeframe between February 14 and February 27 when
usr.sbin/bsnmpd/ including gensnmptree was disconnected from build, so
if you upgraded in this timeframe, you ended up with the 700014 system
but still with a buggy gensnmptree binary. This also means not being
able to buildworld now.

Reported by: jhb
Attention: harti, keramida

Extend coverage of the MK_IPX build option to the following:

- <netipx> headers [1]
- IPX library (libipx)
- IPX support in ifconfig(8)
- IPXrouted(8)
- new MK_NCP option

New MK_NCP build option controls:

- <netncp> and <fs/nwfs> headers
- NCP library (libncp)
- ncplist(1) and ncplogin(1)
- mount_nwfs(8)
- ncp and nwfs kernel modules

User knobs: WITHOUT_IPX, WITHOUT_IPX_SUPPORT, WITHOUT_NCP.

[1] <netsmb/netbios.h> unconditionally uses <netipx> headers
so they are still installed. This needs to be dealt with.

Change the default SUP command to use csup instead of CVSup for
"make update" when SUP_UPDATE is defined.

Convert NO_PROFILE and NO_LIB32 to new style.

Add the new "showconfig" target that displays build configuration.

Suggested by: phk

Reimplementation of world/kernel build options. For details, see:

http://lists.freebsd.org/pipermail/freebsd-current/2006-March/061725.html

The src.conf(5) manpage is to follow in a few days.

Brought to you by: imp, jhb, kris, phk, ru (all bugs are mine)

When building for pc98, don't create stray objects in the
usr.sbin/pcvt/keycap source directory.

Found by: phk
MFC after: 3 days

After revision 1.4 of `src/usr.sbin/bsnmpd/bsnmpd/Makefile' the
gensnmptree utility is required at "make depend" time, because it
is used to generate `oid.h' for bsnmpd.

This means that all versions of HEAD before 2004/01/23 16:22:49 need
gensnmptree as a bootstrap util when it's not already installed as
part of the base system. The first __FreeBSD_version that we can
assume this can work is 700014 which happened after the gensnmptree
change (in -r 1.263 of `src/sys/sys/param.h', at 2006/02/17 14:14:15).

For __FreeBSD_version values before 700014 add gensnmptree to the
bootstrap tools, to allow upgrades from versions of FreeBSD before
that date.

Approved by: ru, harti

Detect that the "audit" group is missing earlier during install.

A file can also be a link, so check not only for a file, but also for a link
in the delete-old and check-old targets.

We don't install a lib (libXY.so.Z) as a link, but an user may have created
something like this. This is dangerous if this link points to a different
version of the lib. So check for a link also in the *-lib targets (an
annoyed user which absolutely wants this redirection of a lib should use
libmap.conf instead of a link).

A directory can also be a link, but in this case just echo a message to
remove it by hand.

Revert the previous change - the lib32 build should work now.

Temporarily disable the kerberos5 lib32 build while I try to fix it
properly.

Add a new extensible GSS-API layer which can support GSS-API plugins,
similar the the Solaris implementation. Repackage the krb5 GSS mechanism
as a plugin library for the new implementation. This also includes a
comprehensive set of manpages for the GSS-API functions with text mostly
taken from the RFC.

Reviewed by: Love Hörnquist Åstrand <lha@it.su.se>, ru (build system), des (openssh parts)

Use cross-compile friendly spelling of CPUTYPE.

Submitted by: ru

Style, no functional changes.

SSE2 comes thru the users choice of CPUTYPE.

Default to producing 'k8' COMPAT_32BIT bits, but allow override.

Back out my sh -> ${SHELL} change, since it will be a few days before
I can fix it better.

Requested by: ru@

Switch BUILD_ARCH in Makefile to use uname -p suggested by ru.
Switch strncpy to strlcpy suggested by gad and issue found by pjd.
Add to uname(3) man page describing:
UNAME_s
UNAME_r
UNAME_v
UNAME_m
Add to getosreldate(3) man page describing:
OSVERSION

Submitted by: ru, pjd/gad
Reviewed by: ru (man pages)

Add support to easily build FreeBSD unpacked in a chroot of another
FreeBSD machine. To do this add the man 1 uname changes to __xuname.c
so we can override the settings it reports. Add OSVERSION override
to getosreldate. Finally which Makefile.inc1 to use uname -m instead
of sysctl -n hw.machine_arch to get the arch. type.

With these change you can put a complete FreeBSD OS image into a
chroot set:
UNAME_s=FreeBSD
UNAME_r=4.7-RELEASE
UNAME_v="FreeBSD $UNAME_r #1: Fri Jul 22 20:32:52 PDT 2005 fake@fake:/usr/obj/usr/src/sys/FAKE"
UNAME_m=i386
UNAME_p=i386
OSVERSION=470000
on an amd64 or i386 and it just work including building ports and using
pkg_add -r etc. The caveat for this example is that these patches
have to be applied to FreeBSD 4.7 and the uname(1) changes need to
be merged. This also addresses issue with libtool.

This is usefull for when a build machine has been trashed for an
old release and we want to do a build on a new machine that FreeBSD
4.7 won't run on ...

In make buildenv, spell sh as ${SHELL}. In this case, we don't want
to hard code /bin/sh since we're forking it for the user to type
commands into. As such, ${SHELL} is the preferred thing users type
commands into.

Revert revision 1.416 and don't create a hierarchy before installing
a kernel. It's slower and is generally only applicable to RELENG_4.

Unbreak installworld.

Get rid of SPECIAL_INSTALLCHECKS variable that isn't settable
by a user. Instead, add individual checks as dependencies to
the main "installcheck" target. Make sure that installkernel
etc. depend on it (including the UID/GID checks).

Fold some common stuff into a macro ${KMAKE}.

Add the NO_INCS knob to bsd.prog.mk and bsd.lib.mk to not include
bsd.incs.mk, and use it when installing 32-bit compat libraries
on amd64. This causes it to *not* overwrite native headers with
i386 versions, which was the case with <fenv.h> and <vgl.h>.

PR: amd64/83806
Prodded by: bde
MFC after: 1 week

Speed up stale catpages hunting by not running sed(1) for every catpage.

Don't check DESTDIR when making distributeworld; the latter expects
DISTDIR.

Reported by: nyan

Add missing shared library interdependencies.

Require DESTDIR to be set when installing for a different architecture.

Serialize access to the info/dir file; needed for parallel installs.

Reported by: scottl

I'm not very fond of using the non-standard lockf(1) here, but I
have no better idea at the moment. NetBSD uses ln(1) to create a
lock file, but this approach can result in a deadlock if make is
interrupted, leaving an orphaned lock file.

Force an upgrade to Groff 1.19.2.

Fix a couple of typos

Fix the "unexpected operator" bug some people are seeing.

First noticed by: kris

Remove cat-pages without corresponding man-pages.

- clean ${LIB32TMP} in _worldtmp
- clean ${OBJTREE}/lib32 in _cleanobj
- ensure that ${WORLDTMP} exists in build32

- Remove duplicate setting from LIB32MAKE definition.
- Install lib32 bits using the real install(1) program.

Make delete-old* and friends useable for cross-builds.

- Only make the delete-old related parts visible if one of the 3 targets
are called (wrapped in ".if make(...)"). This may reduce the amount of
memory needed for all other targets (the file lists are already large
and they will grow further).
- Be verbose in the batch case of the delete-old part too.

Enforce the native environment requirement for {check,delete}-old-* at
run time, not parse time. This unbreaks cross-builds.

Pointy hat to: netchild

Forced commit [1] to add another design goal which I forgot to mention in
the initial commit of the delete-old feature:

One of the design goals of the recently committed way of removing obsolete
files was to *not* remove a file which we never had in the base system (I'm
not sure I mentioned this in the commit log). So if an administrator decides
to move something to somewhere in the base system (for whatever reason), it
will stay where it is (and this is good, since this move was done with a
specific purpose).

Suggested by: Jeremie Le Hen <jeremie@le-hen.org> [1]
Approved by: mentor (joerg; implicit)

- Move recently added dir into the correct section.
- Add a note about the organisation of the sections.
- Expand shell globs (they worked in a previous version of the delete-old
target, but not in this one).
- Use the correct way of checking for a native environment. [1]
- Add some more obsolete files.
- Fix some bad english. [1]

Suggested by: ru [1]
Approved by: mentor (joerg)

Disable thread support in BIND. It appears to reduce performance rather
than increase it, and seems to be the cause of the memory leaks which some
users have reported.

Requested by: dougb
MFC after: 5 days

Add delete-old and delete-old-libs targets:
- removes obsolete files/dirs or libraries.
- works in interactive (default) and batch mode
- respects DISTDIR
- documented in UPDATING and build(7)

The head of the file ObsoleteFiles.inc contains instructions how to add
obsolete files/dirs/libs to the list. Obviously one should add obsolete
files to this list, when he removes a file/dir/lib from the basesystem.

Additionally add check-old target:
- allows re@ to check if a file on the obsolete list resurfaces

Design goals:
- allows full control by the user (default interactive mode)
- possibility of scripted removal of obsolete files (batch mode)
- opt-in removal of files (explicit list of files)
- seperate removal of libs (2 delete targets)

Important design decissions:
- structured list of files to remove instead of a plain text file:
* allows to remove additional files if a NO_foo knob is specified
without the need to change the targets (no NO_foo knob is respected
yet)
- not using mtree like NetBSD does:
* mtree doesn't has an interactive mode

Discussed on: arch (long ago), current (this year)
Additional input from: re (hrs)
Approved by: mentor (joerg)

Make the `distrib-dirs' target official and working properly
when cross-building (installing). (This is useful for NFS
installing world/kernel to another architecture machine.)

Exclude build32 target from the list of toolchain and kernel-toolchain
dependencies. We do not need to build 32bit compat tools to compile
a kernel.

Approved by: re (scottl)
Reported by: jhb

Provide 32-bit runtime support on amd64 as a separate distribution, lib32.

Prodded by: obrien
Nodded by: peter
Approved by: re

Allow to override the compiler, linker, and assembler used to build 32-bit
shim libraries on amd64.

Add install-info to the list of install tools now that we don't
always bootstrap texinfo.

Reported by: scottl

Stop redundantly rebuilding texinfo on the next __FreeBSD_version bump.

Remove lame and ineffectual attempt to fix ipf breakage of tenderbox.

Add some subversive code to cleandir: remove the sbin/ipf tree
entirely to fix the tinderbox machine. This change will be reverted
once the tinderboxen are tindering again.

Merge from RELENG_5: invert WITH_LIB32 into NO_LIB32 so that we build the
32 bit libraries by default. Define NO_LIB32 in make.conf if you do not
want this.

Fix the lib32 build for amd64 after v1.238 of src/include/Makefile (imp).
That change exposed the fact that I'd forgotten to set $MACHINE here.

Fix AMD64 cross-builds when WITH_LIB32 is defined.

(Initially spotted by kris@ on RELENG_5.)

Protect against DESTDIR being set during the build of build32.

Reported by: pav

Add hacks that I use to test cross-builds (by building on
native and foreign architectures and comparing products).
They eliminate most of the differences caused by different
object directory paths, timestamping, and identification.

(Note WORLDTMP was renamed to ${OBJTREE}${.CURDIR}/tmp.)

Bootstrap pwd_mkdb(8) and use its new feature during "make distribute".

Accept the old user interface for NO_CLEAN as it is a POLA violation as
we've eventually changed the user interface of a common command.

Unbreak buildworld.

Make the format of LC_COLLATE files architecture independent.

Bootstrap gencat(1).

OK'ed by: phantom

Added the convenience "distribution" target which calls the
target of the same name from src/etc/Makefile with a proper
environment, suitable to be used during upgrades and cross-
builds.

Make it clear we no longer support source upgrades from
versions prior to 5.3.

Add missing continuation.

As threatened, drop support for source upgrades from pre-5.3.

Inspired by: obrien

Added the "buildenv" target, for developers only!

CAVEAT: if you run tcsh(1) from within this target, this will
reset your PATH making this target mostly useless. Careful!

Submitted by: jmg, ru

Make the format of LC_CTYPE files architecture independent by
introducing the disk formats for _RuneLocale and friends.

The disk formats do not have (useless) pointers and have 32-bit
quantities instead of rune_t and long. (htonl(3) only works
with 32-bit quantities, so there's no loss).

Bootstrap mklocale(1) when necessary. (Bootstrapping from 4.x
would be trivial (verified), but we no longer provide pre-5.3
source upgrades and this is the first commit to actually break
it.)

Add endianness support to cap_mkdb(1), useful for cross builds.

Add lorder(1) to the list of bootstrap-tools.

Allow to cross-build amd64 on non-i386.

In crunchgen(1), when calling make(1), don't redirect stderr to stdout,
just rely on the exit status to detect an error. This makes crunchgen(1)
safe to use with certain make(1) debugging flags.

MFC after: 1 week

Remove the special sparc64 time_t support.
If someone isn't running with a 64-bit time_t by now, they don't
track -CURRENT and would probably go to RELENG_5 vs. 6-CURRENT.

Further fix the case mentioned in rev. 1.302. The
intent was (and still is) that if a user has say
CPUTYPE=i686 set in /etc/make.conf, we don't print
the assignment type warning unless TARGET_CPUTYPE
is overridden.

Unfortunately, the implementation was buggy, and
only recent changes to bsd.cpu.mk that swapped
canonical and alias values of some CPU types made
the bug apparent.

Here's what happens here.

- CPUTYPE=i686 is set in /etc/make.conf,
- bsd.cpu.mk reset it to "pentiumpro",
- Makefile.inc1 compares this canonical value
with the result of the following test,

make -f /dev/null CPUTYPE=pentiumpro -V CPUTYPE

and expects the result to be "pentiumpro" too,
but "i686" is returned, here's why. We have two
CPUTYPE variables, global, set to "i686" in
/etc/make.conf, and command-line (of a higher
precedence), set to "pentiumpro".

The following part of bsd.cpu.mk,

. elif ${CPUTYPE} == "i686"
CPUTYPE = pentiumpro

which is responsible for converting aliases to
canonical values, sees the value of the CPUTYPE
command-line variable first, "pentiumpro", and
no conversion is done -- the net effect is that
CPUTYPE global stays with its old value "i686",
and "make -V CPUTYPE" (which prints variables
in the global context) returns "i686".

The fix was to pass the CPUTYPE in the test above
as an environment variable instead of as a command
line variable, i.e.,

CPUTYPE=pentiumpro make -f /dev/null -V CPUTYPE

This time, CPUTYPE global is still set to "i686"
initially (by /etc/make.conf), and an envieronment
variable CPUTYPE (of a lower precedence) is set
to "pentiumpro". The .elif sees it's set to
"i686" and resets it to "pentiumpro", and so
"make -V" returns "pentiumpro".

NB: these various types of make(1) variables can
be very painful, especially when combined with
"make -V".

NOCLEAN -> NO_CLEAN
NOCLEANDIR -> NO_CLEANDIR

NOSHARE -> NO_SHARE

NOHTML -> NO_HTML

NOGAMES -> NO_GAMES

NOCRYPT -> NO_CRYPT

NOSHARED -> NO_SHARED

NOFSCHG -> NO_FSCHG