MFC SVN r329188,329334,329353,329914,329995-329996: DTrace Enhancements

r329188: Use tabs in io.d, fix alignment issues, remove extra newlines
r329334: Add errno definitions to /usr/lib/dtrace/errno.d
r329353: Add inline to errno.d for translating int to string
r329914: Updates and enhancements to io.d to aid DTrace scripting
r329995: Updates and enhancements to signal.d to aid DTrace scripting
r329996: Consistent casing for fallback SIGCHLD (s/Unknown/unknown/)

MFC SVN r330559-330560,330672,332865-332867,333513-333519: dwatch(1)

r330559: Introduce dwatch(1) as a tool for making DTrace more useful
r330560: Bump dwatch(1) internal version from 1.0-beta-91 to 1.0
r330672: Fix display of wrong pid from dtrace_sched(4)
r332865: Add `-dev' option to aid debugging of profiles
r332866: Add profile for send(2)/recv(2) syscalls
r332867: Remove the line used to demonstrate `-dev' option
r333513: Bugfix, usage displayed with `-1Q'
r333514: Separate default values so `-[BK] num' don't affect usage
r333515: Simplify info message test
r333516: Export ARGV to profiles loaded via load_profile()
r333517: Allow `-E code' to override profile EVENT_DETAILS
r333518: Expose process for ip/tcp/udp
r333519: Refactor sendrecv profile

Reviewed by: markj, gnn, bdrewery (head; earlier version)
Approved by: re (gjb)
Relnotes: yes
Sponsored by: Smule, Inc.
Differential Revision: https://reviews.freebsd.org/D15418

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

zfsd(8), the ZFS fault management daemon

Add zfsd, which deals with hard drive faults in ZFS pools. It manages
hotspares and replements in drive slots that publish physical paths.

cddl/usr.sbin/zfsd
Add zfsd(8) and its unit tests

cddl/usr.sbin/Makefile
Add zfsd to the build

lib/libdevdctl
A C++ library that helps devd clients process events

lib/Makefile
share/mk/bsd.libnames.mk
share/mk/src.libnames.mk
Add libdevdctl to the build. It's a private library, unusable by
out-of-tree software.

etc/defaults/rc.conf
By default, set zfsd_enable to NO

etc/mtree/BSD.include.dist
Add a directory for libdevdctl's include files

etc/mtree/BSD.tests.dist
Add a directory for zfsd's unit tests

etc/mtree/BSD.var.dist
Add /var/db/zfsd/cases, where zfsd stores case files while it's shut
down.

etc/rc.d/Makefile
etc/rc.d/zfsd
Add zfsd's rc script

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c
Fix the resource.fs.zfs.statechange message. It had a number of
problems:

It was only being emitted on a transition to the HEALTHY state.
That made it impossible for zfsd to take actions based on drives
getting sicker.

It compared the new state to vdev_prevstate, which is the state that
the vdev had the last time it was opened. That doesn't make sense,
because a vdev can change state multiple times without being
reopened.

vdev_set_state contains logic that will change the device's new
state based on various conditions. However, the statechange event
was being posted _before_ that logic took effect. Now it's being
posted after.

Submitted by: gibbs, asomers, mav, allanjude
Reviewed by: mav, delphij
Relnotes: yes
Sponsored by: Spectra Logic Corp, iX Systems
Differential Revision: https://reviews.freebsd.org/D6564

Add initial DTrace support for RISC-V.

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

Remove the old version of the DTraceToolkit from the source tree.
The DTraceToolkit is part of the Open DTrace effort and is supported
on FreeBSD as a port (sysutils/DTraceToolkit) which has been updated
to properly track toolkit development upstream.

Sponsored by: DARPA, AFRL

Fully connect cddl/usr.sbin/plockstat.

There seems to be no reason to keep this so private.

Also add missing optional MK_CDDL files.

Sponsored by: EMC / Isilon Storage Division

First cut of DTrace for AArch64.

Reviewed by: andrew, emaste
Sponsored by: ARM Limited
Differential Revision: https://reviews.freebsd.org/D2738

Build cddl/{sbin,usr.bin,usr.sbin} in parallel as all of the applications are
freestanding (they require libraries build via make libraries in buildworld)

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Initial version of DTrace on ARM32.

Submitted by: Howard Su based on work by Oleksandr Tymoshenko
Reviewed by: ian, andrew, rpaulo, markj

Build plockstat if WITH_PLOCKSTAT is defined.

Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.

Add placeholder Kyuafiles for various top-level hierarchies.

This change adds tests/ directories in the source tree to create various
subdirectories in /usr/tests/ and to install placeholder Kyuafiles for
them.

the relevant hierarchies are: cddl, etc, games, gnu and secure.

The reason for this is to simplify the addition of new test programs for
utilities or libraries under any of these directories. Doing so on a
case by case basis is unnecessary and is quite an obscure process.

Implement DTrace for PowerPC. This includes both 32-bit and 64-bit.

There is one known issue: Some probes will display an error message along the
lines of: "Invalid address (0)"

I tested this with both a simple dtrace probe and dtruss on a few different
binaries on 32-bit. I only compiled 64-bit, did not run it, but I don't expect
problems without the modules loaded. Volunteers are welcome.

MFC after: 1 month

Introduce "feature flags" for ZFS pools (bump SPA version to 5000).
Add first feature "com.delphix:async_destroy" (asynchronous destroy
of ZFS datasets).
Implement features support in ZFS boot code.

Illumos revisions merged:
13700:2889e2596bd6
13701:1949b688d5fb
2619 asynchronous destruction of ZFS file systems
2747 SPA versioning with zfs feature flags

References:
https://www.illumos.org/issues/2619
https://www.illumos.org/issues/2747

Obtained from: illumos (issue #2619, #2747)
MFC after: 1 month

Enable build of DTrace-related userland parts for MIPS

Install dtruss.

Sponsored by: The FreeBSD Foundation

Use double quotes when checking the value of MACHINE_ARCH.

Sponsored by: The FreeBSD Foundation

libdtrace, dtrace(1) and lockstat(1) are for i386 and amd64 only right
now.

Sponsored by: The FreeBSD Foundation

Add the OpenSolaris lockstat(1M) command. Requires the dtrace driver,
the lockstat provider, and the ksyms(4) pseudo driver kernel modules.

Approved by: gnn (mentor)

Add the dtrace client app to the build.

Remove options MK_LIBKSE and DEFAULT_THREAD_LIB now that we no longer
build libkse. This should fix WITHOUT_LIBTHR builds as a side effect.

Tweak the handling of "WITHOUT_LIBPTHREAD". Also remove the accidental
treatment of 'LIBKSE' as an "old style" knob.

Submitted by: ru
Approved by: re(kensmith)

Repo copy libpthreads to libkse.
This introduces the WITHOUT_LIBKSE nob,
and changes WITHOUT_LIBPTHREADS to mean with neither threading libs.
Approved by: re(kensmith)

Don't build bits that depend on the pthreads support if a
system was configured without such support.

Approved by: re (kensmith)

- Reduce number of atomic operations needed to be implemented in asm by
implementing some of them using existing ones.
- Allow to compile ZFS on all archs and use atomic operations surrounded
by global mutex on archs we don't have or can't have all atomic
operations needed by ZFS.

Apply a set of style.Makefile(5) changes to src/cddl/ makefiles.
There are some insignificant non-style changes as well.

Not fixed: makefiles use ${LIBTHR} that doesn't exist, thus
breaking "make checkdpadd" and not tracking dependencies
properly.

Approved by: pjd

Build ZFS on amd64 and pc98.

Approved by: pjd@

Please welcome ZFS - The last word in file systems.

ZFS file system was ported from OpenSolaris operating system. The code in under
CDDL license.

I'd like to thank all SUN developers that created this great piece of software.

Supported by: Wheel LTD (http://www.wheel.pl/)
Supported by: The FreeBSD Foundation (http://www.freebsdfoundation.org/)
Supported by: Sentex (http://www.sentex.net/)