Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r229699, r229744, r229778, r229944, r229945, r229946, r230092, r230395,
r230396, r230436, r230457, r230515, r230976:

r229744 (pjd):

fork(2) returns -1 on failure, not some random negative number.

r229699 (pjd):

Constify argument.

r229778 (uqs):

Spelling fixes for sbin/

r229944 (pjd):

Don't touch pidfiles when running in foreground. Before that change we
would create an empty pidfile on start and check if it changed on SIGHUP.

r229945 (pjd):

For functions that return -1 on failure check exactly for -1 and not for
any negative number.

r229946 (pjd):

- Fix a bug where pidfile was removed in SIGHUP when it hasn't changed in
configuration file.
- Log the fact that pidfile has changed.

r230092 (pjd):

Style cleanups.

r230395 (pjd):

Remove unused token 'port'.

r230396 (pjd):

Remove another unused token.

r230436 (pjd):

Fix minor memory leak.

r230457 (pjd):

Free memory that won't be used in child.

r230515 (pjd):

- Fix documentation to note that /etc/hast.conf is the default configuration
file for hastd(8) and hastctl(8) and not hast.conf.
- In copyright statement correct that this file is documentation, not software.
- Bump date.

r230976 (pjd):

Fix typo in comment.

MFC r225773, r225781, r225782, r225783, r225784, 225785, r225786, r225787,
r225830, r225831, r225832, r225835, r226461, r226462, r226463, r226842,
r226851, r226852, r226854, r226855, r226856, r226857, r226859, r226861,
r228542, r228542, r228543, r228544, r228695, r228696:

r225773 (pjd):

Ensure that pjdlog functions don't modify errno.

r225781 (pjd):

No need to use KEEP_ERRNO() macro around pjdlog functions, as they don't
modify errno.

r225782 (pjd):

Prefer PJDLOG_ASSERT() and PJDLOG_ABORT() over assert() and abort().
pjdlog versions will log problem to syslog when application is running in
background.

r225783 (pjd):

Correct two mistakes when converting asserts to PJDLOG_ASSERT()/PJDLOG_ABORT().

r225784 (pjd):

- Convert some impossible conditions into assertions.
- Add missing 'if' in comment.

r225785 (pjd):

Prefer PJDLOG_ASSERT()/PJDLOG_ABORT() over assert().

r225786 (pjd):

No need to wrap pjdlog functions around with KEEP_ERRNO() macro.

r225787 (pjd):

Use PJDLOG_ASSERT() and PJDLOG_ABORT() everywhere instead of assert().

r225830 (pjd):

After every activemap change flush disk's write cache, so that write
reordering won't make the actual write to be committed before marking
the coresponding extent as dirty.

It can be disabled in configuration file.

If BIO_FLUSH is not supported by the underlying file system we log a warning
and never send BIO_FLUSH again to that GEOM provider.

r225831 (pjd):

Break a bit earlier.

r225832 (pjd):

If the underlying provider doesn't support BIO_FLUSH, log it only once
and don't bother trying in the future.

r225835 (pjd):

Correct typo.

r226461 (pjd):

When path to the configuration file is relative, obtain full path,
so we can always find the file, even after daemonizing and changing
working directory to /.

r226462 (pjd):

Remove redundant space.

r226463 (pjd):

Allow to specify pidfile in HAST configuration file.

r226842 (pjd):

Correct comments.

r226851 (pjd):

Delay resuid generation until first connection to secondary, not until first
write. This way on first connection we will synchronize only the extents that
were modified during the lifetime of primary node, not entire GEOM provider.

r226852 (pjd):

Minor cleanups.

r226854 (pjd):

- Eliminate the need for hio_nv.
- Introduce hio_clear() function for clearing hio before returning it
onto free queue.

r226855 (pjd):

Improve comment so it doesn't suggest race is possible, but that we handle
the race.

r226856 (pjd):

Reduce indentation.

r226857 (pjd):

Minor cleanups.

r226859 (pjd):

Implement 'async' mode for HAST.

r226861 (pjd):

Remove redundant space.

r228542 (pjd):

Remove redundant setting of the error variable.

Found by: Clang Static Analyzer

r228543 (pjd):

Simplify code by changing functions types from int to avoid, as the functions
always return 0.

Found by: Clang Static Analyzer

r228544 (pjd):

Remove redundant assignment.

Found by: Clang Static Analyzer

r228695 (pjd):

Don't use function name as format string.

Detected by: clang

r228696 (pjd):

Use lex's standard way of not generating unused function.

Inspired by: r228555

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Compile capsicum support only if HAVE_CAPSICUM is defined.

MFC after: 3 days

Log a warning if we cannot sandbox using capsicum, but only under debug level 1.
It would be too noisy to log it as a proper warning as CAPABILITIES are not
compiled into GENERIC by default.

MFC after: 3 days

To handle BIO_FLUSH and BIO_DELETE requests in secondary worker we need
to use ioctl(2). This is why we can't use capsicum for now to sandbox
secondary. Capsicum is still used to sandbox hastctl.

MFC after: 1 week

Currently we are unable to use capsicum for the primary worker process,
because we need to do ioctl(2)s, which are not permitted in the capability
mode. What we do now is to chroot(2) to /var/empty, which restricts access
to file system name space and we drop privileges to hast user and hast
group.

This still allows to access to other name spaces, like list of processes,
network and sysvipc.

To address that, use jail(2) instead of chroot(2). Using jail(2) will restrict
access to process table, network (we use ip-less jails) and sysvipc (if
security.jail.sysvipc_allowed is turned off). This provides much better
separation.

MFC after: 1 week

When using capsicum to sanbox, still use other methods first, just in case
one of them have some problems.

Add my copyright.

MFC after: 1 week

White space cleanups.

MFC after: 1 week

When dropping privileges prefer capsicum over chroot+setgid+setuid.
We can use capsicum for secondary worker processes and hastctl.
When working as primary we drop privileges using chroot+setgid+setuid
still as we need to send ioctl(2)s to ggate device, for which capsicum
doesn't allow (yet).

X-MFC after: capsicum is merged to stable/8

Add snprlcat() and vsnprlcat() - the functions I'm always missing.
They work as a combination of snprintf(3) and strlcat(3) - the caller
can append a string build based on the given format.

MFC after: 1 week

Let the caller log info about successful privilege drop.
We don't want to log this in hastctl.

MFC after: 1 week

- Use pjdlog for assertions and aborts as this will log assert/abort message
to syslog if we run in background.
- Asserts in proto.c that method we want to call is implemented and remove
dummy methods from protocols implementation that are only there to abort
the program with nice message.

MFC after: 1 week

Implement function that drops privileges by:
- chrooting to /var/empty (user hast home directory),
- setting groups to 'hast' (user hast primary group),
- setting real group id, effective group id and saved group id to 'hast',
- setting real user id, effective user id and saved user id to 'hast'.
At the end verify that those operations where successfull.

MFC after: 1 week

Please welcome HAST - Highly Avalable Storage.

HAST allows to transparently store data on two physically separated machines
connected over the TCP/IP network. HAST works in Primary-Secondary
(Master-Backup, Master-Slave) configuration, which means that only one of the
cluster nodes can be active at any given time. Only Primary node is able to
handle I/O requests to HAST-managed devices. Currently HAST is limited to two
cluster nodes in total.

HAST operates on block level - it provides disk-like devices in /dev/hast/
directory for use by file systems and/or applications. Working on block level
makes it transparent for file systems and applications. There in no difference
between using HAST-provided device and raw disk, partition, etc. All of them
are just regular GEOM providers in FreeBSD.

For more information please consult hastd(8), hastctl(8) and hast.conf(5)
manual pages, as well as http://wiki.FreeBSD.org/HAST.

Sponsored by: FreeBSD Foundation
Sponsored by: OMCnet Internet Service GmbH
Sponsored by: TransIP BV