Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r236507, r237931, r238120, r238538:

r236507 (pjd):

Simplify the code by using snprlcat().

r237931 (pjd):

Check if there is cmsg at all.

r238120 (pjd):

Make use of GEOM Gate direct reads feature. This allows HAST to serve
reads with native speed of the underlying provider.
There are three situations when direct reads are not used:
1. Data is being synchronized and synchronization source is the secondary
node, which means secondary node has more recent data and we should read
from it.
2. Local read failed and we have to try to read from the secondary node.
3. Local component is unavailable and all I/O requests are served from the
secondary node.

Sponsored by: Panzura, http://www.panzura.com

r238538:

Metaflush on/off values don't need quotes.

Reviewed by: pjd

MFC r229699, r229744, r229778, r229944, r229945, r229946, r230092, r230395,
r230396, r230436, r230457, r230515, r230976:

r229744 (pjd):

fork(2) returns -1 on failure, not some random negative number.

r229699 (pjd):

Constify argument.

r229778 (uqs):

Spelling fixes for sbin/

r229944 (pjd):

Don't touch pidfiles when running in foreground. Before that change we
would create an empty pidfile on start and check if it changed on SIGHUP.

r229945 (pjd):

For functions that return -1 on failure check exactly for -1 and not for
any negative number.

r229946 (pjd):

- Fix a bug where pidfile was removed in SIGHUP when it hasn't changed in
configuration file.
- Log the fact that pidfile has changed.

r230092 (pjd):

Style cleanups.

r230395 (pjd):

Remove unused token 'port'.

r230396 (pjd):

Remove another unused token.

r230436 (pjd):

Fix minor memory leak.

r230457 (pjd):

Free memory that won't be used in child.

r230515 (pjd):

- Fix documentation to note that /etc/hast.conf is the default configuration
file for hastd(8) and hastctl(8) and not hast.conf.
- In copyright statement correct that this file is documentation, not software.
- Bump date.

r230976 (pjd):

Fix typo in comment.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Revert r222688.

Requested by: Mikolaj Golub

Read from the socket using the same max buffer size as we use while
sending. What happens otherwise is that the sender splits all the
traffic into 32k chunks, while the receiver is waiting for the whole
packet. Then for a certain packet sizes, particularly 66607 bytes in
my case, the communication stucks to secondary is expecting to
read one chunk of 66607 bytes, while primary is sending two chunks
of 32768 bytes and third chunk of 1071. Probably due to TCP windowing
and buffering the final chunk gets stuck somewhere, so neither server
not client can make any progress.

This patch also protect from short reads, as according to the manual
page there are some cases when MSG_WAITALL can give less data than
expected.

MFC after: 3 days

Handle ENOBUFS on send(2) by retrying for a while and logging the problem.

MFC after: 1 week

When we are operating on blocking socket and get EAGAIN on send(2) or recv(2)
this means that request timed out. Translate the meaningless EAGAIN to
ETIMEDOUT to give administrator a hint that he might need to increase timeout
in configuration file.

MFC after: 1 month

Allow to disable sends or receives on a socket using shutdown(2) by
interpreting NULL 'data' argument passed to proto_common_send() or
proto_common_recv() as a will to do so.

MFC after: 1 month

Remove #include needed for debugging.

MFC after: 1 week

- Rename proto_descriptor_{send,recv}() functions to
proto_connection_{send,recv} and change them to return proto_conn
structure. We don't operate directly on descriptors, but on
proto_conns.
- Add wrap method to wrap descriptor with proto_conn.
- Remove methods to send and receive descriptors and implement this
functionality as additional argument to send and receive methods.

MFC after: 1 week

Fix build on ia64.

I found no way how to use CMSG_NXTHDR() macro on ia64 without alignment
warnings.

MFC after: 1 week

Until I fix the build on ia64 comment out problematic lines.
Those lines are part of the (for now) unused functions.

Implement two new functions for sending descriptor and receving descriptor
over UNIX domain sockets and socket pairs.
This is in preparation for capsicum.

MFC after: 1 week

- Use pjdlog for assertions and aborts as this will log assert/abort message
to syslog if we run in background.
- Asserts in proto.c that method we want to call is implemented and remove
dummy methods from protocols implementation that are only there to abort
the program with nice message.

MFC after: 1 week

For some setups sending data in 128kB chunks makes communication very slow. No
idea why. 32kB on the other hand seems to work properly everywhere.

Reported by: Thomas Steen Rasmussen <thomas@gibfest.dk>
MFC after: 3 weeks

Fix a problem where hastd will stuck in recv(2) after sending request to
secondary, which died between send(2) and recv(2). Do it by adding timeout
to recv(2) for primary incoming and outgoing sockets and secondary outgoing
socket.

Reported by: Mikolaj Golub <to.my.trociny@gmail.com>
Tested by: Mikolaj Golub <to.my.trociny@gmail.com>
MFC after: 3 days

Please welcome HAST - Highly Avalable Storage.

HAST allows to transparently store data on two physically separated machines
connected over the TCP/IP network. HAST works in Primary-Secondary
(Master-Backup, Master-Slave) configuration, which means that only one of the
cluster nodes can be active at any given time. Only Primary node is able to
handle I/O requests to HAST-managed devices. Currently HAST is limited to two
cluster nodes in total.

HAST operates on block level - it provides disk-like devices in /dev/hast/
directory for use by file systems and/or applications. Working on block level
makes it transparent for file systems and applications. There in no difference
between using HAST-provided device and raw disk, partition, etc. All of them
are just regular GEOM providers in FreeBSD.

For more information please consult hastd(8), hastctl(8) and hast.conf(5)
manual pages, as well as http://wiki.FreeBSD.org/HAST.

Sponsored by: FreeBSD Foundation
Sponsored by: OMCnet Internet Service GmbH
Sponsored by: TransIP BV