Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r257155, r257582, r259191, r259192, r259193, r259194, r259195, r259196:

r257155:

Make hastctl list command output current queue sizes.

Reviewed by: pjd

r257582 (pjd):

Correct alignment.

r259191:

For memsync replication, hio_countdown is used not only as an
indication when a request can be moved to done queue, but also for
detecting the current state of memsync request.

This approach has problems, e.g. leaking a request if memsynk ack from
the secondary failed, or racy usage of write_complete, which should be
called only once per write request, but for memsync can be entered by
local_send_thread and ggate_send_thread simultaneously.

So the following approach is implemented instead:

1) Use hio_countdown only for counting components we waiting to
complete, i.e. initially it is always 2 for any replication mode.

2) To distinguish between "memsync ack" and "memsync fin" responses
from the secondary, add and use hio_memsyncacked field.

3) write_complete() in component threads is called only before
releasing hio_countdown (i.e. before the hio may be returned to the
done queue).

4) Add and use hio_writecount refcounter to detect when
write_complete() can be called in memsync case.

Reported by: Pete French petefrench ingresso.co.uk
Tested by: Pete French petefrench ingresso.co.uk

r259192:

Add some macros to make the code more readable (no functional chages).

r259193:

Fix compiler warnings.

r259194:

In remote_send_thread, if sending a request fails don't take the
request back from the receive queue -- it might already be processed
by remote_recv_thread, which lead to crashes like below:

(primary) Unable to receive reply header: Connection reset by peer.
(primary) Unable to send request (Connection reset by peer):
WRITE(954662912, 131072).
(primary) Disconnected from kopusha:7772.
(primary) Increasing localcnt to 1.
(primary) Assertion failed: (old > 0), function refcnt_release,
file refcnt.h, line 62.

Taking the request back was not necessary (it would properly be
processed by the remote_recv_thread) and only complicated things.

r259195:

Send wakeup to threads waiting on empty queue before releasing the
lock to decrease spurious wakeups.

Submitted by: davidxu

r259196:

Check remote protocol version only for the first connection (when it
is actually sent by the remote node).

Otherwise it generated confusing "Negotiated protocol version 1" debug
messages when processing the second connection.

MFC r255714, r255716, r255717:

r255714:

Use cv_broadcast() instead of cv_signal() when waking up threads
waiting on an empty queue as the queue may have several consumers.

Before the fix the following scenario was possible: 2 threads are
waiting on empty queue, 2 threads are inserting simultaneously. The
first inserting thread detects that the queue is empty and is going to
send the signal, but before it sends the second thread inserts
too. When the first sends the signal only one of the waiting threads
receive it while the other one may wait forever.

The scenario above is is believed to be the cause of the observed
cases, when ggate_recv_thread() was getting stuck on taking free
request, while the free queue was not empty.

Reviewed by: pjd
Tested by: Yamagi Burmeister yamagi.org

r255716:

When updating the map of dirty extents, most recently used extents are
kept dirty to reduce the number of on-disk metadata updates. The
sequence of operations is:

1) acquire the activemap lock;
2) update in-memory map;
3) if the list of keepdirty extents is changed, update on-disk metadata;
4) release the lock.

On-disk updates are not frequent in comparison with in-memory updates,
while require much more time. So situations are possible when one
thread is updating on-disk metadata and another one is waiting for the
activemap lock just to update the in-memory map.

Improve this by introducing additional, on-disk map lock: when
in-memory map is updated and it is detected that the on-disk map needs
update too, the on-disk map lock is acquired and the on-memory lock is
released before flushing the map.

Reported by: Yamagi Burmeister yamagi.org
Tested by: Yamagi Burmeister yamagi.org
Reviewed by: pjd

r255717:

Fix comments.

A set of MFCs allowing stable/9 to be built again on head after r250881:

r250227:
Improve compatibility with recent flex from flex.sourceforge.net.

r250914:
Improve compatibility with old flex and fix build with GCC.

r250926 (partial):
Work around build breakages with GCC 4.2.

MFC r246922 (pjd):

- Add support for 'memsync' mode. This is the fastest replication mode that's
why it will now be the default.
- Bump protocol version to 2 and add backward compatibility for version 1.
- Allow to specify hosts by kern.hostid as well (in addition to hostname and
kern.hostuuid) in configuration file.

Sponsored by: Panzura

MFC r247281:

Add i/o error counters to hastd(8) and make hastctl(8) display
them. This may be useful for detecting problems with HAST disks.

Discussed with and reviewed by: pjd

MFC r225773, r225781, r225782, r225783, r225784, 225785, r225786, r225787,
r225830, r225831, r225832, r225835, r226461, r226462, r226463, r226842,
r226851, r226852, r226854, r226855, r226856, r226857, r226859, r226861,
r228542, r228542, r228543, r228544, r228695, r228696:

r225773 (pjd):

Ensure that pjdlog functions don't modify errno.

r225781 (pjd):

No need to use KEEP_ERRNO() macro around pjdlog functions, as they don't
modify errno.

r225782 (pjd):

Prefer PJDLOG_ASSERT() and PJDLOG_ABORT() over assert() and abort().
pjdlog versions will log problem to syslog when application is running in
background.

r225783 (pjd):

Correct two mistakes when converting asserts to PJDLOG_ASSERT()/PJDLOG_ABORT().

r225784 (pjd):

- Convert some impossible conditions into assertions.
- Add missing 'if' in comment.

r225785 (pjd):

Prefer PJDLOG_ASSERT()/PJDLOG_ABORT() over assert().

r225786 (pjd):

No need to wrap pjdlog functions around with KEEP_ERRNO() macro.

r225787 (pjd):

Use PJDLOG_ASSERT() and PJDLOG_ABORT() everywhere instead of assert().

r225830 (pjd):

After every activemap change flush disk's write cache, so that write
reordering won't make the actual write to be committed before marking
the coresponding extent as dirty.

It can be disabled in configuration file.

If BIO_FLUSH is not supported by the underlying file system we log a warning
and never send BIO_FLUSH again to that GEOM provider.

r225831 (pjd):

Break a bit earlier.

r225832 (pjd):

If the underlying provider doesn't support BIO_FLUSH, log it only once
and don't bother trying in the future.

r225835 (pjd):

Correct typo.

r226461 (pjd):

When path to the configuration file is relative, obtain full path,
so we can always find the file, even after daemonizing and changing
working directory to /.

r226462 (pjd):

Remove redundant space.

r226463 (pjd):

Allow to specify pidfile in HAST configuration file.

r226842 (pjd):

Correct comments.

r226851 (pjd):

Delay resuid generation until first connection to secondary, not until first
write. This way on first connection we will synchronize only the extents that
were modified during the lifetime of primary node, not entire GEOM provider.

r226852 (pjd):

Minor cleanups.

r226854 (pjd):

- Eliminate the need for hio_nv.
- Introduce hio_clear() function for clearing hio before returning it
onto free queue.

r226855 (pjd):

Improve comment so it doesn't suggest race is possible, but that we handle
the race.

r226856 (pjd):

Reduce indentation.

r226857 (pjd):

Minor cleanups.

r226859 (pjd):

Implement 'async' mode for HAST.

r226861 (pjd):

Remove redundant space.

r228542 (pjd):

Remove redundant setting of the error variable.

Found by: Clang Static Analyzer

r228543 (pjd):

Simplify code by changing functions types from int to avoid, as the functions
always return 0.

Found by: Clang Static Analyzer

r228544 (pjd):

Remove redundant assignment.

Found by: Clang Static Analyzer

r228695 (pjd):

Don't use function name as format string.

Detected by: clang

r228696 (pjd):

Use lex's standard way of not generating unused function.

Inspired by: r228555

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Keep statistics on number of BIO_READ, BIO_WRITE, BIO_DELETE and BIO_FLUSH
requests as well as number of activemap updates.

Number of BIO_WRITEs and activemap updates are especially interesting, because
if those two are too close to each other, it means that your workload needs
bigger number of dirty extents. Activemap should be updated as rarely as
possible.

MFC after: 1 week

Rename ipv4/ipv6 to tcp4/tcp6.

MFC after: 3 weeks

In preparation for IPv6 support allow to specify multiple addresses to
listen on.

MFC after: 3 weeks

Increase default timeout from 5 seconds to 20 seconds. 5 seconds is definitely
to short under heavy load and I was experiencing those timeouts in my recent
tests.

MFC after: 1 week

The proto API is a general purpose API, so don't use 'hast' in structures or
function names. It can now be used outside of HAST.

MFC after: 1 week

In hast.conf we define the other node's address in 'remote' variable.
This way we know how to connect to secondary node when we are primary.
The same variable is used by the secondary node - it only accepts
connections from the address stored in 'remote' variable.
In cluster configurations it is common that each node has its individual
IP address and there is one addtional shared IP address which is assigned
to primary node. It seems it is possible that if the shared IP address is
from the same network as the individual IP address it might be choosen by
the kernel as a source address for connection with the secondary node.
Such connection will be rejected by secondary, as it doesn't come from
primary node individual IP.

Add 'source' variable that allows to specify source IP address we want to
bind to before connecting to the secondary node.

MFC after: 1 week

For secondary, set 2 * HAST_KEEPALIVE seconds timeout for incoming
connection so the worker will exit if it does not receive packets from
the primary during this interval.

Reported by: Christian Vogt <Christian.Vogt@haw-hamburg.de>
Tested by: Christian Vogt <Christian.Vogt@haw-hamburg.de>
Approved by: pjd (mentor)
MFC after: 1 week

Allow to compress on-the-wire data using two algorithms:
- HOLE - it simply turns all-zero blocks into few bytes header;
it is extremely fast, so it is turned on by default;
it is mostly intended to speed up initial synchronization
where we expect many zeros;
- LZF - very fast algorithm by Marc Alexander Lehmann, which shows
very decent compression ratio and has BSD license.

MFC after: 2 weeks

Allow to checksum on-the-wire data using either CRC32 or SHA256.

MFC after: 2 weeks

Setup another socketpair between parent and child, so that primary sandboxed
worker can ask the main privileged process to connect in worker's behalf
and then we can migrate descriptor using this socketpair to worker.
This is not really needed now, but will be needed once we start to use
capsicum for sandboxing.

MFC after: 1 week

Implement function that drops privileges by:
- chrooting to /var/empty (user hast home directory),
- setting groups to 'hast' (user hast primary group),
- setting real group id, effective group id and saved group id to 'hast',
- setting real user id, effective user id and saved user id to 'hast'.
At the end verify that those operations where successfull.

MFC after: 1 week

Remember created control connection so on fork(2) we can close it in child.

Found with: procstat(1)
MFC after: 1 week

Because it is very hard to make fork(2) from threaded process safe (we are
limited to async-signal safe functions in the child process), move all hooks
execution to the main (non-threaded) process.

Do it by maintaining connection (socketpair) between child and parent
and sending events from the child to parent, so it can execute the hook.

This is step in right direction for others reasons too. For example there is
one less problem to drop privs in worker processes.

MFC after: 2 weeks
Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com

Allow to execute specified program on various HAST events.

MFC after: 2 weeks
Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com

Implement keepalive mechanism inside HAST protocol so we can detect secondary
node failures quickly for HAST resources that are rarely modified.

Remove XXX from a comment now that the guard thread never sleeps infinitely.

MFC after: 2 weeks
Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com

Prepare configuration parsing code to be called multiple times:
- Don't exit on errors if not requested.
- Don't keep configuration in global variable, but allocate memory for
configuration.
- Call yyrestart() before yyparse() so that on error in configuration file
we will start from the begining next time and not from the place we left of.

MFC after: 1 month

Fix a problem where hastd will stuck in recv(2) after sending request to
secondary, which died between send(2) and recv(2). Do it by adding timeout
to recv(2) for primary incoming and outgoing sockets and secondary outgoing
socket.

Reported by: Mikolaj Golub <to.my.trociny@gmail.com>
Tested by: Mikolaj Golub <to.my.trociny@gmail.com>
MFC after: 3 days

Please welcome HAST - Highly Avalable Storage.

HAST allows to transparently store data on two physically separated machines
connected over the TCP/IP network. HAST works in Primary-Secondary
(Master-Backup, Master-Slave) configuration, which means that only one of the
cluster nodes can be active at any given time. Only Primary node is able to
handle I/O requests to HAST-managed devices. Currently HAST is limited to two
cluster nodes in total.

HAST operates on block level - it provides disk-like devices in /dev/hast/
directory for use by file systems and/or applications. Working on block level
makes it transparent for file systems and applications. There in no difference
between using HAST-provided device and raw disk, partition, etc. All of them
are just regular GEOM providers in FreeBSD.

For more information please consult hastd(8), hastctl(8) and hast.conf(5)
manual pages, as well as http://wiki.FreeBSD.org/HAST.

Sponsored by: FreeBSD Foundation
Sponsored by: OMCnet Internet Service GmbH
Sponsored by: TransIP BV