MFC r333322: Keep CARP state as INIT when net.inet.carp.allow=0.

Currently when net.inet.carp.allow=0 CARP state remains as MASTER, which is
not very useful (if there are other masters -- it can lead to split brain,
if there are none -- it makes no sense). Having it as INIT makes it clear
that carp packets are disabled.

Sponsored by: iXsystems, Inc.

MFC r335837 (by kp): carp: Set DSCP value CS7

Update carp to set DSCP value CS7(Network Traffic) in the flowlabel field of
packets by default. Currently carp only sets TOS_LOWDELAY in IPv4 which was
deprecated in 1998. This also implements sysctl that can revert carp back to
it's old behavior if desired.

This will allow implementation of QOS on modern network devices to make sure
carp packets aren't dropped during interface contention.

Submitted by: Nick Wolff <darkfiberiru AT gmail.com>
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D14536

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

The tcpdump is section 1, not 8.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Sort SEE ALSO

Fix various mdoc issues.

Found with: mandoc -Tlint

Change examples to have master skew above 0 to have ability to overwrite this
from the slave (for example, when master is failing on disk IO and could not be
logged into or execute cronjob).

Commented out examples changed too to simplify future merging.

Idea by: kaa@zvuki.ru
Discussed with: glebius
MFC after: 1 week

In the EXAMPLES section: append the subsystem regular expression with a
typical interface name pattern as devd(8) uses full string match.

Discussed with: avg, glebius

Change net.inet.carp.demotion sysctl to add the supplied value
to the current demotion factor instead of assigning it.

This allows external scripts to control demotion factor together
with kernel in a raceless manner.

Give a clue how to force tcpdump(8) to interpret correctly CARP packets.

Catch up to the carp rewrite and refer to vhids instead of interfaces.

Pointy hat to: bjk
Submitted by: glebius
Approved by: hrs (mentor)

Grammar and markup fixes.

PR: docs/159854
Approved by: hrs (mentor)

Use 10.0 instead of 10 for FreeBSD version number. The latter undefined.

Remove trailing whitespace per mdoc lint warning

Disussed with: gavin
No objection from: doc
Approved by: joel
MFC after: 3 days

- Rewrite paragraphs about preemption.
- Avoid word combination "carp interface".

Prodded by: az

Apply Typo-B-Gone 2000 [TM]

Restore a feature that was present in 5.x and 6.x, and was cleared in
7.x, 8.x and 9.x with pf(4) imports: pfsync(4) should suppress CARP
preemption, while it is running its bulk update.

However, reimplement the feature in more elegant manner, that is
partially inspired by newer OpenBSD:

- Rename term "suppression" to "demotion", to match with OpenBSD.
- Keep a global demotion factor, that can be raised by several
conditions, for now these are:
- interface goes down
- carp(4) has problems with ip_output() or ip6_output()
- pfsync performs bulk update
- Unlike in OpenBSD the demotion factor isn't a counter, but
is actual value added to advskew. The adjustment values for
particular error conditions are also configurable, and their
defaults are maximum advskew value, so a single failure bumps
demotion to maximum. This is for POLA compatibility, and should
satisfy most users.
- Demotion factor is a writable sysctl, so user can do
foot shooting, if he desires to.

A major overhaul of the CARP implementation. The ip_carp.c was started
from scratch, copying needed functionality from the old implemenation
on demand, with a thorough review of all code. The main change is that
interface layer has been removed from the CARP. Now redundant addresses
are configured exactly on the interfaces, they run on.

The CARP configuration itself is, as before, configured and read via
SIOCSVH/SIOCGVH ioctls. A new prefix created with SIOCAIFADDR or
SIOCAIFADDR_IN6 may now be configured to a particular virtual host id,
which makes the prefix redundant.

ifconfig(8) semantics has been changed too: now one doesn't need
to clone carpXX interface, he/she should directly configure a vhid
on a Ethernet interface.

To supply vhid data from the kernel to an application the getifaddrs(8)
function had been changed to pass ifam_data with each address. [1]

The new implementation definitely closes all PRs related to carp(4)
being an interface, and may close several others. It also allows
to run a single redundant IP per interface.

Big thanks to Bjoern Zeeb for his help with inet6 part of patch, for
idea on using ifam_data and for several rounds of reviewing!

PR: kern/117000, kern/126945, kern/126714, kern/120130, kern/117448
Reviewed by: bz
Submitted by: bz [1]

Fix a couple of issues in last commit.

Submitted by: maxim
Approved by: re (kib)

Describe how carp(4) status changes can be processed with
help of devd(8).

Submitted by: "Alexander V. Chernikov" <melifaro ipfw.ru>
Approved by: re (kib)

Complete the swap of carp(4) log levels and document the change.

MFC after: 3 days

Add section describing how ARP load balancing works and its
limitations:
- Can't load balance behind router
- ARP balancing + pfsync probably won't work

PR: kern/93829
Wording by: scottl

Typo: s/adverisement/advertisement/.

Submitted by: Fredrik Lindberg <fli+freebsd@shapeshifter.se>
PR: docs/85050

(Mostly) markup fixes.

Approved by: re (blanket)

- Document net.inet.carp.suppress_preempt MIB.
- Xref pfsync(4)

s/foobar/mekmitasdigoat/g

Requested by: ceri, des

- Be more informative about advbase and advskew. [1]
- Fix ifconfig commands. Replace 'mekmitasdigoat' with
'foobar'. While the former is more cool, the latter
makes example lines shorter.

Wording by: scottl
MFC after: 3 days

1. Annotate the rc.conf reference with a hint to the "cloned_interfaces"
variable, because it might be not obvious how to configure carp(4)
devices in rc.conf.

2. Remove the sentence about the not implemented "carpdev" option (this
was not imported from OpenBSD according to our source code) to avoid
confusion.

Reviewed by: glebius@
MFC after: 3 days

Update description of net.inet.carp.log

mdoc(7) cleanup

Submitted by: ru

Manpage for CARP from .Ox, with minor modifications.

Obtained from: OpenBSD