pfsync: Performance improvement

pfsync code is called for every new state, state update and state
deletion in pf. While pf itself can operate on multiple states at the
same time (on different cores, assuming the states hash to a different
hashrow), pfsync only had a single lock.
This greatly reduced throughput on multicore systems.

Address this by splitting the pfsync queues into buckets, based on the
state id. This ensures that updates for a given connection always end up
in the same bucket, which allows pfsync to still collapse multiple
updates into one, while allowing multiple cores to proceed at the same
time.

The number of buckets is tunable, but defaults to 2 x number of cpus.
Benchmarking has shown improvement, depending on hardware and setup, from ~30%
to ~100%.

Sponsored by: Orange Business Services

MFC r322663:

Correct multicast address used in pfsync(4)
As stated in src/sys/netinet/in.h L395, address used is 224.0.0.240.

PR: 216885

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Remove duplicate sentence from pfsync.8.

PR: 209539
Submitted by: olivier@
Reviewed by: timmoore88@gmail.com
Sponsored by: iXsystems

Sort SEE ALSO

Fix various mdoc issues.

Found with: mandoc -Tlint

o Create directory sys/netpfil, where all packet filters should
reside, and move there ipfw(4) and pf(4).

o Move most modified parts of pf out of contrib.

Actual movements:

sys/contrib/pf/net/*.c -> sys/netpfil/pf/
sys/contrib/pf/net/*.h -> sys/net/
contrib/pf/pfctl/*.c -> sbin/pfctl
contrib/pf/pfctl/*.h -> sbin/pfctl
contrib/pf/pfctl/pfctl.8 -> sbin/pfctl
contrib/pf/pfctl/*.4 -> share/man/man4
contrib/pf/pfctl/*.5 -> share/man/man5

sys/netinet/ipfw -> sys/netpfil/ipfw

The arguable movement is pf/net/*.h -> sys/net. There are
future plans to refactor pf includes, so I decided not to
break things twice.

Not modified bits of pf left in contrib: authpf, ftp-proxy,
tftp-proxy, pflogd.

The ipfw(4) movement is planned to be merged to stable/9,
to make head and stable match.

Discussed with: bz, luigi

Restore a feature that was present in 5.x and 6.x, and was cleared in
7.x, 8.x and 9.x with pf(4) imports: pfsync(4) should suppress CARP
preemption, while it is running its bulk update.

However, reimplement the feature in more elegant manner, that is
partially inspired by newer OpenBSD:

- Rename term "suppression" to "demotion", to match with OpenBSD.
- Keep a global demotion factor, that can be raised by several
conditions, for now these are:
- interface goes down
- carp(4) has problems with ip_output() or ip6_output()
- pfsync performs bulk update
- Unlike in OpenBSD the demotion factor isn't a counter, but
is actual value added to advskew. The adjustment values for
particular error conditions are also configurable, and their
defaults are maximum advskew value, so a single failure bumps
demotion to maximum. This is for POLA compatibility, and should
satisfy most users.
- Demotion factor is a writable sysctl, so user can do
foot shooting, if he desires to.

- Fix examples to show new CARP style.
- Remove OpenBSDisms, add FreeBSDisms.

Update packet filter (pf) code to OpenBSD 4.5.

You need to update userland (world and ports) tools
to be in sync with the kernel.

Submitted by: mlaier
Submitted by: eri

Commit resolved import of OpenBSD 4.1 pf userland from perforce.

Approved by: re (kensmith)

- Note that the synchronisation interface needs to be up and have
an IP address assigned.
- Add "quick" keyword to pf.conf example.

PR: docs/85209

Add an rc.d script to start pfsync at the right moment of the
system boot, and hook it up in the system.

The separate script is needed because in the presence of various
interface lists in rc.conf ($network_interfaces, $cloned_interfaces,
$sppp_interfaces, $gif_interfaces, more to come) it is hard to start
them orderly, so that pfsync is brought up after its syncdev, which
is required for the proper startup of pfsync.

Discussed with: mlaier on -pf
MFC after: 5 days

More tcpdump 8->1 cleanup.

Approved by: mlaier
MFC after: 3 days

- Remove MLINKS to nonexistant manpages
- Change some section numbers to match reality
- For MLINKS to manpages from ports, mention which port installs them

MFC after: 3 days

Resolve conflicts created during the import of pf 3.7 Some features are
missing and will be implemented in a second step. This is functional as is.

Tested by: freebsd-pf, pfsense.org
Obtained from: OpenBSD

- remove OpenBSDisms, add FreeBSDisms
- comment out feature, we do not have yet: tcpdumping on pfsync,
add a BUGS section
- reference carp.4
- dereference bpf(4), tcpdump(7), hostname.if(5)
- sort references
- tell when pfsync appeared in FreeBSD

Reviewed by: mlaier
MFC after: 1 week

Commit userland part of pf version 3.5 from OpenBSD (OPENBSD_3_5_BASE).

FreeBSD-if .4 manpages for pf/pflog/pfsync.

PR: docs/65687
Submitted by: Sergey Matveychuk
Approved by: bms(mentor)

This commit was generated by cvs2svn to compensate for changes in r126353,
which included commits to RCS files with non-trunk default branches.

Vendor import of OpenBSD's pf userland as of OpenBSD 3.4

Approved by: bms(mentor), core(in general)