MFC r359130:

pfctl: improve rule load times with thousands of interfaces

r343287 / D18759 introduced ifa_add_groups_to_map() which is now run by
ifa_load/ifa_lookup/host_if. When loading an anchor or ruleset via pfctl that
does NOT contain ifnames as hosts, host() still ends up iterating all
interfaces twice, grabbing SIOCGIFGROUP ioctl twice for each. This adds an
unnecessary amount of time on systems with thousands or tens of thousands of
interfaces.

Prioritize the IPv4/6 check over the interface name lookup, which skips loading
the iftab and iterating all interfaces when the configuration does not contain
interface names.

Submitted by: Nick Rogers
Sponsored by: RG Nets

MFC r340264:

pfctl: Populate ifname in ifa_lookup()

pfctl_adjust_skip_ifaces() relies on this name.

MFC r334726:

pf: Return non-zero from 'status' if pf is not enabled

In the pf rc.d script the output of `/etc/rc.d/pf status` or `/etc/rc.d/pf
onestatus` always provided an exit status of zero. This made it fiddly to
programmatically determine if pf was running or not.

Return a non-zero status if the pf module is not loaded, extend pfctl to have
an option to return an error status if pf is not enabled.

PR: 228632
Submitted by: James Park-Watt <jimmypw AT gmail.com>

MFC r326276:

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

pf: Filter on and set vlan PCP values

Adopt the OpenBSD syntax for setting and filtering on VLAN PCP values. This
introduces two new keywords: 'set prio' to set the PCP value, and 'prio' to
filter on it.

Reviewed by: allanjude, araujo
Approved by: re (gjb)
Obtained from: OpenBSD (mostly)
Differential Revision: https://reviews.freebsd.org/D6786

Use nitems() from sys/param.h.

MFC after: 2 weeks.

pf: Remove support for 'scrub fragment crop|drop-ovl'

The crop/drop-ovl fragment scrub modes are not very useful and likely to confuse
users into making poor choices.
It's also a fairly large amount of complex code, so just remove the support
altogether.

Users who have 'scrub fragment crop|drop-ovl' in their pf configuration will be
implicitly converted to 'scrub fragment reassemble'.

Reviewed by: gnn, eri
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D3466

Use feature_present(3) to determine whether to open an INET or an
INET6 socket when needed to allow pfctl to work on noinet and noinet6
kernels (and try to provide a fallback using AF_LINK as best effort).
Adjust the Makefile to also respect relevant src.conf(5) options
for compile time decisions on INET and INET6 support.

Reviewed by: glebius (no objections)
MFC after: 1 week

o Create directory sys/netpfil, where all packet filters should
reside, and move there ipfw(4) and pf(4).

o Move most modified parts of pf out of contrib.

Actual movements:

sys/contrib/pf/net/*.c -> sys/netpfil/pf/
sys/contrib/pf/net/*.h -> sys/net/
contrib/pf/pfctl/*.c -> sbin/pfctl
contrib/pf/pfctl/*.h -> sbin/pfctl
contrib/pf/pfctl/pfctl.8 -> sbin/pfctl
contrib/pf/pfctl/*.4 -> share/man/man4
contrib/pf/pfctl/*.5 -> share/man/man5

sys/netinet/ipfw -> sys/netpfil/ipfw

The arguable movement is pf/net/*.h -> sys/net. There are
future plans to refactor pf includes, so I decided not to
break things twice.

Not modified bits of pf left in contrib: authpf, ftp-proxy,
tftp-proxy, pflogd.

The ipfw(4) movement is planned to be merged to stable/9,
to make head and stable match.

Discussed with: bz, luigi

Merge the projects/pf/head branch, that was worked on for last six months,
into head. The most significant achievements in the new code:

o Fine grained locking, thus much better performance.
o Fixes to many problems in pf, that were specific to FreeBSD port.

New code doesn't have that many ifdefs and much less OpenBSDisms, thus
is more attractive to our developers.

Those interested in details, can browse through SVN log of the
projects/pf/head branch. And for reference, here is exact list of
revisions merged:

r232043, r232044, r232062, r232148, r232149, r232150, r232298, r232330,
r232332, r232340, r232386, r232390, r232391, r232605, r232655, r232656,
r232661, r232662, r232663, r232664, r232673, r232691, r233309, r233782,
r233829, r233830, r233834, r233835, r233836, r233865, r233866, r233868,
r233873, r234056, r234096, r234100, r234108, r234175, r234187, r234223,
r234271, r234272, r234282, r234307, r234309, r234382, r234384, r234456,
r234486, r234606, r234640, r234641, r234642, r234644, r234651, r235505,
r235506, r235535, r235605, r235606, r235826, r235991, r235993, r236168,
r236173, r236179, r236180, r236181, r236186, r236223, r236227, r236230,
r236252, r236254, r236298, r236299, r236300, r236301, r236397, r236398,
r236399, r236499, r236512, r236513, r236525, r236526, r236545, r236548,
r236553, r236554, r236556, r236557, r236561, r236570, r236630, r236672,
r236673, r236679, r236706, r236710, r236718, r237154, r237155, r237169,
r237314, r237363, r237364, r237368, r237369, r237376, r237440, r237442,
r237751, r237783, r237784, r237785, r237788, r237791, r238421, r238522,
r238523, r238524, r238525, r239173, r239186, r239644, r239652, r239661,
r239773, r240125, r240130, r240131, r240136, r240186, r240196, r240212.

I'd like to thank people who participated in early testing:

Tested by: Florian Smeets <flo freebsd.org>
Tested by: Chekaluk Vitaly <artemrts ukr.net>
Tested by: Ben Wilber <ben desync.com>
Tested by: Ian FREISLICH <ianf cloudseed.co.za>

Update packet filter (pf) code to OpenBSD 4.5.

You need to update userland (world and ports) tools
to be in sync with the kernel.

Submitted by: mlaier
Submitted by: eri

Add a new option -P to suppress getservbyport(3) calls when printing rules.
This allows one to force consistent printing of numeric port numbers like
we do with -n for other tools like netstat (just that -n was already taken)
rather than the service names.

-P is currently unused in OpenBSD so the change is eligible for upstreaming.

PR: misc/151015
Submitted by: Matt Koivisto (mkoivisto sandvine.com)
Sponsored by: Sandvine Incorporated
MFC after: 1 week

Adapt OpenBSD pf's "sloopy" TCP state machine which is useful for Direct
Server Return mode, where not all packets would be visible to the load
balancer or gateway.

This commit should be reverted when we merge future pf versions. The
benefit it would provide is that this version does not break any existing
public interface and thus won't be a problem if we want to MFC it to
earlier FreeBSD releases.

Discussed with: mlaier
Obtained from: OpenBSD
Sponsored by: iXsystems, Inc.
MFC after: 1 month

Commit resolved import of OpenBSD 4.1 pf userland from perforce.

Approved by: re (kensmith)

Resolve conflicts created during the import of pf 3.7 Some features are
missing and will be implemented in a second step. This is functional as is.

Tested by: freebsd-pf, pfsense.org
Obtained from: OpenBSD

Fix printing of u_int64_t with a cast to unsigned long long.

Found-by: tinderbox(amd64)

Commit userland part of pf version 3.5 from OpenBSD (OPENBSD_3_5_BASE).

Fix $FreeBSD$ ids.

Fix some style(9) related issues after discussion with/education from bde:
- Add <sys/param.h> and <limits.h> where required (do not depend on other
headers pulling it in).
- __dead -> __dead2
- #if defined() -> #ifdef
- Remove ugly PRIu64 macros and use %llu w/ (unsigned long long) cast.

All changes looped back to OpenBSD (where applicable) for easier sync in the
future.

Requested by: bde
Approved by: bms(mentor)

Apply diff from the port.
Rather small diff for the userland (in contrast to the kernel):
- Some header file location/differences
- Clean compilation on 64bit arch (identified by bento a long time ago)
- ALTQ not (yet) available. Leave a switch for patchsets and future ...
- most files can be used from the vendor branch

Approved by: bms(in general)

This commit was generated by cvs2svn to compensate for changes in r126353,
which included commits to RCS files with non-trunk default branches.

Vendor import of OpenBSD's pf userland as of OpenBSD 3.4

Approved by: bms(mentor), core(in general)