pf tests: ensure that pflog shows malformed packets as blocked, not passed

Sponsored by: Rubicon Communications, LLC ("Netgate")

pf tests: Add option to send fragmented packets

Add option to send fragmented packets and to properly sniff them by
reassembling them by the sniffer itself.

Reviewed by: kp
Sponsored by: InnoGames GmbH
Differential Revision: https://reviews.freebsd.org/D42354

netpfil tests: improve pft_ping.py

Multiple improvements to pft_ping.py:

* Automatically use IPv6 when IPv6 addresses are used, --ip6 is not needed.
* Building of ping requests and parsing of ping replies is done layer by
layer. This way most arguments are available both for IPv6 and IPv4,
for ICMP and TCP.
* Use argument groups for improved readability.
* Change ToS and TTL argument name to TC and HL to reflect the modern
IPv6 nomenclature. The argument still set related IPv4 header fields
properly.
* Instead of sniffing for the very specific case of duplicated packets,
allow for sniffing on multiple interfaces.
* Report which sniffer has failed by setting bits of error code.
* Raise meaningful exceptions when irrecoverable errors happen.
* Make IPv4 fragmentation flags configurable.
* Make IPv6 HL / IPv4 TTL configurable.
* Make TCP MSS configurable.
* Make TCP sequence number configurable.
* Make ICMP payload size configurable.
* Add debug output.
* Move command line argument parsing out of network functions.
* Make the code somehow PEP-8 compliant.

MFC after: 1 week
Sponsored by: InnoGames GmbH
Differential Revision: https://reviews.freebsd.org/D38122

netpfil tests: improve sniffer.py

Multiple improvements to sniffer.py:

* Remove ambiguity of configuring recvif, it must be now explicitly specified.
* Don't catch exceptions around creating the sniffer, let it properly
fail and display the whole stack trace.
* Count correct packets so that duplicates can be found.

MFC after: 1 week
Sponsored by: InnoGames GmbH
Differential Revision: https://reviews.freebsd.org/D38120

pf: implement set-tos for IPv6

Extend the existing set-tos keyword to also be able to set traffic class
on IPv6 traffic.
Add tests for this as well.

Reviewed by: kp
Differential Revision: https://reviews.freebsd.org/D31564

tests: Only log critical errors from scapy

Since 2.4.5 scapy started issuing warnings about a few different
configurations during our tests. These are harmless, but they generate
stderr output, which upsets atf_check.

Configure scapy to only log critical errors (and thus not warnings) to
fix these tests.

MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")

pf tests: IPv6 test case for the 'kill state(s)' feature

Reviewed by: donner
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D29797

pf tests: Explicitly ask for python3

If we install the scapy package (which we do list as a dependency) we
don't automatically install python (but we do have python3).

MFC after: 1 week
Sponsored by: Rubicon Communications, LLC (“Netgate”’)

pf tests: Test unicast reverse path forwarding check

Ensure that pf's urpf-failed keyword works as expected.

PR: 253479
MFC after: 1 week
Reviewed by: melifaro@
Differential Revision: https://reviews.freebsd.org/D28694

pf tests: Test that dup-to doesn't produce extra duplicate packets

pf tests: Verify (tcp) checksum modification on unaligned options

It turns out pf incorrectly updates the TCP checksum if the TCP option
we're modifying is not 2-byte algined with respect to the start of the
packet.

Create a TCP packet with such an option and throw it through a scrub
rule, which will update timestamps and modify the packet.

PR: 240416
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D27688

netpfil tests: Add missing copyright & license statements

Follow r354121 to fix some python3 errors in sys.netpfil.*

stderr:

Traceback (most recent call last):
File "/usr/tests/sys/netpfil/common/pft_ping.py", line 135, in <module>
main()
File "/usr/tests/sys/netpfil/common/pft_ping.py", line 124, in main
ping(args.sendif[0], args.to[0], args)
File "/usr/tests/sys/netpfil/common/pft_ping.py", line 74, in ping
raw = sp.raw(str(PAYLOAD_MAGIC))
File "/usr/local/lib/python3.6/site-packages/scapy/compat.py", line 52, in raw
return bytes(x)
TypeError: string argument without an encoding

MFC with: r354121
Sponsored by: The FreeBSD Foundation

Upgrade (scapy) py2 tests to work on py3.

In order to move python2 out of the test framework to avoid py2 vs. py3
confusions upgrade the remaining test cases using scapy to work with py3.
That means only one version of scapy needs to be installed in the CI system.
It also gives a path forward for testing i386 issues observed in the CI
system with some of these tests.

Fixes are:
- Use default python from environment (which is 3.x these days).
- properly ident some lines as common for the rest of the file to avoid
errors.
- cast the calculated offset to an int as the division result is considered
a float which is not accepted input.
- when comparing payload to a magic number make sure we always add the
payload properly to the packet and do not try to compare string in
the result but convert the data payload back into an integer.
- fix print formating.

Discussed with: lwhsu, kp (taking it off his todo :)
MFC after: 2 weeks

netpfil tests: Move pft_ping.py and sniffer.py to the common test directory

The pft_ping.py and sniffer.py tool is moved from tests/sys/netpfil/pf to
tests/sys/netpfil/common directory because these tools are to be used in
common for all the firewalls.

Submitted by: Ahsan Barkati
Reviewed by: kp, thj
Sponsored by: Google, Inc. (GSoC 2019)
Differential Revision: https://reviews.freebsd.org/D21276