wg: uma_zcreate() does not fail

No functional change intended.

MFC after: 1 week

wg: Add netmap support

When in netmap (emulated) mode, wireguard interfaces prepend or strip a
dummy ethernet header when interfacing with netmap. The netmap
application thus sees unencrypted, de-encapsulated frames with a fixed
header.

In this mode, netmap hooks the if_input and if_transmit routines of the
ifnet. Packets from the host TX ring are handled by wg_if_input(),
which simply hands them to the netisr layer; packets which would
otherwise be tunneled are intercepted in wg_output() and placed in the
host RX ring.

The "physical" TX ring is processed by wg_transmit(), which behaves
identically to wg_output() when netmap is not enabled, and packets
appear in the "physical" RX ring by hooking wg_deliver_in().

Reviewed by: vmaffione
MFC after: 1 month
Sponsored by: Klara, Inc.
Sponsored by: Zenarmor
Differential Revision: https://reviews.freebsd.org/D43460

bpf: Make BPF interop consistent with if_loop

The pseudo_AF_HDRCMPLT check is already being done in if_loop and
just needed to be ported over to if_ic, if_wg, if_disc, if_gif,
if_gre, if_me, if_tuntap and ng_iface. This is needed in order to
allow these interfaces to work properly with e.g., tcpreplay.

PR: 256587
Reviewed by: markj
MFC after: 2 weeks
Pull Request: https://github.com/freebsd/freebsd-src/pull/876

wg: Use ENETUNREACH when transmitting to a non-existent peer

The old errno value used is specifically for Capsicum and shouldn't be
co-opted in this way. It has special handling in the generic syscall
layer (see syscallret()). OpenBSD returns ENETUNREACH in this case;
let's do the same thing.

Reviewed by: kevans, imp
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D44582

if_wg: use proper barriers around pkt->p_state

Without appropriate load-synchronization to pair with store barriers in
wg_encrypt() and wg_decrypt(), the compiler and hardware are often
allowed to reorder these loads in wg_deliver_out() and wg_deliver_in()
such that we end up with a garbage or intermediate mbuf that we try to
pass on. The issue is particularly prevalent with the weaker
memory models of !x86 platforms.

Switch from the big-hammer wmb() to more explicit acq/rel atomics to
both make it obvious what we're syncing up with, and to avoid somewhat
hefty fences on platforms that don't necessarily need this.

With this patch, my dual-iperf3 reproducer is dramatically more stable
than it is without on aarch64.

PR: 264115
MFC after: 1 week
Reviewed by: andrew, zlei
Differential Revision: https://reviews.freebsd.org/D44283

wg: detach bpf upon destroy as well

bpfattach() is called in wg_clone_create(), but the bpfdetach() is
missing from wg_close_destroy(). Add the missing bpfdetach() to avoid
leaking both the associated bpf bits as well as the ifnet that bpf will
hold a reference to.

PR: 276526
MFC after: 3 days

if_wg: fix erroneous calculation in calculate_padding() for p_mtu == 0

In practice this is harmless; only keepalive packets may realistically have
p_mtu == 0, and they'll also have no payload so the math works out the same
either way. Still, let's prefer technical accuracy and calculate the amount
of padding needed rather than the padded length...

PR: 276363

sockets: don't malloc/free sockaddr memory on getpeername/getsockname

Just like it was done for accept(2) in cfb1e92912b4, use same approach
for two simplier syscalls that return socket addresses. Although,
these two syscalls aren't performance critical, this change generalizes
some code between 3 syscalls trimming code size.

Following example of accept(2), provide VNET-aware and INVARIANT-checking
wrappers sopeeraddr() and sosockaddr() around protosw methods.

Reviewed by: tuexen
Differential Revision: https://reviews.freebsd.org/D42694

if_wg: Missing radix unlock can cause deadlock

In function 'wg_aip_add()', the error path of returning ENOMEM when
(node == NULL) is forgetting to unlock the radix tree, and thus may lead
to a deadlock.

PR: 275001
Reviewed by: kp
MFC after: 1 week

wg: fix a number of issues with module load failure handling

If MOD_LOAD fails, then MOD_UNLOAD will be called to unwind module
state, but wg_module_init() will have already deinitialized everything
it needs to in a manner that renders it unsafe to call MOD_UNLOAD
after (e.g., freed zone not reset to NULL, wg_osd_jail_slot not reset
to 0). Let's simply stop trying to handle freeing everything in
wg_module_init() to simplify it; let the subsequent MOD_UNLOAD deal with
it, and let's make that robust against partially-constructed state.

jhb@ notes that MOD_UNLOAD being called if MOD_LOAD fails is kind of an
anomaly that doesn't match other paradigms in the kernel; e.g., if
device_attach() fails, we don't invoke device_detach(). It's likely
that a future commit will revert this and instead stop calling
MOD_UNLOAD if MOD_LOAD fails, expecting modules to clean up after
themselves in MOD_LOAD upon failure. Some other modules already do this
and may see similar problems to the wg module (see: carp). The proper
fix is decidedly a bit too invasive to do this close to 14 branching,
and it requires auditing all kmods (base + ports) for potential leaks.

PR: 272089
Reviewed by: emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D40708

wg: fix MOD_LOAD to fail properly if cookie_init() fails

Previously we'd jump to the `free_crypto` label, but never set `ret` to
a failure value -- it would retain success from the call just prior.

Set ret up properly.

This is part of D40708, but not the main point of the change.

wg: change module name to if_wg

Other virtual interface drivers (e.g. if_gif, if_stf, if_ovpn) all start
with if_. The wireguard file is also named if_wg, but the module name
was 'wg'.

Fix this inconsistency.

Reported by: Christian McDonald <cmcdonald@netgate.com>
Reviewed by: zlei, kevans
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D39853

Mechanically convert wg(4) to IfAPI

Reviewed By: jhb
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D38307

Switch wg(4) to the new if_clone KPI

MFC after: 2 weeks
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D37740

wg: Use NET_EPOCH_DRAIN_CALLBACKS macro

Reviewed by: jhb, kp
Approved by: kp (mentor)
Differential Revision: https://reviews.freebsd.org/D37734

wg: Retire now unused support.h.

Reviewed by: kevans, markj, emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36912

wg: Use zfree.

Reviewed by: kevans, markj, emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36911

wg: Use atomic(9) instead of concurrency-kit atomics.

Kernel sanitizers only support atomic(9) operations.

Reviewed by: kevans, markj, emaste
Reported by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36910

Import the WireGuard driver from zx2c4.com.

This commit brings back the driver from FreeBSD commit
f187d6dfbf633665ba6740fe22742aec60ce02a2 plus subsequent fixes from
upstream.

Relative to upstream this commit includes a few other small fixes such
as additional INET and INET6 #ifdef's, #include cleanups, and updates
for recent API changes in main.

Reviewed by: pauamma, gbe, kevans, emaste
Obtained from: git@git.zx2c4.com:wireguard-freebsd @ 3cc22b2
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36909