ntb_hw_plx: Workaround read-only scratchpad registers

On several systems we've noticed that when NTB link goes down, the
Physical Layer User Test Pattern registers we use as additional
scratchpad registers (that is explicitly allowed by the chip specs)
become read-only for about 100us. I see no explanation for this in
the chip specs, neither why it was not seen before, may be a race.
Since we do need these registers, workaround it by repeating writes
until we succeed or 1ms timeout expire.

MFC after: 1 week

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

ntb: Remove unused devclass arguments to DRIVER_MODULE.

newbus: Move from bus_child_{pnpinfo,location}_src to bus_child_{pnpinfo,location} with sbuf

Now that the upper layers all go through a layer to tie into these
information functions that translates an sbuf into char * and len. The
current interface suffers issues of what to do in cases of truncation,
etc. Instead, migrate all these functions to using struct sbuf and these
issues go away. The caller is also in charge of any memory allocation
and/or expansion that's needed during this process.

Create a bus_generic_child_{pnpinfo,location} and make it default. It
just returns success. This is for those busses that have no information
for these items. Migrate the now-empty routines to using this as
appropriate.

Document these new interfaces with man pages, and oversight from before.

Reviewed by: jhb, bcr
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D29937

o Don't include headers from iommu.h, include them from the header
consumers instead;
o Order includes properly.

Reviewed by: kib
Sponsored by: DARPA/AFRL
Differential Revision: https://reviews.freebsd.org/D25878

Fix build.

Reviewed by: kib
Sponsored by: DARPA/AFRL
Differential Revision: https://reviews.freebsd.org/D25879

o Move the buswide_ctxs bitmap to iommu_unit and rename related functions.
o Rename bus_dma_dmar_load_ident() as well.

Reviewed by: kib
Sponsored by: DARPA/AFRL
Differential Revision: https://reviews.freebsd.org/D25852

Call bus_dma_dmar_set_buswide(9) added in r354830.

PLX NTB sends translated DMA requests not only from itsels, but from all
slots and functions of its bus. By default DMAR blocks those additional.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Make ntb(4) send bus_get_dma_tag() requests to parent buses passing real
bus' child pointers instead of grandchilds.

DMAR does not like requests from devices not parented directly by PCI.

MFC after: 2 weeks

Allow splitting PLX NTB BAR2 into several memory windows.

Address Lookup Table (A-LUT) being enabled allows to specify separate
translation for each 1/128th or 1/256th of the BAR2. Previously it was
used only to limit effective window size by blocking access through some
of A-LUT elements. This change allows A-LUT elements to also point
different memory locations, providing to upper layers several (up to 128)
independent memory windows. A-LUT hardware allows even more flexible
configurations than this, but NTB KPI have no way to manage that now.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Implement new methods for Intel and PLX NTB.

This restores parity with AMD NTB driver. Though without any drivers
supporting more then one peer and respective KPI modification to pass
peer index to most of the calls this addition is pretty useless now.

MFC after: 2 weeks

Allow I/OAT of present Xeon E5/E7 to work thorugh PLX NTB.

Its a hack, we can't know/list all DMA engines, but this covers all
I/OAT of Xeon E5/E7 at least from Sandy Bridge till Skylake I saw.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Add initial support for Address Lookup Table (A-LUT).

When enabled by EEPROM, use it to relax translation address/size alignment
requirements for BAR2 window by 128 or 256 times.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Add second entry to LUT on a link side in B2B mode.

Each of two entries on a virtual side should have its counterpart on a
peer's link side.

MFC after: 1 week

Make NTB drivers report more info via NewBus methods.

MFC after: 12 days

Link Interface has no Link Error registers.

MFC after: 13 days

Remove unneeded pmap_change_attr() calls.

Reported by: kib
MFC after: 13 days

Add/polish some defines.

MFC after: 13 days

Fix port control for PEX 8749.

That chip has three Station Ports, so previous address math was incorrect.

MFC after: 13 days
Sponsored by: iXsystems, Inc.

Add NTB driver for PLX/Avago/Broadcom PCIe switches.

This driver supports both NTB-to-NTB and NTB-to-Root Port modes (though
the second with predictable complications on hot-plug and reboot events).
I tested it with PEX 8717 and PEX 8733 chips, but expect it should work
with many other compatible ones too. It supports up to two NT bridges
per chip, each of which can have up to 2 64-bit or 4 32-bit memory windows,
6 or 12 scratchpad registers and 16 doorbells. There are also 4 DMA engines
in those chips, but they are not yet supported.

While there, rename Intel NTB driver from generic ntb_hw(4) to more specific
ntb_hw_intel(4), so now it is on par with this new ntb_hw_plx(4) driver and
alike to Linux naming.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.