MFC r312882, r321983 and r321984:
Use the busdma API to allocate all DMA-able memory.

The MLX5 driver has four different types of DMA allocations which are
now allocated using busdma:

1) The 4K firmware DMA-able blocks. One busdma object per 4K allocation.
2) Data for firmware commands use the 4K firmware blocks split into four 1K blocks.
3) The 4K firmware blocks are also used for doorbell pages.
4) The RQ-, SQ- and CQ- DMA rings. One busdma object per allocation.

After this patch the mlx5en driver can be used with DMAR enabled in
the FreeBSD kernel.

Sponsored by: Mellanox Technologies

MFC r312881:
Add support for device surprise removal and other PCI errors.

- When device disappears from PCI indicate error device state and:
1) Trigger command completion for all pending commands
2) Prevent new commands from executing and return:
- success for modify and remove/cleanup commands
- failure for create/query commands
3) When reclaiming pages for a device in error state don't ask FW to
return all given pages, just release the allocated memory

Sponsored by: Mellanox Technologies

MFC r312877 and r312878:
Minor code refactor as a preparation step for suprise removal of CX-4
PCI device(s), changes:
- alloc_entry() now clears bit for page slot entry aswell
- update of cmd->ent_arr[] is now under cmd->alloc_lock
- complete command if alloc_entry() fails

Sponsored by: Mellanox Technologies

MFC r312876:
Use ffs() to scan for first bit instead of using a for() loop.
Minor code refactor while at it.

Sponsored by: Mellanox Technologies

MFC r308409:
When a firmware command times out do not free the command structure to
avoid use after free.

Sponsored by: Mellanox Technologies

MFC r305867:
Update the MLX5 core module:
- Add new firmware commands and update existing ones.
- Add more firmware related structures and update existing ones.
- Some minor fixes, like adding missing \n to some prints.

Sponsored by: Mellanox Technologies

MFC r290650, r290709, r291067, r291068, r291069 and r291070:
Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be
done later.

The code added is not compiled by default, which will be done by a
separate commit.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4177
Differential Revision: https://reviews.freebsd.org/D4178
Differential Revision: https://reviews.freebsd.org/D4179
Differential Revision: https://reviews.freebsd.org/D4180

Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be
done later.

The code added is not compiled by default, which will be done by a
separate commit.

Sponsored by: Mellanox Technologies
MFC after: 2 weeks