MFC r306325, r306329-r306330, r306333, r306620-r306622, r307544, r307550, r318137, r319125, r319295

r306325 by marcel:
Replace the use of linker sets with constructors for both the
formats and schemes. Formats and schemes are registered at
runtime now, rather than collected at link time.

r306329 by marcel:
Eliminate the use of EDOOFUS. The error code was used to signal
programming errors, but is really a poor substitute for assert.
And less portable as well.

r306330 by marcel:
Avoid depending on the <sys/endian.h> header for le*enc and be*enc.
Not only is the header unportable, the encoding/decoding functions
are as well. Instead, duplicate the handful of small inlines we
need into a private header called endian.h.

Aside: an alternative approach is to move the encoding/decoding
functions to a separate system header. While the header is still
nonportable, such an approach would make it possible to re-use the
definitions by playing games with include paths. This may be the
preferred approach if more (build) utilities need this. This
change does not preclude that. In fact, it makes it easier.

r306333 by marcel:
Portability changes:
1. macOS nor Linux have MAP_NOCORE nor MAP_NOSYNC. Define as 0.
2. macOS doesn't have SEEK_DATA nor SEEK_HOLE. Define as -1
so that lseek will return -1 (with errno set to EINVAL).
3. gcc correctly warns that error is assigned but not used in
image_copyout_region(). Fix by returning on the first error.

r306620 by marcel:
Replace STAILQ with TAILQ. TAILQs are portable enough that they can
be used on both macOS and Linux. STAILQs are not. In particular,
STAILQ_LAST does not next on Linux. Since neither STAILQ_FOREACH_SAFE
nor TAILQ_FOREACH_SAFE exist on Linux, replace its use with a regular
TAILQ_FOREACH. The _SAFE variant was only used for having the next
pointer in a local variable.

r306621 by marcel:
Prefer <stdint.h> over <sys/types.h>. While here remove redundant
inclusion of <sys/queue.h>.

Move the inclusion of the disk partitioning headers out of order
and inbetween standard headers and local header. They will change
in a subsequent commit.

r306622 by marcel:
Replace OFF_MAX with INT64_MAX. The former is defined on Linux.

r307544 by marcel:
o Provide a private definition for UUIDs (mkimg_uuid_t) because
UUIDs are not portable.
o Move mkimg_uuid() to a new file and merge both gpt_uuid_enc()
and vhd_uuid_enc() into a single mkimg_uuid_enc() that lives
in the same file.
o Move the OS-specific implementation of generating a UUID to
osdep_uuidgen() and provide the implementations for FreeBSD,
macOS and Linux.
o Expect the partitioning scheme headers to be found by having
a search to the directory in which the headers live. This
avoids conflicts on non-FreeBSD machines.

r307550 by imp:
Add a new flag to mkimg (-a num) to specify the active partition for
those partitioning schemes that have this concept. Implement it as an
override for mbr's setting 0x80 in the flags for the first partition
when we have boot code.

Differential Revision: https://reviews.freebsd.org/D4403

r318137:
mkimg: Add -C argument to specify maximum capacity

Add a -C option to specify a maximum capacity for the final image file.
It is useful to control the size of the generated image for sdcard or
when we will add dynamic size partition.

Add --capacity which is a shorthand to define min and max capacity at
the same time.

Reviewed by: bapt, marcel, wblock (manpages)
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D10509

r319125:
mkimg: Correct an off by one error in the PMBR size

The PMBR last sector should be number of sector - 1 (As stated in UEFI Spec
2.6 page 118 table 17).
This fixes warning printed by linux tools like parted or fdisk.

Sponsored by: Gandi.net

r319295 by ngie:
Update the usr.bin/mkimg golden test output files after ^/head@r319125

^/head@r319125 changed the location of the backup pmbr, requiring the
output files to be regenerated, since they're binary disk dumps.

The output files were regenerated with "make rebase"--fixed in
^/head@r319294.

MFC with: r319125, r319294
PR: 219673
Sponsored by: Dell EMC Isilon

Revert r328974, r307387 was not supposed to be part of it.

MFC r306325, r306329-r306330, r306333, r306620-r306622, r307387, r307544, r307550, r318137, r319125, r319295

r306325 by marcel:
Replace the use of linker sets with constructors for both the
formats and schemes. Formats and schemes are registered at
runtime now, rather than collected at link time.

r306329 by marcel:
Eliminate the use of EDOOFUS. The error code was used to signal
programming errors, but is really a poor substitute for assert.
And less portable as well.

r306330 by marcel:
Avoid depending on the <sys/endian.h> header for le*enc and be*enc.
Not only is the header unportable, the encoding/decoding functions
are as well. Instead, duplicate the handful of small inlines we
need into a private header called endian.h.

Aside: an alternative approach is to move the encoding/decoding
functions to a separate system header. While the header is still
nonportable, such an approach would make it possible to re-use the
definitions by playing games with include paths. This may be the
preferred approach if more (build) utilities need this. This
change does not preclude that. In fact, it makes it easier.

r306333 by marcel:
Portability changes:
1. macOS nor Linux have MAP_NOCORE nor MAP_NOSYNC. Define as 0.
2. macOS doesn't have SEEK_DATA nor SEEK_HOLE. Define as -1
so that lseek will return -1 (with errno set to EINVAL).
3. gcc correctly warns that error is assigned but not used in
image_copyout_region(). Fix by returning on the first error.

r306620 by marcel:
Replace STAILQ with TAILQ. TAILQs are portable enough that they can
be used on both macOS and Linux. STAILQs are not. In particular,
STAILQ_LAST does not next on Linux. Since neither STAILQ_FOREACH_SAFE
nor TAILQ_FOREACH_SAFE exist on Linux, replace its use with a regular
TAILQ_FOREACH. The _SAFE variant was only used for having the next
pointer in a local variable.

r306621 by marcel:
Prefer <stdint.h> over <sys/types.h>. While here remove redundant
inclusion of <sys/queue.h>.

Move the inclusion of the disk partitioning headers out of order
and inbetween standard headers and local header. They will change
in a subsequent commit.

r306622 by marcel:
Replace OFF_MAX with INT64_MAX. The former is defined on Linux.

r307387 by marcel:
Switch to using the portable partition scheme headers.

r307544 by marcel:
o Provide a private definition for UUIDs (mkimg_uuid_t) because
UUIDs are not portable.
o Move mkimg_uuid() to a new file and merge both gpt_uuid_enc()
and vhd_uuid_enc() into a single mkimg_uuid_enc() that lives
in the same file.
o Move the OS-specific implementation of generating a UUID to
osdep_uuidgen() and provide the implementations for FreeBSD,
macOS and Linux.
o Expect the partitioning scheme headers to be found by having
a search to the directory in which the headers live. This
avoids conflicts on non-FreeBSD machines.

r307550 by imp:
Add a new flag to mkimg (-a num) to specify the active partition for
those partitioning schemes that have this concept. Implement it as an
override for mbr's setting 0x80 in the flags for the first partition
when we have boot code.

Differential Revision: https://reviews.freebsd.org/D4403

r318137:
mkimg: Add -C argument to specify maximum capacity

Add a -C option to specify a maximum capacity for the final image file.
It is useful to control the size of the generated image for sdcard or
when we will add dynamic size partition.

Add --capacity which is a shorthand to define min and max capacity at
the same time.

Reviewed by: bapt, marcel, wblock (manpages)
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D10509

r319125:
mkimg: Correct an off by one error in the PMBR size

The PMBR last sector should be number of sector - 1 (As stated in UEFI Spec
2.6 page 118 table 17).
This fixes warning printed by linux tools like parted or fdisk.

Sponsored by: Gandi.net

r319295 by ngie:
Update the usr.bin/mkimg golden test output files after ^/head@r319125

^/head@r319125 changed the location of the backup pmbr, requiring the
output files to be regenerated, since they're binary disk dumps.

The output files were regenerated with "make rebase"--fixed in
^/head@r319294.

MFC with: r319125, r319294
PR: 219673
Sponsored by: Dell EMC Isilon

MFC r315212, r315213, r315214, r315215:
mkimg(1): let calloc(3) do the multiplication.
nscd(8): let calloc(3) do the multiplying.
mpsutil(8): let calloc(3) do the multiplying.
ypbind(8): let calloc(3) do the multiplying.

MFC after: 1 week

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

The size of the first level reference count table is given in terms of the
number of clusters it occupies. It's not the number of entries in the table,
as it is for the L1 cluster table.

For small images, the two are the same. With the unit tests based on small
images, this change has therefore no effect on the unit test. For larger
images (like the FreeBSD 10.1-RELEASE image), this gives a discrepancy that
actually shows up when running "qemu-img check".

Bump the version number of mkimg.

While here, fix a white-space bug.

MFC after: 1 week

Finish QCOW version 2 and stop making it conditional.

We have a different ordering for the RC block(s) and L2 tables.
This is expected to be a non-issue, because everything is found
through file offsets in the corresponding RC table and L1 table.
Files that grow organically have RC blocks and L2 tables scattered
all over the place anyway.

The reason for the difference is that mkimg needs to be able to
write to a pipe. We can't seek forward and backward to fill in
the bits in non-sequential order.

Fix the creation of the L2 cluster table for version 1. The blkofs
variable was assigned the image offset in bytes and not in blocks
(i.e. sectors). This had image_data() return FALSE, which meant that
we didn't assign a cluster when we needed and also meant that we
didn't write parts of the L2 table when we should have. The result
being that the actual data clusters were written at the wrong offset.

Improve support for QCOW version 2. We're having the right layout
and even know how many refcnt blocks we need. All we need to do is
populate the refcnt blocks for every cluster we write and allocate
a cluster when we need a new refcnt block. The allocation part is
tricky in that it'll interleave with the assignment of clusters to
L2 tables and data. Since version 2 is not quite done, keep it
compiled out for now.

Add support for QCOW version 1. Version 2 is partially implemented.
And because of that, it's entirely disabled for now. Both versions
are similar enough that a single header definition works for both
of them. The only "diverting" side-effect is that the union of the
two is larger than the official V1 header.

What this means for our V1 support is that we can't put the L1 table
adjacent to the V1 header (i.e. at offset 0x30 in the file), unless
we revert to hackery and klugery. Let's not. Instead, we align the L1
table at the cluster boundary. This is in line with the V2 layout and
perfectly ok for V1 anyway (ok -- as far as I've seen so far).
Due to the alignment, our V1 image seems to be 1 cluster larger than
the V1 image created by qemu-img (on average).

Compression of the clusters is not supported at this time.

MFC after: 2 months