#
362771 |
|
29-Jun-2020 |
gonzo |
MFC r361935:
Add VHDX support to mkimg(1)
VHDX is the successor of Microsoft's VHD file format. It increases maximum capacity of the virtual drive to 64TB and introduces features to better handle power/system failures.
VHDX is the required format for 2nd generation Hyper-V VMs.
Reviewed by: marcel Differential Revision: https://reviews.freebsd.org/D25184
|
#
329059 |
|
09-Feb-2018 |
manu |
MFC r306325, r306329-r306330, r306333, r306620-r306622, r307544, r307550, r318137, r319125, r319295
r306325 by marcel: Replace the use of linker sets with constructors for both the formats and schemes. Formats and schemes are registered at runtime now, rather than collected at link time.
r306329 by marcel: Eliminate the use of EDOOFUS. The error code was used to signal programming errors, but is really a poor substitute for assert. And less portable as well.
r306330 by marcel: Avoid depending on the <sys/endian.h> header for le*enc and be*enc. Not only is the header unportable, the encoding/decoding functions are as well. Instead, duplicate the handful of small inlines we need into a private header called endian.h.
Aside: an alternative approach is to move the encoding/decoding functions to a separate system header. While the header is still nonportable, such an approach would make it possible to re-use the definitions by playing games with include paths. This may be the preferred approach if more (build) utilities need this. This change does not preclude that. In fact, it makes it easier.
r306333 by marcel: Portability changes: 1. macOS nor Linux have MAP_NOCORE nor MAP_NOSYNC. Define as 0. 2. macOS doesn't have SEEK_DATA nor SEEK_HOLE. Define as -1 so that lseek will return -1 (with errno set to EINVAL). 3. gcc correctly warns that error is assigned but not used in image_copyout_region(). Fix by returning on the first error.
r306620 by marcel: Replace STAILQ with TAILQ. TAILQs are portable enough that they can be used on both macOS and Linux. STAILQs are not. In particular, STAILQ_LAST does not next on Linux. Since neither STAILQ_FOREACH_SAFE nor TAILQ_FOREACH_SAFE exist on Linux, replace its use with a regular TAILQ_FOREACH. The _SAFE variant was only used for having the next pointer in a local variable.
r306621 by marcel: Prefer <stdint.h> over <sys/types.h>. While here remove redundant inclusion of <sys/queue.h>.
Move the inclusion of the disk partitioning headers out of order and inbetween standard headers and local header. They will change in a subsequent commit.
r306622 by marcel: Replace OFF_MAX with INT64_MAX. The former is defined on Linux.
r307544 by marcel: o Provide a private definition for UUIDs (mkimg_uuid_t) because UUIDs are not portable. o Move mkimg_uuid() to a new file and merge both gpt_uuid_enc() and vhd_uuid_enc() into a single mkimg_uuid_enc() that lives in the same file. o Move the OS-specific implementation of generating a UUID to osdep_uuidgen() and provide the implementations for FreeBSD, macOS and Linux. o Expect the partitioning scheme headers to be found by having a search to the directory in which the headers live. This avoids conflicts on non-FreeBSD machines.
r307550 by imp: Add a new flag to mkimg (-a num) to specify the active partition for those partitioning schemes that have this concept. Implement it as an override for mbr's setting 0x80 in the flags for the first partition when we have boot code.
Differential Revision: https://reviews.freebsd.org/D4403
r318137: mkimg: Add -C argument to specify maximum capacity
Add a -C option to specify a maximum capacity for the final image file. It is useful to control the size of the generated image for sdcard or when we will add dynamic size partition.
Add --capacity which is a shorthand to define min and max capacity at the same time.
Reviewed by: bapt, marcel, wblock (manpages) Sponsored by: Gandi.net Differential Revision: https://reviews.freebsd.org/D10509
r319125: mkimg: Correct an off by one error in the PMBR size
The PMBR last sector should be number of sector - 1 (As stated in UEFI Spec 2.6 page 118 table 17). This fixes warning printed by linux tools like parted or fdisk.
Sponsored by: Gandi.net
r319295 by ngie: Update the usr.bin/mkimg golden test output files after ^/head@r319125
^/head@r319125 changed the location of the backup pmbr, requiring the output files to be regenerated, since they're binary disk dumps.
The output files were regenerated with "make rebase"--fixed in ^/head@r319294.
MFC with: r319125, r319294 PR: 219673 Sponsored by: Dell EMC Isilon
|
#
328976 |
|
07-Feb-2018 |
manu |
Revert r328974, r307387 was not supposed to be part of it.
|
#
328974 |
|
07-Feb-2018 |
manu |
MFC r306325, r306329-r306330, r306333, r306620-r306622, r307387, r307544, r307550, r318137, r319125, r319295
r306325 by marcel: Replace the use of linker sets with constructors for both the formats and schemes. Formats and schemes are registered at runtime now, rather than collected at link time.
r306329 by marcel: Eliminate the use of EDOOFUS. The error code was used to signal programming errors, but is really a poor substitute for assert. And less portable as well.
r306330 by marcel: Avoid depending on the <sys/endian.h> header for le*enc and be*enc. Not only is the header unportable, the encoding/decoding functions are as well. Instead, duplicate the handful of small inlines we need into a private header called endian.h.
Aside: an alternative approach is to move the encoding/decoding functions to a separate system header. While the header is still nonportable, such an approach would make it possible to re-use the definitions by playing games with include paths. This may be the preferred approach if more (build) utilities need this. This change does not preclude that. In fact, it makes it easier.
r306333 by marcel: Portability changes: 1. macOS nor Linux have MAP_NOCORE nor MAP_NOSYNC. Define as 0. 2. macOS doesn't have SEEK_DATA nor SEEK_HOLE. Define as -1 so that lseek will return -1 (with errno set to EINVAL). 3. gcc correctly warns that error is assigned but not used in image_copyout_region(). Fix by returning on the first error.
r306620 by marcel: Replace STAILQ with TAILQ. TAILQs are portable enough that they can be used on both macOS and Linux. STAILQs are not. In particular, STAILQ_LAST does not next on Linux. Since neither STAILQ_FOREACH_SAFE nor TAILQ_FOREACH_SAFE exist on Linux, replace its use with a regular TAILQ_FOREACH. The _SAFE variant was only used for having the next pointer in a local variable.
r306621 by marcel: Prefer <stdint.h> over <sys/types.h>. While here remove redundant inclusion of <sys/queue.h>.
Move the inclusion of the disk partitioning headers out of order and inbetween standard headers and local header. They will change in a subsequent commit.
r306622 by marcel: Replace OFF_MAX with INT64_MAX. The former is defined on Linux.
r307387 by marcel: Switch to using the portable partition scheme headers.
r307544 by marcel: o Provide a private definition for UUIDs (mkimg_uuid_t) because UUIDs are not portable. o Move mkimg_uuid() to a new file and merge both gpt_uuid_enc() and vhd_uuid_enc() into a single mkimg_uuid_enc() that lives in the same file. o Move the OS-specific implementation of generating a UUID to osdep_uuidgen() and provide the implementations for FreeBSD, macOS and Linux. o Expect the partitioning scheme headers to be found by having a search to the directory in which the headers live. This avoids conflicts on non-FreeBSD machines.
r307550 by imp: Add a new flag to mkimg (-a num) to specify the active partition for those partitioning schemes that have this concept. Implement it as an override for mbr's setting 0x80 in the flags for the first partition when we have boot code.
Differential Revision: https://reviews.freebsd.org/D4403
r318137: mkimg: Add -C argument to specify maximum capacity
Add a -C option to specify a maximum capacity for the final image file. It is useful to control the size of the generated image for sdcard or when we will add dynamic size partition.
Add --capacity which is a shorthand to define min and max capacity at the same time.
Reviewed by: bapt, marcel, wblock (manpages) Sponsored by: Gandi.net Differential Revision: https://reviews.freebsd.org/D10509
r319125: mkimg: Correct an off by one error in the PMBR size
The PMBR last sector should be number of sector - 1 (As stated in UEFI Spec 2.6 page 118 table 17). This fixes warning printed by linux tools like parted or fdisk.
Sponsored by: Gandi.net
r319295 by ngie: Update the usr.bin/mkimg golden test output files after ^/head@r319125
^/head@r319125 changed the location of the backup pmbr, requiring the output files to be regenerated, since they're binary disk dumps.
The output files were regenerated with "make rebase"--fixed in ^/head@r319294.
MFC with: r319125, r319294 PR: 219673 Sponsored by: Dell EMC Isilon
|
#
302408 |
|
07-Jul-2016 |
gjb |
Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle. Prune svn:mergeinfo from the new branch, as nothing has been merged here.
Additional commits post-branch will follow.
Approved by: re (implicit) Sponsored by: The FreeBSD Foundation |
#
301999 |
|
17-Jun-2016 |
emaste |
mkimg: bump version to 20151211 after r292082
mkimg has had a number of functional additions after the last time the version was incremented. Do so now, to r292082's commit date, so that users can determine what is supported.
Reviewed by: marcel Approved by: re (gjb) Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D6882
|
#
284658 |
|
21-Jun-2015 |
marcel |
Microsoft Azure expects the creator OS to be "Wi2k" and not "FBSD". The image is not accepted for provisioning otherwise. Bump the VHD creator tool version and the version of mkimg to signify our success in provisioning.
Note that this also imapcts the dynamic VHD images.
Tested by: gjb@
|
#
279139 |
|
22-Feb-2015 |
marcel |
Add the -c option for specifying the capacity of the disk image. When a capcity is given, no partitions are required. When no partitions are given, no scheme needs to be specified either. This makes it possible to create an entirely empty disk image. To add an empty partitioning table, specify the scheme.
Bump the version to 20150222.
|
#
275721 |
|
12-Dec-2014 |
marcel |
The size of the first level reference count table is given in terms of the number of clusters it occupies. It's not the number of entries in the table, as it is for the L1 cluster table.
For small images, the two are the same. With the unit tests based on small images, this change has therefore no effect on the unit test. For larger images (like the FreeBSD 10.1-RELEASE image), this gives a discrepancy that actually shows up when running "qemu-img check".
Bump the version number of mkimg.
While here, fix a white-space bug.
MFC after: 1 week
|
#
275042 |
|
25-Nov-2014 |
bapt |
Convert to usr.bin/ to LIBADD Reduce overlinking
|
#
272485 |
|
03-Oct-2014 |
marcel |
Add mkimg_chs() for those schemes that need the LBA broken down into cylinder, head and track numbers. Return ~0U for these values when mkimg wasn't given both -T and -H (i.e. no geometry) or the cylinder would be larger than the provided maximum.
Use mkimgs_chs() for the EBR, MBR and PC98 schemes to fill in the appropriate fields. Make sure to use a "rounded" size so that the partition is always a multiple of the track size. We reserved the room for it in the metadata callback so that's a valid thing to do.
Bump the mkimg version number. While doing that again: have mkimg.o depend on the Makefile so that a version change triggers a rebuild as needed.
|
#
272384 |
|
01-Oct-2014 |
marcel |
Improve performance of mking(1) by keeping a list of "chunks" in memory, that keeps track of a particular region of the image. In particular the image_data() function needs to return to the caller whether a region contains data or is all zeroes. This required reading the region from the temporary file and comparing the bytes. When image_data() is used multiple times for the same region, this will get painful fast.
With a chunk describing a region of the image, we now also have a way to refer to the image provided on the command line. This means we don't need to copy the image into a temporary file. We just keep track of the file descriptor and offset within the source file on a per-chunk basis.
For streams (pipes, sockets, fifos, etc) we now use the temporary file as a swap file. We read from the input file and create a chunk of type "zeroes" for each sequence of zeroes that's a multiple of the sector size. Otherwise, we allocte from the swap file, mmap(2) it, read into the mmap(2)'d memory and create a chunk representing data.
For regular files, we use SEEK_HOLE and SEEK_DATA to handle sparse files eficiently and create a chunk of type zeroes for holes and a chunk of type data for data regions. For data regions, we still compare the bytes we read to handle differences between a file system's block size and our sector size.
After reading all files, image_write() is used by schemes to scribble in the reserved sectors. Since this never amounts to much, keep this data in memory in chunks of exactly 1 sector.
The output image is created by looking using the chunk list to find the data and write it out to the output file. For chunks of type "zeroes" we prefer to seek, but fall back to writing zeroes to handle pipes. For chunks of type "file" and "memoty" we simply write.
The net effect of this is that for reasonably large images the execution time drops from 1-2 minutes to 10-20 seconds. A typical speedup is about 5 to 8 times, depending on partition sizes, output format whether in input files are sparse or not.
Bump version to 20141001.
|
#
272217 |
|
27-Sep-2014 |
marcel |
Update the usage message and the man page to account for the new long options. Bump the version number to 20140927. While here, use explicit fputc() calls to skip a line in the output. This to avoid having to hunt for extra '\n' characters in the printf format strings.
MFC after: 1 week Relnotes: yes
|
#
272198 |
|
27-Sep-2014 |
marcel |
Add 3 long options for getting information about mkimg itself: --version print the version of mkimg and also whether it's 64- or 32-bit. --formats list the supported output formats separated by space. --schemes list the supported partitioning schemes separated by space.
Inspired by a patch from: gjb@
MFC after: 1 week Relnotes: yes
|
#
271965 |
|
22-Sep-2014 |
marcel |
Add support for QCOW version 1. Version 2 is partially implemented. And because of that, it's entirely disabled for now. Both versions are similar enough that a single header definition works for both of them. The only "diverting" side-effect is that the union of the two is larger than the official V1 header.
What this means for our V1 support is that we can't put the L1 table adjacent to the V1 header (i.e. at offset 0x30 in the file), unless we revert to hackery and klugery. Let's not. Instead, we align the L1 table at the cluster boundary. This is in line with the V2 layout and perfectly ok for V1 anyway (ok -- as far as I've seen so far). Due to the alignment, our V1 image seems to be 1 cluster larger than the V1 image created by qemu-img (on average).
Compression of the clusters is not supported at this time.
MFC after: 2 months
|
#
271911 |
|
20-Sep-2014 |
marcel |
Add unit tests for mkimg(1):
|
#
268236 |
|
03-Jul-2014 |
marcel |
Add VHD support to mkimg(1). VHD is used by Xen and Microsoft's Hyper-V among others.
Add an undocumented option for unit testing (-y). When given, the image will have UUIDs and timestamps synthesized in a way that gives identical results across runs. As such, UUIDs stop being unique, globally or otherwise.
VHD support requested by: gjb@
|
#
266176 |
|
15-May-2014 |
marcel |
MFuser/marcel/mkimg: Add support for different output formats: 1. The output file that was previously written is now called the raw format. 2. Add the vmdk output format to create VMDK images.
When the format is not given, the raw output format is assumed.
|
#
263918 |
|
29-Mar-2014 |
marcel |
Add mkimg, a utility for making disk images from raw partition contents. The partitioning scheme can be one of the schemes supported by gpart.
Reviewed by: sjg Obtained from: Juniper Networks, Inc.
|
#
263867 |
|
28-Mar-2014 |
marcel |
Make mkimg a general command and not a maintenance command.
|
#
263866 |
|
28-Mar-2014 |
marcel |
Define SPARSE_WRITE to enable writing sparse files. Output compares ok...
|
#
263409 |
|
19-Mar-2014 |
marcel |
Use linker sets to provide build-time selection of the schemes that need to be supported. Each scheme provides data and callbacks for handling the scheme-specifics. To this end, put each scheme in its own source file.
|
#
263381 |
|
19-Mar-2014 |
marcel |
Set BINDIR & WARNS while we're outside the source tree.
|
#
253923 |
|
04-Aug-2013 |
marcel |
Safe WIP: mkimg is a user-space utility for creating disk images. In its current form and shape, it creates images with the correct partition contents and overall image layout. What it doesn't do yet is actually scribble the metadata, such as partition tables and boot code. This is where I'd like to leverage code from other places, such as geom_part, and if possible/feasible. Also, the utility should be a little bit smarter about files and pipes so that we create temporary files only when needed (e.g. when we don't know the size of a partition's contents in advance while wrting the image to stdout).
|