Bump the UFS partition in the EC2 AMI up to 2.9 GB; this doesn't affect
the final image size (which still rounds up to 3 GB) but gives us enough
space to ensure that we don't run out while installing packages.

Direct commit to stable/11 (HEAD and stable/12 were expanded to 3.9 GB
last year).

Approved by: re (gjb)
Sponsored by: https://www.patreon.com/cperciva

Include the shells/bash port on Vagrant images, which prevents
a shell issue during startup.

Approved by: re (kib, insta-MFC)
PR: 245051
Sponsored by: Rubicon Communications, LLC (netgate.com)

MFC r348439:
Increase the size of the Vagrant default image size, as the 3GB
size is too small to bootstrap the firstboot_pkgs list.

While here, add the growfs(8) startup script to /etc/rc.conf,
as Vagrant images can be resized by modifying the Vagrantfile.

Approved by: re (kib)
Sponsored by: The FreeBSD Foundation

MFC r347056:
Increase the default size of the GCE disk image from 3GB to 20GB,
as 3GB is too small as discovered in last week's snapshot builds.

Sponsored by: The FreeBSD Foundation

MFC r346959:
Reduce the default image size for virtual machine disk images from
30GB to 3GB. The raw images can be resized using truncate(1), and
other formats can be resized with tools included in hypervisors.

Enable the growfs(8) rc(8) at firstboot if the disk was resized
prior to booting the virtual machine for the first time.

Sponsored by: The FreeBSD Foundation

Fix cross-building VMs with a non-/usr/src source directory.

The path /usr/src was hard-coded here, resulting in mkimg not being able
to find the bootfiles.

Direct commit to stable/11 because the code in HEAD and stable/12 is
completely different due to changes in OBJDIR layout.

MFC r345316, r345317, r345858:

r345316: Register ARM64 EC2 AMIs as being for the ARM64 architecture.

r345317: Don't install amazon-ssm-agent package into ARM64 AMIs.

r345858: Add support for cross-building cloudware images.

With these changes it is possible to build ARM64 EC2 AMIs on stable/12 --
said images do not *work* yet, however.

MFC r344315:
Fix sed script to insert Amazon NTP server into ntp.conf once rather
than twice.

Reminded by: rgrimes

MFC Loader Interpreter Coexistance: r337806, r337868, r337914, r337927,
r337989, r338052, r338064, r338407, r338418, r338474

r337806:
Create a loader for each interpreter for x86 BIOS and all EFI

Create loader_{4th,lua,simp}{,.efi}. All of these are installed by
default. Create LOADER_DEFAULT_INTERP to specify the default
interpreter when no other is specified. LOADER_INTERP is the current
interpreter language building. Turn building of lua on by default to
match 4th. simploader is a simplified loader build w/o any interpreter
language (but with a simple loader). This is the historic behavir you
got with WITHOUT_FORTH. Make a hard link to the default loader. This
has to be a hard link rather than the more desirable soft link because
older zfsboot blocks don't support symlinks.

r337868:
stand: Use -Oz/-Os for all loader/stand builds.

While we're not super size constrained, the x86 BIOS /boot/loader has
to be less than about 520k-530k to be reliable. The LUA loader is at
this size today. -Oz saves 15-20% on the size, keeping us safely small
enough (comparable to where we were with the 4th loader). This will
also help with sjg's work on bringing in bearssl, though we may again
be looking for space in the LUA loader.

Size table for clang 6.0.0:
default -O1 -Os -Oz
4th 442368 417792 389120 376832
lua 524288 479232 446464 430080

r337914:
Install links for loader.efi.

r337927:
Add ashldi3 and ashrdi3 to mips.

Now that we're using -Os, mips needs these routines.

r337989:
Copy the boot loader from the new location for the co-existing
loaders.

r338052:
libsa: Add lshrdi3.c for powerpc* and mips

r338064:
Turn back the clock just a little: make userboot.so always be 4th

Turns out there was a hidden dependency we hasn't counted upon. The
host load /boot/userboot.so to boot the VMs it runs. This means that
the change to lua meant suddently that nobody could run their older
VMs because LUA wasn't in 10.0, last month's HardenedBSD, 11.2 or
whatever. Even more than for the /boot/loader* binaries, we need a
good coexistance strategy for this. While that's being designed and
implemented, drop back to always 4th for userboot.so. This will fail
safe in all but the most extreme environments (but lua-only hacks
to .lua files won't be processes in VMs until we fix it).

r338407:
lualoader: Print error messages from command failures at the prompt

Previously lualoader would remain silent, rather than printing
command_errmsg or noting that a command had failed or was not found.

r338418:
userboot: handle guest interpreter mismatches more intelligently

The switch to lualoader creates a problem with userboot: the host is
inclined to build userboot with Lua, but the host userboot's interpreter
must match what's available on the guest. For almost all FreeBSD guests in
the wild, Lua is not yet available and a Lua-based userboot will fail.

This revision updates userboot protocol to version 5, which adds a
swap_interpreter callback to request a different interpreter, and tries to
determine the proper interpreter to be used based on how the guest
/boot/loader is compiled. This is still a bit of a guess, but it's likely
the best possible guess we can make in order to get it right. The
interpreter is now embedded in the resulting executable, so we can open
/boot/loader on the guest and hunt that down to derive the interpreter it
was built with.

Using -l with bhyveload will not allow an intepreter swap, even if the
loader specified happens to be a userboot with the wrong interpreter. We'll
simply complain about the mismatch and bail out.

For legacy guests without the interpreter marker, we assume they're 4th.
For new guests with the interpreter marker, we'll read it and swap over
to the proper interpreter if it doesn't match what the userboot we're using
was compiled with.

Both flavors of userboot are installed by default, userboot_4th.so and
userboot_lua.so. This fixes the build WITHOUT_FORTH as a coincidence, which
was broken by userboot being forced to 4th.

r338474:
Be a little conservative about when to force size optimizations.

Reports have come in that there's issue with powerpc and sparc64 since
we've switched to using -Oz / -Os. We don't strictly need them for
!x86, so be conservative about when we enable them.

MFC r340983:
Fix NTP query on GCE due to unresolved hostname.

PR: 232456
Submitted by: Lucas Kanashiro
Sponsored by: The FreeBSD Foundation

MFC r339684:
Reduce the GCE image size to 27G to be lower than the free
quota limit.

PR: 232313
Sponsored by: The FreeBSD Foundation

MFC r337717, r337718:

r337717:
Add lang/python2, lang/python3, and lang/python to GCE images
to help avoid hard-coding 'python<MAJOR>.<MINOR>' in several
scripts in the client-side scripts. [1]

r337718:
Add a space between a variable and escaped new line.

PR: 230248 [1]
Sponsored by: The FreeBSD Foundation

MFC r337555, r337556:
r337555:
Update and replace old rc daemons for GCE images.

r337556:
Invoke the growfs rc script for each boot on GCE.

PR: 229000, 230275
Sponsored by: The FreeBSD Foundation

As part of r336741, BOOTFILES needs special handling when
cross building on stable/11, where the path should be:

/usr/obj/TARGET.TARGET_ARCH/[...]

The change in r336741, which was a direct commit to stable/11
was not complete enough to evaluate where BOOTFILES should be.

This, as was r336741, is a direct commit to stable/11.

Tested with: stable/11 BANANAPI build
Sponsored by: The FreeBSD Foundation

Following r336726, explicitly invoke the 'obj' target when
setting BOOTFILES. On stable/11, without this change, the
.OBJDIR expands to /usr/src/stand instead /usr/obj/<foo>.

This is a piece of duct tape for now until I figure out why
the correct directory is not being located.

This is a direct commit to stable/11.

Sponsored by: The FreeBSD Foundation

MFC r336598-r336600, r336721

r336598:
release: Uniform release arm image size

Create a 50m fat16 for all boards.

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D16238

r336599:
release: Add arm_install_boot to install the commit boot bits

This reduce the per-board arm_install_uboot to just install u-boot.
While here remove the installation of rpi.dtb and rpi2.dtb as we load
them from the UFS partition via ubldr.

Reviewed by: gjb, imp (older version)
Differential Revision: https://reviews.freebsd.org/D16239

r336600:
release: Add rpi firmware dtb and mmc overlays for RPI-B and RPI2

This is needed with new u-boot that uses the rpi-firmware dtbs.

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D16240

r336721:
release: Add OL_DIR forgot in r336600

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D16439

Approved by: gjb (for insta-mfc of r336721)

MFC r333493:

Set kldxref_enable="YES" for ARM images. Without it, the images are missing
the /boot/kernel/linker.hints file, which breaks loading some of the modules
with dependencies, eg cfiscsi.ko.

This is a minimal fix for ARM images, in order to safely MFC it before
11.2-RELEASE. Afterwards, however, I believe we should actually just change
the default (as in, etc/defaults/rc.conf). The reason is that it's required
for every image that's being cross-built, as kldxref(1) cannot handle files
for non-native architectures. For the one that is not - amd64 - having it
on by default doesn't change anything - the script is noop if the linker.hints
already exists.

The long-term solution would be to rewrite kldxref(1) to handle other
architectures, and generate linker.hints at build time.

Approved by: re (marius@)
Sponsored by: DARPA, AFRL

MFC r333262, r333264:

r333262:
Ensure the ports and src trees are available on GCE images,
satisfying a requirement to allow FreeBSD to be considered
a top-tier supported OS in Google Compute Engine.

r333264:
Fix a typo.

Approved by: re (kib)
Sponsored by: The FreeBSD Foundation

MFC r332663: Move panic-related settings from sysctl.conf to loader.conf so
that they apply if an EC2 instance panics while booting.

Sponsored by: https://www.patreon.com/cperciva

MFC r331559:
Escape trailing newlines in a long variable list for consistency.

Sponsored by: The FreeBSD Foundation

MFC r331364:
Remove google_accounts_manager from VM_RC_LIST in the GCE configuration
file, no longer needed.

PR: 221714
Sponsored by: The FreeBSD Foundation

MFC r322794:
Use py-google-compute-engine instead for releasing Google Compute
Engine (GCE) images with an updated version of Google's tools.

PR: 221714
Sponsored by: The FreeBSD Foundation

MFC r325834,r325997,326502: Move sys/boot to stand/

This is effectively a direct commit to stable/11, due to differences between
stable/11 and head. Changes to DTS in sys/boot/fdt/dts were often
accompanied by kernel changes. Many of these were also risc-v updates that
likely had many more dependencies to MFC.

Because of this, sys/boot/fdt/dts remains as-is while everything else in
sys/boot relocates to stand/.

r325834: Move sys/boot to stand. Fix all references to new location

r325997: Remove empty directories.

r326502: Document the sys/boot -> stand move in hier.7 and the top-level README.

MFC r326565: Make EC2 instances use Amazon's NTP service for time
synchronization.

Relnotes: EC2 instances now keep their clocks synchronized using
the Amazon Time Sync Service (aka. NTP).

MFC r326068:
Remove /etc/resolv.conf from virtual machine images, which is
copied from the build host. It is renamed to /etc/resolv.conf.bak
on boot, so never used anyway.

Sponsored by: The FreeBSD Foundation

MFC r320252, r320686, r325769:
r320252:
In release/release.sh:
- Rename chroot_arm_armv6_build_release() to chroot_arm_build_release()
and make it hardware agnostic (such as armv6 -vs- armv7 -vs- arm64).
- Evaluate EMBEDDED_TARGET differently so release/tools/arm.subr can
be used for arm/armv6 and arm64/aarch64.
- Update comments and copyright.

In release/tools/arm.subr:
- In arm_create_disk(), change the default alignment from 63 to 512k,
fixing a boot issue on arm64 and EFI. [1]
- Update comments and copyright.

r320686:
Fix the ftp-stage target for RPI3 images by loosening the
constraints on the TARGET and TARGET_ARCH variables.

r325769:
Update the GUMSTIX image build to use arm/arm TARGET/TARGET_ARCH.
Update the TARGET/TARGET_ARCH matching in release/release.sh and
release/Makefile.mirrors for simplification.

Note: The RPI3.conf addition from r320252 is not included, as the
11-STABLE image fails to boot in my testing.

Sponsored by: The FreeBSD Foundation

MFC r325156:
Set a default hostname for virtual machine images.

Sponsored by: The FreeBSD Foundation

MFC r323924:
Revert r323812 from release/tools/arm.subr, which has broken the
build on arm/armv6 images.

Pointyhat: gjb (myself)
MFC note: releng/10.4 has broken because of this
Sponsored by: The FreeBSD Foundation

MFC r323812:
Bootstrap etcupdate(8) and mergemaster(8) databases when creating
virtual machine images and embedded images, similar to what is
done when extracting base.txz to the target root filesystem in
a new installation.

Approved by: re (marius, insta-MFC)
Sponsored by: The FreeBSD Foundation

MFC r318894: Enable Amazon's "Elastic Network Adapter" in EC2 AMIs.

MFC r318872:
Enable DHCP and IPv6 autoconfig on non-cloud VM images.

PR: 203653
Sponsored by: The FreeBSD Foundation

MFC r314935 (thompsa):
Change ec2.conf to use the pkg tool from a chroot rather than trying
to bootstrap it and fail from the livecd readonly filesystem.

Sponsored by: The FreeBSD Foundation

MFC r307008:

Add the ability to override the size of the swap partition when building
VM images. The default continues to be 1G.

Sponsored by: Chelsio Communications

MFC r317169:
Trim trailing '/release/..' when setting _OBJDIR so arm64/aarch64
boot1.efifat is properly located when creating virtual machine images.

Note, the underlying issue has no direct impact against stable/10,
as arm64/aarch64 virtual machine images are not created for 10-STABLE.

Sponsored by: The FreeBSD Foundation

MFC r314561:
Increase the EC2 image size. The recent snapshot builds of EC2
images for 12-CURRENT (and now 11-STABLE) failed due to a full
filesystem on the md(4) device during creation.

Sponsored by: The FreeBSD Foundation

MFC 308106:
- Use virtualbox-ose-additions-nox11 for vagrant image to reduce size

MFC r312214: Enable IPv6 on EC2 AMIs.

MFC r302929: Now that potentially buggy versions of Xen are automatically
detected (see r302635, MFCed as r302895), there is no need to force msix
interrupt migration off via loader.conf.

Reverts: r302184
Approved by: re (gjb)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Enable indirect segment I/Os by default in the Xen blkfront driver when
running on EC2. Due to improvements in EC2, the performance penalty which
was present on some EC2 instances no longer exists, and enabling this
feature now consistently yields ~20% higher throughput with equal or lower
latency.

Reverts: r286063
Approved by: re (gjb)
MFC after: 2 weeks
Relnotes: Improved disk throughput on EC2

Turn off MSI-X interrupt migration by default in EC2 instances; this works
around a bug in older versions of Xen and unbreaks SR-IOV (aka. "EC2
Enhanced Networking").

Approved by: re (gjb)
Thanks to: jhb, Jeremiah Lott

Bump /tmp from 30m to 50m for ARM release images.
30m isn't enough for pkg anymore to extract packagesite.txz.
40m is fine for now but let's take a safer way as we don't know when pkg will need more.

Reported by: many
Approved by: re (gjb), andrew (mentor)

Switch from console="comconsole" to boot_multicons="YES" in EC2.

Amazon recently introduced an API for capturing screenshots of an
emulated VGA device; this commit makes that (somewhat) useful.

MFC after: 3 weeks

Turn off nfsv4acls for arm/armv6 builds, which has an impact
on performance, especially with SD cards on certain SoCs.

Requested by: trasz
Discussed with: ian, kientzle

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Fix tmpfs(5) '/tmp' mount point.

A stray trailing space snuck in with one of the recent
changes, making r290550 and r290573 effectively no-op.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Remove noatime from tmpfs(5) fstab(5) entry.

Submitted by: Howard Su
MFC after: 3 days
X-MFC-With: r290550
Sponsored by: The FreeBSD Foundation

Use tmpfs(5) instead of md(4) for '/tmp' mount.

Submitted by: Nikolai Lifanov
Differential Revision: D3506
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

In vm_copy_base(), turn off SU+J on the resultant filesystem,
leaving only SU enabled.

Discussed with: kib (a few weeks ago)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Disable blkif indirect segment I/Os in EC2 by default due to performance
issues on some EC2 instance types. Users may want to experiment with
removing this from loader.conf and measuring the performance impact on
the EC2 instances they are using.

Fix an out-of-order execution issue regarding pkg(8):
- pkg(8) cannot be removed before subsequent reinvocations
- The PKG_CACHEDIR cannot be cleaned after the repo*.sqlite
has been removed
- pkg(8) cannot be removed as a precursor to any of the other
steps involved here

MFC after: 3 days
X-MFC-With: r285722
X-MFC-Before: 10.2-{BETA3,RC1} (whichever happens next)
Sponsored by: The FreeBSD Foundation

Add support for building VirtualBox Vagrant images.
Abstract the build, package and upload to handle building either type.

Approved by: re (gjb)

Do not mount /var/log and /var/tmp as md(4)-backed devices.

Discussed with: ian
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Revert r285018, r285019, r285076, r285078, r285082.

The true cause of the missing UFS/MSDOSFS labels has been
identified, and only affects stable/10 at the moment.

An request for commit to stable/10 will be pending RE approval
after this commit.

MFC after: 1 day
X-MFC-Note: never
X-MFC-Never: r285018, r285019, r285076, r285078, r285082
Sponsored by: The FreeBSD Foundation

Test for loader.rc existence before attempting to create
the symlink from loader.rc.sample.

Fix paths relative to the CHROOTDIR.

MFC after: 3 days
X-MFC-With: r285076, r285078
X-MFC-Before: 10.2-BETA1
Sponsored by: The FreeBSD Foundation

Revert r285018 and r285019, which attempted to resolve
UFS/MSDOSFS label issues on FreeBSD/arm builds, however
the real problem was addressed in r285076, which is due
to two separate issues, unrelated to md(4) stale device
existence.

MFC after: 3 days
X-MFC-With: r285076
X-MFC-Before: 10.2-BETA1
Sponsored by: The FreeBSD Foundation

Fix the real problem with missing UFS/MSDOSFS labels on the
FreeBSD/arm builds. The problem stems from the loader.rc file
not existing, as well as geom_label not being loaded at boot.

For now, add the geom_label_load entry to loader.conf, and
symlink loader.rc.sample to loader.rc, both of which allowed
my BeagleBone Black to boot fine with a UFS label reference in
fstab(5).

MFC after: 3 days
X-MFC-Before: 10.2-BETA1
Sponsored by: The FreeBSD Foundation

Fix the path to the UFS/MSDOSFS filesystem labels, since they
cannot possibly exist within the chroot(8) before the target
filesystem actually exists.

MFC after: 3 days
X-MFC-With: r285018
Sponsored by: The FreeBSD Foundation

Implement an evil workaround that prevents UFS/MSDOS labels from being
written to disk with newfs(8) and newfs_msdosfs(8).

When iterating through snapshot builds in serial, it is possible for
a build failure to leave stale md(4) devices behind, in some cases, they
could have a UFS or MSDOS filesystem label assigned.

If the md(4) is not destroyed (or not able to be destroyed, as has
happened recently due to my own fault), the filesystem label that
already exists can interfere with a new md(4) device that is targeted to
have the same label.

This behavior, although admittedly a logic error in the wrapper build
scripts, has caused intermittent reports (in particular with the armv6
builds) of missing UFS/MSDOSFS labels, causing the image to fallback to
the mountroot prompt. This appears to only happen when the backing
md(4) device is destroyed before the calling umount(8) on the target
mount, after which the UFS/MSDOSFS label persists.

The workaround is this: If EVERYTHINGISFINE is set to non-empty value,
check for an existing ufs/rootfs and msdosfs/MSDOSBOOT filesystem label
in arm_create_disk(), and rm(1) them if they exist.

The EVERYTHINGISFINE variable is chosen because it is used in exactly
one other place - release/Makefile.mirrors - and there are big scary
warnings at the top of that file as well that it should *not* be used
under normal circumstances. This should not destroy a build machine
that also uses '/dev/ufs/rootfs' as the UFS label, and I have verified
in extensive local testing that the destroyed label is recreated when
the md(4) is unmounted/mounted, but this really should not be enabled
by anyone.

Having said all that, I absolutely *do* plan MFC this to stable/10 for
the 10.2-RELEASE cycle, as so far, I have only observed this behavior
on stable/10, but this is a temporary solution until I can unravel all
of the failure paths to properly trap them.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Add initial support for building Vagrant images for VMWare. Next steps will be adding Virtualbox support and uploading to Hashicorp Atlas for others to consume.

Approved by: re (gjb)

Add initial support for automatically uploading GCE virtual
machine images to the Google Compute Engine platform.

By default, gcutil/gsutil requires an Oauth2 login generated
from a URL that must be opened in a browser, a verification
code copied back to the terminal from which it was invoked,
etc., etc., making it near impossible for automation.

I've hacked together an evil solution to work around this,
so unless GCE_LOGIN_SKIP is set to a non-empty value, this
Makefile will not do anything useful.

As a result of this commit, remove the gce-package.sh script
that was never, nor will ever be, used.

MFC after: 3 days
X-MFC-Note: (hopefully)
Sponsored by: The FreeBSD Foundation

Ensure ASSUME_ALWAYS_YES is set when deleting pkg(8) from the
DESTDIR, otherwise it can get stuck in ttyin.

This means no 10-STABLE EC2 images this week.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Chase r284656 and r284658:

Remove the Azure-local vm_extra_create_disk(), since we no longer
need qemu-img to convert the final VHD image to an Azure-compatible
format.

Although the waagent utility is installed from ports, create the
symlink to /usr/sbin, pending investigation on where this is
hard-coded, so it can be reported upstream. In the meantime, this
is good enough.

MFC after: 3 days
X-MFC-Needs: r284269, r284270, r284271, r284655,
r284656, r284657, r284658, r284659
X-MFC-Note: Required for 10.2-RELEASE, marcel@ has
implicit approval for the required changes
Sponsored by: The FreeBSD Foundation

Fix path to waagent.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Work around a potential bug in pw(8) when '-m' is specified.
According to the manual page, '-m' should create the user home
directory, however rigorous testing suggests it does not, and
it is unclear if this is an implementation or expectation issue.

Sponsored by: The FreeBSD Foundation

Make sure /home exists within the DESTDIR before creating the
'freebsd' user account.

Sponsored by: The FreeBSD Foundation

Use the new pw(8) '-R' option to create the users/groups.

MFC after: 1 week
X-MFC-after: r283961
Sponsored by: The FreeBSD Foundation

Pass the '-b' argument to pw(8) so the '/home' directory is created
in the correct place.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

In arm_create_disk(), disable soft updates journaling.

Disabling soft updates journaling appears to resolve issues
with kernel panics, and may also be generally bad to have
enabled for SD cards.

Requested by: ian
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Re-enable arm_create_user(), and pass the '-V DESTDIR/etc' to
pw(8) to set the correct /etc directory for the user/group
files.

Provided by: ian (thanks!)
MFC after: 3 days
X-MFC-with: r283894
Sponsored by: The FreeBSD Foundation

Disable arm_create_user(). As-is, this does not create the
user in the userland for the target image, but creates the
user in the build chroot.

Before this is re-enabled, I want to figure out a clean way
to do this without requiring the overhead of third-party
utilities (such as qemu).

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Fix virtual machine disk format creating by passing VMFORMAT
to mkimg(1)

PR: 200068
Submitted by: Jeremy Norris
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Merge ^/projects/release-arm-redux into ^/head.

Of note:

- This commit adds native FreeBSD/arm release build support without
requiring out-of-tree utilities.

- Part of this merge removes the WANDBOARD-{SOLO,DUAL,QUAD} kernel
configuration files, for which the IMX6 kernel configuration file
should be used instead.

- The resulting images have a 'freebsd' user (password 'freebsd'),
to allow ssh(1) access when console access is not available (VGA
or serial). The default 'root' user password is set to 'root'.

- The /etc/ttys file for arm images now enable both ttyv0 and ttyu0
by default.

Help from: many (boot testing, feedback, etc.)
Sponsored by: The FreeBSD Foundation

Fix building BEAGLEBONE images with Crochet using the
sysutils/u-boot-beaglebone port:

- In arm/BEAGLEBONE.conf, set EMBEDDEDPORTS to the
sysutils/u-boot-beaglebone port.
- In arm/release.sh, remove BEAGLEBONE from setting WANT_UBOOT
- In tools/arm/crochet-BEAGLEBONE.conf, override the
beaglebone_check_uboot(), and set BEAGLEBONE_UBOOT to
/tmp/external/u-boot-beaglebone, and create symlinks to the
u-boot files in /usr/local/share/u-boot-beaglebone and the
uEnv.txt file in crochet/board/Beaglebone/files.

Sponsored by: The FreeBSD Foundation

Add initial support for building RPI2 images.

In release.sh, allow overriding buildenv_setup() before
the handoff to arm/release.sh.

Copy arm/RPI-B.conf -> arm/RPI2.conf, set UBOOT_PORT and
the correct KERNEL, and add the buildenv_setup() override
to install the sysutils/u-boot-rpi2 port/package.

Copy tools/arm/crochet-RPI-B.conf -> tools/arm/crochet-RPI2.conf,
and set the correct entries for the RaspberryPi2 board.

Thanks to: loos@
Sponsored by: The FreeBSD Foundation

Fix a comment.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Create the arm64/aarch64 VM disk image as MBR instead of
GPT scheme. UEFI needs to know the unique partition GUID
with GPT, which changes each time. Specifically, the QEMU
EFI BIOS file has this hard-coded.[1]

Since the GPT labels are now unavailable, unconditionally
label the root filesystem as 'rootfs' with newfs(8), since
it does not hurt anything anywhere else. For the arm64 case,
'/' is mounted from /dev/ufs/rootfs; for all other VM images,
'/' is mounted from /dev/gpt/rootfs.

Unfortunately, since the /dev/gpt/swapfs label is also lost,
set NOSWAP=1 for the arm64/aarch64 images. This is temporary,
until I figure out a scalable solution to this. But, a certain
piece of softare was written "very fast", and ended up living
for 15 years. We can deal with this for a week or so.

Information from: andrew, emaste [1]
Sponsored by: The FreeBSD Foundation

Revert r281809, which did more harm than good, and apply
a more proper fix when attempting to locate the /boot
files.

Sponsored by: The FreeBSD Foundation

Simplify variable expansion in attempt to fix the vm-image build.

Sponsored by: The FreeBSD Foundation

Merge the following from ^/projects/release-arm64 to allow
building FreeBSD/arm64 VM images and memstick.img installation
medium:
r281786, r281788, r281792:

r281786:
Add support for building arm64/aarch64 virtual machine images.

r281788:
Copy amd64/make-memstick.sh to arm64/make-memstick.sh for
aarch64 memory stick images.

Although arm64 does not yet have USB support, the memstick
image should be bootable with certain virtualization tools,
such as qemu.

r281792:

Add a buildenv_setup() prototype, intended to be overridden as
needed.

For example, the arm64/aarch64 build needs devel/aarch64-binutils,
so buildenv_setup() in the release.conf for this architecture
handles the installation of the port before buildworld/buildkernel.

Sponsored by: The FreeBSD Foundation

When building VM disk images, vm_copy_base() uses tar(1) to
copy the userland from one md(4)-mounted filesystem to a clean
filesystem to prevent remnants of files that were added and
removed from resulting in an unclean filesystem. When newfs(8)
creates the first filesystem with journaled soft-updates enabled,
the /.sujournal file in the new filesystem cannot be overwritten
by the /.sujournal in the original filesystem.

To avoid this particular error case, do not enable journaled
soft-updates when creating the md(4)-backed filesystems, and
instead use tunefs(8) to enable journaled soft-updates after
the new filesystem is populated in vm_copy_base().

While here, fix a long standing bug where the build environment
/boot files were used by mkimg(1) when creating the VM disk
images by using the files in .OBJDIR.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

It would help if I committed the right patch...

Partially revert r278118 now that the required logic for deciding whether
freebsd-update can be useful has moved into the firstboot_freebsd_update
script.

Add bits for building EC2 disk images. Make logic for uploading these
to create EC2 AMIs will come in a later commit.

Sigh. s/AutoSize/Growfs/ following upstream commit r761.

MFH: 3 days
Sponsored by: The FreeBSD Foundation

Improve check for whether ${DESTDIR}/dev is mounted.

Submitted by: gcooper

Clean up filesystem unmounting in vmimage builds:
* Remove vm_umount_base function which is currently unused.
* Add umount_loop function which loops attempting to unmount one filesystem.
* Replace calls to umount with calls to umount_loop.
* Don't attempt to unmount ${DESTDIR}/dev if it isn't mounted.

The looping is necessary because sometimes umount fails due to filesystems
being busy. The most common cause of such busyness is periodic(8) jobs
running `find / ...`.

Reviewed by: gjb

When creating VM images, copy the contents of the created filesystem into
a new filesystem before packaging it into a disk image. This prevents
"remnants" of deleted files from showing up in the VM images, and reduces
their compressed size (by about 10% for the cloudware images) as a result.

Looks good to: gjb

Add a CLOUDPACKAGE variable to contain a list of targets for
post-install packaging cloud provider images.

Add a 'gce-package.sh' script to generate the final output
image ready for upload to the GCE platform. Right now, this
is the only image that has a specific output format (GNU-tar),
and this implementation is expected to be temporary.

This is not directly connected to the other release targets.

MFC after: 1 week
X-MFC-with: r279249, r279250
Sponsored by: The FreeBSD Foundation

Use ifconfig_DEFAULT="" instead of hard-coding vtnet(4)

MFC after: 2 weeks
X-MFC-with: r277458, r277536, r277606, r277609,
r277836, r278118
Sponsored by: The FreeBSD Foundation

Move service_enable="YES" rc.conf(5) additions to
VM_RC_LIST and deduplicate.

Evaluate if firstboot_freebsd_update should be enabled
based on UNAME_r, because it is not supported for the
-CURRENT or -STABLE branches.

MFC after: 2 weeks
X-MFC-with: r277458, r277536, r277606, r277609,
r277836
Sponsored by: The FreeBSD Foundation

Merge the following revisions from ^/projects/release-vmimage:
r273823-r273826, r273833, r273836, r273944, r274069-r274071,
r274134, r274211, r274280-r274285, r274287-r274288, r274292,
r274296-r274297, r274356, r274533, r274725, r274726, r274729,
r274734, r274771, r274945-r274946, r277180, r277183-r277184,
r277186-r277187, r277250-r277253, r277263-r277264, r277383-r277384,
r277393-r277395, r277438-r277439, r277447, r277455:

r273823:
Move virtual machine / cloud provider targets and
options from release/Makefile to their own Makefile.

r273824:
Add glue to allow enabling building cloud provider VM images
by default.

When WITH_CLOUDWARE is not empty, add CLOUDTARGETS to the
release/Makefile 'release' target.

r273825:
Avoid hard-coding the Azure image file format. While here,
avoid using OSRELEASE for the output file name.

r273826:
Remove a few vestiges of passing an exit code to panic().

r273833:
Initial commit providing a mechanism to create openstack images
as part of the release build.

r273836:
Fix output file name for openstack images. No further conversion
is necessary for this VM file target, so there is no need to append
the '.raw' suffix here.

r273944:
Uncomment the cloudinit rc.conf(5) line.

r274069:
Add line continuation so OPENSTACKCONF is actually included in the env(1).

r274070:
Add a 'vm-cloudware' target, used to drive all targets in CLOUDTARGETS.

r274071:
Add examples for WITH_CLOUDWARE to release.conf.sample.
Add WITH_CLOUDWARE evaluation to RELEASE_RMAKEFLAGS.

r274134:
Initial rewrite to consolidate VM image build scripts into one.

r274211:
Add write_partition_layout() used to populate the final image.

Fix duplicated mkimg(1) call in vm_create_disk().

Add primitive (untested) PowerPC/PowerPC64 VM image support.

Note: As it is currently written, the /boot/pmbr and
/boot/{gptboot,boot1.hfs} use the build host and not the target
build. Fixing this is likely going to be a hack in itself.

r274280:
Return if vm_create_disk() is unsuccessful.

r274281:
Add CLEANFILES entry for VM targets

r274282:
Add vm_extra_pre_umount() prototype to vmimage.subr.

r274283:
Fix DESTDIR for installworld, and make sure it is created before use.

r274284:
Move usage() from vmimage.subr to mk-vmimage.sh, in case vmimage.subr
has not been sourced.

r274285:
Spell 'OPTARG' correctly. Actually call vm_create_base().

r274287:
Fix line continuation in write_partition_layout().
Remove variable test that is no longer needed.

r274288:
Fix scheme flag to mkimg(1).

r274292:
mount(8) and umount(8) devfs(5) as needed.

r274296:
Change path for mk-vmimage.sh from ${TARGET}/ to scripts/ now that
it is consolidated into one file.

Fix paths for the base image and output disk image files.

r274297:
Call cleanup() after everything is done.

r274356:
Remove a stray directory from CLEANFILES.

r274533:
Set the boot partition type to 'apple-boot' for powerpc.

r274725:
In vm_install_base(), copy the host resolv.conf into
the build chroot before attempting to do anything that
requires working DNS (i.e., pkg bootstrap).

In vm_extra_pre_umount(), remove the resolv.conf before
the disk image is unmounted from the backing md(4).

r274726 (cperciva):
Silence errors when umounting the chroot's /dev, since it
probably doesn't exist when we're running this.

Unmount filesystems before attempting to destroy the md which
holds them.

r274729 (cperciva):
Unmount filesystem and destroy md before we read the vnode from
disk and package it into a disk image. Otherwise we end up
packaging an unclean filesystem.

r274734 (cperciva):
Merge duplicative vm-CLOUDTYPE targets before additional duplication
gets added by the impending arrival of ec2 and gcloud.

r274771 (cperciva):
Add NOSWAP option which can be set by a vmimage.conf file to specify
that no swap space should be created in the image. This will be used
by EC2 builds, since FreeBSD/EC2 allocates swap space on "ephemeral"
disks which are physically attached to the Xen host node.

r274945:
In vm_extra_install_packages(), only bootstrap pkg(8) if
VM_EXTRA_PACKAGES is empty.

In vm_extra_pre_umount(), cleanup downloaded packages if pkg(8) was
bootstrapped earlier.

r274946:
Fix indentation nit.

r277180:
In vm_extra_install_base(), do not install waagent in the openstack
image, because it is not used. This appears to be a copy mistake.

Remove vm_extra_install_base() from the openstack.conf entirely,
since it does not need to be overridden.

r277183:
Enable the textmode console by default for VM images, since there is
no way to tell if the environment will be able to use the
graphics-mode console.

r277184:
Enable password-less sudo for openstack images.

r277186:
Update the VM_EXTRA_PACKAGES list for the openstack images.

The documentation suggests doing a "just fetch this and run it"-style
bootstrap, from which the list of dependencies was obtained (in
github, at: pellaeon/bsd-cloudinit-installer)

There is one Python dependency unmet, oslo.config, which is not in
the Ports Collection.

r277187:
Add a comment to note that setting hw.vga.textmode=1 is temporary.

r277250:
Remove vm_extra_install_base() for the Azure image, now that the
waagent exists in the ports tree.

Add sysutils/azure-agent to the VM_EXTRA_PACKAGES list.

In vm_extra_pre_umount(), remove the explicit pkg(8) install
list, as dependencies are resolved by sysutils/azure-agent.

r277251:
Add a 'list-cloudware' target to print the list of supported CLOUDWARE
values and a description.

Add the AZURE_DESC and OPENSTACK_DESC descriptions.

r277252:
Update release(7)

r277253:
Add 'list-vmtargets' target, which produces a list of all supported
VM and cloud provider images.

Add VHD_DESC, VMDK_DESC, QCOW2_DESC, RAW_DESC image descriptions.

Format the output to make a bit more readable.

Update release(7) to document the list-vmtargets target.

r277263:
Add initial support for the GCE (Google Compute Engine) cloud hosting
provider image.

r277264:
Style and line length cleanup.

r277383:
Remove the console setting from rc.conf(5), which is not used there.
While here, set console to include vidconsole in the loader.conf(5).

r277384:
Fix an indentation nit.
No functional changes.

r277393:
Remove the pkg-clean(8) call from vm_extra_pre_umount() since the
function is often overridden.

Add vm_extra_pkg_rmcache() to call pkg-clean(8) to avoid duplicated
code.

r277394:
Move resolv.conf(5) removal back to vm_extra_pre_umount() where it
belongs.

The GCE image needs resolv.conf(5) to exist (created as part of the
image setup), so it cannot be removed.

r277395:
Comment the line that configures ttys(5) to 'off', which makes it
impossible to test that the image boots.

Add a note explaining why the line is commented, and not (yet) removed
entirely.

r277438:
Move the 'install' bits that are specific to virtual machine images
from the Makefile to Makefile.vm.

Rename the 'install' target to 'release-install', and add a new
'vm-install' target.

Add a new 'install' target that invokes the new targets.

r277439:
Add WITH_CLOUDWARE to the list of make(1) variables for the release
build.

r277447:
Remove hw.vga.textmode=1 from the VM image loader.conf, which was
included during test builds and not intended to be included when
merging this project branch back to head.

r277455:
Remove mk-azure.sh, which is no longer needed.

MFC after: 1 month
X-MFC-To: stable/10 (requires mkimg(1))
Help from: cperciva, swills
Relnotes: yes
Sponsored by: The FreeBSD Foundation

Merge the following from ^/projects/release-vmimage:
r273076, r273077, r273079, r273095:

r273076:
Add a separate make(1) target to release/Makefile to
build FreeBSD virtual machine disk images for use on
the Microsoft Azure service.

For now, this target is not directly connected to the
build, however can be manually invoked.

The 'vm-azure' target invokes {amd64,i386}/mk-azure.sh,
which does the heavy lifting to produce proper VHDs.
mk-azure.sh uses a configuration file, defaulting to
tools/azure.conf if otherwise unset.

r273077:
Clear VM_RC_LIST.

r273079:
Fix signal list to trigger umount(8).

r273095:
Output an informational message when mkimg(1) runs, so it
does not appear that the process has stopped while waiting
for a 'y/n' response when waagent is deprovisioned.

Tested on: releng/10.1@r272876
MFC after: 3 days
X-MFC-10.1: yes
Sponsored by: The FreeBSD Foundation

Merge r262907 from ^/projects/release-embedded:
Add ZEDBOARD support for release builds.

Sponsored by: The FreeBSD Foundation

Update board definition for WANDBOARD-QUAD to match crochet.

Sponsored by: The FreeBSD Foundation

Merge ^/projects/release-embedded into ^/head.

After several months of testing and fixing (and breaking)
various parts of release/release.sh changes, it is now
possible to build FreeBSD/arm images as part of the release
process.

When EMBEDDEDBUILD is set in the release.conf file, release.sh
will create the build environment, then run a separate script
in release/${XDEV}/release.sh [1]. Currently, only arm is
supported.

The release/${XDEV}/release.sh configures the build environment
specific for the target image, such as installing gcc(1),
installing additional third-party software from the ports tree,
and fetching external sources.

Once the build environment is set up, release/${XDEV}/release.sh
runs Crochet, written by Tim Kientzle, which builds the userland
and kernel, and creates an image that can be written to an SD
card with dd(1). Many thanks to Tim for his work on Crochet.

Sample configurations for FreeBSD/arm boards are in the
release/arm/ directory, and Crochet configuration files for each
board are located in release/tools/arm/. Supported boards at this
time are: BEAGLEBONE, PANDABOARD, RPI-B, and WANDBOARD-QUAD.

Adding support for additional boards will continue in the
projects/release-embedded/ branch, and incrementally merged back
to head/.

Many thanks to the FreeBSD Foundation for the support and
sponsorship of this project.

[1] XDEV is used in order to keep the various configurations
organized by architecture, but since TARGET and TARGET_ARCH
are used to build the chroot, the values of those variables
cannot be used.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Merge the following:
^/user/gjb/hacking/release-embedded:
259994,260000,260895-260896,261139,261152,
261174,261176,261210,261221,261237,261239,
261448,261489
^/user/gjb/release-embedded:
262305,262307

svn:mergeinfo is intentionally not included in the commit, so it does
not propagate to head/.

Sponsored by: The FreeBSD Foundation

Move tools/release/${XDEV}/ to release/tools/${XDEV}/.

In general, the workflow used for the latest releases has been
to checkout head/release/ to a directory outside of any build
environments, modify the release.conf accordingly, and run:

# ./release.sh -c ./release.conf

By moving tools/release/${XDEV}/ (for RPI-B build testing, in
this case), any dependency on a different branch of the source
tree can be removed. In particular, release.sh expects the
tools/release/${XDEV}/crochet-${KERNEL}.conf file to exist, so
by moving the crochet configuration file to a directory in
release/, it becomes possible to build images for branches that
do not yet have the necessary configuration files and/or scripts.

Sponsored by: The FreeBSD Foundation