Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

release: Support GNU stat in mkisoimages.sh

BSD stat and GNU stat differ significantly when it comes to using a
custom format string, both in the option name and in the format string
itself. Handle both here (assuming Linux means GNU stat rather than BSD
stat).

Reviewed by: brooks, gjb
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D35814

release: Add support for building on non-FreeBSD

This requires two sets of changes. Firstly, for non-FreeBSD, we do not
know where tools are in PATH (and it is likely that some are not in
system directories and have been built as bootstrap tools during the
build), so we should leave PATH alone and trust the user. Secondly,
makefs needs a master.passwd for building images from a METALOG file, so
pass the directory in the image tree to makefs's -N option in order to
pick up a valid FreeBSD master.passwd; this is unnecessary on FreeBSD
(except in the edge case of building an image that refers to users or
groups not present in the host's database, which is unlikely but
technically possible) but harmless so can be done unconditionally.

Reviewed by: brooks, emaste, gjb
Differential Revision: https://reviews.freebsd.org/D34001

release: Support -DNO_ROOT image building

This requires a bunch of METALOG mangling to include the files we inject
into the tree. The mkisoimages.sh and make-memstick.sh scripts are now
called with the current directory inside the tree so that the relative
paths in the METALOG match up with the current directory. The scripts do
not require this when not using a METALOG, but for simplicity we always
do so. The Makefile mangles the real METALOG created from the install,
as those files are shared across all uses of the tree, but the shell
scripts create a temporary copy of the METALOG that they mangle as their
tree modifications are specific to that image. We also need to pass -D
to makefs to turn any duplicate METALOG entry errors into warnings, as
we have many (harmless) instances of those.

Whilst dvd1.iso should work, the !NOPKG code will need more work to
support this.

All media will also lack mergemaster and etcupdate trees, since more
work is needed to add -DNO_ROOT modes to them. Users of install media
built this way will have to manually bootstrap them.

Reviewed by: brooks, gjb
Differential Revision: https://reviews.freebsd.org/D33999

mkisoimages.sh: Avoid creating temporary files in the current directory

Currently the current directory is the parent of the rootfs directory,
but this will change in order to support NO_ROOT builds that use a
metalog manifest, since those need to have the current directory be the
rootfs itself in order for the relative paths to be correct, and we do
not want the non-METALOG case (which passes the directory to makefs) to
pick up leftover temporary .img files from a previous failed build.

Reviewed by: brooks, emaste, gjb
Differential Revision: https://reviews.freebsd.org/D33998

release: amd64: Fix ISO/USB hybrid image

Recent mkimg changes forces to have partitions given in explicit order.
This is so we can have the first partition starting at a specific offset
and the next ones starting after without having to specify an offset.
Switch the partition in the mkisoimage.sh script so the first one created
is the isoboot one.

PR: 254490
Reported by: Michael Dexter <editor@callfortesting.org
Tested by: Vincent Milum Jr <freebsd@darkain.com>
MFC after: Right now

Bump the ISO EFI partition size from 1024 to 2048, following r366732.

Suggested by: imp
Sponsored by: Rubicon Communications, LLC (netgate.com)

Increase the amd64 ISO ESP file size from 800KB to 1024KB.

At some poing over the last week, the bootx64.efi file has grown
past the 800KB threshold, resulting in being unable to copy it to
the EFI/BOOT directory.

# stat -f %z efiboot.znWo7m
819200
# stat -f %z stand-test.PIEugN/EFI/BOOT/bootx64.efi
842752

The comment in the script that creates the ISOs suggests that 800KB
is the maximum allowed for the boot code, however I was able to
boot an ISO with a 1024KB boot partition. Additionally, I verified
against an ISO from OtherOS, where the boot EFI partition is 2.4MB.

Sponsored by: Rubicon Communications, LLC (netgate.com)

Use makefs -t msdos in make_esp_file

With this last piece in place, make -C /usr/src/release release.iso is
finally able to run in a jail. This was not possible before because
msdosfs cannot be mounted inside a jail.

Submitted by: ryan@ixsystems.com
Reviewed by: emaste@, imp@, gjb@
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D21385

Rework UEFI ESP generation

Currently, the installer uses pre-created 800KB FAT12 filesystems that
it dd's onto the ESP partition.
This changeset improves that by having the installer generate a FAT32
filesystem directly onto the ESP using newfs_msdos and then copying
loader.efi into /EFI/freebsd.
For live installs it then runs efibootmgr to add a FreeBSD boot entry
in the BIOS.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D17947

mkisoimages.sh: don't use -p flag when copying loader.efi to msdosfs.

This fixes 'cdrom' target in the case when world was built by user,
and not root.

Reviewed by: imp
Differential revision: https://reviews.freebsd.org/D18414

release: set -e to exit on error in iso image scripts

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D17651

- Once we have shifted arguments up to thrice, base-bits-dir is $1 rather
than $4. Introduce $BASEBITSDIR for clarity and to avoid repeating this
mistake in the future. Fixing this ensures that we pick up newly built
boot bits native to the target rather for/from the host.
- Apply some of the argument quoting fixes done in r287635 but missing in
later revisions.

Allow etdump, makefs and mkimg to be overridden.

Recent changes to makefs and mkimg have led to situations where the
disconnect between this script and the versions installed on the host cause
failures. Provide a way to work around this that doesn't require the
installation of new versions to the host system if that's not desired.

With this change mkisoimages.sh will honour the $ETDUMP, $MAKEFS and $MKIMG
environment variables but fall back to the previous behaviour of finding them
within $PATH.

Reviewed by: gjb
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D15181

Generate hybrid ISO images for amd64.

This keeps the existing El Torito entries for BIOS and UEFI boot code and
adds a GPT in the ISO image's System Area containing boot code for BIOS that
will load /boot/loader from the ISO filesystem and execute it. We then use
etdump to find the EFI System Partition image in the El Torito catalog and
add an entry to the GPT that allows EFI to find it.

Reviewed by: gjb, imp
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D15051

Fix the position of $bootable so that -o platformid=efi applies correctly.

Sponsored by: iXsystems, Inc.
Pointy hat to: benno (again)

Fix makefs invocation after r331843.

Reported by: gjb
Reviewed by: gjb
Sponsored by: iXsystems, Inc.
Pointy hat to: benno

Allow makefs to properly tag UEFI El Torito boot images. Use them in amd64 ISOs.

UEFI booting requires an EFI System Partition (ESP). On most storage devices
this will be in a specific partition type. To allow booting from CD/ISO
filesystems, UEFI will look for an ESP in the form of a FAT filesystem image
embedded in the image. Historically FreeBSD has added one of these to its
amd64 ISO images but marked it as simply another i386 boot image. Luckily for
us most UEFI implementations are rather forgiving and work this out for us.

This change adds the ability to mark a boot image as being a UEFI image. It
also modifies our ISO generation to use this marking for the UEFI image we
embed.

Reported by: Thomas Schmitt <scdbackup@gmx.net>
Reviewed by: emaste, imp
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D14809

Update several more URLs

- Primarily http -> https
- Primarily FreeBSD project URLs

mkisoimages.sh: remove obsolete x$var convention

Ancient shells had trouble with empty variables but this has not been
relevant for FreeBSD for a very long time (decades?).

Increase efiboot.img size used in ISO creation

Due to recent and upcoming changes to add additional functionality to
the EFI loader its now bigger than the space allocates for efiboot.img
so increase this in line with boot1.efifat.

MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay

Merge ^/projects/release-install-debug:

- Rework MANIFEST generation and parsing via bsdinstall(8).
- Allow selecting debugging distribution sets during install.
- Rework bsdinstall(8) to fetch remote debug distribution sets
when they are not available on the local install medium.
- Allow selecting additional non-GENERIC kernels during install.
At present, GENERIC is still required, and installed by default.

Tested with: head@r293203
Sponsored by: The FreeBSD Foundation

Fix argument quoting and remove unnecessary braces

MFC after: 3 weeks
X-MFC-to: stable/10

Now that GENERIC can boot on UEFI systems (r268158), switch the build to use
UEFI-compatible images. These will boot as before on BIOS systems, but will
boot using the UEFI loader on UEFI-aware systems.

- Add CHECKSUM.* support in Makefile[1].
- Use ln -fs to create a symlink.
- Remove pkgadd for docports.
- Use WITHOUT_JADETEX=yes instead of WITH_JADETEX=no.
- Add {WORLD,KERNEL}_FLAGS to [BTWK]MAKE.
- Use makefs(8) and gpart(8) for sparc64 ISO image[2].
- Add publisher option to makefs(8)[2].

Based on work by: gjb[1]
Discussed with: marius, nwhitehorn[2]

ISO 9660 specification allows only "d-characters" and "a-characters" in the
Volume Descriptor (section 7.4). In short, upper-case alphanumeric + some
symbols only. While the makefs utility automatically converts the characters,
$LABEL should be consistent in the scripts.

Garbage collect some unused variables and stray debug code.

Submitted by: marius

Repair release CD generation on PC98 and sun4v after release building
changes, and backport the new logic (ISO images are TARGET dependant, not
TARGET_CPUARCH dependant) to Makefile.sysinstall. While modifying ISO
image scripts, change several archs to use makefs (from base) instead of
mkisofs (from ports) which makes release CD generation both faster and
self-hosting.

Use labels to find release media instead of hard-coded device paths. This
makes booting more reliable (and working at all on USB sticks). While here,
move responsibility for setting up fstab into the various platform mk-*.sh
scripts.

Suggested by: many

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

- Import the HEAD csup code which is the basis for the cvsmode work.

The current version of mkisofs is warning us that the -P option will
be changing with the next version, and we should be using -publisher
instead.

Also warned by: trhodes
MFC after: 1 week

Add a publisher variable with the URL for the FreeBSD Project, and put
this into the ISO headers by specifying the -P option to mkisofs.

Obtained from: share/examples/worm/makecdfs.sh

Set BATCH=yes to avoid bumping into interactivity in the port builds.

XXX /usr/ports is still hardcoded, should be ${PORTSDIR}

mkisofs(8) is now installed by sysutils/cdrtools.

This will be instantly MFC'ed.

Switch to using cdboot for booting i386 CD's by default.

Approved by: murray

Providing long filename support on the Macintosh requires over 30
megabytes of additional space on the install CD. Given our premium
for space, I don't think this is a wise use of 30MB. HFS support was
not provided on the FreeBSD 4.4 CDs and I received a grand total of 0
complaints.

Try to build `mkisofs' first, and only `pkg_add -r' if you cannot.

Fix my backwards logic.

Submitted by: dirk

+ do not hard code where a port may be installed to
+ uses `mkisofs' rather than `mkhybrid' as the new mkisofs is the merger
of the two
+ checks for `setcdboot's existsance and tries to pkg_add it if needed
+ removes English contraction

Approved by: jkh

Add scripts and Makefile support for building ISO images automatically
as part of the release if MAKE_ISOS is set. Will also build the
first CD with packages (in addition to the "minimal" CD) if CD_EXTRA_BITS
points to them. This probably need a bit more work to get fully useful,
but it at least covers the basics for now.