MFC freebsd-update: unconditionally regenerate passwd/login.conf files

The existing logic is nice in theory, but in practice freebsd-update will
not preserve the timestamps on these files. When doing a major upgrade, e.g.
from 12.1-RELEASE -> 12.2-RELEASE, pwd.mkdb et al. appear in the INDEX and
we clobber the timestamp several times in the process of packaging up the
existing system into /var/db/freebsd-update/files and extracting for
comparisons. This leads to these files not getting regenerated when they're
most likely to be needed.

Measures could be taken to preserve timestamps, but it's unclear whether
the complexity and overhead of doing so is really outweighed by the marginal
benefit.

I observed this issue when pkg subsequently failed to install a package that
wanted to add a user, claiming that the user was removed in the process.
bapt@ pointed to this pre-existing bug with freebsd-update as the cause.

PR: 234014, 232921

(cherry picked from commit ebebc41e4cfe44b8e8fd881badf2fa2c4be65aa4)

Git Hash: cd7da1deb581122c94c3735b78fafdd04ce77b67
Git Author: kevans@FreeBSD.org

MFC r367678:

Document the PAGER environment variable

Sometimes users want to use freebsd-update(8) in a non-interactive way and
what they often miss is that they have to set PAGER to cat(1) in order to
avoid interactive prompts from less(1).

MFC r364396: Unbreak `freebsd-update updatesready'.

The command would only work if PWD happened to be WORKDIR.
Also, exit 1 in case WORKDIR exists, but isn't accessible
by the current user.

PR: 242709
Reported by: Max Fiedler

MFC r360287: freebsd-update: rehash certs

With the inclusion of caroot bits, we'll need to also rehash on update as we
do in mergemaster/etcupdate.

If certctl's installed on the system, just unconditionally rehash. This
isn't an expensive operation, and we can refine it to compare
INDEX-{OLD,NEW} later if we really want to.

MFC r352513, r352514, r352662:

As suggested by koobs, MFC recent freebsd-update changes to
have a consistent user experience on all supported
versions.

r352513:
freebsd-update.8: appease igor

igor follows American style guides in the belief that abbreviations i.e.
and e.g. are always followed by a comma. Make that change now so that
future updates to freebsd-update.8 do not complain about this.

r352514:
freebsd-update: make usage output consistent

Drop trailing . which appeared only on description of IDS.

r352662:
freebsd-update: Add `updatesready' and `showconfig' commands

`freebsd-update updatesready' can be used to check if there are any pending
fetched updates that can be installed.

`freebsd-update showconfig' writes freebsd-update's configuration to
stdout.

This also changes the exit code of `freebsd-update install' to 2 in case
there are no updates pending to be installed and there wasn't a fetch phase
in the same invocation. This allows scripts to tell apart these error
conditions without breaking existing jail managers.

PR: 240757, 240177, 229346
Reviewed by: manpages (bcr), secteam (emaste), yuripv
Differential Revision: https://reviews.freebsd.org/D21473

MFC r352542: freebsd-update: Fix src component detection

PR: 224048, 238558, 23999
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D21579

MFC r344280: freebsd-update: Clarify help text

Improve help text to include example release numbers for reference
and clarify the -F option.

PR: 231185, 214619
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: delphij, rgrimes
Sponsored by: The FreeBSD Foundation

MFC r34505: freebsd-update: restore old exit code when no updates are available
locally

This unbreaks ezjail and iocell, which get into an endless loop trying to
figure out how many times "freebsd-update install" needs to be called.

PR: 229346
Submitted by: Mike Cole <mcole36 at gmail.com>
Approved by: bapt

MFC r344818: freebsd-update.8: update example to contemporary versions

PR: 235761
Reported by: linimon

MFC freebsd-update: Clarify unsupported release upgrade error message

MFC r343122:

freebsd-update: Clarify unsupported release upgrade error message

Notify users that upgrading from -CURRENT or -STABLE is unsupported by
freebsd-update.

Also ensure --currently-running provides a correctly formatted release
(as done by -r).

Reported by: yuri
Reviewed by: bcran

MFC r343268:

frebsd-update: fix --currently-running after r343122

Reported by: Brandon Schneider

MFC r343271:

freebsd-update: Allow upgrade from pre-release builds

Update r343122 to include -ALPHA, -BETA and -RC releases as
upgrade-able via freebsd-update.

Reported by: delphij, des
Reviewed by: delphij

MFC r343410:

freebsd-update: fix style from r343271 change

PR: 234771
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Sponsored by: The FreeBSD Foundation

freebsd-update: Use BASEDIR when checking for src componentfreebsd-update: Use BMFC r343152: freebsd-update: Use BASEDIR when checking for src component

src could potentially be installed under the based dir
and not under the root or vice versa.

PR: 224048
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: delphij
Sponsored by: The FreeBSD Foundation

MFC r343270: freebsd-update: Update /etc/passwd after password db changes

Add -p to pwd_mkdb in order to ensure password db changes are also
included in /etc/passwd.

PR: 165954, 232921, 229487
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: jilles
Sponsored by: The FreeBSD Foundation

MFC r343411: freebsd-update: Clarify unsupported upgrade message

PR: 204115
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: delphij
Sponsored by: The FreeBSD Foundation

MFC r343412: Fix a typo/wordsmith a description modified in [MFC of r343407]

MFC r343407: freebsd-update: open $PAGER only if necessary

PR: 194547, 208497
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: delphij
Sponsored by: The FreeBSD Foundation

MFC r343409: freebsd-update: Stop installing empty component sets

Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reported by: rgrimes
Reviewed by: delphij
Sponsored by: The FreeBSD Foundation

MFC r343589: freebsd-update: regenerate man page database after update

These are currently not reproducible because they're built by the
makewhatis on the freebsd-update build host, not the one in the tree.
Regenerate after update, and later we can avoid including it in
freebsd-update data.

PR: 214545, 217389
Reviewed by: delphij
Sponsored by: The FreeBSD Foundation

MFC r343153: freebsd-update.8: mandoc -Tlint fixes

PR: 185389

MFC r324441:

Fix freebsd-update(8) erroneous message and exit status when "fetch install" used.

PR: 190660

MFC r326276:

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

MFC r307156:

MFportsnap r264740: Use case insensitive match when parsing host(1) output.

Some DNS caches turn "FreeBSD.org" into "freebsd.org", which was causing
the printed SRV records to not match our regex.

PR: 170503

MFC r326082: freebsd-update: do not duplicate patchlist entries

PR: 221079
Submitted by: Masachika ISHIZUKA
Submitted by: ota@j.email.ne.jp
Reviewed by: cperciva

MFC r319954:
Modernize FreeBSD version numbers in freebsd-update(8).
While here, expand a contraction to make textproc/igor happy.

Approved by: re (kib)
Sponsored by: The FreeBSD Foundation

MFC r302534:

Allow - in distribution names. This is needed for freebsd-update to
work with 11.0+, where the debugging symbols use a new naming scheme
for release distribution files.

Errata candidate.

Approved by: re (gjb)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Provide a future release as an example, instead of a historical one.

Add debug file extension to freebsd-update(8) after r288176

Reviewed by: delphij
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3784

Add --currently-running <release> option to freebsd-update.

This option tells freebsd-update to act as if it is running a specific
release instead of querying the kernel. In particular, this can be
useful when upgrading jails.

Requested by: EuroBSDCon devsummit jails session
Tested by: allanjude
MFC after: 1 week

Skip src component if /usr/src is empty.

Differential Revision: https://reviews.freebsd.org/D2364
Submitted by: kczekirda
Reviewed by: cperciva, delphij, nwhitehorn, allanjude
MFC after: 2 weeks

Add META_MODE support.

Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision: D2796
Reviewed by: brooks imp

Fix syntax errors in conditions for new features in freebsd-update

Differential Revision: https://reviews.freebsd.org/D1550
Submitted by: kmoore
Approved by: delphij
Obtained from: PCBSD
MFC after: 1 week
X-MFC-With: 279571
Sponsored by: ScaleEngine Inc.

- Include commas in valid file-name chars freebsd-update will support

Approved by: cperciva

Add a new safetly belt to freebsd-update to prevent a user doing a minor update (-pX) while having an unfinished major upgrade (9.x to 9.y)
Safetly belt can be disabled with the -F flag

Additionally, add the --not-running-from-cron flag they bypasses the TTY requirement, and allows freebsd-update to be invoked by orchestration frameworks, scripts, or otherwise.

PR: 196760
Differential Revision: https://reviews.freebsd.org/D1550
Reviewed by: cperciva, delphij
Approved by: bcr (mentor), rodrigc (src)
MFC after: 1 month
Relnotes: yes
Sponsored by: ScaleEngine Inc.

(backup_kernel_finddir, backup_kernel, install_files):
Add missing references to $BASEDIR, in order to allow correct operation
when updating a system mounted at another location than / (e.g. when
updating an alternate Boot Environment).

Reviewed by: cperciva
MFC after: 1 week

When upgrading, install the ELF runtime linkers before libraries.

This is required to prevent problems with nss modules that use libthr when
upgrading from releases prior to 10.1.

PR: 197366
Sponsored by: DARPA, AFRL
Differential Revision: D1790
Reviewed by: cperciva
MFC after: 3 days

Use "RCS tag" instead of "$FreeBSD$ tag", since svn will obediently
expand the latter.

MFC after: 3 days

Strip trailing / characters from paths in "not present" index entries, not
just "directory" entries.

Prior to this commit, if / was added as part of a security update (how? In
the most recent case, because lib32 was accidentally omitted and was then
re-added, and every installer distribution set gets its own paths) then
the code which was supposed to filter out updates to deleted parts of the
base system (if someone decides to delete / then we shouldn't re-create it
for them) would instead get confused and decided that while / should exist,
// should not exist and needs to be removed.

This fixes the bug which caused freebsd-update to want to delete / (which is
harmless, since `rm /` fails, but scary nonetheless). A workaround is being
applied to the update bits in order to avoid triggering the bug on unpatched
systems.

PR: 196055, 196091, 196147

Re-work non-persistent filesystem detection as it was not possible to
detect /dev/md backed mfs filesystems that way.

Differential Revision: https://reviews.freebsd.org/D1163
Approved by: ian

Add logic for detecting non-persistent filesystems being utilized by
workdir which would break the upgrade process upon reboot.

Currently we check for tmpfs and mdmfs.

PR: 195006
Differential Revision: https://reviews.freebsd.org/D1163
Approved by: cperciva

use .Mt to mark up email addresses consistently (part2)

PR: 191174
Submitted by: Franco Fichtner <franco@lastsummer.de>

Fix typo in r256646: We want to generate lists of directories in INDEX-OLD
and INDEX-NEW and compare them, not generate the same list of directories
from INDEX-OLD twice...

Pointy hats to: cperciva & everybody who didn't proofread EN-13:04 enough

Be more selective when filtering for lib*.so.N files. These are deleted
at the end of the upgrade process, after warning users to upgrade any
3rd party software (e.g., from the ports tree) which might link to the
libraries being removed.

Prior to this commit, the line
/usr/lib/libc.so|...|/lib/libc.so.7
matched the regex, which -- upgrading from 9.x to 10.x, where libc.so is
a regular file and thus was not part of a line which matched the regex --
resulted in freebsd-update thinking that /usr/lib/libc.so was a shared
library which was being removed as part of the upgrade. This had some
unfortunate consequences.

This will be part of an upcoming Errata Notice.

Allow ~ in file names so libtool droppings in contrib don't break updates.
It has happened twice now, and is likely to happen again.

Errata notice candidate.

When installing updates, install new directories first and remove old
directories last.

This is generally handled by the fact that the list of filesystem objects
is sorted, but this sorting is broken by code which moves .so files ahead
(so that they're present before any binaries which use them)... that code
also moved .so files ahead of directories, which is a problem for upgrading
to 10.0 where there's a new directory containing new .so files.

Errata Notice Candidate.

Speed up `freebsd-update IDS` by using IFS to split fields instead of
forking lots of processes to run echo|cut. In one test this reduced
the CPU time from 980s to 134s and the wallclock time from 806s to
132s.

Submitted by: Oleg Ginzburg

mdoc sweep.

Fixes to man8 groff mandoc style, usage mistakes, or typos.

PR: 168016
Submitted by: Nobuyuki Koganemaru
Approved by: gjb
MFC after: 3 days

Remove trailing whitespace per mdoc lint warning

Disussed with: gavin
No objection from: doc
Approved by: joel
MFC after: 3 days

Add '%' and '@' to the set of characters which can appear in path names.
Without this change, freebsd-update refuses to accept 9.0 metadata files.

Make freebsd-update(8) smarter in how it handles $FreeBSD$ tags in
configuration files.

If the current file differs from the canonical version from the old release
only due to differences in the $FreeBSD$ tag (which can happen if the system
was installed from source code, depending on how the src tree was checked out)
then freebsd-update will treat the file as "unmodified" and silently update
it to the "clean" version in the new release.

If the only change being made to a configuration file is in the $FreeBSD$
tag (e.g., for any configuration files which have been modified locally, now
that we're using SVN and the $FreeBSD$ tag changes when a branch is created),
freebsd-update will no longer print the diff and prompt "Does this look
reasonable (y/n)?".

Nagged by: pgollucci
MFC after: 1 month

Honor $PAGER instead of always calling more.

Approved by: colin@
MFC after: 1 week

Typo fix in a comment.

Reviewed by: cperciva

mdoc: drop redundant .Pp and .LP calls

They have no effect when coming in pairs, or before .Bl/.Bd

In backup_kernel(), support backing up subdirectories and handle files
with spaces correctly.

Approved by: cperciva
MFC after: 1 month

"freebsd-update fetch -r NEWRELEASE" is meaningless and probably a typo
for "freebsd-update upgrade -r NEWRELEASE". Error out and suggest what
the user probably meant.

Submitted by: James Seward
MFC after: 1 month

Add warning about freebsd-update upgrade needing lots of disk space.

Submitted by: jpaetzel
MFC after: 1 month

Remind the user that he needs to run 'freebsd-update install' to install
new bits after downloading them using 'freebsd-update upgrade'.

Submitted by: bapt
MFC after: 1 month

Fix typos, spelling, formatting and mdoc mistakes found by Nobuyuki while
translating these manual pages. Minor corrections by me.

Submitted by: Nobuyuki Koganemaru <n-kogane@syd.odn.ne.jp>

Update some release version numbers. The patch is based on the one
from the PR, but the version numbers reflect the newer ones from
http://security.freebsd.org/#sup

PR: docs/145227
Submitted by: Glen Barber (glen dot j dot barber at gmail dot com)
Reviewed by: cperciva
Mentored by: jkois
MFC after: 1 week

Fix several typos in macros or macro misusage.

Found by: make manlint
Reviewed by: ru
Approved by: philip (mentor)

Disable SSL renegotiation in order to protect against a serious
protocol flaw. [09:15]

Correctly handle failures from unsetenv resulting from a corrupt
environment in rtld-elf. [09:16]

Fix permissions in freebsd-update in order to prevent leakage of
sensitive files. [09:17]

Approved by: so (cperciva)
Security: FreeBSD-SA-09:15.ssl
Security: FreeBSD-SA-09:16.rtld
Security: FreeBSD-SA-09:17.freebsd-udpate

Special-case "-r X" where X is [0-9.]+ to mean "-r X-RELEASE".

Tripped over by: too many people to count
MFC after: 1 month

Add support for backing up the old kernel when installing a new kernel
using freebsd-update. This applies to using freebsd-update in "upgrade
mode" and normal freebsd-update on a security branch.

The backup kernel will be written to /boot/kernel.old, if the directory
does not exist, or the directory was created by freebsd-update in a
previous backup. Otherwise freebsd-update will generate a new directory
name for use by the backup. By default symbol files are not backed up
to save diskspace and avoid filling up the root partition.

This feature is fully configurable in the freebsd-update config file,
but defaults to enabled.

MFC after: 1 week (stable/7)
Reviewed by: cperciva
Approved by: re (kib)

fix grammar...

Reviewed by: cperciva

In `freebsd-update IDS`, strip out file flags before we look for
non-matching index lines. This fixes a bug where bogus warnings would
be printed file has the wrong file flags AND has been updated by
FreeBSD Update.

Reported by: Royce Williams

Add "IDS" command to freebsd-update. This was present in the original
version of freebsd-update, but I took it out when I rewrote everything
and added FreeBSD Update to the base system because I didn't think it
was useful. It turns out that quite a few people liked it and wanted
it back.

Requested by: Royce Williams + others
MFC after: 2 weeks

Adjust recognize-shared-libraries regex to avoid matching symlinks to
shared libraries.

This fixes a problem which resulted in 6.x->7.x upgrades having the
/usr/lib/libpthread.so -> libthr.so symlink missing; what happened was
that the old libpthread.so symlink pointed to /lib/libpthread.so.2 --
which matched the "/lib/*\.so\.[0-9]+" regex -- but the new symlink
didn't, so FreeBSD Update got confused and deleted the symlink as part
of its "remove old shared libraries" step.

To recreate the symlink (which I understand is necessary for ports like
KDE to build) on a 7.x system which FreeBSD Update upgraded from 6.x:
# ln -s libthr.so /usr/lib/libpthread.so

Reported by: Dmitry RCL Rekman
Help diagnosing bug from: kris
MFC after: 7 days

When updating the install list for files which have had local changes
merged with upgrade changes, don't try to compute the SHA256 hash of
files which don't exist.

Reported by: Jaakko Heinonen
MFC after: 1 week

Add change missing from previous commit: Remove temporary file.

Fix "freebsd-update rollback" applied to minor-version upgrades: Old
shared object files which have the same name as currently-installed
shared object files should be reinstalled after binaries are rolled
back. The order for rolling back updates is therefore
1. Install any old shared object files which can be installed without
overwriting a new shared object file.
2. Rollback everything which isn't a shared object or kernel file.
3. Rollback any shared object files which we didn't deal with in (1).
4. Rollback to the old kernel.

Bug reported by: Jan Henrik Sylvester
MFC after: 3 days

Add support for "freebsd-update -r newrelease upgrade" -- binary
upgrading to new releases. Important parts of this code include
* automatically determining which optional components (e.g., src,
info, proflibs) are installed.
* merging changes in files which are modified locally and have
changed between the currently running and new release.
* prompting the user to rebuild all 3rd party software before
deleting old shared libraries.

Yes, this is compatible with "freebsd-update rollback" -- you can
test a new -BETA and roll back to the old release if you don't
like it.

Subject to re@ approval, this will be MFCed before 7.0-BETA3 and
6.3-RC1.

MFC after: 2 days

Two minor improvements uncovered by work on upgrading between releases:
* When installing updates, make sure that securelevel <= 0. Otherwise
we can't remove the schg flag from files.
* When preparing to download updates, check to see if we already have
them sitting in the /files/ directory. This saves bandwidth if users
run "freebsd-update fetch" more than once without installing updates
in between.

While I'm here, bump the copyright date.

MFC after: 3 days

Change the time of the first "EoL is coming soon, you should upgrade" warning
from EoL minus 6 months to EoL minus 3 months, in order to increase the odds
of there actually being a more recent release to which users can upgrade.
(In particular, for releases which are only supported for 12 months, it's
quite likely that the next release will occur between 6 and 9 months later.)

Discussed with: kensmith
Approved by: re (bmah)
MFC after: 3 days

When storing old versions of files for use in generating new files via
patching and for rolling back updates, don't copy a file if it has already
been stored. This provides a significant speedup to the "Preparing to
download files" stage of "freebsd-update fetch" if many updates have already
been applied or if a file being updated is linked many times (such as
/rescue/*).

Reported by: Paul Dekkers
MFC after: 1 week
Approved by: re (bmah)

Add some missing '${BASEDIR}/'s. Prior to this commit, FreeBSD Update
operating with the "-b basedir" option would not correctly update files
which had flags set or were hardlinked.

Submitted by: Karsten Schmidt
Pointy hat to: cperciva
MFC after: 1 week

Fix problems resulting from SMP kernels (mis-)identifying themselves as
"SMP-GENERIC" (i386) or "GENERIC" (amd64).

FreeBSD 6.2 Errata candidate.

MFC after: 3 days
Pointy hat to: cperciva

"block is set to UTC" -> "clock is set to UTC"

Submitted by: csjp

Add a sentence to the DESCRIPTION noting that updates aren't available
for -STABLE or -CURRENT.

Inspired by submission from: Scott Robbins
MFC after: 3 days

Three minor bug fixes:
1. When downloading metadata files, make sure we only download each
file once; without this fix, "freebsd-update fetch" will fail the first
time it is run if there have been no updates yet for the installed
release.
2. If the FOO kernel is installed in /boot/kernel instead of /boot/FOO
and the /boot/FOO directory does not exist, don't try to update
/boot/FOO. This is an issue only where an update involves adding a new
kernel module.
3. When removing files and directories, operate in reverse
lexographical order, in order to ensure that files are removed before
the directory which contains them.

MFC after: 3 days

Set LC_ALL=C in order to avoid problems with character ranges and
sorting.

PR: bin/104505
MFC after: 3 days

Conform to wider English usage.

Submitted by: Royce Williams

It's spelled MAN.

Add FreeBSD Update 2.0 client code. The build code is in the projects
repository.

Sponsored by: FreeBSD security development fundraiser