Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r250916: coexist with /etc/localtime being a symlink

MFC r255428:

Pass -n (do not emit comments) when saving mtree information for future
mergemaster(8) runs.

MFC r250463:
Permit relative paths with mergemaster -m

PR: bin/173929

MFC r249906:

Use the system MAKEOBJDIRPREFIX when running make targets in
mergemaster. This allows bootstrap verions of tools to be used.

Add a note to UPDATING about this change.

This commit is a partial backout of r248531 in that it removes an
accidentally committed change to mergemaster.

Discussed with: jhb
Sponsored by: DARPA, AFRL

Revert r248352:

> Replace all known uses of ln in the build process with appropriate
> install -l invocations via new INSTALL_LINK and INSTALL_SYMLINK
> variables.

This breaks the assumption mergemaster and etcupdate that no new
features of tools will be required to run "make distribution" or it's
prerequisites.

Plotting a way forward will take some time and nothing in 9-STABLE
depends on INSTALL_LINK and INSTALL_SYMLINK yet so backing this out seem
like the right short-term approach.

PR: misc/177055

MFC r243891:
Add ability to mergemaster to permit the user to type
the absolute path to PAGER if mergemaster can not find
the one already set.

Approved by: cperciva (implicit)

MFC r243892:
Remove pointless check for the existence of /usr/bin/less which exists
in all configurations.

Approved by: cperciva (implicit)

MFC r241218:

Remove references to CVS so that people will stop bringing it up

For -p:
The localtime update should have been excluded in the first place

The make.conf comparison has been OBE for some time now, and there
is no src.conf equivalent to share/examples/make.conf, so remove
the whole thing.

Update copyright

MFC r228122:

If using DESTDIR we need to be sure to create a
${DESTDIR}/var/db/zoneinfo

Approved by: re (kensmith)

MFC r227013:

1. Add a new option, --run-updates, to always or never run the newalises,
pwd_mkdb, etc. updates at the end of the comparison.

2. Add an update to the end to handle /etc/localtime, if it exists.
If /var/db/zoneinfo exists, automatically update /etc/localtime,
which should (hopefully) be safe to do. If not, prompt the user
to run tzsetup.

3. Update run_it_now(), the function that handles input for the updates,
to make sure that we got a valid answer, and to handle the --run-updates
option if supplied.

Approved by: re (kensmith)

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Make clear that the -U option can be dangerous. [1]

In -p mode install the group file with the same permissions as normally
done in etc/Makefile.

Update the copyright.

Requested by: peter [1]

Approved by: re (kib)

Hide the creation and population of the temproot

Add support run services_mkdb(8).

Approved by: dougb, imp (mentor)
Reviewed by: ume
MFC after: 2 weeks

Make it more clear in the docs that -a is not compatible with -iFU,
and enforce this in the code. Apparently a lot of users mistakenly
combine -a with these flags and are then mystified that no changes
were made.

While I'm here, fix a trailing space in mergemaster.8

Make -U once again honor -D after my change to consolidate setting
of MTREEDB with DESTDIR.

PR: bin/143089
Submitted by: Anton Yuzhaninov <citrin@citrin.ru>

Oops, deleted one line too many. Add back the null case for IGNORE_MOTD

Now that we've officially arrived at FreeBSD 9, remove the IGNORE_MOTD
compat shim as advertised. This change should not be MFC'ed.

Update copyright date

Update delete_temproot() to include the error message if it fails,
and clean up the places where it's called.

If there are no files left in temproot when the comparison is done
delete it without prompting. This should make "automated" runs of
mergemaster without -a a little easier.

Document the new behavior in the man page.

If we are using -p it does not make any sense to even create the
MTREENEW file since it will never be used.

Add some patches contributed by jhb:
1. Don't prompt the user for "-U but no db" error if we're using -a
2. Add an option to delete stale rc.d files automatically if the user
has DELETE_STALE_RC_FILES in their rc file. Lack of command line option
for this is not an oversight.
3. Add []'s around the terminal $ for the $FreeBSD$ test for -F

For one bug raised by jhb I did a more thorough solution:
There were a lot of things that "snuck in" between the end of the test
for -r and the start of the comparison. One of them is the creation of
the mtree db, as pointed out by jhb. Fix this problem more thoroughly
by moving the end of the test down to where it should/used to be, right
before the comparison. As a result, indent the interloping code to match.

Fix a problem with how mergemaster handles the hard links for /.cshrc
and /.profile. The problem is that install(1) will unlink the old file
before it installs the new one, which means that in the best case we
have to compare the changes for the old file twice.

So, change the logic to first test to see if the link exists, then
install the file. Then if the link was there and we're using -i, just
create the link in /root and be done with it. Otherwise display the
message to the user and give them the option.

Because we are now sorting things before doing the comparison we can
know conclusively that the files in / should be the sources, and the
files in /root will be the targets, so adjust the paths accordingly.

While I'm here, split a too-long error message into two lines and
just return at the end of handling these files instead of setting
the variable that says "do nothing" and then returning at the end
of the function anyway.

Fix an indentation issue, no functional changes

In the places where find is used that the user may see the results,
first pipe it to sort so that order of processing will be deterministic
and like things will be grouped together.

Over time things that used to be files/directories/links can change
to something else. So add code to detect when things don't match and
give the user choices about how to fix it.

If we're using -P and something in the above check needs to be moved
we need to have the directory there for it, so create it at the
beginning and delete empty versions of it at the end.

The case where something used to be a file or link and now is supposed
to be a directory (e.g., /etc/security) is especially dangerous, so
make failure to install a necessary directory in $DESTDIR a fatal error.

Simplify handling of MTREEFILE relative to DESTDIR

Make the message about a missing MTREEFILE combined with -U
more informative

1. Update the message that prints out for -U when the mtree database does
not exist to let the user know that it will be created for the next run.

2. Delete more stuff we're not going to use from the temproot prior to
creating the mtree database to dramatically reduce its size (162K -> 37K).

3. We've been deleting the zero-size files from temproot for a long time
now, so remove the spurious "-size +0" from the find command in the
comparison loop, and remove what is now a really stale comment.

Enhance the safety of the -U option:
1. In several places make sure that the mtree database is not empty using
the -s argument to test instead of -f. (I thought I'd already changed this,
but obviously not.)
2. When deriving the list of changed files use a colon-delimited list.
3. If the list of changed files is empty, unset the variable, and test
for a non-empty CHANGED in diff_loop() before entering that routine.

Enhance the speed of the -U option by using an internal case statement
instead of echoing the variable to an external grep for every file.

Fix indentation on one line.

Add -L to the mtree invocation to chase symbolic links that are present
in "Service jails."

Submitted by: Jeremie Le Hen <jeremie@le-hen.org>

Don't tempt svn to expand the example $FreeBSD strings

Add a new -F option to automatically install files that differ
only by version control Id strings

1. Clean up usage() output a bit by grouping options that take an argument
2. Fix a comment to refer to the right loop

When using the -D option:
1. The new mtree file should be created in the "host" system /tmp
2. The existing mtree file in the "host" system should not be deleted

Submitted by: scf

Instead of using obj and all targets which are not cross-build aware,
use _obj and everything which are.

Submitted by: ru

Simplify the code a little by moving the working part of
ARCHSTRING into the new MM_MAKE variable.

We only need to check for the presence of the target of $PAGER
if that variable is actually set. [1]

Pointed out by: Steve Kargl <sgk@troutmask.apl.washington.edu> [1]

Only check for 'nodev' in fstab if that file exists

Submitted by: sam

For IGNORE_FILES delete the version in the TEMPROOT, not the base.

Submitted by: clemens fischer <ino-news@spotteswoode.dnsalias.org>

Maintenance and updates
=======================
1. Various improvements to the mtree (-U) feature:
a. Seperate the notion of directory and file (user can override db path)
b. Only check for the existence of the mtree file if -U is set
c. Use mktemp to create the new version of the file
d. More safely install the new file
e. Standardize error messages a bit
2. Remove the last of the MAKEDEV stuff (RIP)

New Features
============
1. Switch to using the top level (e.g., /usr/src) Makefile, and specify
that we should use the *.mk files from the source directory instead of
the installed versions. [1][2] This allows easier cross builds and
simplifies (or in some cases permits) upgrading.
2. Check for the deprecated 'nodev' option in /etc/fstab [3]
3. Add support for the IGNORE_FILES variable [4] and deprecate IGNORE_MOTD
accordingly.
4. Before installing a file check to make sure that the target does not
already exist as a directory [5]
5. Check to be sure that the file installed and error out if not

PR: bin/96528 [1]
Submitted by: ru [1]
PR: bin/129639 [2]
Submitted by: sam [2]
PR: bin/122282 [3]
Submitted by: Eygene Ryabinkin <rea-fbsd@codelabs.ru> [3]
PR: bin/106642 [4]
Submitted by: Henrik Brix Andersen <henrik@brixandersen.dk> [4]
PR: bin/108183 [5]
Submitted by: Riccardo Torrini <riccardo@torrini.org> [5]

Revert 184781, 184804, and 184832 (automatic installation of files
that differ only by VCS Id) for the following reasons:
1. It was added without my consent, review, or even a heads up
2. It is something that I've repeatedly said I do not want, and certainly
do not want as the default
3. It is poorly implemented (much too complex, produces false positives
e.g., /etc/mail/helpfile)

Given that this is a situation that comes up very infrequently (usually
only for a major version upgrade) and can usually be handled simply
enough on a one-off basis, I will once again point out that I think
this is a Bad Idea. I would be willing to consider a better implementation
as an option that is off by default.

No need to run rm ${COMPFILE} after mm_install() - mm_install()
does it for us.

Improve on 184781 - instead of ignoring the file when the only difference
is CVS Id, replace the old one with the new one automatically. While I
don't see much difference, some people think it's somehow better that way.

Ignore files that only differ in CVS Id tag.

MFC after: 2 weeks

Remove ISDN4BSD (I4B) from HEAD as it is not MPSAFE and
parts relied on the now removed NET_NEEDS_GIANT.
Most of I4B has been disconnected from the build
since July 2007 in HEAD/RELENG_7.

This is what was removed:
- configuration in /etc/isdn
- examples
- man pages
- kernel configuration
- sys/i4b (drivers, layers, include files)
- user space tools
- i4b support from ppp
- further documentation

Discussed with: rwatson, re

In the following scenario:
1. Start mergemaster
2. Interrupt it
3. < Somehow the temproot directory disappears >
4. mergemaster -r
Many bad things can happen, especially if the -i option is in use.

Therefore, add a check to make sure it still exists before we start
comparing files.

Brought to my attention by:
PR: bin/40538
Submitted by: Cyrille Lefevre <cyrille.lefevre@laposte.net>

Add auto upgrade capability to mergemaster.

An mtree description of all non-zero files that make
distribution installs (only size and md5) is built from the
temproot. When the user completes a mergemaster run, the
mtree description file gets installed into /var/db for
safe-keeping.

When the user then decides to do a subsequent upgrade (with
the -U flag), the existing mtree description from /var/db
is called into service looking for files that are different in
DESTDIR. This is stashed away until a file that would normally
end up prompting the user to look at changes is encountered.
Since there are no user modified changes, the new file is
installed without bothering the user.

Looked at by: dougb
MFC after: 6 weeks

Add a -A argument to mergemaster to allow explicitly specifying an
architecture to pass through to the underlying makefiles. This is
quite useful when building on an i386 box to populate an amd64 NFS
root.

Head nod: dougb
MFC after: 1 week

Change the default for deleting stale files from yes (which violates
the general principle that mm should do nothing by default) to no.

PR: bin/85449
Submitted by: Kevin Oberman <oberman@es.net>

In the case where /etc/rc.d/ is empty, do not bogusly ask the user if
they want to delete '*'. It turns out that there is one valid case where
this might happen, installing to an empty DESTDIR.

Patch submitted by: schweikh

Set MAKEOBJDIRPREFIX in the environment, like God intended. This
only worked because src/Makefile.inc1 is wrapped by src/Makefile.

Silence from: dougb

mergemaster -p (pre-world) doesn't populate /etc/rc.d, which led to a
false positive for every file according to the new stale rc.d checker.

Friendly reminder by: dhw

* Update copyright
* Delete a trailing space
* Add a 'stale /etc/rc.d file checker', offer to delete any stale
files for the user.

Use the actual build target (all), since relying on the build as a
side effect of the distribution target was apparently just good luck
all these years.

Submitted by: ru

Preserve directory structure in PRESERVED_FILES_DIR to avoid
filename collisions.

Submitted by: des

rc.sendmail should not have been removed yet.
Don't include it in the rcOG removal prompt.

* Add a -P (Preserve) option, which saves a copy of files that you replace.
The idea for this option came from Marc Schneiders <marc@schneiders.org>

* I've been meaning to remove the code that checked for the existence of
/etc/sysconfig for a while now, since that file was last seen in 2.2.2.

* Replace the above with a check for elements of the old rc system.
Offer to move these files out of DESTDIR/etc for the user's convenience.

Put back the deletion of TEMPROOT/etc/passwd.

Noticed by: maxim

Better handle the recent (desirable) change to create login.conf.db at
install time, and make sure to catch any future .db files as well.

* Update copyright year.
* Give back one line of ws when showing a diff.
This was requested by several.
* Un-quote DIFF_FLAG, and add DIFF_OPTIONS. This will allow the user to
do more creative things in a mergemaster rc file. Un-quoting is necessary
in order to handle things like CVS Id tags that look like shell variables.
* Simplify terminal width determination.

Suggestions related to ignoring the CVS Id's were taken from several, with
dillon and gad providing particularly helpful feedback.

Backout last commit by request.

Commit simple mergemaster changes to make the UI more deterministic
(far easier to use) by limiting the initial display to the size of the
window.

Don't use NO_MAKEDEV* option, its the default now.

Suppress spurious warning messages about rm -rf TEMPROOT failing
before chflags, and exit non-zero if the old TEMPROOT dir
can't be deleted for whatever reason.

Fix a couple DESTDIR related problems. Credit for the "Only in" section
goes to James <James_Bond_79@yahoo.com>, and Cyrille Lefevre
<cyrille.lefevre@laposte.net>. The make variables problem was found
by jon.

* Handle all flags, not just schg
* Make chflags -R more efficient by deleting everything we can first

Add 'chflags -R noschg' to the deletion process for the TEMPROOT
directory so that /var/empty, and any other such items that are
added in the future can be deleted effectively.

Go back to taking the user's umask into account. Thanks to Alfred for
hatching the idea of using dc, and Giorgos (keramida) for incubating it.

This also reverses most of the previous commit which took out or
modified the text about umask stuff.

* Use the new stat(1) to get mode information from the files we
are installing.
* Since this means that for now we can't accomodate non-standard
umask's, warn the user accordingly.
* Convert the "press enter to continue" prompt into a function.

Actually accept the default argument to SCREEN_WIDTH.

Submitted by: brian

* Add some niceties to make the -p option quieter, and better behaved
* Take a simple idea about setting the screen width using stty (for sdiff's
purposes) thoughtfully suggested by Joerg to an absurd conclusion.

Solve the problem of spuriously comparing files that were created
in the ${TEMPROOT}/usr/obj directory by deleting them first.

Improve grep'ing for variables in make.conf and rc.conf*.
Thanks to cjc for the idea.

* Add a feature to do a "pre {build|install}world" check of (so far)
master.passwd, group, and make.conf
* Add a feature to check variables in rc.conf[.local] to their
counterparts in /etc/defaults/rc.conf after a run
* Twiddle whitespace a little
* Change some "[ -f file ] && rm file" to "rm -f file"

* Update (c)
* Fix a problem with files that have no CVS $Id's. Thanks to naddy for
spotting this one. It wasn't a _huge_ problem since almost all the files
we install (except motd) have one, but still, it's a bug.
* Add a divider between diff outputs, which is helpful both for logs,
and for giving a good visual clue for diffs that are smaller than
$LINES. Another helpful suggestion from Gary W. Swearingen, swear@blarg.net.

Hopefully squish the problem of mm creating sendmail files in
the source tree once and for all.

Pointed out by: A cast of thousands

Reviewed by: asmodai

Work around the problem in RELENG_4 where the file doesn't actually
exist, and therefore mm_install is returning the "fail" value of
the test instead of the "success" value for install.

This change is a no-op on HEAD, but since the only harm on RELENG_4
ATM is a spurious warning it can follow the usual MFC practice.

Submitted by: A cast of thousands :-/

Add the last of the compatibility mods from openbsd. Neatly, the
"install && rm" change fits in with the new FreeBSD default of
copy instead of move for install. Changing the order of the deletion
of the spurious password files doesn't affect FreeBSD functionality,
but it's done in such a way as not to matter.

Obtained from: Christian Weisgerber <naddy@mips.inka.de>

Various fixes, including suggestions from some thoughtful folks, thanks.

* Minor umask portability change (Christian Weisgerber <naddy@mips.inka.de>)
* Clarify default value of the "run it now" option (bmah)
* Make "run it now" $DESTDIR safe (bsd)
* Handle installation of hard links for /.profile and /.cshrc properly
when the auto-install option is selected
* Fix some more whitespace issues

Be more consistent with the terminology, referring to "temporary and
installed" instead of "old and new". Inspired by the somewhat
non-linear PR, which really didn't have a fix, per se.

PR: conf/27235 Roelof Osinga <roelof@eboa.com>

Fix a problem when user selects -a (auto run) but we encounter a file
missing on the system. Instead of passing it by, mm was prompting...
bad mm, no cookie!

Brought to my attention by the PR, but the fix needed to be tweaked to
handle the auto-install option as well.

PR: misc/25731 Gilbert Gong <ggong@cal.alumni.berkeley.edu>

Whitespace only changes to try and fit a little better into 80 columns.

Change NO_MAKEDEV to a finer granularity method:
NO_MAKEDEV_INSTALL and NO_MAKEDEV_RUN. The former implying the latter.
The names imply what they do. The last commit by DES based on a PR defeated
the original idea behind NO_MAKEDEV, which was not to run MAKEDEV, but to do
the installation of MAKEDEV. This should satisfy both parties on the MAKEDEV
challenge.
Reflect this in mergemaster, this might later on be decided to be set to
NO_MAKEDEV_INSTALL, for now I kept to the old behaviour.

* Add two requested features, source'ing of /etc/mergemaster.rc, and
asking the user to actually run the recommended commands related
to installation of files such as aliases or login.conf.

* Return to using grep for CVS $Id comparison. Using ident caused too
many problems for people with local CVS/RCS tags in their stuff.
Attempt to make portability a little easier to maintain in spite of
this change by defining the name of the tag to search for. This
is a slightly different change that solves the problem in the PR.

PR: bin/24564
Submitted by: Tony Finch <dot@dotat.at>

* Reduce diffs to RELENG_4 by specifying a full path to sysctl
* Adjust a little whitespace
* Make the distrib-dirs/mtree on DESTDIR conditional on user
actually specifying a DESTDIR. This seemed like a safe
way to get the right directories and permissions in the
installed tree since 'make installworld' does the same
thing, but in practice too many people have custom hacks
that we should leave unmolested. Still need to find a way
to deal with 'install -d' and permissions on nonexistent
directories in the middle of the path, but this is at
least no worse that it was before.

PR: bin/22661

Reorder one of the new features based on experience. Since the number
of files auto-installed during an upgrade from a really old system
can get quite long, and it's piped to the PAGER already, print
that first, then print any of the 4 two-line messages that might
apply.

Add several new features, reorder some code, and continue with the
process of making the script more cross platform friendly.

* Add -i option to automatically install files that do not exist
on the system already.
* Add the ability to specify DESTDIR.
* Allow the user to specify scripts to run right before the
comparison starts, and when mm is done. This will
allow the user to specify customized local behavior, and
implement features such as automatically deleting files.
* Document the above changes in the man page.
* Switch to using 'ident' for the CVS Id comparison, which
should help with portability, and makes it faster.
* Reorder, and in one case fix some code by doing things in
ways that make more sense.
* Check to see if the file exists on the system before doing
the comparisons. This saves CPU cycles, and streamlines
the auto-install process.

I used bits and pieces of suggestions and patches from various
people, ultimately too numerous to name. Which is not to say
that they were not both appreciated, and helpful in achieving
the ultimate result.

More changes that don't modify functionality.

* More whitespace
* Change read -p to echo -n/read to help support portability
* Genericize an informational message regarding /.cshrc and /.profile
for the same reason

Whitespace only. Make the style of the contributed sections consistent with
the rest of the file. (Read, put it back the way I like it.)

Make mergemaster ignore files in /dev if /dev is a devfs.

Approved by: Doug Barton <Doug@gorean.org>

Use new aliases path (/etc/mail/aliases)

Approved by: Doug Barton <DougB@gorean.org>

Be more careful about selecting a PAGER.
Improve the question asked when the PAGER is invalid.

Submitted by: Doug Barton <DougB@gorean.org>

Actually accept 'm' as a PAGER query response to mean 'more'.

Make mergemaster only take single letters for options as well as loop
asking a question again if given an invalid input instead of assuming
what the user wants. /etc is not the place to make assumptions when
given invalid input.

Reformat some of the more convoluted code into seperate functions instead
of being inline using tabs instead of space indents.

Allow the user to view merged files they created with sdiff.

Allow the user to redisplay the diff between the installed file and
the new file again.

Time wasted waiting for review: 1 month 2 weeks

Use the new (Hi rwatson!) NO_MAKEDEV when making our temproot.

Submitted by: Doug Barton <Doug@gorean.org> (maintainer)
Approved by: you-know-who.

Allow $PAGER to have arguments.

Submitted by: brian
Reviewed by: Doug Barton <Doug@gorean.org>
Approved by: jkh

Add IGNORE_MOTD to allow the sysadmin to ignore (likely) differences in
the FreeBSD default motd and the system motd

Submitted by: author

Use $FreeBSD$ (instead of the author's $Revision$) to report the
version number now that we have it.

Reviewed by: author

Add $FreeBSD$

This commit was generated by cvs2svn to compensate for changes in r52400,
which included commits to RCS files with non-trunk default branches.

Import mergemaster into the tree, a product of freebsdcon.

Submitted by: Doug Barton <doug@gorean.org>