MFC r352261,r352262,r352265:

r352261:
mtree: Fix -f -f not considering type changes.
r352262:
mtree -c: Fix username logic when getlogin(3) fails.
r352265:
mtree -O: Fix not descending on hash collisions

Relnotes: yes

MFC r314659,r314676:

r314659:

usr.sbin: normalize paths using SRCTOP-relative paths or :H when possible

This simplifies make logic/output

r314676:

Fix build after r314656

Some of the changes I introduced to use .ALLSRC were correct in spirit,
but incorrect in reality -- in particular, ../Makefile.inc hadn't been
pulled in via bsd.init.mk (via bsd.lib.mk, bsd.prog.mk), so the value
of .ALLSRC (evaluated immediately) was empty. .include bsd.init.mk
explicitly so we can be certain that the values used as dependencies in
the targets are defined when the target recipe has been evaluated.

Reminder: thou shalt separate out separate functional changes before
committing them.

(YUGE) Pointyhat to: ngie
In collaboration with: bdrewery

MFC r311522:
Use type-independent formats for printing nlink_t and ino_t.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Use NULL instead of 0 for pointers.

fopen(3) returns NULL in case it can't open the STREAM.
fgetln(3) returns NULL if it can't get a line from a STREAM.
malloc returns NULL if it can't allocate memory.

Rename usr.sbin/mtree to usr.sbin/fmtree since it is not what /usr/sbin/mtree
actually is (which is usr.sbin/nmtree).

This has tricked me numerous times.

Add META_MODE support.

Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision: D2796
Reviewed by: brooks imp

Add a ${CP} alias for copying files in the build.

Some users build FreeBSD as non-root in Perforce workspaces. By default,
Perforce sets files read-only unless they're explicitly being edited.
As a result, the -f argument must be used to cp in order to override the
read-only flag when copying source files to object directories. Bare use of
'cp' should be avoided in the future.

Update all current users of 'cp' in the src tree.

Reviewed by: emaste
MFC after: 1 week
Sponsored by: Spectra Logic

-u and -U are supposed to affect existing files too.

Reviewed by: marcel

Convert usr.sbin to LIBADD
Reduce overlinking

Fix incremental builds involving non-root users with read-only source files.

Makefiles should not assume that source files can be overwritten. This is the
common case for Perforce source trees.

This is a followup commit to r211243 in the same vein.

MFC after: 1 month
Sponsored by: Spectra Logic
MFSpectraBSD: r1036319 on 2014/01/29, r1046711 on 2014/03/06

Merge from CheriBSD:
commit 70b8f0c127db6b80411789d237b403cc64a93573
Author: Brooks Davis <brooks@one-eyed-alien.net>
Date: Mon Jan 27 22:53:57 2014 +0000

Move mtree.5 to usr.sbin/nmtree.
Remove note that mtree 2.0 format files aren't supported.

MFC after: 4 weeks
Sponsored by: DARPA, AFRL

Merge from CheriBSD:
commit 6b569451b92c48ccf1768da32e7e89189e1aa253
Author: Brooks Davis <brooks@one-eyed-alien.net>
Date: Mon Jan 27 22:50:46 2014 +0000

Always install nmtree as mtree.
For compability, link mtree to nmtree.

X-MFC after: never
Sponsored by: DARPA, AFRL

Do not emit size for non-regular files. There is nothing that
mtree(1) can do in this situation and would cause confusion.

MFC candidate.

Approved by: re (hrs)

Always install our mtree as /usr/sbin/fmtree and link it as
/usr/sbin/mtree by default.

Add a src.conf option WITH_NMTREE that causes NetBSD's mtree to be linked
as /usr/sbin/mtree as well as /usr/sbin/nmtree.

More -Wmissing-variable-declarations fixes.

In addition to adding missing `static' keywords:
- bin/dd: Pull in `extern.h' to guarantee consistency with source file.
- libexec/rpc.rusersd: Move shared globals into an extern.h.
- libexec/talkd: Move `debug' and `hostname' into extern.h.
- usr.bin/cksum: Put counters in extern.h, as they are used by ckdist/mtree.
- usr.bin/m4: Move `end_result' into extern.h.
- usr.sbin/services_mkdb: Move shared globals into an extern.h.

Remove superfluous paragraph macro.

Replace index() and rindex() calls with strchr() and strrchr().

The index() and rindex() functions were marked LEGACY in the 2001
revision of POSIX and were subsequently removed from the 2008 revision.
The strchr() and strrchr() functions are part of the C standard.

This makes the source code a lot more consistent, as most of these C
files also call into other str*() routines. In fact, about a dozen
already perform strchr() calls.

- fix duplicate "a a" in some comments

Submitted by: eadler
Approved by: simon
MFC after: 3 days

Wording fixes.

PR: 143454
Submitted by: Warren Block <wblock@wonkity.com>
MFC after: 3 days

mdoc: move CAVEATS, BUGS and SECURITY CONSIDERATIONS sections to the
bottom of the manpages and order them consistently.

GNU groff doesn't care about the ordering, and doesn't even mention
CAVEATS and SECURITY CONSIDERATIONS as common sections and where to put
them.

Found by: mdocml lint run
Reviewed by: ru

Keep the list of default keywords sorted so that paranoid folks like me
don't try to find a hidden meaning in the strange order. The list used
to be sorted in rev. 1.1 from 4.4BSD but the order was broken as soon as
in rev. 1.2 by a single-character fix.

MFC after: 3 days

o Remove a duplicate reference to -f flag in the SYNOPSYS section.
o Improve wording.

PR: docs/144407
Submitted by: gcooper
MFC after: 1 week

Change all our own code to use st_*tim instead of st_*timespec.

Also remove some local patches to diff(1) which are now unneeded.

Fix common misspelling of hierarchy

Pointed out by: bf1783 at gmail
Approved by: np (cxgb), kientzle (tar, etc.), philip (mentor)

The last big commit: let usr.sbin/ use WARNS=6 by default.

Write timestamps with exactly 9 digits after the period.
This ensures that the value written is both compatible with
older mtree versions (which expect the value after the period
to be an integer count of nanoseconds after the whole second)
and is a correct floating-point value.

Leave the parsing code unchanged so it will continue to read
older files.

Accept integer times. Previously, the field "time=1233294539" would be
rejected as invalid.

Add the mtree.5 manpage. I'll come back soon and
remove the format specification from mtree.8.
I also need to reconcile a few issues between this
mtree.5 and what is actually implemented in FreeBSD's
mtree utility.

MFC after: 30 days

When applying a spec, traverse the existing directory tree in lexical
order. This allows direct comparison of the output of two different
runs, regardless of the order in which readdir(2) returns directory
entries.

MFC after: 3 weeks

mtree should use getcwd() instead of getwd(). getcwd() is safer,
because it includes checks for the length of the buffer it uses.

Reviewed by: phk, cperciva
MFC after: 3 days

o Add an example how to create /etc/mtree style mtree(8) files.

PR: docs/113667
Submitted by: edwin
MFC after: 1 week

Markup fixes.

o Add an 'optional' keyword, which allows files to be in the
specification, but not in the file hierarchy.

PR: bin/99531
Submitted by: skv
Obtained from: NetBSD, originally from Ed Symanzik
Regress. test: test/test05.sh
MFC after: 1 month

If we fail in chown(2), try to just change the group and continue on to
change the permissions. Failures are still recorded.

This allows mtree to do a generally better job of things when uid != 0.

Sponsored by: ActiveState/Sophos
Partially submitted by: neilw at ActiveState dot com
Reviewed by: neilw at ActiveState dot com
MFC after: 3 weeks

Fix all the spelling mistakes I could find in the man pages for words
that have at least 3 characters.

MFC after: 1 week
Thanks to: Music band ``Chingon''
for keeping me company while searching for these.

Markup fixes.

Approved by: re (blanket)

Adds sha256 support. While it is a good idea to also add sha384 and
sha512, I did not do that since it is not entirely clear where "the one
true place" to hold their implementations is going to be. Sha256 is
different since mtree already links against libmd.

Make recommended procedure for integrity checking in the manpage
consistent.

Fix a bug with -f spec1 -f spec2 comparison, which prevented
test/tes03.sh from running successfully.

Reviewed by: phk, cperciva

Expand *n't contractions.

Sort sections.

Added the EXIT STATUS section where appropriate.

Remove extra "*/".

Per letter dated July 22, 1999 remove 3rd clause of Berkeley derived software
(with permission of addtional copyright holders where appropriate)

Eliminated double whitespace.

Mechanically kill hard sentence breaks.

Improve spec to spec comparison output.

Added forgotten -w to the usage().

Added forgotten -w to the SYNOPSIS.
Trim whitespace at EOL.

Get rid of "lineno: 0" in error messages from create mode.

Add -w(arning) flag which makes failure to look up uid/gid
a nonfatal warning instead of an error.

collect all "extern bla" stuff in extern.h in order to get cross-file
type checking. No bugs found.

Do not emit comments when -n is specified.

The new "spec against spec" comparison code was yet another time
victim of the special sort order employed where files come before
directories and alphabetic inside these two groups.

Approved by: re@

Off by one error in malloc.

Approved by: re@

Fix a bug relating to the "files before directories" sort order when
comparing two spec files.

Add a small testcase for the new -f-f mode.

Give mtree(8) the ability to take [two -f spec] arguments and compare
the two specifications to each other.

Rename spec() and verify() functions to have mtree_ prefix and take
explicit file descriptor instead of assuming stdin.

Take advantage of the fact that we know the err(3) function does not
return to eliminate some else clauses and gain some "outdenting".

Fixed breakage of my world in rev.1.30. #include <sys/time.h> instead of
depending on namespace pollution in <sys/stat.h> for the declarations of
struct timeval and utimes().

Fixed some style bugs in rev.1.30 and some nearby style bugs (mainly
unsorting and missing or extra blank lines).

Removed a wrong comment that was obtained from NetBSD in rev.1.14. It said
that chflags() reset the times that were set "above" by utimes(), but
utimes wasn't "above" in FreeBSD until rev.1.30, and chflags() does't
actually reset the times.

Make mtree's update function update the modification time too.

Submitted by: Dan Nelson <dnelson@allantgroup.com>
PR: 53063

Update test-case to also fail on filenames containing '#'

Use vis(3) to encode/decode the target for symlinks.

Make vis(3) decode errors fatal for both symlink targets and fienames.

Add another test-script for mtree(8): Check symlinks with space in them.

Fix bug in mtree(8)'s create mode where files or directories with had
names containing glob(3) expressions would appear verbatim in the
output.

If such an mtree file were used by mtree in update mode, wrong things
would happen.

Add a test script which Mtree fails.

Remove clause 3 in the copyrights (according to permission in src/COPYRIGHT)
Remove trailing whitespace.

Remove "register" keywords.
Use newspeak functions definitions.

Get rid of duplicates.

Use __FBSDID over rcsid[]. Protect copyright[] where needed.

The -s option should not assume -U.

MFC After: 2 weeks

- Align the function prototype of the external `crc' function with
how `crc' is actually defined.
- Remove an unnecessary `extern' variable declaration.
Data type corrections:
- Define a variable which contains a file byte offset value as type
off_t as required by the `crc' function.
- Change the type of a variable carrying a CRC checksum from `u_long'
to `uint32_t'.
- Substitute the wrong `extern' variable declaration of `crc_total'
by putting a correct one in the shared header extern.h.
`crc_total' is defined as an `uint32_t', thus fixing
incorrect mtree checksums on big-endian LP64 machines.

Fix two minor mismatch reporting output errors.

Reviewed by: phk
MFC After: 3 weeks

Uniformly refer to a file system as "file system".

Approved by: re

mtree(8) wasn't a bootstrap tool since 2000/07/23 (Makefile.inc1,v 1.161).

Make the threatened fts(3) ABI fix. FTSENT now avoids the use of the struct
hack, thereby allowing future extensions to the structure (e.g., for extended
attributes) without rebreaking the ABI. FTSENT now contains a pointer to the
parent stream, which fts_compar() can then take advantage of, avoiding the
undefined behavior previously warned about. As a consequence of this change,
the prototype of the comparison function passed to fts_open() has changed
to reflect the required amount of constness for its use. All callers in the
tree are updated to use the correct prototype.

Comparison functions can now make use of the new parent pointer to access
the new stream-specific private data pointer, which is intended to assist
creation of reentrant library routines which use fts(3) internally.

Not objected to in spirit by: -arch

Get rid of %qd, and unbreak Alpha build.

The .Nm utility

fixup for WARNS=4.

de-__P()

Assume __STDC__, remove non-__STDC__ code.

Consistancy check: s/file system/filesystem/g

Typo.

Only compile with strtofflags.c from libc if bootstrapping.

Fix the bootstrapping aid from revision 1.9.
WORLD was renamed to BOOTSTRAPPING.

mdoc(7) police: utilize the new .Ex macro.

can not -> cannot

Perform a major cleanup of the usr.sbin Makefiles.
These are not perfectly in agreement with each other style-wise, but they
are orders of orders of magnitude more consistent style-wise than before.

Remove whitespace at EOL.

Mark a static function as __printflike()

MFC After: 1 week

- Backout botched attempt to introduce MANSECT feature.
- MAN[1-9] -> MAN.

Set the default manual section for usr.sbin/ to 8.

Convert to use the <sys/queue.h> macros rather than fiddling with the queue
structure internals.

Reviewed by: markm

Prepare for mdoc(7)NG.

mdoc(7) police: use the new features of the Nm macro.

Avoid use of direct troff requests in mdoc(7) manual pages.

Don't try to compare other attributes if the type doesn't match, it's
nothing but trouble, and no use anyway.

Don't whine about modes on symlinks.

Make the output of mtree(8) more systematic and machine readable.

The new format is:

filename {changed,missing,extra}
$field expected $foo found $bar
...

Fix various bugs along the way:
Don't complain about directory sizes differing.
Correctly check flags.

Re-add -P option (for POSIX-like command line interface)

Restore standard symlink default, return -L, remove -P, add missing -q to getopt
This is part of whole subsystem fixing

Reviewed by: imp

Pass a useful pathname to rlink(). Many thanks to the PR originator
for all the debugging work.

PR: 21017
Reported by: Gerhard Sittig <Gerhard.Sittig@gmx.net>
Submitted by: Jos Backus <josb@cncdsl.com>,

Add -q. this will cause the warning messages about missing
directories to not be printed. This is from OpenBSD (and I think
NetBSD also) and makes our mtree more compatible with other BSDs.
This makes cross compilation easier than it was before. Other changes
will be needed to allow NetBSD or OpenBSD to cross build on FreeBSD,
but this is a start.

Reviewed by: andrey
Obtained from: OpenBSD
Concentric Red Circles by: My own stupidity

Revert the reversion of the default mtree behaviour. The mtree
command now follows symlinks again.

Requested by: jkh

First attempt to fix a buildworld from -stable machine. Explicitly
include strtofflags.c from lib/libc/gen when building mtree until the
current dependency mess with mtree is fixed.

Correct the usage() message as per rev 1.25 of mtree.8 .

* Order options in the SYNOPSIS correctly.
* Re-order the list of options in teh DESCRIPTION as per
the SYNOPSIS.
* Move the description of exit conditions from the
DESCRIPTION section to a new DIAGNOSTICS section.
* Typo fix: "effect" -> "affect" when used as a verb.
* Clear the Nm macro as appropriate.
* Typo fix: "consider" -> "considers" for a singular subject.
* Use Nx instead of NetBSD.

Fix getopt string I forget

Revert back to original mtree default behaviour to not follow symlinks, to be
compatible with other *BSD camp. Add -L option to follow symlinks, so remove
-P option which is now default. The next step will be to add -L to building
process.

Asked-by: bde

Fix the NAME section. Order the options which do not take arguments,
add the -P option (introduced in rev 1.22), and use mdoc word keeps
(Bk -words / Ek) for those options which do take arguments.

Use -P instead of -S to specify that no symlinks should be followed.
There's no reason to make it different from cp(1), chmod(1), chown(1)
etc.

Requested by: sheldonh

Allow symlinks to be created if they appear in the mtree specification.

PR: bin/14171

Added a flag (-S) to mtree to specify that symlinks shouldn't be
followed. This allows type=link matches to work properly and match
the link itself, instead of the file/directory that the link is
pointing to.

Switch over to using the new fflagstostr and strtofflags library calls.

Use .Fx

Back out the previous change to the queue(3) interface.
It was not discussed and should probably not happen.

Requested by: msmith and others

Change the way that the queue(3) structures are declared; don't assume that
the type argument to *_HEAD and *_ENTRY is a struct.

Suggested by: phk
Reviewed by: phk
Approved by: mdodd

Punctuation fix.

Add used includes.

Add excludes.c (missed in previous commit).

PR: 16944

Give mtree the ability to exclude files and directories from its traversal.

PR: 16944

Revert part of the last commit, remove {g|s}etflags from the libc
interface, and statically link them to the programs using them.
These functions, upon reflection and discussion, are too generically
named for a library interface with such specific functionality.
Also the api that they use, whilst ok for private use, isn't good
enough for a libc function.

Additionally there were complications with the build/install-world
process. It depends heavily upon xinstall, which got broken by
the change in api, and caused bootstrap problems and general mayhem.

There is work in progress to address future problems that may be
caused by changes in install-chain tools, and better names for
{g|s}etflags can be derived when some future program requires them.
For now the code has been left in src/lib/libc/gen (it started off
in src/bin/ls).

It's important to provide library functions for manipulating file
flag strings if we ever want this interface to be adopted outside
of the source tree, but now isn't necessarily the right moment
with 4.0-release just around the corner.

Approved: jkh

Historically file flags (schg, uschg, etc) have been converted from
string to u_long and back using two functions, flags_to_string and
string_to_flags, which co-existed with 'ls'. As time has progressed
more and more other tools have used these private functions to
manipulate the file flags.

Recently I moved these functions from /usr/src/bin/ls to libutil,
but after some discussion with bde it's been decided that they
really ought to go in libc.

There are two already existing libc functions for manipulating file
modes: setmode and getmode. In keeping with these flags_to_string
has been renamed getflags and string_to_flags to setflags.

The manual page could probably be improved upon ;)

Added missing DPADD's. Removed unrequired SRCS's.

Obtained from: bde

Moved flags_to_string and string_to_flags into libutil. It's used in
many places nowadays.

Added support for file flags, mostly merged from the NetBSD version.
The way is now open to schg and sappnd key files and directories in
our tree. There are recommendations in bin/15229.

PR: bin/15229
Reviewed by: imp, brian

-Wall fixes.

Submitted by: nrahlstr

$Id$ -> $FreeBSD$

When bootstrapping mtree, don't depend on there being a libmd at all.

mtree is a "bootstrap tool", don't use the fancy new world-breaking features
if we're building it as part of the bootstrap toolset.

Add support for SHA-1 and RIPEMD160, now that libmd includes them. Make
all of the hashes (including MD5) conditionalized in case we want
to turn one of them off later.

Move to a uniform '\ooo' (VIS_OCTAL) encoding for special characters.
The previous commit broke mtree(8) when file names using certain 8-bit
characters ended up being encoded with '/', '*', and other shell
metacharacters.

PR: bin/9538
Submitted by: "Eugene M. Kim" <astralblue@usa.net>
Reviewed by: jkoshy
Bug-missed-last-time-by: jkoshy

Teach mtree(8) to handle whitespace in filenames. Uses strvis(3).

PR: bin/3246
Submitted by: Eugene M. Kim <astralblue@usa.net>
Reviewed by: jkoshy, wollman

Free memory from getmode.

Obtained from: OpenBSD

Fixed printf format errors.

There is no need to make nochange imply ignore as well.

Suggested by: bde

Use correct size for readlink buffer. This is the size of the
buffer -1 since readlink adds its own NUL to the end.

Inspired by: Similar changes in OpenBSD

Create a new flag, "nochange", which will specify that a directory is
to be created if it's missing, otherwise completely ignore it's modes and
owners. Primary intended targets: /usr/src and /usr/obj.

Adjust the 'not created: File exists' message to mention that it's a
directory that's the problem, otherwise it doesn't make sense.

I had created chown-style -L and -P flag to control logical/physical mode
(ie: whether symlinks were followed), but the nochange flag is enough to
get the blasted thing out of my hair so I took them back out.

Use err(3) instead of local redefinition.

Change FTS_PHYSICAL (not follow symlinks but return them)
to FTS_LOGICAL (follow symlinks and return their targets)

Because
1) In the real system there is a lot of symlinks nowdays over
/var, /spool, etc. Mtree can't do anything for them, i.e. change
permissions/owner. I.e. if you have /var/mail -> /somewhere
it will never be changed to group "mail" 775 like -current does now,
symlink totally confuses mtree.

2) Mtree can't deal with symlinks properly in any case (they are created
separately in other places now), so we lost no functionality by making
them invisible for mtree.

compare return value from getopt against -1 rather than EOF, per the final
posix standard on the topic.

Revert $FreeBSD$ to $Id$

Sort cross references.

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

ts_sec -> tv_sec
ts_nsec -> tv_nsec

Pointed out by: bde

Correct a bunch of man page cross references and generally
try and silence "manck".

ncurses, rpc, and some of the gnu stuff are still a big mess, however.

Fix a bunch of spelling errors in a bunch of man pages.

st_size in struct stat has type off_t (long long) and not unsigned long
change strtoul() to strtoq()

Use p->fts_level instead of unsuccessfully trying to keep track of the
level ourself. We failed for unreadable directories. E.g.,
`mtree -d -f /etc/mtree/BSD.usr.dist -p /usr' run by `nobody' was
confused after it couldn't descend into /usr/games/hide. It looked
for /usr/include and subsequent directories in /usr/games.

Don't search for `extra' files when the spec depth is less than the
fts level. The spec depth isn't incremented for leaf nodes because
that would give a NULL level pointer and make it inconvenient to go
back to the parent level. Leaf nodes are built for directories that
are empty in the spec. Since they are empty in the spec, all files
in them are extra. The search looked for files one spec level
too high, so for `mtree -d -f /etc/mtree/BSD.usr.dist -p /usr',
obj/sbin matched ./sbin and wasn't considered extra, so it was
descended into and lots of bogus extra things in it were found.
This was harmful for `mtree -U' (as reported in pr623) and worse
for `mtree -r'.

Use rmdir(), not unlink(), to remove `extra' directories. unlink()
succeeds for root but unlinking directories normally damages the
file system.

Report `fts_errno' instead of `errno' when the former applies.

Report `fts_errno' instead of `errno' when the former applies.

Fix to match new MD5 api, faster.

Remove trailing whitespace.

mtree * MD5 = better security

Idea from: Dr. John T. Wroclawski

Add -U option which does the same things as -u except exits with 0
instead of 2 on MISMATCH.

Fixed bug caused by missing set of paranthesis when checking the modification
time. This caused differences in the nanosecond portion of the time to
screwup the logic and made mtree think it should compare the modtime even
though "time" wasn't specified. The end result is that mtree couldn't deal
with procfs because procfs sets the mtime with high accuracy via microtime().

My optimization to eliminate /set records was a little too aggressive, it
optimized away the first one :-(.

1. Remove commented out CFLAGS+=-DDEBUG from Makefile

2. Clean up code so it compiles -Wall (except for sccsid's and copyright).
This included fixing several printf formats that where not correct,
and changing the data types of a few things.

3. Implement new option -i that produces indented mtree output files.

4. Implement new option -n that turns off directory comments.

5. Only emit /set records if something has changed since the last one.

fixed typos
Reviewed by:
Submitted by:

Get rid of update. Make man page installation work with our scheme
(and rename a few in the process).

This commit was generated by cvs2svn to compensate for changes in r1553,
which included commits to RCS files with non-trunk default branches.