Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r241010: libc/fts: Use O_CLOEXEC for internal file descriptors.

Because fts keeps internal file descriptors open across calls, making such
descriptors close-on-exec helps not only multi-threaded applications but
also single-threaded applications.

In particular, this prevents passing a temporary file descriptor for saving
the current directory to processes created via find -exec.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

From fts.c comment:

The "FTS_NOSTAT" option can avoid a lot of calls to stat(2) if it knows that a
directory could not possibly have subdirectories. This is decided by looking at
the link count: a subdirectory would increment its parent's link count by
virtue of its own ".." entry. This assumption only holds for UFS-like
filesystems that implement links and directories this way, so we must punt for
others.

It looks like ZFS is a UFS-like file system, as the above also holds for ZFS.
Add ZFS to the list of file systems that allow for such optimization.

MFC after: 1 month

Our fts(3) API, as inherited from 4.4BSD, suffers from integer
fields in FTS and FTSENT structs being too narrow. In addition,
the narrow types creep from there into fts.c. As a result, fts(3)
consumers, e.g., find(1) or rm(1), can't handle file trees an ordinary
user can create, which can have security implications.

To fix the historic implementation of fts(3), OpenBSD and NetBSD
have already changed <fts.h> in somewhat incompatible ways, so we
are free to do so, too. This change is a superset of changes from
the other BSDs with a few more improvements. It doesn't touch
fts(3) functionality; it just extends integer types used by it to
match modern reality and the C standard.

Here are its points:

o For C object sizes, use size_t unless it's 100% certain that
the object will be really small. (Note that fts(3) can construct
pathnames _much_ longer than PATH_MAX for its consumers.)

o Avoid the short types because on modern platforms using them
results in larger and slower code. Change shorts to ints as
follows:

- For variables than count simple, limited things like states,
use plain vanilla `int' as it's the type of choice in C.

- For a limited number of bit flags use `unsigned' because signed
bit-wise operations are implementation-defined, i.e., unportable,
in C.

o For things that should be at least 64 bits wide, use long long
and not int64_t, as the latter is an optional type. See
FTSENT.fts_number aka FTS.fts_bignum. Extending fts_number `to
satisfy future needs' is pointless because there is fts_pointer,
which can be used to link to arbitrary data from an FTSENT.
However, there already are fts(3) consumers that require fts_number,
or fts_bignum, have at least 64 bits in it, so we must allow for them.

o For the tree depth, use `long'. This is a trade-off between making
this field too wide and allowing for 64-bit inode numbers and/or
chain-mounted filesystems. On the one hand, `long' is almost
enough for 32-bit filesystems on a 32-bit platform (our ino_t is
uint32_t now). On the other hand, platforms with a 64-bit (or
wider) `long' will be ready for 64-bit inode numbers, as well as
for several 32-bit filesystems mounted one under another. Note
that fts_level has to be signed because -1 is a magic value for it,
FTS_ROOTPARENTLEVEL.

o For the `nlinks' local var in fts_build(), use `long'. The logic
in fts_build() requires that `nlinks' be signed, but our nlink_t
currently is uint16_t. Therefore let's make the signed var wide
enough to be able to represent 2^16-1 in pure C99, and even 2^32-1
on a 64-bit platform. Perhaps the logic should be changed just
to use nlink_t, but it can be done later w/o breaking fts(3) ABI
any more because `nlinks' is just a local var.

This commit also inludes supporting stuff for the fts change:

o Preserve the old versions of fts(3) functions through libc symbol
versioning because the old versions appeared in all our former releases.

o Bump __FreeBSD_version just in case. There is a small chance that
some ill-written 3-rd party apps may fail to build or work correctly
if compiled after this change.

o Update the fts(3) manpage accordingly. In particular, remove
references to fts_bignum, which was a FreeBSD-specific hack to work
around the too narrow types of FTSENT members. Now fts_number is
at least 64 bits wide (long long) and fts_bignum is an undocumented
alias for fts_number kept around for compatibility reasons. According
to Google Code Search, the only big consumers of fts_bignum are in
our own source tree, so they can be fixed easily to use fts_number.

o Mention the change in src/UPDATING.

PR: bin/104458
Approved by: re (quite a while ago)
Discussed with: deischen (the symbol versioning part)
Reviewed by: -arch (mostly silence); das (generally OK, but we didn't
agree on some types used; assuming that no objections on
-arch let me to stick to my opinion)

Forced commit to note repo-copy:

These files have been repo-copied from src/include/fts.h
and src/lib/libc/gen/fts.c to serve as a base for 4.4BSD
compatible versions of fts(3) functions to be preserved
through libc symbol versioning while the default versions
undergo ABI-breaking extension to support big file trees.

Approved by: re (kensmith)

Per Regents of the University of Calfornia letter, remove advertising
clause.

# If I've done so improperly on a file, please let me know.

In fts_build(), if we try to chdir and fail (e.g. due to lack of search
permission), try to continue in FTS_DONTCHDIR mode. Of course this
won't work for long paths, but we can't descend more than one pathname
component beyond the directory anyway if we lack search permission.

Here is a transcript demonstrating the change, where oldls is ls(1)
linked with the old fts(3):

das@VARK:~> mkdir t && touch t/{a,b,c} && chmod u-x t
das@VARK:~> oldls t
a b c
das@VARK:~> oldls -l t
das@VARK:~> \ls t
a b c
das@VARK:~> \ls -l t
ls: a: Permission denied
ls: b: Permission denied
ls: c: Permission denied

I had forgotten about this patch until bde reminded me. He reports
using it without problems for over a year.

PR: 45723

Fixed some minor style bugs.

Fix some^Wseveral style bugs from last commit.

Remove "sys/types.h" as "sys/param.h" is already included

Use cast rather than back-pointer to convert from public to private
version of FTS data, and so avoid littering fts.h with any of the
details.

Pointed out By: bde, kientzle

The FTS_NOSTAT option is an optimisation that reduces the number
of stat(2) calls by keeping an eye of the number of links a directory
has. It assumes that each subdirectory will have a hard link to its
parent, to represent the ".." node, and stops calling stat(2) when
all links are accounted for in a given directory.

This assumption is really only valid for UNIX-like filesystems: A
concrete example is NTFS. The NTFS "i-node" does contain a link
count, but most/all directories have a link count between 0 and 2
inclusive. The end result is that find on an NTFS volume won't
actually traverse the entire hierarchy of the directories passed
to it. (Those with a link count of two are not traversed at all)

The fix checks the "UFSness" of the filesystem before enabling the
optimisation.

Reviewed By: Tim Kientzle (kientzle@)

A minor refactoring to simplify portability: assign the filename
length to a separate variable so that it will be easier to adapt to
systems that don't have d_namlen in struct dirent.

Remove an unused variable: `sb' in fts_read().

Make the threatened fts(3) ABI fix. FTSENT now avoids the use of the struct
hack, thereby allowing future extensions to the structure (e.g., for extended
attributes) without rebreaking the ABI. FTSENT now contains a pointer to the
parent stream, which fts_compar() can then take advantage of, avoiding the
undefined behavior previously warned about. As a consequence of this change,
the prototype of the comparison function passed to fts_open() has changed
to reflect the required amount of constness for its use. All callers in the
tree are updated to use the correct prototype.

Comparison functions can now make use of the new parent pointer to access
the new stream-specific private data pointer, which is intended to assist
creation of reentrant library routines which use fts(3) internally.

Not objected to in spirit by: -arch

* Remove __P and convert to ANSI prototypes.
* Remove 'register'. (some functions had 7+ register functions...)
* Fix SCM ID's.

Removed duplicate VCS ID tags, as per style(9).

Fix another unprotected instance of chdir() by extending the
fts_safe_changedir() function and using that instead for both of the
chdir()s.

Partially submitted by: Todd Miller <millert@OpenBSD.org>, bde

When doing the chdir("..") in the !FTS_NOCHDIR case, stat() after we get
there and compare the inode and device numbers to the values we remember,
to guard against the directory having been moved around in the meantime.

Reported by: Nick Cleaton <nick@cleaton.net>

Remove _THREAD_SAFE and make libc thread-safe by default by
adding (weak definitions to) stubs for some of the pthread
functions. If the threads library is linked in, the real
pthread functions will pulled in.

Use the following convention for system calls wrapped by the
threads library:
__sys_foo - actual system call
_foo - weak definition to __sys_foo
foo - weak definition to __sys_foo

Change all libc uses of system calls wrapped by the threads
library from foo to _foo. In order to define the prototypes
for _foo(), we introduce namespace.h and un-namespace.h
(suggested by bde). All files that need to reference these
system calls, should include namespace.h before any standard
includes, then include un-namespace.h after the standard
includes and before any local includes. <db.h> is an exception
and shouldn't be included in between namespace.h and
un-namespace.h namespace.h will define foo to _foo, and
un-namespace.h will undefine foo.

Try to eliminate some of the recursive calls to MT-safe
functions in libc/stdio in preparation for adding a mutex
to FILE. We have recursive mutexes, but would like to avoid
using them if possible.

Remove uneeded includes of <errno.h> from a few files.

Add $FreeBSD$ to a few files in order to pass commitprep.

Approved by: -arch

Fix style bugs (including ones introduced from OpenBSD).

Simplify sytem call renaming. Instead of _foo() <-- _libc_foo <-- foo(),
just use _foo() <-- foo(). In the case of a libpthread that doesn't do
call conversion (such as linuxthreads and our upcoming libpthread), this
is adequate. In the case of libc_r, we still need three names, which are
now _thread_sys_foo() <-- _foo() <-- foo().

Convert all internal libc usage of: aio_suspend(), close(), fsync(), msync(),
nanosleep(), open(), fcntl(), read(), and write() to _foo() instead of foo().

Remove all internal libc usage of: creat(), pause(), sleep(), system(),
tcdrain(), wait(), and waitpid().

Make thread cancellation fully POSIX-compliant.

Suggested by: deischen

Add three-tier symbol naming in support of POSIX thread cancellation
points. For library functions, the pattern is __sleep() <--
_libc_sleep() <-- sleep(). The arrows represent weak aliases. For
system calls, the pattern is _read() <-- _libc_read() <-- read().

Switch over to the OpenBSD fts.c, fixing lots of things.

Obtained from: OpenBSD

Fix the root cause of the fts buffer overflow. This is a temporary
patch to stop the core dumps while others come up with a better
reviewed patch which may also fix other problems. We do illegal
pointer arithmetic, but it should be OK since FreeBSD only supports
machines with flat address spaces.

Submitted by: bde

Reverted to revision 1.8 as previous fix causes fts_open with with a
path name argument with a trailing '/' to fail.

Reviewed by: phk

Reviewed by: phk
When fts_open is used with option FTS_NOCHDIR the full
path entry of type FTS_DP is returned with a trailing
'/' if the final directory is empty.
This fix coresponds to netbsd's __fts13.c v. 1.16

Replace memory leaking instances of realloc with non-leaking reallocf.
In some cases replace if (a == null) a = malloc(x); else a =
realloc(a, x); with simple reallocf(a, x). Per ANSI-C, this is
guaranteed to be the same thing.

I've been running these on my system here w/o ill effects for some
time. However, the CTM-express is at part 6 of 34 for the CAM
changes, so I've not been able to do a build world with the CAM in the
tree with these changes. Shouldn't impact anything, but...

fts_close calls free(sp), the ISSET(FTS_NOCHDIR) which is a macro that
references sp. The free needs to follow ISSET

PR: 7148
Reviewed by: phk
Submitted by: Ken Mayer <kmayer@freegate.com>

Apply fts() fix from PR#4593

Submitted by: Dmitrij Tejblum <dima@tejblum.dnttm.rssi.ru>

Merge in the safe chdir changes from Todd Miller's mods to the OpenBSD
tree. Also merge in fix to NetBSD PR #1495. These represent 1.3-1.9 in
the OpenBSD tree. Make minor KNF changes to new code (which is in the
OpenBSD as 1.10). This avoids the symlink race problems.

These patches should go into 2.2.5 before the ship if they don't
break anything in -current.
Reviewed by: Bruce Evans
Obtained from: OpenBSD

Merge from Lite2:
filesystem include updates, duplicate group suppression, cleanups,
filesystem whiteout support (unionfs), bidir popen().

General -Wall warning cleanup, part I.
Submitted-By: Kent Vander Velden <graphix@iastate.edu>

Remove trailing whitespace.

BSD 4.4 Lite Lib Sources