Add scandirat(3); from freebsd

To be used in httpd(8) shortly to prevent toctu issues.

This makes __fdopendir internally accessible to avoid unnecessary
syscalls in scandirat(3). Suggested & diff by guenther

suggested by & OK millert
tweak & OK guenther
OK tb, jca

This rides the libc crank.

Delete casts to off_t and size_t that are implied by assignments
or prototypes. Ditto for some of the char* and void* casts too.

verified no change to instructions on ILP32 (i386) and LP64 (amd64)
ok natano@ abluhm@ deraadt@ millert@

Wrap <dirent.h> so that internal calls go direct and they're all weak symbols

use 4x buffer size for opendir. speeds up large directory reading, and
increased memory use is minimal.
ok deraadt logan

Optimization, no functional change:
Save a cookie pointing to the very first entry in the DIR.dd_buf
cache buffer and use that to speed up seekdir(3) when rewinding
to that first entry.
No libc bump because DIR is an opaque struct.

When the optimization applies, which in particular it always does
for rewinddir(3) after reading less than about 500 entries,
seekdir(3) execution time drops from 100 to 0.05 milliseconds
on my i386 notebook. Other cases are not slowed down.

Based on an idea from and ok by guenther@.

Search the userland buffer of dirent structures before falling back to
getdents(2). No functional change, but considerable speedup in many
cases, see http://marc.info/?l=openbsd-tech&m=138369623117934 for some
approximate numbers.
ok guenther@, "good diagnosis" deraadt@

fdopendir() needs to set dd_curpos so that an immediate telldir() works.

pointed out by and ok matthew@

Switch time_t, ino_t, clock_t, and struct kevent's ident and data
members to 64bit types. Assign new syscall numbers for (almost
all) the syscalls that involve the affected types, including anything
with time_t, timeval, itimerval, timespec, rusage, dirent, stat,
or kevent arguments. Add a d_off member to struct dirent and replace
getdirentries() with getdents(), thus immensely simplifying and
accelerating telldir/seekdir. Build perl with -DBIG_TIME.

Bump the major on every single base library: the compat bits included
here are only good enough to make the transition; the T32 compat
option will be burned as soon as we've reached the new world are
are happy with the snapshots for all architectures.

DANGER: ABI incompatibility. Updating to this kernel requires extra
work or you won't be able to login: install a snapshot instead.

Much assistance in fixing userland issues from deraadt@ and tedu@
and build assistance from todd@ and otto@

Expose a bunch of new functionality from POSIX 2008: openat(2),
fchmodat(2), fstatat(2), mkdirat(2), mkfifoat(2), mknodat(2),
faccessat(2), fchownat(2), linkat(2), readlinkat(2), renameat(2),
symlinkat(2), unlinkat(2), utimensat(2), futimens(2), and
fdopendir(3).

"Minor" libc bump.

Tested in a bulk build by naddy@
Much help from guenther@, thib@, tedu@, oga@, and others.
ok deraadt@, naddy@

Implement fdopendir(3) and refactor opendir(3) and fdopendir(3) to use
a common __fdopendir() function. Also, take advantage of the new
O_DIRECTORY and O_CLOEXEC flags in opendir(3).

(Currently fdopendir(3) is commented out; it will be enabled shortly
alongside openat(2) et al.)

Tested by naddy@ in a bulk build.
tweaks and ok guenther@; stupid POSIX nit pointed out by oga@

__opendir2, DTF_NODUP, and __DTF_READALL can die. struct dirent
dd_flags is renamed to the placeholder position dd_unused so that
we can spot "broken software" which assumes we have Jan Simon Pendry's
union mounts (we don't have them, and won't have them ever again).
__opendir2 question spotted by matthew
verified to not break ports by sthen

Leave errno as-is if fstat(2) fails instead of masking the real
errno with ENOTDIR. From Tim van der Molen.

_FD_LOCK/UNLOCK() is libpthread specific and not needed for librthread, so
isolate its usage to libpthread only and replace with generic non-static
mutex support in the one place it is needed:
- remove _FD_LOCK/UNLOCK from lseek and ftruncate in libc and make the
functions weak so that libpthread can override with its own new
versions that do the locking.
- remove _thread_fd_lock/unlock() weak functions from libc and adjust
libpthread for the change.
- add generic _thread_mutex_lock/unlock/destroy() weak functions in libc
to support non-static mutexes in libc and add libpthread and librthread
implementations for them. libc can utilize non-static mutexes via the
new _MUTEX_LOCK/UNLOCK/DESTROY() macros. Actually these new macros can
support both static and non-static mutexes but currently only using
them for non-static.
- make opendir/closedir/readdir/readdir_r/seekdir/telldir() thread-safe
for both thread libraries by using a non-static mutex in the struct
_dirdesc (typedef DIR), utilizing it in the *dir functions and remove
remaining and incorrect _FD_LOCK/UNLOCK() use in libc.
- add comments to both thread libraries to indicate libc depends on the
current implementation of static mutex initialization. suggested by
marc@
- major bump libc and libpthread due to function removal, structure
change and weak symbol conversions.
okay marc@, tedu@

The buffer passed to getdirentries() must be at least as large as
the filesystem block size. Previously we used a constant (1024)
rounded to the page size. Now we use the filesystem's block size,
rounded up to the nearest page.

telldir/seekdir optimization, avoid scanning the complete list,
while still returning the correct index for seekdir(); telldir()
Tested by Fred Crowson and others; "put it in" deraadt@

- Plug huge mem leak; mostly samba was suffering.
- Fix semantics: seekdir(pos); telldir() shoud return pos. The code
that implements this will be made faster in a later commit.
- We loose documented behaviour (after closedir() the telldir()
positions are not valid anymore). This was never in Posix, and most
other systems have nothing like it.

Diff originally from Paul Thorn, rewritten by me using some FreeBSD
code. "slap it in" deraadt@

fix a few warnings, keep the crypt stuff for later.
okay otto@

zap remaining rcsid.

Kill old files that are no longer compiled.

okay theo

bye bye whiteouts

union is gone, ok millert@

Set errno right before return(), so that it does not risk being reset
by any other functions we will invoke in-between.

Fix from Andrey Matveev <andrushock@korovino.net>

ansify function definitions and zap some `register'

ok millert@

Remove the advertising clause in the UCB license which Berkeley
rescinded 22 July 1999. Proofed by myself and Theo.

malloc paranoia; it is unlikely that any filesystem will support
enough directory entries to cause a problem but it is good form anyway.
deraadt@ OK.

Use qsort() instead of mergesort() since the latter can fail due
to malloc(). opendir() requires a stable sort so we rig the compare
routine to never return 0. From Lars J. Buitinck

fix realloc patch

realloc repair

Clean up some -Wall complaints.

Understand about union mounts as well as union file system

Fix RCS ids
Make sure everything uses {SYS,}LIBC_SCCS properly

branches: 1.1.1;
Initial revision

Delete casts to off_t and size_t that are implied by assignments
or prototypes. Ditto for some of the char* and void* casts too.

verified no change to instructions on ILP32 (i386) and LP64 (amd64)
ok natano@ abluhm@ deraadt@ millert@

Wrap <dirent.h> so that internal calls go direct and they're all weak symbols

use 4x buffer size for opendir. speeds up large directory reading, and
increased memory use is minimal.
ok deraadt logan

Optimization, no functional change:
Save a cookie pointing to the very first entry in the DIR.dd_buf
cache buffer and use that to speed up seekdir(3) when rewinding
to that first entry.
No libc bump because DIR is an opaque struct.

When the optimization applies, which in particular it always does
for rewinddir(3) after reading less than about 500 entries,
seekdir(3) execution time drops from 100 to 0.05 milliseconds
on my i386 notebook. Other cases are not slowed down.

Based on an idea from and ok by guenther@.

Search the userland buffer of dirent structures before falling back to
getdents(2). No functional change, but considerable speedup in many
cases, see http://marc.info/?l=openbsd-tech&m=138369623117934 for some
approximate numbers.
ok guenther@, "good diagnosis" deraadt@

fdopendir() needs to set dd_curpos so that an immediate telldir() works.

pointed out by and ok matthew@

Switch time_t, ino_t, clock_t, and struct kevent's ident and data
members to 64bit types. Assign new syscall numbers for (almost
all) the syscalls that involve the affected types, including anything
with time_t, timeval, itimerval, timespec, rusage, dirent, stat,
or kevent arguments. Add a d_off member to struct dirent and replace
getdirentries() with getdents(), thus immensely simplifying and
accelerating telldir/seekdir. Build perl with -DBIG_TIME.

Bump the major on every single base library: the compat bits included
here are only good enough to make the transition; the T32 compat
option will be burned as soon as we've reached the new world are
are happy with the snapshots for all architectures.

DANGER: ABI incompatibility. Updating to this kernel requires extra
work or you won't be able to login: install a snapshot instead.

Much assistance in fixing userland issues from deraadt@ and tedu@
and build assistance from todd@ and otto@

Expose a bunch of new functionality from POSIX 2008: openat(2),
fchmodat(2), fstatat(2), mkdirat(2), mkfifoat(2), mknodat(2),
faccessat(2), fchownat(2), linkat(2), readlinkat(2), renameat(2),
symlinkat(2), unlinkat(2), utimensat(2), futimens(2), and
fdopendir(3).

"Minor" libc bump.

Tested in a bulk build by naddy@
Much help from guenther@, thib@, tedu@, oga@, and others.
ok deraadt@, naddy@

Implement fdopendir(3) and refactor opendir(3) and fdopendir(3) to use
a common __fdopendir() function. Also, take advantage of the new
O_DIRECTORY and O_CLOEXEC flags in opendir(3).

(Currently fdopendir(3) is commented out; it will be enabled shortly
alongside openat(2) et al.)

Tested by naddy@ in a bulk build.
tweaks and ok guenther@; stupid POSIX nit pointed out by oga@

__opendir2, DTF_NODUP, and __DTF_READALL can die. struct dirent
dd_flags is renamed to the placeholder position dd_unused so that
we can spot "broken software" which assumes we have Jan Simon Pendry's
union mounts (we don't have them, and won't have them ever again).
__opendir2 question spotted by matthew
verified to not break ports by sthen

Leave errno as-is if fstat(2) fails instead of masking the real
errno with ENOTDIR. From Tim van der Molen.

_FD_LOCK/UNLOCK() is libpthread specific and not needed for librthread, so
isolate its usage to libpthread only and replace with generic non-static
mutex support in the one place it is needed:
- remove _FD_LOCK/UNLOCK from lseek and ftruncate in libc and make the
functions weak so that libpthread can override with its own new
versions that do the locking.
- remove _thread_fd_lock/unlock() weak functions from libc and adjust
libpthread for the change.
- add generic _thread_mutex_lock/unlock/destroy() weak functions in libc
to support non-static mutexes in libc and add libpthread and librthread
implementations for them. libc can utilize non-static mutexes via the
new _MUTEX_LOCK/UNLOCK/DESTROY() macros. Actually these new macros can
support both static and non-static mutexes but currently only using
them for non-static.
- make opendir/closedir/readdir/readdir_r/seekdir/telldir() thread-safe
for both thread libraries by using a non-static mutex in the struct
_dirdesc (typedef DIR), utilizing it in the *dir functions and remove
remaining and incorrect _FD_LOCK/UNLOCK() use in libc.
- add comments to both thread libraries to indicate libc depends on the
current implementation of static mutex initialization. suggested by
marc@
- major bump libc and libpthread due to function removal, structure
change and weak symbol conversions.
okay marc@, tedu@

The buffer passed to getdirentries() must be at least as large as
the filesystem block size. Previously we used a constant (1024)
rounded to the page size. Now we use the filesystem's block size,
rounded up to the nearest page.

telldir/seekdir optimization, avoid scanning the complete list,
while still returning the correct index for seekdir(); telldir()
Tested by Fred Crowson and others; "put it in" deraadt@

- Plug huge mem leak; mostly samba was suffering.
- Fix semantics: seekdir(pos); telldir() shoud return pos. The code
that implements this will be made faster in a later commit.
- We loose documented behaviour (after closedir() the telldir()
positions are not valid anymore). This was never in Posix, and most
other systems have nothing like it.

Diff originally from Paul Thorn, rewritten by me using some FreeBSD
code. "slap it in" deraadt@

fix a few warnings, keep the crypt stuff for later.
okay otto@

zap remaining rcsid.

Kill old files that are no longer compiled.

okay theo

bye bye whiteouts

union is gone, ok millert@

Set errno right before return(), so that it does not risk being reset
by any other functions we will invoke in-between.

Fix from Andrey Matveev <andrushock@korovino.net>

ansify function definitions and zap some `register'

ok millert@

Remove the advertising clause in the UCB license which Berkeley
rescinded 22 July 1999. Proofed by myself and Theo.

malloc paranoia; it is unlikely that any filesystem will support
enough directory entries to cause a problem but it is good form anyway.
deraadt@ OK.

Use qsort() instead of mergesort() since the latter can fail due
to malloc(). opendir() requires a stable sort so we rig the compare
routine to never return 0. From Lars J. Buitinck

fix realloc patch

realloc repair

Clean up some -Wall complaints.

Understand about union mounts as well as union file system

Fix RCS ids
Make sure everything uses {SYS,}LIBC_SCCS properly

branches: 1.1.1;
Initial revision