Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r239347, 240295, 240296 and 253325:

r239347 | davidxu | 2012-08-17 10:26:31 +0800 (Fri, 17 Aug 2012) | 7 lines

Implement syscall clock_getcpuclockid2, so we can get a clock id
for process, thread or others we want to support.
Use the syscall to implement POSIX API clock_getcpuclock and
pthread_getcpuclockid.

PR: 168417

------------------------------------------------------------------------
r240295 | davidxu | 2012-09-10 13:00:29 +0800 (Mon, 10 Sep 2012) | 2 lines

Add missing prototype for clock_getcpuclockid.

------------------------------------------------------------------------
r240296 | davidxu | 2012-09-10 13:09:39 +0800 (Mon, 10 Sep 2012) | 2 lines

Process CPU-Time Clocks option is supported, define _POSIX_CPUTIME.

------------------------------------------------------------------------
r253325 | kib | 2013-07-14 03:32:50 +0800 (Sun, 14 Jul 2013) | 6 lines

Allow to call clock_gettime() on the clock id for zombie process.

Reported by: Petr Salinger <Petr.Salinger@seznam.cz>
PR: threads/180496
Sponsored by: The FreeBSD Foundation

MFC r251047:
Avoid a nested frame for getcontext() call in check_deferred_signal().

MFC r248302:

Update to the latest (un)vis(3) sources from NetBSD. This adds
multibyte support[0] and the new functions strenvisx and strsenvisx.

Add MLINKS for vis(3) functions add by this and the initial import from
NetBSD[1].

PR: bin/166364, bin/175418
Submitted by: "J.R. Oldroyd" <fbsd@opal.com>[0]
stefanf[1]
Obtained from: NetBSD

MFC r244401,245305,245308:

Replace our implementation of the vis(3) and unvis(3) APIs with
NetBSD's. This output size limited versions of vis and unvis functions
as well as a set of vis variants that allow arbitrary characters to be
specified for encoding.

Finally, MIME Quoted-Printable encoding as described in RFC 2045 is
supported.

MFC r241731:

Replace our version of the pwcache(3) API with NetBSD's implementation.

This adds two features:
* uid_from_user() and gid_from_group() as the reverse of user_from_uid()
and groups_from_gid().
* pwcache_userdb() and pwcache_groupdb() which allow alternative lookup
functions to be used. For example lookups from passwd and group
databases in a non-standard location.

MFC r242960:
Implement the waitid() SUSv4 function using wait6() system call.

PR: standards/170346

MFC r230429:
Add API for obtaining extended machine context states that cannot be
fit into existing mcontext_t.

On i386 and amd64 do return the extended FPU states using
getcontextx(3). For other architectures, getcontextx(3) returns the
same information as getcontext(2).

MFC r230864:
Make the sys/ucontext.h self-contained by changing the return type
of __getcontextx_size(3) from size_t to int.

MFC r229768:
Implement fdlopen(3), an rtld interface to load shared object by file
descriptor.

Fix a problem whereby a corrupt DNS record can cause named to crash. [11:06]

Add an API for alerting internal libc routines to the presence of
"unsafe" paths post-chroot, and use it in ftpd. [11:07]

Fix a buffer overflow in telnetd. [11:08]

Make pam_ssh ignore unpassphrased keys unless the "nullok" option is
specified. [11:09]

Add sanity checking of service names in pam_start. [11:10]

Approved by: so (cperciva)
Approved by: re (bz)
Security: FreeBSD-SA-11:06.bind
Security: FreeBSD-SA-11:07.chroot
Security: FreeBSD-SA-11:08.telnetd
Security: FreeBSD-SA-11:09.pam_ssh
Security: FreeBSD-SA-11:10.pam

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Implement __pthread_map_stacks_exec() callback for libc, to change the
stack protection to allow execution for single-threaded processes.

Add a new libc function: cfmakesane(3).

I've noticed various terminal emulators that need to obtain a sane
default termios structure use very complex `hacks'. Even though POSIX
doesn't provide any functionality for this, extend our termios API with
cfmakesane(3), which is similar to the commonly supported cfmakeraw(3),
except that it fills the termios structure with sane defaults.

Change all code in our base system to use this function, instead of
depending on <sys/ttydefaults.h> to provide TTYDEF_*.

To support stack unwinding for cancellation points, add -fexceptions flag
for them, two functions _pthread_cancel_enter and _pthread_cancel_leave
are added to let thread enter and leave a cancellation point, it also
makes it possible that other functions can be cancellation points in
libraries without having to be rewritten in libthr.

On shared object unload, in __cxa_finalize, call and clear all installed
atexit and __cxa_atexit handlers that are either installed by unloaded
dso, or points to the functions provided by the dso.

Use _rtld_addr_phdr to locate segment information from the address of
private variable belonging to the dso, supplied by crtstuff.c. Provide
utility function __elf_phdr_match_addr to do the match of address against
dso executable segment.

Call back into libthr from __cxa_finalize using weak
__pthread_cxa_finalize symbol to remove any atfork handler which
function points into unloaded object.

The rtld needs private __pthread_cxa_finalize symbol to not require
resolution of the weak undefined symbol at initialization time. This
cannot work, since rtld is relocated before sym_zero is set up.

Idea by: kan
Reviewed by: kan (previous version)
MFC after: 3 weeks

Introduce implementation-private rtld interface _rtld_addr_phdr, which
fills struct dl_phdr_info for the shared object that contains the
specified address, if any.

Idea and reviewed by: kan
MFC after: 3 weeks

Use aux vector to get values for SSP canary, pagesize, pagesizes array,
number of host CPUs and osreldate.

This eliminates the last sysctl(2) calls from the dynamically linked image
startup.

No objections from: kan
Tested by: marius (sparc64)
MFC after: 1 month

Revert r202447 by re-exposing the old uname(3) function.

It makes hardly any sense to expose a symbol which should only be
provided for binary compatibility, but it seems we don't have a lot of
choice here. There are many autoconf scripts out there that try to
create a binary that links against the old symbol to see whether
uname(3) is present. These scripts fail to detect uname(3) now.

It should be noted that the behaviour we implement is not against the
standards:

| The following shall be declared as a function and may also be defined
| as a macro:
|
| int uname(struct utsname *);

Unexpose the old uname(3) function.

Nowadays uname(3) is an inline function around __xuname(3). Prevent
linkage of new binaries against this compatibility function, similar to
what I did with ttyslot(3).

Remove ttyslot from Symbol.map anyway.

Requested by: kan

Revert the change to Symbol.map, made in r202274.

Even though we use __sym_compat(), we should list the symbol in
Symbol.map.

ttyslot() is now listed as follows, which seems to do the right thing:

| Symbol table '.dynsym' contains 2755 entries:
| Num: Value Size Type Bind Vis Ndx Name
| 613: 00000000000477b0 121 FUNC GLOBAL DEFAULT 10 ttyslot@FBSD_1.0

Reported by: kib

Phase out ttyslot(3).

The ttyslot() function was originally part for SUSv1, marked LEGACY in
SUSv2 and removed later on. This function only makes sense when using
utmp(5), because it was used to determine the offset of the record for
the controlling TTY. It makes little sense to keep it here, because the
new utmpx file format doesn't index based on TTY slots.

Implement <utmpx.h>.

The utmpx interface is the standardized interface of the user accounting
database. The standard only defines a subset of the functions that were
present in System V-like systems.

I'd like to highlight some of the traits my implementation has:

- The standard allows the on-disk format to be different than the
in-memory representation (struct utmpx). Most operating systems don't
do this, but we do. This allows us to keep our ABI more stable, while
giving us the opportunity to modify the on-disk format. It also allows
us to use a common file format across different architectures (i.e.
byte ordering).

- Our implementation of pututxline() also updates wtmp and lastlog (now
called utx.log and utx.lastlogin). This means the databases are more
likely to be in sync.

- Care must be taken that our implementation discard any fields that are
not applicable. For example, our DEAD_PROCESS records do not hold a
TTY name. Just a time stamp, a record identifier and a process
identifier. It also guarantees that strings (ut_host, ut_line and
ut_user) are null terminated. ut_id is obviously not null terminated,
because it's not a string.

- The API and its behaviour should be conformant to POSIX, but there may
be things that slightly deviate from the standard. This implementation
uses separate file descriptors when writing to the log files. It also
doesn't use getutxid() to search for a field to overwrite. It uses an
allocation strategy similar to getutxid(), but prevents DEAD_PROCESS
records from accumulating.

Make sure libulog doesn't overwrite the manpages shipped with our C
library. Also keep the symbol list in Symbol.map sorted.

I'll bump __FreeBSD_version later this evening. I first want to convert
everything to <utmpx.h> and get rid of <utmp.h>.

More cleanup, remove _libc prefix because libthr no longer has stubs
referencing them.

Use umtx to implement process sharable semaphore, to make this work,
now type sema_t is a structure which can be put in a shared memory area,
and multiple processes can operate it concurrently.
User can either use mmap(MAP_SHARED) + sem_init(pshared=1) or use sem_open()
to initialize a shared semaphore.
Named semaphore uses file system and is located in /tmp directory, and its
file name is prefixed with 'SEMD', so now it is chroot or jail friendly.
In simplist cases, both for named and un-named semaphore, userland code
does not have to enter kernel to reduce/increase semaphore's count.
The semaphore is designed to be crash-safe, it means even if an application
is crashed in the middle of operating semaphore, the semaphore state is
still safely recovered by later use, there is no waiter counter maintained
by userland code.
The main semaphore code is in libc and libthr only has some necessary stubs,
this makes it possible that a non-threaded application can use semaphore
without linking to thread library.
Old semaphore implementation is kept libc to maintain binary compatibility.
The kernel ksem API is no longer used in the new implemenation.

Discussed on: threads@

Commit libc files missed in r198508

Add basename_r(3) to complement basename(3). basename_r(3) which accepts
a caller-allocated buffer of at least MAXPATHLEN, rather than using a
global buffer.

MFC after: 1 month
Sponsored by: Google

Add getpagesizes(3). This functions either the number of supported page
sizes or some number of the sizes themselves. It is functionally
compatible with a function by the same name under Solaris.

Reviewed by: jhb

Change the ABI of some of the structures used by the SYSV IPC API:
- The uid/cuid members of struct ipc_perm are now uid_t instead of unsigned
short.
- The gid/cgid members of struct ipc_perm are now gid_t instead of unsigned
short.
- The mode member of struct ipc_perm is now mode_t instead of unsigned short
(this is merely a style bug).
- The rather dubious padding fields for ABI compat with SV/I386 have been
removed from struct msqid_ds and struct semid_ds.
- The shm_segsz member of struct shmid_ds is now a size_t instead of an
int. This removes the need for the shm_bsegsz member in struct
shmid_kernel and should allow for complete support of SYSV SHM regions
>= 2GB.
- The shm_nattch member of struct shmid_ds is now an int instead of a
short.
- The shm_internal member of struct shmid_ds is now gone. The internal
VM object pointer for SHM regions has been moved into struct
shmid_kernel.
- The existing __semctl(), msgctl(), and shmctl() system call entries are
now marked COMPAT7 and new versions of those system calls which support
the new ABI are now present.
- The new system calls are assigned to the FBSD-1.1 version in libc. The
FBSD-1.0 symbols in libc now refer to the old COMPAT7 system calls.
- A simplistic framework for tagging system calls with compatibility
symbol versions has been added to libc. Version tags are added to
system calls by adding an appropriate __sym_compat() entry to
src/lib/libc/incldue/compat.h. [1]

PR: kern/16195 kern/113218 bin/129855
Reviewed by: arch@, rwatson
Discussed with: kan, kib [1]

Add tcsetsid(3).

The entire world seems to use the non-standard TIOCSCTTY ioctl to make a
TTY a controlling terminal of a session. Even though tcsetsid(3) is also
non-standard, I think it's a lot better to use in our own source code,
mainly because it's similar to tcsetpgrp(), tcgetpgrp() and tcgetsid().

I stole the idea from QNX. They do it the other way around; their
TIOCSCTTY is just a wrapper around tcsetsid(). tcsetsid() then calls
into an IPC framework.

Fix whitespace and sorting in Symbol.map.

Properly update the shm_open/shm_unlink symbol versioning metadata after
these functions were moved into the kernel:
- Move the version entries from gen/ to sys/. Since the ABI of the actual
routines did not change, I'm still exporting them as FBSD 1.0 on purpose.
- Add FBSD-private versions for the _ and __sys_ variants.

Add two new routines: fdevname() and fdevname_r().

A more elegant way of obtaining a name of a character device by its file
descriptor on FreeBSD, is to use the FIODGNAME ioctl. Because a valid
file descriptor implies a file descriptor is visible in /dev, it will
always resolve a valid device name.

I'm adding a more friendly wrapper for this ioctl, called fdevname(). It
is a lot easier to use than devname() and also has better error
handling. When a device name cannot be resolved, it will just return
NULL instead of a generated device name that makes no sense.

Discussed with: kib

Add two rtld exported symbols, _rtld_atfork_pre and _rtld_atfork_post.
Threading library calls _pre before the fork, allowing the rtld to
lock itself to ensure that other threads of the process are out of
dynamic linker. _post releases the locks.

This allows the rtld to have consistent state in the child. Although
child may legitimately call only async-safe functions, the call may
need plt relocation resolution, and this requires working rtld.

Reported and debugging help by: rink
Reviewed by: kan, davidxu
MFC after: 1 month (anyway, not before 7.1 is out)

Add arc4random_uniform() function (to avoid "modulo bias")

Obtained from: OpenBSD

Add feature_present(3) to the FBSD 1.1 symbol map.

Add arc4random_buf to FBSD_1.1 space

Turn execvpe() into an internal libc routine.

Adding exevpe() has caused some ports to break. Even though execvpe() is
a useful routine, it does not conform to any standards.

This patch is a little bit different from the patch sent to the mailing
list. I forgot to remove execvpe from the Symbol.map (which does not
seem to miscompile libc, though).

Reviewed by: davidxu
Approved by: philip

Add POSIX routines called posix_spawn() and posix_spawnp(), which
can be used as replacements for exec/fork in a lot of cases. This
change also added execvpe() which allows environment variable
PATH to be used for searching executable file, it is used for
implementing posix_spawnp().

PR: standards/122051

Implement fdopendir(3) by splitting __opendir2() into two parts, the upper part
deals with the usual __opendir2() calls, and the rest part with an interface
translator to expose fdopendir(3) functionality. Manual page was obtained from
kib@'s work for *at(2) system calls.

Implement POSIX function tcgetsid() which returns session id.

PR: stand/107561

Our fts(3) API, as inherited from 4.4BSD, suffers from integer
fields in FTS and FTSENT structs being too narrow. In addition,
the narrow types creep from there into fts.c. As a result, fts(3)
consumers, e.g., find(1) or rm(1), can't handle file trees an ordinary
user can create, which can have security implications.

To fix the historic implementation of fts(3), OpenBSD and NetBSD
have already changed <fts.h> in somewhat incompatible ways, so we
are free to do so, too. This change is a superset of changes from
the other BSDs with a few more improvements. It doesn't touch
fts(3) functionality; it just extends integer types used by it to
match modern reality and the C standard.

Here are its points:

o For C object sizes, use size_t unless it's 100% certain that
the object will be really small. (Note that fts(3) can construct
pathnames _much_ longer than PATH_MAX for its consumers.)

o Avoid the short types because on modern platforms using them
results in larger and slower code. Change shorts to ints as
follows:

- For variables than count simple, limited things like states,
use plain vanilla `int' as it's the type of choice in C.

- For a limited number of bit flags use `unsigned' because signed
bit-wise operations are implementation-defined, i.e., unportable,
in C.

o For things that should be at least 64 bits wide, use long long
and not int64_t, as the latter is an optional type. See
FTSENT.fts_number aka FTS.fts_bignum. Extending fts_number `to
satisfy future needs' is pointless because there is fts_pointer,
which can be used to link to arbitrary data from an FTSENT.
However, there already are fts(3) consumers that require fts_number,
or fts_bignum, have at least 64 bits in it, so we must allow for them.

o For the tree depth, use `long'. This is a trade-off between making
this field too wide and allowing for 64-bit inode numbers and/or
chain-mounted filesystems. On the one hand, `long' is almost
enough for 32-bit filesystems on a 32-bit platform (our ino_t is
uint32_t now). On the other hand, platforms with a 64-bit (or
wider) `long' will be ready for 64-bit inode numbers, as well as
for several 32-bit filesystems mounted one under another. Note
that fts_level has to be signed because -1 is a magic value for it,
FTS_ROOTPARENTLEVEL.

o For the `nlinks' local var in fts_build(), use `long'. The logic
in fts_build() requires that `nlinks' be signed, but our nlink_t
currently is uint16_t. Therefore let's make the signed var wide
enough to be able to represent 2^16-1 in pure C99, and even 2^32-1
on a 64-bit platform. Perhaps the logic should be changed just
to use nlink_t, but it can be done later w/o breaking fts(3) ABI
any more because `nlinks' is just a local var.

This commit also inludes supporting stuff for the fts change:

o Preserve the old versions of fts(3) functions through libc symbol
versioning because the old versions appeared in all our former releases.

o Bump __FreeBSD_version just in case. There is a small chance that
some ill-written 3-rd party apps may fail to build or work correctly
if compiled after this change.

o Update the fts(3) manpage accordingly. In particular, remove
references to fts_bignum, which was a FreeBSD-specific hack to work
around the too narrow types of FTSENT members. Now fts_number is
at least 64 bits wide (long long) and fts_bignum is an undocumented
alias for fts_number kept around for compatibility reasons. According
to Google Code Search, the only big consumers of fts_bignum are in
our own source tree, so they can be fixed easily to use fts_number.

o Mention the change in src/UPDATING.

PR: bin/104458
Approved by: re (quite a while ago)
Discussed with: deischen (the symbol versioning part)
Reviewed by: -arch (mostly silence); das (generally OK, but we didn't
agree on some types used; assuming that no objections on
-arch let me to stick to my opinion)

Add _pthread_mutex_init_calloc_cb to libc's map, for which malloc defines
a stub.

Some libc symbol map cleanups.

net: endhostdnsent is named _endhostdnsent and is
private to netdb family of functions.

posix1e: acl_size.c has been never compiled in,
so there's no "acl_size".

rpc: "getnetid" is a static function.

stdtime: "gtime" is #ifdef'ed out in the source.

some symbols are specific only to some architectures,
e.g., ___tls_get_addr is only defined on i386.

__htonl, __htons, __ntohl and __ntohs are no longer
functions, they are now (internal) defines in
<machine/endian.h>.

Submitted by: ru

Use C comments since we now preprocess these files with CPP.

Clean-ip TLS symbol versions. [_]__tls_get_addr function is part of
the platform ABI and as such does not belong in FBSDprivate.

__libc_tls_* functions do not have to be visible to outside world
at all.

Add entry for dl_iterate_phdr.

Remove getobjformat() from libc's symbol map. It probably should have been
in the private area anyway. Nothing in FreeBSD uses it any more anyway.

Add symbol maps and initial symbol version definitions to libc.

Reviewed by: davidxu