- Copy stable/10 (r259064) to releng/10.0 as part of the
10.0-RELEASE cycle.
- Update __FreeBSD_version [1]
- Set branch name to -RC1

[1] 10.0-CURRENT __FreeBSD_version value ended at '55', so
start releng/10.0 at '100' so the branch is started with
a value ending in zero.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

libc/stdio: Allow fopen/freopen modes in any order (except initial r/w/a).

Austin Group issue #411 requires 'e' to be accepted before and after 'x',
and encourages accepting the characters in any order, except the initial
'r', 'w' or 'a'.

Given that glibc accepts the characters after r/w/a in any order and that
diagnosing this problem may be hard, change our libc to behave that way as
well.

libc: Always use our own copy of sys_errlist and sys_nerr (.so only).

This ensures strerror() and friends continue to work correctly even if a
(non-PIE) executable linked against an older libc imports sys_errlist (which
causes sys_errlist to refer to the executable's copy with a size fixed when
that executable was linked).

The executable's use of sys_errlist remains broken because it uses the
current value of sys_nerr and may access past the bounds of the array.

Different from the message "Using sys_errlist from executables is not
ABI-stable" on freebsd-arch, this change does not affect the static library.
There seems no reason to prevent overriding the error messages in the static
library.

Add mkostemp() and mkostemps().

These are like mkstemp() and mkstemps() but allow passing open(2) flags like
O_CLOEXEC.

Typo corrected.

mktemp(3): Add standards section. Prefer standard header.

mktemp(), mkstemp() and mkdtemp() are available in standard <stdlib.h> and
also in <unistd.h>. Encourage use of the former by listing it in the
synopsis.

Convert libc/stdio from K&R to ANSI C

And add '__restrict' where it appeared in the header prototypes

Renumber clauses to reduce diffs to other versions

NetBSD, OpenBSD, and Android's Bionic all number the clauses 1 through 3,
so follow suit to make comparison easier.

Acked-by: imp@

Spelling correction

Remove unused atomic header

mdoc: add missing El.

Add an implementation of open_memstream() and open_wmemstream(). These
routines provide write-only stdio FILE objects that store their data in a
dynamically allocated buffer. They are a string builder interface somewhat
akin to a completely dynamic sbuf.

Reviewed by: bde, jilles (earlier versions)
MFC after: 1 month

setbuf(3): Restore a BUGS section about setbuf().

The brokenness of setbuf() is not specific to 4.2BSD and 4.3BSD but inherent
in the API definition.

Reported by: bde

setbuf(3): Remove bugs section about ancient versions of BSD.

mdoc: Remove EOL whitespace.

- Fix more style(9)-related issues (copyright header, spaces after function
names, unnecessary casts)
- Change type of boolean variable from char to bool

Suggested by: jhb, zont, jmallett
Reviewed by: cognet
Approved by: cognet

- Remove underscores from the internal structure name, as it doesn't collide
with the user's namespace.

- Correct size and position variables type from long to size_t.

- Do not set errno to ENOMEM on malloc failure, as malloc already does so.

- Implement the concept of "buffer data length", which mandates what SEEK_END
refers to and the allowed extent for a read.

- Use NULL as read-callback if the buffer is opened in write-only mode.
Conversely, use NULL as write-callback when opened in read-only mode.

- Implement the handling of the ``b'' character in the mode argument. A binary
buffer differs from a text buffer (default mode if ``b'' is omitted) in that
NULL bytes are never appended to writes and that the "buffer data length"
equals to the size of the buffer.

- Remove shall from the man page. Use indicative instead. Also, specify that
the ``b'' flag does not conform with POSIX but is supported by glibc.

- Update the regression test so that the ``b'' functionality and the "buffer
data length" concepts are tested.

- Minor style(9) corrections.

Suggested by: jilles
Reviewed by: cognet
Approved by: cognet

Add fmemopen(3), an interface to get a FILE * from a buffer in memory, along
with the respective regression test.
See http://pubs.opengroup.org/onlinepubs/9699919799/functions/fmemopen.html

Reviewed by: cognet
Approved by: cognet

The getline function returns the number of characters read, not
written. Use clearer text for this.

PR: docs/174023
Submitted by: Paul Procacci <pprocacci@gmail.com>
Approved by: bcr (mentor)
MFC after: 1 week

libc: Allow setting close-on-exec in fopen/freopen/fdopen.

This commit adds a new mode option 'e' that must follow any 'b', '+' and/or
'x' options. C11 is clear about the 'x' needing to follow 'b' and/or '+' and
that is what we implement; therefore, require a strict position for 'e' as
well.

For freopen() with a non-NULL path argument and fopen(), the close-on-exec
flag is set iff the 'e' mode option is specified. For freopen() with a NULL
path argument and fdopen(), the close-on-exec flag is turned on if the 'e'
mode option is specified and remains unchanged otherwise.

Although the same behaviour for fopen() can be obtained by open(O_CLOEXEC)
and fdopen(), this needlessly complicates the calling code.

Apart from the ordering requirement, the new option matches glibc.

PR: kern/169320

Clarify that the ' flag is an apostrophe.

MFC after: 2 weeks

fopen(3): Mention that the "x" mode option is from C11.

MFC after: 1 week

Correct double "the the"

Approved by: cperciva
MFC after: 3 days

The register_printf_render_std() function expects regular string.
Change argument type from 'const unsigned char *' to 'const char *'.

MFC after: 2 weeks

Only set _w to 0 when the file stream is not currently reading. Without
this fflush may fail to write data in the buffer.

PR: kern/137819
Submitted by: Eric Blake <ebb9@byu.net>
Reviewed by: theraven
Approved by: cperciva
MFC after: 2 weeks

Add two new locale-specific man pages:
- libc/stdio/scanf_l.3
- libc/stdio/printf_l.3

Reviewed by: theraven
Approved by: gabor (mentor)
MFC after: 5 days

Minor mdoc nits.

Remove reference to non-existent FreeBSD Security Architecture

Approved by: cperciva
MFC after: 3 days

Remove incorrect __restrict qualifier on several pointers

The typical case was:
static __inline int
convert_ccl(FILE *fp, char * __restrict p, [...])
{
[...]

if (p == SUPPRESS_PTR) {
[...]
} else {
[...]
}

[...]
}

This qualifier says that the pointer is the only one at that time
pointing to the resource.

Here, clang considers that "p" will never match "SUPPRESS_PTR" and
optimize the if{} block out. This leads to segfaults in programs calling
vfscanf(3) and vfwscanf(3) with just the format string (no arguments
following it).

The following softwares were reported to abort with segmentation fault
and this patch fixes it:
o cmake
o smartd
o devel/ORBit2

dim@ opened an LLVM PR to discuss this clang optimization:
http://llvm.org/bugs/show_bug.cgi?id=12656

Tested by: bsam@

Fix a bug in *wscanf's handling of non-wide %s, %c, and %[
conversions. Both the specification and the documentation say the
width is interpreted as the max number of wide characters to read, but
the implementation was interpreting it as the number of bytes to convert.
(See also r105317.)

This change has security implications for any applications that depend
on the buggy behavior, but the impact in practice is probably nil.
Any such application would already be buggy on other platforms that
get the semantics right. Also, these conversions are rarely used;
%ls, %lc, and %l[ are more appropriate.

Previously, vfscanf()'s wide character processing functions were
reading wide characters manually. With this change, they now use
fgetwc(). To make this work, we use an internal version of fgetwc()
with a few extensions: it takes an mbstate * because non-wide streams
don't have a built-in mbstate, and it indicates the number of bytes
read.

vfscanf() now resembles vfwscanf() more closely. Minor functional
improvements include working xlocale support in vfscanf(), setting the
stream error indicator on encoding errors, and proper handling of
shift-based encodings. (Actually, making shift-based encodings work
with non-wide streams is hopeless, but the implementation now matches
the broken specification.)

Fix small documentation mistakes.

Submitted by: brueffer
Approved by: kib (mentor)

Import stdbuf(1) and the shared library it relies on.
This tool changes the default buffering behaviour of standard
stdio streams.

It only works on dynamic binaries. To make it work for static
ones it would require cluttering stdio because there no single
entry point.

PR: 166660
Reviewed by: current@, jhb
Approved by: kib (mentor)
MFC after: 1 week

Take the spinlock around clearing of the fp->_flags in fclose(3), which
indicates the avaliability of FILE, to prevent possible reordering of
the writes as seen by other CPUs.

Reported by: Fengwei yin <yfw.bsd gmail com>
Reviewed by: jhb
MFC after: 1 week

Bugfix: Include whitespace characters in the count of the number of
characters read.

Bugfix: Correctly count the number of characters read for %l[ conversions.

Bugfix: %n doesn't count as a conversion, so
sscanf("abc", "ab%ncd", &i) returns EOF, not 0.

Refactor scanf to improve modularity. Conversions are now performed
by separate conversion functions. This will hopefully make bugs more
noticeable (I noticed several already) and provide opportunities to
reduce code duplication.

As noted by Peter Jeremy, r234528 only partially fixed the infinite
loop bug introduced in r187302. This completes the fix.

PR: 167039
MFC after: 3 days

If the size passed to {,v}s{w,n}printf is larger than INT_MAX+1
(i.e., the return value would overflow), set errno to EOVERFLOW
and return an error. This improves the chances that buggy
applications -- for instance, ones that pass in a negative integer
as the size due to a bogus calculation -- will fail in safe ways.
Returning an error in these situations is specified by POSIX, but
POSIX appears to have an off-by-one error that isn't duplicated in
this change.

Previously, some of these functions would silently cap the size at
INT_MAX+1, and others would exit with an error after writing more
than INT_MAX characters.

PR: 39256
MFC after: 2 weeks

- Fix the claim that the output is always null-terminated. This isn't
true if the size is zero.
- Fix a claim that sprintf() is the same as snprintf() with an
infinite size. It's equivalent to snprintf() with a size of
INT_MAX + 1.
- Document the return values in the return values section.
- Document the possible errno value of EOVERFLOW.

MFC after: 2 weeks

Ensure that the {,v}swprintf functions always null-terminate the
output string, even if an encoding error or malloc failure occurs.

Fix a bug introduced in r187302 that was causing fputws() to enter an
infinite loop pretty much unconditionally. It's remarkable that the
patch that introduced the bug was never tested, but even more
remarkable that nobody noticed for over two years.

PR: 167039
MFC after: 3 days

Remove trailing whitespace per mdoc lint warning

Disussed with: gavin
No objection from: doc
Approved by: joel
MFC after: 3 days

Remove outdated comment of seven years

PR: docs/116116
Approved by: cperciva
MFC after: 1 week

Remove reference to gcc's non-standard -fwritable-strings, which
doesn't exist in recent releases (and is bad advice anyway)

PR: docs/163119
Submitted by: Yuri Pankov <yuri.pankov@gmail.com>
Approved by: cperciva
MFC after: 1 week

Implement xlocale APIs from Darwin, mainly for use by libc++. This adds a
load of _l suffixed versions of various standard library functions that use
the global locale, making them take an explicit locale parameter. Also
adds support for per-thread locales. This work was funded by the FreeBSD
Foundation.

Please test any code you have that uses the C standard locale functions!

Reviewed by: das (gdtoa changes)
Approved by: dim (mentor)

Add support for the 'x' mode option in fopen() as specified in the C1X
draft standard. The option is equivalent to O_EXCL.

MFC after: 1 month

Because we call __printf_out() with a on-stack buffer, also call
__printf_flush() so we are sure it won't be referenced after we return.

MFC after: 2 weeks

Fix various issues in how %#T is handled:
- If precision is 0, don't print period followed by no digits.
- If precision is 0 stop printing units as soon as possible
(eg. if we have three years and five days and precision is 0
print only 3y5d).
- If precision is not 0, print all units (eg. 3y0d0h0m0s.00).

MFC after: 2 weeks

When reopening a stream backed by an open file descriptor, do not close
the existing file descriptor. Instead, let dup2() atomically close the
old file descriptor when assigning the newly opened file to the same
descriptor. This closes a race in a multithreaded application where a
concurrent open() could allocate the existing file descriptor in between
the calls to close() and dup2().

PR: threads/79887
Submitted by: Dmitrij Tejblum tejblum of yandex-team.ru
Reviewed by: davidxu
MFC after: 1 week

Remove two unused variables, left over from the refactoring in r180104.

PR: bin/152551
Submitted by: Henning Petersen <henning.petersen t-online.de>
MFC after: 2 weeks

Update the documentation to reflect changes to the implementation in
r197752, which is related to handling of null buffer pointers. Also
make a few minor wording changes.

Reported by: jh@

Revert changes of 'assure' to 'ensure' made in r211936.

Approved by: rrs (mentor)

Fix incorrect usage of 'assure' and 'insure'.

Approved by: rrs (mentor)

mdoc: move CAVEATS, BUGS and SECURITY CONSIDERATIONS sections to the
bottom of the manpages and order them consistently.

GNU groff doesn't care about the ordering, and doesn't even mention
CAVEATS and SECURITY CONSIDERATIONS as common sections and where to put
them.

Found by: mdocml lint run
Reviewed by: ru

mdoc: fix parenthesis

Reviewed by: brueffer

mdoc: use macro for +- that is understood by mdocml

Reviewed by: brueffer

I feel this wording of the history is more clear.
ANSIfy vasprintf() while I'm here.

- Use an initializer macro to initialize fields in 'fake' FILE objects used
by *sprintf(), etc.
- Explicitly initialize _fl_mutex to PTHREAD_MUTEX_INITIALIZER for all FILE
objects. This is currently a nop on FreeBSD, but is import for other
platforms (or in the future) where PTHREAD_MUTEX_INITIALIZER is not simply
zero.

PR: threads/141198
Reported by: Jeremy Huddleston @ Apple
MFC after: 2 weeks

In _gettemp(), check that the length of the path doesn't exceed
MAXPATHLEN. Otherwise the path name (or part of it) may not fit to
carrybuf causing a buffer overflow.

PR: bin/140228
Suggested by: jilles

%U was macroized in mdoc(7), escape.

Give a less silly response to a silly request.

Prior to this commit, fread/fwrite calls with size * nmemb > SIZE_MAX
were handled by reading or writing (size_t)(size * nmemb) bytes; for
example, on 32-bit platforms, fread(ptr, 641, 6700417, f) would read 1
byte and indicate that the requested 6700417 blocks had been read.

This commit adds a check for such integer overflows, and treats them as
if an overly large request was passed to read/write; i.e., it sets errno
to EINVAL, sets the error indicator on the file, and returns a short
object count (0, to be specific).

The overflow check involves an integer division, so as a performance
optimization we check first to see if both size and nmemb are less than
2^16; if they are, no overflow is possible and we avoid the division.
We assume here that size_t is at least 32 bits; this appears to be true
on all platforms FreeBSD supports.

Although this commit fixes an integer overflow, it is not likely to have
any security implications, since any program which would be affected by
this bug fix is quite clearly already very confused.

Reviewed by: kib
MFC after: 1 month

Remove unnecessary quoting and markup, add missing punctuation.

PR: 140494
Submitted by: Jeremy Huddleston <jeremyhu@apple.com>, bde
MFC after: 1 week

Use vsprintf instead of rolling our own.

PR: bin/140496
Submitted by: Jeremy Huddleston <jeremyhu apple.com>
MFC after: 1 month

Use vsscanf instead of rolling our own.

PR: bin/140530
Submitted by: Jeremy Huddleston <jeremyhu apple.com>
MFC after: 1 month

K&R -> ANSI prototype.

MFC after: 1 month

Fix many "function declaration isn't a prototype" warnings in libc.

I've only fixed code that seems to be written by `us'. There are still
many warnings like this present in resolv/, rpc/, stdtime/ and yp/.

Fix the dprintf() prototype.

PR: 141087
Submitted by: Jeremy Huddleston <jeremyhu@apple.com>
MFC after: 3 days

Eliminate more dead stores.

Found by: Clang static analyzer
MFC after: 7 days

Make all three if conditions look similar by always initializing nsec
and moving the default initialization of prec into the else clause.
The clang static analyzer erroneously thought that nsec can be used
uninitialized here; it was not actually possible, but better to make
the code clearer. (Clang can't know that sprintf() won't modify *pi
behind the scenes.)

In __mbsconv(), if prec was zero, nconv could have been used
uninitialized. Initialize it to a safe value so that there's no
chance of returning an error if stack garbage happens to be equal to
(size_t)-1 or (size_t)-2.

Found by: Clang static analyzer
MFC after: 7 days

Eliminate dead store.

Found by: Clang static analyzer
MFC after: 7 days

Purge some duplicate mergeinfo.

Better glibc compatibility for getline/getdelim:

- Tolerate applications that pass a NULL pointer for the buffer and
claim that the capacity of the buffer is nonzero.

- If an application passes in a non-NULL buffer pointer and claims the
buffer has zero capacity, we should free (well, realloc) it
anyway. It could have been obtained from malloc(0), so failing to
free it would be a small memory leak.

MFC After: 2 weeks
Reported by: naddy
PR: ports/138320

Make the description of `b' a little better.

If you have a one-byte sequence, `w', `b' is the second character. Not
the third.

Submitted by: Christoph Mallon

Fix fwrite() to return 0 when size or nmemb are zero.

Right now nmemb is returned when size is 0. In newer versions of the
standards, it is explicitly required that fwrite() should return 0.

Submitted by: Christoph Mallon
Approved by: re (kib)

Return -1 instead of 0 upon reaching EOF. This is somewhat ill-advised
because it means getdelim() returns -1 for both error and EOF, and
never returns 0. However, this is what the original GNU implementation
does, and POSIX inherited the bug.

Reported by: marcus@

Add dprintf() and vdprintf() from POSIX.1-2008. Like getline(),
dprintf() is a simple wrapper around another function, so we may as
well implement it. But also like getline(), we can't prototype it by
default right now because it would break too many ports.

Rewrite asprintf() as a wrapper around vasprintf(), thus reducing the
number of functions that have an incestuous relationship with the
arcane innards of stdio.

The argument corresponding to %zn is supposed to be an ssize_t *, not
a size_t *, although the distinction is moot in practice.

Use C99-style initializers. No functional change.

Reviewed by: md5(1)

Replace a dozen lines of code with a call to strnlen() / wcsnlen().

- Add getdelim(), getline(), stpncpy(), strnlen(), wcsnlen(),
wcscasecmp(), and wcsncasecmp().
- Make some previously non-standard extensions visible
if POSIX_VISIBLE >= 200809.
- Use restrict qualifiers in stpcpy().
- Declare off_t and size_t in stdio.h.
- Bump __FreeBSD_version in case the new symbols (particularly
getline()) cause issues with ports.

Reviewed by: standards@

Make sure %zd treats negative arguments properly on 32-bit platforms.
Fix harmless but related bugs in %_$zd and %_$tu.

PR: 131880
MFC after: 1 week

Better wording for clearing EOF indicator.

Submitted by: keramida and jhb

Remove another comment about clearing EOF indicator.

Noticed by: bde

Update the manpage to reflect r145172.

Remove comment about clearerr() being the only method of clearing
the EOF indicator, fseek() may also be used for this.

Bump document date.

PR: 76333
Submitted by: Yoshihiko Sarumaru <mistral@imasy.or.jp>

Add support for multibyte thousands_sep encodings, e.g., U+066C.
The integer thousands' separator code is rewritten in order to
avoid having to preallocate a buffer for the largest possible
digit string with the most possible instances of the longest
possible multibyte thousands' separator. The new version inserts
thousands' separators for integers using the same code as floating point.

- Add support for multibyte decimal_point encodings, e.g., U+066B.
A forthcoming gdtoa import is needed to make this fully work.
- Improve the way "nan(...)" is parsed.

Add support for multibyte decimal_point encodings, e.g., U+066B.

When f[w]printf() is called on an unbuffered file like stdout, it
sets up a fake buffered FILE and then effectively calls itself
recursively. Unfortunately, gcc doesn't know how to do tail call
elimination in this case, and actually makes things worse by
inlining __sbprintf(). This means that f[w]printf() to stderr was
allocating about 5k of stack on 64-bit platforms, much of which was
never used.

I've reorganized things to eliminate the waste. In addition to saving
some stack space, this improves performance in my tests by anywhere
from 5% to 17% (depending on the test) when -fstack-protector is
enabled. I found no statistically significant performance difference
when stack protection is turned off. (The tests redirected stderr to
/dev/null.)

Simplify printf's inlined output buffering routines. On amd64, this
reduces the code size by about 10% and improves performance slightly.

Introduce a local variable and use it instead of passed in parameter
to get rid of restrict qualifier discarding. This lets libc compile
cleanly in gnu99 mode.

Suggested by: kib, christoph.mallon at gmx.de
Approved by: kib (mentor)

Reduce code duplication by moving functions that are identical in both
vfprintf.c and vfwprintf.c (except for char/wchar_t differences) to a
common header file.

Convert the insidious macros that handle printf()'s buffering into
slightly less evil inline functions, and move the buffering state into
a struct. This will make it possible for helper routines to produce
output for printf() directly, making it possible to untangle the code
somewhat.

In wprintf(), use the same buffering mechanism to reduce diffs to
printf(). This has the side-effect of causing wprintf() to catch write
errors that it previously ignored.

Set the error indicator on an attempt to write to a read-only stream.

PR: 127335
MFC after: 2 weeks

Consolidate some variable initializations. No substantive change.

Move the xprintf hook to where it belongs; it shouldn't be in the
middle of vfprintf's variable declarations.

Fix fread() to return a correct value on platforms where sizeof(int) !=
sizeof(size_t), i.e. on all 64-bit platforms.

Reported by: Andrey V. Elsukov
MFC after: 3 days

Initialize "nconv" to a reasonable value in all code paths. Prior to
this commit, sprintf("%s", "") could fail depending on what happened
to be on the stack.

Found by: LLVM/Clang Static Checker

Fix a few bugs with the _gettemp() routine which implements mkstemp(),
mkstemps(), and mkdtemp().
- Add proper range checking for the 'slen' parameter passed to mkstemps().
- Try all possible permutations of a template if a collision is encountered.
Previously, once a single template character reached 'z', it would not wrap
around to '0' and keep going until it encountered the original starting
letter. In the edge case that the randomly generated starting name used
all 'z' characters, only that single name would be tried before giving up.

PR: standards/66531
Submitted by: Jim Luther
Obtained from: Apple
MFC after: 1 week

Use arc4random_uniform(3) since modulo size is not power of 2

Fix a bogon in the previous commit and add some missing error checks.

Correctly handle malloc() failure. While here, reduce the code size a
bit by removing some calls to the inline function addtype().

Factor out the code that builds the argument table. We don't need separate
normal and wide character versions of it.

No functional change.

Reduce the level of duplication between vfprintf() and vfwprintf()
by moving the positional argument handling code to a new file,
printf-pos.c, and moving common definitions to printflocal.h.
No functional change intended.

Begin de-spaghettifying the code that handles positional arguments.
In particular, encapsulate the state of the type table in a struct,
and add inline functions to initialize, free, and manipulate that
state. This replaces some ugly macros that made proper error handling
impossible.

While here, remove an unneeded test for NULL and a variable that is
initialized (many times!) but never used. The compiler didn't catch
these because of rampant use of the same variable to mean different
things in different places.

This commit should not cause any changes in functionality.

Remove useless call to getdtablesize(2) in fdopen(3) and its useless
variable nofile.

PR: 123109
Submitted by: Christoph Mallon
Approved by: rwatson (mentor)
MFC after: 1 month

Retire the __fgetcookie(), __fgetpendout(), and __fsetfileno() accessors
as we aren't hiding FILE's internals anymore.

Expose FILE's internals to the world again in all their glory. Restore
all the previous inline optimizations as well. FILE is back to using
__mbstate_t, struct pthread *, and struct pthread_mutex *.

Add __fgetcookie(), __fgetpendout() and __fsetfileno() to the private
name space.

Unbreak build: gnu sort has been configured to grope inside struct
__sFILE. It's opaque now, so add a function that returns the pending
output bytes.

Pointy hat: jhb

Unbreak build: libftpio gropes inside struct __sFILE. Implement
accessor functions for its benefit now thaat FILE is opaque.
I'm sure there's a better way. I leave that for people to work
on in a src tree that isn't broken.

Pointy hat: jhb

Next round of stdio changes: Remove all inlining of stdio operations and
move the definition of the type backing FILE (struct __sFILE) into an
internal header.
- Remove macros to inline certain operations from stdio.h. Applications
will now always call the functions instead.
- Move the various foo_unlocked() functions from unlocked.c into foo.c.
This lets some of the inlining macros (e.g. __sfeof()) move into
foo.c.
- Update a few comments.
- struct __sFILE can now go back to using mbstate_t, pthread_t, and
pthread_mutex_t instead of knowing about their private, backing types.

MFC after: 1 month
Reviewed by: kan

Fix a leak in the recent fixes for file descriptors > SHRT_MAX. In the
case of a file descriptor we can't handle, clear the FILE structure's flags
so it can be reused.

MFC after: 1 week
Reported by: otto @ OpenBSD

Next stage of stdio cleanup: Retire __sFILEX and merge the fields back into
__sFILE. This was supposed to be done in 6.0. Some notes:
- Where possible I restored the various lines to their pre-__sFILEX state.
- Retire INITEXTRA() and just initialize the wchar bits (orientation and
mbstate) explicitly instead. The various places that used INITEXTRA
didn't need the locking fields or _up initialized. (Some places needed
_up to exist and not be off the end of a NULL or garbage pointer, but
they didn't require it to be initialized to a specific value.)
- For now, stdio.h "knows" that pthread_t is a 'struct pthread *' to
avoid namespace pollution of including all the pthread types in stdio.h.
Once we remove all the inlines and make __sFILE private it can go back
to using pthread_t, etc.
- This does not remove any of the inlines currently and does not change
any of the public ABI of 'FILE'.

MFC after: 1 month
Reviewed by: peter

Updates for changes in the way printf() handles hex floating point
numbers.

Add <limits.h> for SHRT_MAX.

Pointy hat to: jhb

File descriptors are an int, but our stdio FILE object uses a short to hold
them. Thus, any fd whose value is greater than SHRT_MAX is handled
incorrectly (the short value is sign-extended when converted to an int).
An unpleasant side effect is that if fopen() opens a file and gets a
backing fd that is greater than SHRT_MAX, fclose() will fail and the file
descriptor will be leaked. Better handle this by fixing fopen(), fdopen(),
and freopen() to fail attempts to use a fd greater than SHRT_MAX with
EMFILE.

At some point in the future we should look at expanding the file descriptor
in FILE to an int, but that is a bit complicated due to ABI issues.

MFC after: 1 week
Discussed on: arch
Reviewed by: wollman

Move all the xprintf-related symbols to FBSDprivate_1.0.

Discussed with: deischen, kan, phk

Catch up with vfprintf.c,v 1.77.

Remove some test instrumentation. (The Symbol.map changes broke it anyway.)

Add rewind() to the list of functions which may fail and set errno

Add mbstate clear missed in one of the cases.
Move overflow check for fseek as early as needed.

We should never zero-pad INF or NaN (yielding silly strings like "00inf")
even if the programmer asks for zero padding.

Use C comments since we now preprocess these files with CPP.

Grammer correction from: emaste

Loosen return value spec for fputs to be standards compliant so people
don't program against our more restrictive behaviour.

o Add ENVIRONMENT section and mention there that TMPDIR is ignored
when issetugid(3) is true.

PR: docs/108346
Obtained from: OpenBSD
MFC after: 1 week

Remove 3rd clause, renumber, ok per email

Per Regents of the University of Calfornia letter, remove advertising
clause.

# If I've done so improperly on a file, please let me know.

Fix rounding of 0xf for hex fp formats.

PR: 90333

Revert the rev. 1.4, it causes breakage on most arches except i386.

Remove the const qualifier from ap argument for __v2printf, that induced
that breakage, and seems to be the real reason for bad code. ap is modified
inside the __v2printf body by va_arg macro.

Pointy hat to: kib
Approved by: pjd (mentor)

Workaround for (what seems to be) compiler error for gcc 3.4.6. On
i386 with default optimization level (-O2), va_list pointer ap in the
__v2printf function is advanced before the use. That cause argument
shift and garbage instead last argument in printf-family when xprintf is
activated.

The nsswitch is easy victim of the bug.

Reviewed by: kan
Approved by: kan (mentor)
MFC after: 1 week

file == NULL:
Issue __sflush() before possible setting O_APPEND mode or ftruncate(),
write to wrong place may occurse oserwise.
Use simplified _sseek() to the start, if no O_APPEND is set, instead
of _fseeko() (_sseek() to the end, if O_APPEND, occurse later, as for
file != NULL).
Don't check seek error return, as original fopen() and freopen() never
does.

file != NULL:
Add missing _sseek() to the end.

Honor errno obtained from __sflags
Move errno restoring after FUNLOCKFILE in one case

Back out ftruncate error handling. There can some file tipes
f.e. sockets when ftruncate normally fail.

Do real seek
Catch ftruncate errors

PR: 104295
Submitted by: ru (seek)

Use correct type in va_arg argument.

Markup fixes.

Remove alpha left-overs.

Oops, correct the weak reference (s/fclose/fcloseall).

Spotted by: Antoine Brodin (antoine _dot_ brodin _at_ laposte _dot_ net)

Now that libc has fcloseall(), remove _cleanup() from the list
of FreeBSD private symbols.

Add fcloseall() to libc. This removes the need to export _cleanup().
Linux also provides an fcloseall() implementation.

Discussed on: arch

Add missing #if's for NO_FLOATING_POINT

Add __collate_load_error and __collate_range_cmp to the list of
FBSDprivate locale symbols. These functions are needed by
libcompat.

Add _cleanup to the list of stdio FBSDprivate symbols. Some
third party applications use this. This will be removed and
replaced by fcloseall() once libc version is bumped.

Add _res to the list of resolv symbols.

Found by: portbuilder runs (thanks Kris!)

Add each directory's symbol map file to SYM_MAPS.

Add symbol maps and initial symbol version definitions to libc.

Reviewed by: davidxu

Fix the %Q printf extension to behave as expected

Remove spurious "union arg" from printf.h

Make sure to always print something in the alternate time format.

Add missing 's' suffix on alternate rendition of time.

Make the %V{is} extension handle a NULL pointer like %s does: output "(null)"

Add %M{essage} extension which prints an errno value as the
corresponding string if possible or numerically otherwise.

It is not currently possible to do the syslog(3) like %m extension
because errno would need to get capatured on entry to the first
function in the printf family, so %M requires you to supply errno
as an argument.

Add %Q{uote} extension which will print a string in double quotes with
appropriate back-slash escapes (only) if necessary.

Explicitely use a "signed char" instead of a "char", for those archs where
char defaults to unsigned.

Add an extensible version of our *printf(3) implementation to libc
on probationary terms: it may go away again if it transpires it is
a bad idea.

This extensible printf version will only be used if either
environment variable USE_XPRINTF is defined
or
one of the extension functions are called.
or
the global variable __use_xprintf is set greater than zero.

In all other cases our traditional printf implementation will
be used.

The extensible version is slower than the default printf, mostly
because less opportunity for combining I/O operation exists when
faced with extensions. The default printf on the other hand
is a bad case of spaghetti code.

The extension API has a GLIBC compatible part and a FreeBSD version
of same. The FreeBSD version exists because the GLIBC version may
run afoul of our FILE * locking in multithreaded programs and it
even further eliminate the opportunities for combining I/O operations.

Include three demo extensions which can be enabled if desired: time
(%T), hexdump (%H) and strvis (%V).

%T can format time_t (%T), struct timeval (%lT) and struct timespec (%llT)
in one of two human readable duration formats:
"%.3llT" -> "20349.245"
"%#.3llT" -> "5h39m9.245"

%H will hexdump a sequence of bytes and takes a pointer and a length
argument. The width specifies number of bytes per line.
"%4H" -> "65 72 20 65"
"%+4H" -> "0000 65 72 20 65"
"%#4H" -> "65 72 20 65 |er e|"
"%+#4H" -> "0000 65 72 20 65 |er e|"

%V will dump a string in strvis format.
"%V" -> "Hello\tWor\377ld" (C-style)
"%0V" -> "Hello\011Wor\377ld" (octal)
"%+V" -> "Hello%09Wor%FFld" (http-style)

Tests, comments, bugreports etc are most welcome.

With current pthread implementations, a mutex initialization will
allocate a memory block. sscanf calls __svfscanf which in turn calls
fread, fread triggers mutex initialization but the mutex is not
destroyed in sscanf, this leads to memory leak. To avoid the memory
leak and performance issue, we create a none MT-safe version of fread:
__fread, and instead let __svfscanf call __fread.

PR: threads/90392
Patch submitted by: dhartmei
MFC after: 7 days

/* You're not supposed to hit this problem */

For some denormalized long double values, a bug in __hldtoa() (called
from *printf()'s %A format) results in a base 16 digit being rounded
up from 0xf to 0x10.

When this digit is subsequently converted to string format, an index
of 10 reaches past the end of the uppper-case hex/char array, picking
up whatever the code segment happen to contain at that address.

This mostly seem to be some character from the upper half of the
byte range.

When using the %a format instead of %A, the first character past
the end of the lowercase hex/char table happens to be index 0 in
the uppercase hex/char table hextable and therefore the string
representation features a '0', which is supposedly correct.

This leads me to belive that the proper fix _may_ be as simple as
masking all but the lower four bits off after incrementing a hex-digit
in libc/gdtoa/_hdtoa.c:roundup(). I worry however that the upper
bit in 0x10 indicates a carry not carried.

Until das@ or bde@ finds time to visit this issue, extend the
hexdigit arrays with a 17th index containing '?' so that we get a
invalid but consistent and printable output in both %a and %A formats
whenever this bug strikes.

This unmasks the bug in the %a format therefore solving the real
issue may both become easier and more urgent.

Possibly related to: PR 85080
With help by: bde@

Fix prototype.

Use the correct function name as .Nm argument.

PR: 86169
Submitted by: Toby Peterson <toby@apple.com>
MFC after: 3 days

Move the declaration of __cleanup to libc_private.h as it is used in both
stdio/ and stdlib/. Don't define __cleanup twice.

Remove references to nonexistent "FreeBSD Security Architecture" document.

Include <sys/types.h> and <limits.h> ourselves, don't assume they are included
through <pthread.h>.

gen/sem.c: Prerequisite for <_semaphore.h>
net/getprotoent.c: USHRT_MAX
net/getservent.c: USHRT_MAX
stdio/ungetwc.c: MB_LEN_MAX
stdio/vfwscanf.c: MB_LEN_MAX

Speed up __wcsconv() (and hence the printf() %ls format):
- use wcsrtombs() instead of a wcrtomb() loop where possible.
- avoid wcrtomb() loop when output precision is small.

The header glue.h should provide just a declaration for the variable
__sglue, not a definition.

PR: 80378
Submitted by: John Engelhart <johne@zang.com>
MFC after: 1 week

The correct description for mode "w" is
(((truncate to zero length) or (create)) (text file)) (for writing)
and not
((truncate file to zero length) or (create text file)) (for writing)

MFC after: 1 week

Be bug-for-bug compatible with the C standard with respect to
printf("%#.0o", 0). Cite an amusing passage from a defect report.

Remove unused variable.

Fix EOVERFLOW detection in vswprintf(3)

Reviewed by: tjr
MFC after: 2 weeks

Sort sections.

Scheduled mdoc(7) sweep.

Document that the length modifier l is ignored for floating point
conversion specifiers (a, A, e, E, f, F, g and G).

Don't add integers to void pointers.

Don't forget to va_end() the va_list we get from va_copy().

Submitted by: Sean McNeil <sean@mcneil.com>
MFC after: 3 days

note that it is the caller's responsibility to free any buffer passed
to setvbuf(3) and friends.

Fix an off-by-one bug that caused the first character of the buffer to
be uninitialized.

Read directly from the stdio buffer using the new __mbsnrtowcs() interface
instead of making repeated calls to __fgetwc().

Implement the GNU extensions of mbsnrtowcs() and wcsnrtombs(). These are
convenient when the source string isn't null-terminated.

Implement the other conversion functions (mbstowcs(), mbsrtowcs(), wcstombs(),
wcsrtombs()) in terms of these new functions.

Use __wcsrtombs() and __sfvwrite() to convert and write the wide character
string instead of multiple calls to __fputwc().

Call __mbrtowc() and __wcrtomb() directly instead of taking detours
through mbrtowc() and wcrtomb().

Add a cross reference to fgetwln(3).

Add fgetwln(), a wide character version of fgetln().

Rename slbexpand() to __slbexpand() and make it available outside
of fgetln.c (non-static).

Slightly reorganize and simplify.

Add commentary explaining why we return EBADF upon attempts to fflush() a
read-only file.

Discussed on: -current

Eliminate double whitespace.

Mechanically kill hard sentence breaks.

Markup, grammar, and spelling fixes.

Spelling fixes.

The third operand of the conditional operator should have type void too.

Approved by: das (mentor)

Rename cantwrite() to prepwrite(). The latter is less confusing,
since the macro isn't really a predicate, and it has side-effects.
Also, don't set errno if prepwrite() fails, since this is done in
prepwrite() now.

Rename cantwrite() to prepwrite(). The latter is less confusing,
since the macro isn't really a predicate, and it has side-effects.

Set errno to EBADF on attempts to write to a stream that is not
writable. Affected callers include fwrite(), put?(), and *printf().
The issue of whether this is the right errno for funopened streams is
unresolved, but that's an obscure case, and some errno is better than
no errno.

Discussed with: bde, jkh

Provide trivial macro implementations of getwc(), getwchar(), putwc() and
putwchar() to reduce function call overhead.

Perform conversions straight from the stream buffer instead of scanning
through byte by byte with mbrtowc(). In the usual case (buffer is big
enough to contain the multibyte character, character does not straddle
buffer boundary) this results in only one call to mbrtowc() for each
wide character read.

Associate a multibyte conversion state object with each stream. Reset it
to the initial state when a stream is opened or seeked upon. Use the
stream's conversion state object instead of a freshly-zeroed one in
fgetwc(), fputwc() and ungetwc().

This is only a performance improvement for now, but it would also be
required in order to support state-dependent encodings.

Remove bogus FBSDID.

Fix damaged FBSDID.

Merge vfscanf.c, v1.37:
- s/#ifdef FLOATING_POINT/#ifndef NO_FLOATING_POINT/

Merge vfprintf.c, v1.65:
- s/#ifdef FLOATING_POINT/#ifndef NO_FLOATING_POINT/
- Remove HEXFLOAT

- To make it easier to compile *printf() and *scanf() without
floating-point support, remove default definition of FLOATING_POINT
from the source, and change the compile-time option to
NO_FLOATING_POINT.
- Remove the HEXFLOAT option. It saves an insignificant amount of
space (<0.1% of the size of libc on i386) and complicates vfprintf()
and checkfmt().

Use the correct size to allocate, copy and clear argument type tables
after their change from an array of char to an array of enum.
This fixes problems that occurred when using positional arguments in
format strings, particularly with more than STATIC_ARG_TBL_SIZE (8)
of them.

PR: 65841
Submitted by: Steven Smith (mostly)

Prepare to handle trivial state-dependent encodings. Full support for
state-dependent encodings with locking shifts will come later if there
is demand for it.

Mention that funopen() uses fpos_t incorrectly in the BUGS section.

Improve documentation for fgetpos() and fsetpos(), and discourage
users from assuming that fpos_t is an integral type.

Do not redundantly set the stream orientation in getc(), putc(), and
related functions - __sgetc() and __sputc() will set it when necessary.

Update list of macros defined in <stdio.h>.

Re-add description of putc() macro (back out rev. 1.13.)

Re-add text that says getc() is a macro (back out rev. 1.16.)

Re-add macro versions of getc(), getchar(), putc(), putchar(), feof(),
ferror(), fileno() and clearerr(), using the value of __isthreaded to
decide between the fast inline single-threaded code and the more
general function equivalent. This gives most of the performance
benefits of the old unsafe macros while preserving thread safety.

Refer to "wide characters" instead of "wide-characters".

Set stream orientation in ungetc() instead of __ungetc(). This avoids
setting it redundantly when called from ungetwc(), vfscanf() etc.,
which already set the orientation.

Remove duplicate check for EOF from ungetc(); __ungetc() already checks.

Call __sputc() directly in fputc() instead of taking an expensive
detour through putc().

Call __sgetc() directly in getchar() instead of taking an expensive
detour through getc().

Set the stream orientation explicitly in fgetln() instead of relying on
__srefill() to do it.

Set stream orientation in puts().

Backout the previous revision due to objections.

On 4.X it was possible for an application to initialise a local FILE
structure and call stdio functions. In 5.X this was broken when FILE
locking was introduced into libc.

This change makes most (relevant) stdio functions work again when the
_extra file in FILE isn't initialised (and can't be without a libc
function to do it since the __sFILEX structure is private to libc).

If handed a file pointer we can't write to, set errno properly to EBADF
in order to get SUSv2 conformant behavior in higher level calls like
fputs() and puts().

Reviewed by: bde

Document dependence of mktemp(3) on the non-reentrant arc4random(3).

Merge vfscanf.c,v 1.35.

Fix the handling of negative hexadecimal numbers in integer formats.
Note that this bug is unrelated to recent work in this area; it seems
to have been present since revision 1.1.

Obtained from: NetBSD

Merge vfprintf.c, v1.61 and 1.62. For compatibility with __hdtoa()
and to reduce diffs between vfprintf.c and vfwprintf.c, declare xdigs*
to be char arrays rather than wchar_t arrays.

In collaboration with: tjr

Bring the *printf(3) documentation up to date with the code:

- Update and improve the documentation for %[aA]
o Like %[eE], %[aA] may round the result if a precision is specified.
o Grammar police: Fix a split infinitive.
o The FreeBSD implementation does better than the minimum required
by C99 (literal translation of the mantissa). The digit before
the hexadecimal-point is never 0 unless the number itself is 0.
o Clarify that the exponent field represents a decimal exponent of 2.
o Discuss the fact that multiple valid representations are possible.
o Remove the entry in the BUGS section claiming that %[aA] is not
implemented.

- Remove the entry in the BUGS section claiming that the ' flag for
printing thousands separators is unimplemented for floating-point.

- Remove the entry in the BUGS section claiming that the L modifier
reduces the precision to "double" before conversion.

Implement __hdtoa() and __hldtoa() and enable printf() support for %a
and %A, which print floating-point numbers in hexadecimal.

Prototype __hdtoa() and __hldtoa().

Fix some bugs affecting the %a and %A format specifiers. Since
these are not fully implemented and ifdef'd out, the bugs have
never manifested themselves. Specifically:

- Fix a memory leak in the case where %a follows another
floating-point format.
- Make the %a/%A code behave like %e/%E with respect to
precision.
- It is no longer valid to assume that '-' and '0x' are
mutually exclusive.
- Address other minor issues.

Make intentions explicit with additional parenthesis.

Remove unused variables and function declarations. Add missing headers.

Fix prototype for getchar_unlocked().

Use __sfvwrite() instead of __sputc() via __fputwc() to write to fake
string files (__SSTR flag set). This is necessary because __sputc()
does not respect the __SALC flag, and crashes trying to flush the buffer
instead of resizing it.

PR: 59167

Pass NULL instead of a pointer to a zeroed mbstate_t object.

Pass mbrtowc() and wcrtomb() NULL instead of a pointer to a freshly zeroed
mbstate_t object that they ignore. The zeroing is fairly expensive, and it
will never be necessary in these functions; when we support state-dependent
encodings, we will pass in a pointer to the file's mbstate_t object, and
only zero it at the time the file gets opened.

Fix a typo that caused the optimized single-byte locale path not to be taken.

mdoc(7): Use the new feature of the .In macro.

Add more useful cross-references to the SEE ALSO section.

Catch up with recent FP-related changes to scanf.3 and vfwscanf.c.

Fix two incorrect uses of sizeof: we need to divide the size of the buffer
by sizeof(wchar_t) to get the number of wide characters it contains.
Remove the !hardway micro-optimisation from the CT_INT case to avoid
having to fix it for wide characters.

Merge recent floating point conversion changes from vfscanf.c.

When size is 1 should just null terminate the string. The dummy variable
is made an array of two, to explicitly avoid stack corruption due to
null-terminating (which is doesn't actually happen due to stack alignment
padding).

Submitted by: Ed Moy <emoy@apple.com>
Obtained from: Apple Computer, Inc.

Update to reflect changes in vfscanf.c,v 1.32. Remove bogus
documentation for %a, and document it correctly instead.
s/one of aefg/one of a, e, f, or g/

Reviewed by: standards@

Revamp scanf's floating-point-parsing algorithm to support
[+|-]Inf, [+|-]NaN, nan(...), and hexidecimal FP constants.
While here, add %a and %A, which are aliases for %e, and
add support for long doubles.

Reviewed by: standards@

Add or correct section number in .Xr. Use .Vt or .Fn
instead of .Xr when needed

Assorted mdoc(7) fixes.

Approved by: re (blanket)

%E-like %g and %G conversions should remove trailing zeroes unless
the # flag is present. Implement this behavior and add a comment
describing it.

Noticed by: Enache Adrian <enache@rdslink.ro>
Pointy hat to: das

Merge in vfprintf.c rev. 1.58.

Fix a bug where printf was erroneously printing a decimal point for
%f and sufficiently short %g specifiers where the precision was
explicitly zero, no '#' flag was specified, and the floating point
argument was > 0 and <= 0.5. While at it, add some comments to better
explain the relevant bits of code.

Noticed by: Christoph Kukulies <kuku@physik.rwth-aachen.de>

Catch up with recent vfprintf.c changes.

__wcsconv(): free(convbuf) before returning NULL

Today just isn't my day. Remove some old commented out code that snuck
into the last commit.

Noticed by: mike

- %e conversions with precision 0 should not cause a decimal point to
be printed.
- Fix %f conversions where the number of significant digits is < expt.
This would be a one-line change were it not for thousands separators.
Noticed by tjr.
- Remove some unnecessary code in the parsing of precision specifiers.

Rework the floating point code in printf(). Significant changes:

- We used to round long double arguments to double. Now we print
them properly.

- Bugs involving '%F', corner cases of '#' and 'g' format
specifiers, and the '.*' precision specifier have been
fixed.

- Added support for the "'" specifier to print thousands' grouping
characters in a locale-dependent manner.

- Implement the __vfprintf() side of hexadecimal floating point
support. All that is still needed is a routine to convert the
mantissa to hex digits one nibble at a time in the style of ultoa().

Reviewed by: silence on standards@

Add missing #include to unbreak previous commit.

Correct some buffer sizes.

- __vfprintf()'s 'buf' has never been used for floating point, so
don't define it in terms of (incorrect) constants describing
floating point numbers. The actual size needed depends on
sizeof(uintmax_t) and locale details, so I slightly overestimated.

- We don't need a 308-character buffer to store the string "308".
With long doubles and %a we need more than three characters, though.

Merge vfprintf.c revision 1.52.

Kludge around a bug that results from printf() assuming that
dtoa() is buggy. The bug would cause incorrect output to be
generated when format strings such as '%5.0f' were used with
nonzero numbers whose magnitude is less than 1.

Reported by: df(1) by way of periodic(8)
Reviewed by: mike

Unexpand RCS tag.

MFp4: Catch up to recent __dtoa() interface changes and removal of cvt()'s
last argument.

Replace our ancient dtoa/strtod implementation with the gdtoa
package, a more recent, generalized set of routines. Among the
changes:
- Declare strtof() and strtold() in stdlib.h.
- Add glue to libc to support these routines for all kinds
of ``long double''.
- Update printf() to reflect the fact that dtoa works slightly
differently now.

As soon as I see that nothing has blown up, I will kill
src/lib/libc/stdlib/strtod.c. Soon printf() will be able
to use the new routines to output long doubles without loss
of precision, but numerous bugs in the existing code must
be addressed first.

Reviewed by: bde (briefly), mike (mentor), obrien

MFp4: Add the standard "the {fgetws,fputws} function will fail if" text
to the Errors section.

mdoc(7) police: markup laundry.

Fix the description for mkdtemp(), which creates directories, not files.

Submitted by: Murray S. Kucherawy <msk@blackops.org>
X-MFC after: re approval

Eliminate 61 warnings emitted at WARNS=2 (leaving 53 to go).
Only warnings that could be fixed without changing the generated object
code and without restructuring the source code have been handled.

Reviewed by: /sbin/md5

Fix use of an uninitialized pointer introduced in a previous revision.

Approved by: markm (mentor)(implicit)

Back out previous. Many people disagreed with removing the warning.

Remove runtime warning about gets().

Lock stdin on entry, unlock on return, use __sgetc() instead of getchar()
to avoid locking the stream for each character and to avoid input being
scattered among multiple threads.

Remove part of my stateful locale patch that slipped into the previous rev.

Initial implementation of the C99 feature whereby calling freopen() with
a NULL filename argument allows a stream's mode to be changed. At the
moment it just recycles the old file descriptor instead of storing the
filename somewhere and using that to reopen the file, as the standard
seems to require. Strictly conforming C99 applications probably can't
tell the difference but POSIX ones can.

PR: 46791

Lock the stream before calling __sfileno() to retrieve the file descriptor.
1003.1-2001 requires that fileno() behave as if it locks the stream.

Add missing word to "Return Values" section.

The macro versions of putc() and putchar() are gone.

There is no macro version of getc() anymore.

Bring the list of function-like macros up to date with reality.

spell multiple correctly.

Document clearerr_unlocked(), feof_unlocked(), ferror_unlocked()
and fileno_unlocked().

Add a manual page for flockfile(), ftrylockfile(), and funlockfile().

Document getc_unlocked(), getchar_unlocked(), putc_unlocked(),
and putchar_unlocked().

Add function versions of getchar_unlocked(), getc_unlocked(),
putchar_unlocked(), putc_unlocked(), feof_unlocked(), ferror_unlocked(),
clearerr_unlocked(), and fileno_unlocked(). The first four are required
by POSIX. The rest are provided for consistency.

Remove an unused variable: mbresult.

#include <stdlib.h> for free()'s prototype.

Note that the printf(3) and scanf(3) family of functions don't deal with
multibyte characters in the format string correctly.

#include "local.h" to make __svfscanf()'s prototype visible.

Reset the stream orientation to 0 (unoriented) in freopen(), as required
by the C standard.

mdoc(7) police: Deal with self-xrefs.

Document the fact that the printf() family of functions return negative
values (EOF in our case) on error, and some of the possible errno values
in an Errors section.

PR: 39257

C99 standardised the vscanf() family of functions, update Standards
section to reflect this.

Write the message to stderr, not file descriptor 2, so that perror()
writes to the correct stream if stderr has been redirected with freopen().

Use strerror_r() to format the error message so that strerror()'s static
buffer does not get clobbered.

ISO/IEC 9899:1999 7.21.6.2 3:
"The implementation shall behave as if no library function calls the
strerror function."

mdoc(7) police: "The .Fa argument.".

mdoc(7) police: Fixed abuses of the .Ar and .Em macros.

mdoc(7) police: "The .Fn function".

mdoc(7) police: tiny nits.

Approved by: re

Consistently mark std(in|out|err) with .Dv, because that's how they
are marked up in stdio(3), and because they are defined expressions
of type "FILE *".

Approved by: re

mdoc(7) police: markup overhaul.

Approved by: re

mdoc(7) police: formatting nits.

Approved by: re

libc_r wasn't so tied to libc for 22 months.

bde points out that the LIBC_MAJOR macro doesn't exist and requests
that we not use it here. In its place I've put a comment about the
current state of play.

Submitted by: bde

Restore Peter's version of static __sF. There's too much pain for it
to be static for 5.0. I may remove this for 5.1 or 5.2. No more
binaries or libarires will be generated with __sF starting as of
yesterday. Originally the plan had been to eliminate this for 5.0,
but we didn't get the __std{in,out,err}p changes merged into -stable
until yesterday (rather than in September 2001 like it should have
been). Given that didn't happen on time, we can't do the other part
of the scheme now.

# Please do not change this without talking to me first.

Backout "compatibility hack" for __sF.

Requested by: Steve Kargl <sgk@troutmask.apl.washington.edu> (submitter)

Provide a hook to make __sF visible outside of libc for commercial apps
if WANT_COMPAT4_STDIO is defined when compiling libc.

Submitted by: Steve Kargl <sgk@troutmask.apl.washington.edu>

No need to include floatio.h here: vfscanf() no longer uses anything
it defines.

Make __sF static. This can not be allowed to exist in 5.x.

Cross-reference putc(3).

The ORIENTLOCK macro is no longer needed since all functions use
FLOCKFILE/FUNLOCKFILE explicitly.

Cross-reference fmtcheck(3).

Give in on the __sF stuff. I have a better fix in mind that is future
proof, but this should buy me some time for now.

Indent code example with one tab, not two, for consistency with the rest.

Fix off-by-one error when pushing back a multibyte sequence in
wide character class (%l[) and wide string (%ls) conversions.

Make part of the previous change clearer; check flags for SUPPRESS directly
instead of checking whether we're using a temporary buffer.

The field width for single-byte string conversions (%c, %s, %[) is the
maximum number of bytes that may be stored in the array, not the maximum
number of wide characters to read. The wording of the standard unfortunately
does not make this clear.

Count field width correctly for suppressed multibyte fields (%*lc,
%*ls, %*l[).

Set the error bit on the stream if an encoding error occurs. Improve
handling of multibyte sequences representing null wide characters.

Add support for the XSI %C and %S formats, which are the same as %lc
and %ls.

Use a weak reference instead of a macro to make vfscanf an alias
for __vfscanf.

Ignore TMPDIR if the application is setugid.

MFC after: 2 weeks

Add restrict type-qualifier.

Describe the restrictions on seeking on wide character streams, and also
point out that fseek() clears the ungetwc() buffer.

Zap the early-adopter transition aid before we get into serious
5.0-R territory, as threatened. This only affects antique 5.0
systems that have not had a 'make world' done for well over a year.

Cross-reference between byte and wide-character stdio functions. Remove
references to fputwc() and fgetwc() being macros while I'm at it.

Remove masking macros for getwc(), putwc(), putwchar() and getwchar().
Although there was nothing wrong with getwc() and putwc(), getwchar()
and putwchar() assumed that <stdio.h> had been included before <wchar.h>,
which is not allowed by the standard.

Back out previous, free the buffer when __vfprintf() fails and don't bother
trying to shrink the buffer with realloc() before returning it.

Back out previous and solve the problems a different way: move va_start/
va_end closer to the __vfprintf() call, free the buffer when __vfprintf()
fails and don't bother trying to shrink the buffer with realloc() before
returning it.

Submitted by: bde

Correctly handle the case where __vfwprintf() fails because it runs out
of memory.

Sync with OpenBSD: avoid memory leak when __vfprintf() fails because it
runs out of memory, always call va_end.

Add cross-references between wide character and single-byte character
versions of printf() and scanf().

Remove an unneeded call to _sfrefill() that was missed in the conversion
from vfscanf() to vfwscanf(). It doesn't hurt to have it there, but it's
redundant since __fgetwc() will refill the buffer if it needs to.

Use the new va_copy macro to copy variable argument lists instead of
assignment. This is needed on powerpc but is also more correct for the
other ports.

Submitted by: grehan
Tested on: alpha, i386, sparc64

Add implementations of wscanf() and related functions: fwscanf(), swscanf(),
vfwscanf(), vswscanf(), vwscanf(). As the name suggests, these are wide-
character versions of the scanf() family of functions.

Implement the %lc, %ls and %[ conversions, which read sequences of wide
characters, non-whitespace wide character strings and wide character
strings in a scanset.

The character argument for __ungetwc() should be wint_t instead of wchar_t.

Add an unlocked version of ungetwc(), __ungetwc(), that __vfwscanf()
will need to use.

Delete stray reference to vsnprintf().

Add implementations of the wprintf() family of functions, which perform
formatted wide-character output.

Lock the file once per call and use the unlocked fgetwc()/fputwc() variants.

Lock and unlock the file once per call and use the unlocked version of
ungetc() instead of having ungetc() recurse on the lock.

Introduce unlocked versions of fputwc() and fgetwc() called __fputwc()
and __fgetwc() which can be used when we know the file is locked.

Implement the %ls and %lc conversions for printing wide character strings
and wide characters. These were already documented in the manual page,
with an entry mentioning that they were not implemented yet. The XSI
%S and %C synoyms have not been added.

Optimise the common case where no special encoding is in use (LC_CTYPE is "C"
or "POSIX", other European locales). Use __sgetc() and __sputc() where
possible to avoid a wasteful lock and unlock for each byte and to avoid
function call overhead.

Logic error in previous: don't exit the loop when an incomplete multibyte
sequence is detected.

Reimplement the functionality of fgetrune(), fputrune(), and fungetrune()
here in terms of mbrtowc(), wcrtomb(), and the single-byte I/O functions.
The rune I/O functions are about to become deprecated in favour of the
ones provided by ISO C90 Amd. 1 and C99.

Fix vsnprintf(3) memory leak for size == 0.

PR: bin/36175
Obtained from: OpenBSD
Reviewed by: silence on -audit
MFC after: 5 days

Convert eight space indents to tabs in the "*" format handling code.

Style: One space between "restrict" qualifier and "*".

Print a '-' sign for negative zero. Tested with

#include <stdio.h>
int main(void)
{
printf("%+f\n", -0.0);
printf("%+f\n", +0.0);
printf("%+f\n", 0.0);
return 0;
}

to output
-0.000000
+0.000000
+0.000000

PR: bin/41823
Submitted by: GOTO Kentaro <gotoken@notwork.org>
Liked by: bde
MFC after: 3 weeks

o Merge <machine/ansi.h> and <machine/types.h> into a new header
called <machine/_types.h>.
o <machine/ansi.h> will continue to live so it can define MD clock
macros, which are only MD because of gratuitous differences between
architectures.
o Change all headers to make use of this. This mainly involves
changing:
#ifdef _BSD_FOO_T_
typedef _BSD_FOO_T_ foo_t;
#undef _BSD_FOO_T_
#endif
to:
#ifndef _FOO_T_DECLARED
typedef __foo_t foo_t;
#define _FOO_T_DECLARED
#endif

Concept by: bde
Reviewed by: jake, obrien

Leave room for a trailing NUL not a NULL, that's not an ASCII character.

- For compliance with IEEE Std 1003.1-2001, add the 'restrict'
qualifier to function prototypes and definitions where
appropriate using the '__restrict' macro.
- Update the manual page.

- Introduce the 'restrict' qualifier to function prototypes and
definitions to comply with IEEE Std 1003.1-2001.
- Update the manual pages.

- Add the 'restrict' qualifier to the function definitions and
public prototypes of setbuf(3) and setvbuf(3) using the
'__restrict' macro from <sys/cdefs.h> to be compliant with
IEEE Std 1003.1-2001.
- Replace the K&R with ANSI-C function definitions.
- Bring the manual page up-to-date.

Unbreak one of the most confusing breaks of the tree I've seen.

The last commit cannot possibly have been tested.

Tidy up SRCS and MAN assignments.

Remove the Bugs section, which is not relevant to FreeBSD: there is no
one-character ungetwc(3) buffer limit.

FreeBSD-ify: use In macro for header files in Synopsis, St C-99 instead of
St C99 in Standards section.

Basic support for wide character I/O: getwc(), fgetwc(), getwchar(),
putwc(), fputwc(), putwchar(), ungetwc(), fwide().

Manual pages for fwide(), getwc(), fgetwc(), getwchar(), putwc(), fputwc(),
putwchar(), ungetwc() from NetBSD and Citrus Project, unmodified except
for the addition of $FreeBSD$.

Obtained from: NetBSD, Citrus Project

Fix typos; each file has at least one s/seperat/separat/
(I skipped those in contrib/, gnu/ and crypto/)
While I was at it, fixed a lot more found by ispell that I
could identify with certainty to be errors. All of these
were in comments or text, not in actual code.

Suggested by: bde
MFC after: 3 days

mdoc(7) police: punctuation.

Don't claim to fully implement C99 in the STANDARDS section and then disclaim
compliance in the BUGS section immediately below.

Note that fseeko() and ftello() are standard in 1003.1-2001.
(Prefer the more-encompassing POSIX standard to SUSv2.)

Fix style and wording bugs introduced in my last commit.

Sponsored by: DARPA, NAI Labs

o Move more information from BUGS into SECURITY CONSIDERATIONS and
condense the redundant bits.
o Provide an example for using snprintf over sprintf. This may be
supplemented with an asprintf() example soon.

Sponsored by: DARPA, NAI Labs

Include information on the dangers of passing a user-supplied string as
a format string. This will later on be changed to a reference to the
FreeBSD Security Architecture after it has been committed.

PR: docs/39320
Sposnored by: DARPA, NAI Labs

Grammar fix: "contents" is plural.

MFC after: 1 day

mdoc(7) police: markup nits.

mdoc(7) police: fix markup for types.

Assume __STDC__, remove non-__STDC__ code.

Submitted by: keramida

Correct a check for NUL.

Spotted by: bde

- Move the loop conditional into the "for" header.
- Remove redundant "? :" construct.
style(9):
- Place a space after return statements.
- Compare pointers to NULL.
- Do not use ! to compare a character to nul.

Avoid wandering over the beginning of the actual buffer
if the passed template string contains only 'X' characters.

Submitted by: Mark Andrews <marka@isc.org> (patch modified)
PR: 38402

Clarify that the value of getc() etc. is an unsigned char converted
to an int (as per the C standard) i.e. it can be passed straight to
isalpha() etc.

Approved by: dwmalone (mentor)
MFC after: 3 days

Implement several of the c99 updates to scanf(3):
- New length modifiers: hh, j, ll, t, z.

Still to do:
- %C, %S, %lc, %ls (wide character support)
- %a/%A (exact hex representation of floating-point numbers)

Removed old compatability equivalents:
- %D for %ld, %O for %lo, %X for %lx, %E and %F for %le & %lf (these
were buggy anyway, since they should have represented %Le & %Lf).
- %[unknown uppercase char] for %ld, %[unknown lowercase char] for %d

scanf.3 has an obsolete ``this release''.

PR: 35610
MFC after: 2 days

Fix the style of the SCM ID's.
I believe have made all of libc .h's as consistent as possible.

Fix the style of the SCM ID's.
I believe have made all of libc .c's as consistent as possible.

Remove multi-line __P() usage.

Remove __P() usage.

Remove 'register' keyword.

Don't refer the reader to look for a word that doesn't exist anywhere
else in the page.

PR: 35623
Submitted by: Gary W. Swearingen <swear@blarg.net>

mdoc(7) police:

Stop abusing the .%J macro for where the .Pa macro should have been used.

All information from the "BUGS" section not belonging in "SECURITY
CONSIDERATIONS" moved to "COMPATIBILITY".

Obtained from: TrustedBSD Project
Sponsored by: DARPA, NAI Labs

Add new "SECURITY CONSIDERATIONS" sections.

Obtained from: TrustedBSD Project
Sponsored by: DARPA, NAI Labs

Fix some style bugs

Prompted by: mike

Also fix cases when thousands separator should be put before number. For
example before for grouping sequence "\003\003" number 123456 was formated
as ",123,456", now "123,456".

Respect locale while handling of \' flag.

In original version grouping was hardcoded. It assumed that thousands
separator should be inserted to separate each 3 numbers. I.e. grouping
string "\003" was assumed for all cases. In correct case (per POSIX)
vfprintf should respect locale defined non-monetary (LC_NUMERIC
category) grouping sequence.

Also simplify thousands_sep handling.

mdoc(7) police: Minor formatting nits and optimizations to rev. 1.34.

* localeconv() usage is not FLOATING_POINT specific anymore (due to "'" flag
addition) so move locale.h inclusion out of FLOATING_POINT ifdef's.
* add more comments

Remove blank line.

Implement several of the c99 updates to printf(3):
- New length modifiers: hh, j, t, z.
- New flag: '. Note that %'f is not yet implemented.
- Use "inf"/"nan" for efg formats, "INF"/"NAN" for EFG formats.
- Implemented %q in terms of %ll; if "quad_t" is not "long long"
%q will break.

Still to do:
- %C, %S, %lc, %ls (wide character support)
- %'f (thousands in integer portion of %f)
- %a/%A (exact hex representation of floating-point numbers)

Garrett Wollman wrote the first version of the vfprintf.c update;
Mike Barcroft wrote the first version of the printf.3 changes.

Back out national digits support, POSIX explicetely disallows it:

The definition of character class digit requires that only ten characters
-the ones defining digits- can be specified; alternate digits (for
example, Hindi or Kanji) cannot be specified here. However, the encoding
may vary if an implementation supports more than one encoding.

The definition of character class xdigit requires that the characters
included in character class digit are included here also and allows for
different symbols for the hexadecimal digits 10 through 15.

Don't ever assume that isdigit() is always subset of isxdigit()

Allow national (non-ASCII) digits

mdoc(7) police: sort xrefs.

Cross-reference the fdopen and fileno manpages.

PR: docs/31866
Submitted by: W. Campbell <wcampbel@botbay.net>

In > LONG_MAX test use sseek return value and not _offset which can be not
active.

Help to recover from bad seek (i.e. negative or too big) happens beyond
our pre-check control. Do the same way as refill.c does when it set __SERR,
i.e. clear read and ungetc buffers. Clear EOF flag too.

Back out read buffer invalidating via __SMOD.

It was correct, but not needed because internal buffer cleared on each seek
outside of it.

Change comment explaining another usage of __SMOD

Disallow fseek() optimization in internal read buffer, if pointer is moved by
seek. It means that beginning of read buffer becomes not the same as current
file position.

Fixed style bugs in previous commit.

Make this compile on ia64.

mdoc(7) police: Use the new .In macro for #include statements.

1) If __SAPP stream is not seekable, remove __SAPP flag on first call instead
of repeating unsuccessful lseek call on each write (original stdio bug).

2) Save errno accross _sseek call in _swrite to not touch it in case write
success (original stdio bug).

3) Add _sseek error checking back, but only for __SOPT mode now.

For now just back out seek error checking in __SAPP case, it cause problems
with non-seekable streams. Now here is what here was originally, but it is
ugly, producing unneded seek syscall on each non-seekable stream write. I'll
think about proper solution later.

When __SOPT is cleared, clear __SOFF too.

NOTE: original stdio bug.

Re-arrange my funopen(3) fix to minimize differences with original stdio code,
no functional changes.

Add fp->_offset optimization in _SAPP+_SOPT case

Internal seeks are overoptimized. They should remember fp->_offset only for
plain regular files, i.e. files with __SOPT flag set. Fix it, so ftell(stdout)
always returns the same as lseek(1, 0, 1) now.

NOTE: this bug was in original stdio code

Fix bug in off_t overflow checking: if fp->_offset overflows, just remove
__SOFF flag (i.e. we don't have offset) instead of returning EOVERFLOW.
It allows again continious reading from non-stop stream.

Typo seek(2) -> lseek(2)

Move all stdio internal flags processing and setting out of __sread(),
__swrite() and __sseek() to higher level. According to funopen(3) they all
are just wrappers to something like standard read(2), write(2) and
lseek(2), i.e. must not touch stdio internals because they are replaceable
with any other functions knows nothing about stdio internals. See example
of funopen(3) usage in sendmail sources f.e.

NOTE: this is original stdio bug, not result of my range checkin added.

Save errno before function call and restore it on success (because many
internal functions there may fail and set (i.e. overwrite) errno in normal
(not error) situation). In original variant errno testing after call
(as POSIX suggest) is wrong when errno overwrite happens.

Remove even more unneded checks, original code can't overflows in that place

Remove two checks unneeded now (can't happens)

Make fseek(... SEEK_CUR) fails if current file-position is unspecified.

Note that prev. commit addition is for ftell/ftello

Describe ESPIPE as result of unspecified file-position indicator value.
Add more to SEE ALSO section.

Describe file-position behaviour from POSIX

Strict in the POSIX sence, if file position is unspecified after ungetc() at
0, return that we can't specify it, i.e. error with ESPIPE.
(hint from: "Peter S. Housel" <housel@acm.org>)

Back out sinit() addition, not needed after various code simplifications.

If lseek to wrong value sucessfully happens despite all pre-checks, set __SERR
to indicate that stream becomes inconsistent.

If position is underflowed, don't try to hide that fact by recovery, just
return EIO and set __SERR to mark stream as inconsistent.

Back out disabling ungetc() at 0, use different solution:
keep negative offset internally, but return 0 externally in ftell*()
I.e. use 0 now as 'unspecified value' per POSIX ungetc() description.

Add originally missing __sinit() call.

Simplify offset underflow checks even more

Describe that we disallow ungetc at offset 0 now.

Disallow ungetc at offset 0 (to prevent negative offset happens), so simplify
checks in ftell.

Drop buffer first, _then_ ask for real position

The same big piece of ftell code repeated in 3 places. Simplify things moving
it into one subfunction instead.
Try to use real offset in strange cases.

If file offset is smaller than internal buffer character left count, just drop
internal buffer and trust offset, not return error.

Initialize _offset to 0 in fopen(), it helps to optimize fseek/ftell

Detect fp->_offset overflow on read
Use errno to catch negative seek with -1 offset

Use ``.Rv -std'' wherever possible.

Submitted by: yar

Try to discard some ungetc data in saved internal buffer checks too,
if offset tends to be negative.

goto dumb; if can't obtain curoff for whence != SEEK_CUR cases, as supposed

Add more EOVERFLOW checks.

When file offset tends to be negative due to internal and ungetc buffers
additions counted, try to discard some ungetc data first, then return EBADF.
Later one can happens if lseek(fileno(fd),...) called f.e. POSIX says that
ungetc beyond beginning of the file results are undefined, so we can just
discard some of ungetc data in that case.

Don't rely on gcc cast when checking for overflow, use OFF_MAX.

Cosmetique.

Cosmetique fixes from bde

Cosmetique: correct English in comments

Mark some functions as __printflike() and/or taking const char * arguments
instead of char *.

MFC after: 2 weeks

Add about rewind+errno, describe ESPIPE, minor formatting.

Simplify overflow calculations a bit

Remove extra check, already done in upper level caller, i.e. in
_fseeko()

Mention ftell & ftello in EOVERFLOW section too.

Add more overflow checks in case of fseek()

Don't clear "we have offset" flag even if long is overflow for fseek(),
there is no harm to have it, it will reduce next call efforts.

fseek.c:
Resulting fseek() offset must fit in long, required by POSIX (pointed by bde),
so add LONG_MAX and final tests for it.

rewind.c:
1) add missing __sinit() as in fseek() it pretends to be.
2) use clearerr_unlocked() since we already lock stream before _fseeko()
3) don't zero errno at the end, it explicitely required by POSIX as the
only one method to test rewind() error condition.
4) don't clearerr() if error happens in _fseeko()

Explain the relation of getchar() to getc() in less words.

Submitted by: ru

Use smarter overflow tests

Suggested by: bde

1) Disallow negative seek as POSIX require for fseek{o} (but not for lseek):

"[EINVAL] ... The resulting file-position indicator would be set to a
negative value."

Moreover, in real life negative seek in stdio cause EOF indicator cleared
and not set again forever even if EOF returned.

2) Catch few possible off_t overflows.

Reviewed by: arch discussion

Rip out the old __stdin/out/err stuff. It was completely 100% useless. :-(
It was foiled because of dynamic copy relocations that caused compile-time
space to be reserved in .bss and at run time a blob of data was copied to
that space and everything used the .bss version.. The problem is that
the space is reserved at compile time, not runtime... So we *still* could
not change the size of FILE. Sigh. :-(

Replace it with something that does actually work and really does let us
make 'FILE' extendable. It also happens to be the same as Linux does in
glibc, but has the slight cost of a pointer. Note that this is the
same cost that 'fp = fopen(), fprintf(fp, ...); fclose(fp);' has.
Fortunately, actual references to stdin/out/err are not all that common
since we have implicit stdin/out/err-using versions of functions
(printf() vs. fprintf()).

Use .Fn, .Fa, and .Dv where appropriate.

Use the ``.Rv -std'' mdoc(7) macro in appropriate cases.

Reviewed by: ru

mdoc(7) police:

Avoid using parenthesis enclosure macros (.Pq and .Po/.Pc) with plain text.
Not only this slows down the mdoc(7) processing significantly, but it also
has an undesired (in this case) effect of disabling hyphenation within the
entire enclosed block.

Avoid any chance of being misunderstood as having libelled developers
or developers' vendors without compromising the importance of warning
against bad practice.

Reported by: mjacob
MFC after: 1 week

Remove whitespace at EOL.

mdoc(7) police: removed HISTORY info from the .Os call.

mdoc(7) police: remove extraneous .Pp before and/or after .Sh.

Attempt to use the environment variable TMPDIR for the temporary
directory, defaulting to /tmp.

PR: bin/16924
Reviewed by: dd
MFC after: 2 weeks

mdoc(7) police: fixed/simplified formatting.

(cantwrite): do not allocate memory for a NULL string

PR: misc/26044

MFC after: 1 week

Submitted by: bde

revert freeing of memory that gets allocated when str == NULL
(this will be fixed in a better way)

PR: misc/26044

free memory that gets allocated by vfprintf when str == NULL

PR: misc/26044

MFC after: 1 week

Add description for the 'll' modifier for long long. In essence a copy
from the 'l' description with s/long/long long/g.

PR: 27017
Submitted by: Guy Harris <guy@alum.mit.edu>
MFC after: 2 weeks

Back out my changes describing how snprintf nul terminates. It
was from the iso standard. Keep the sentence that says it is always
NUL terminated to make sure that people understand that.

Requested by: bde

Minor improvements:
o Explain snprintf's return value better.
o Document snprintf, et al, were defined in C-99
o Warn against %n.

Properly initialize a temporary FILE that is allocated off the stack.
The change to reuse _up from FILE (to allow FILE to grow without changing
size) overlooked FILE being allocated off the stack.

Approved by: sobomax

Correct the documentation for snprintf() and vsprintf() which actually
return the number of characters that would have been in the new string.

Obtained from: OpenBSD
MFC after: 3 days

Add warnings about trusting user-supplied data.

Reviewed by: ru
Approved by: murray
Obtained from: OpenBSD

mdoc(7) police: fix markup.

To be consistent, use the __weak_reference macro from <sys/cdefs.h>
instead of #pragma weak to create weak definitions. This macro is
improperly named, though, since a weak definition is not the same
thing as a weak reference.

Suggested by: bde

MAN[1-9] -> MAN.

Fix style nit.

Fix style breakage.

Hide the definition of struct __sFILEX and add the needed
lock definitions to it. flockfile state is now allocated
along with the rest of FILE. This eliminates the need for a
separate allocation of flockfile state as well as eliminating
the mutex/lock used to serialize its allocation.

.St -ansiC -> .St -isoC

Correctly handle the race itself, too (don't leave it locked).
This is about to be replaced anyway by initialization explicitly
instead of lazily, and reducing the complexity of it. As it is
now, this will work fine, however.

Use the right names to call pthread_mutex_{,un}lock so that things
work in both the libc only and libc/libc_r case.

Fix the problems I (and others, undoubtedly) have been having for a
while with threaded software in -CURRENT acting very "weird". It has
seemed, for example, in Mozilla that threads attempting to do host
lookups have been locking up. That's exactly the case.

There was a race condition in the implementation of the initialization
of the mutex used to protect FILE operations, first of all: multiple
instances of FLOCKFILE() in libc could occur on the same FILE at
the same time and cause strange behavior by overwriting eachothers'
creation of the mutex and the rest of the file lock.

Secondly, it's not appropriate to test the "validity" of the file
descriptor referenced by the FILE; if the code is calling FLOCKFILE()
or FUNLOCKFILE(), it wants the FILE to be locked or unlocked, not
to be locked or unlocked on the condition that _file is >= 0. This
also could quite easily cause leaks by failing to perform the lock or
unlock operation when it actually is needed.

Mozilla now works again on -CURRENT when linked to libc_r.so.5 and
libc.so.5.

Place some hooks (__stdin, __stdout, __stderr) into libc for a future
ABI change. There is some serious evilness here to work around some
gcc weaknesses. We need to know the sizeof(FILE) manually until __sF
goes away in the next major bump. We have the size for Alpha and i386,
missing is ia64, ppc and sparc* (and i386 with 64 bit longs).
At some point down the track we can change the stdin etc #defines to
stop hard coding the size of FILE into application binaries.

Lots of head scratching and ideas and testing by: green, imp

Extra needs to be initialized for our usual pool of FILEs. This was
causing some versions of as to dump core. This survived make
buildworld/installworld and the building gettext port afterwards.

Submitted by: <nnd@mail.nsk.ru> "N.Dudorov"
Reviewed by: "Daniel M. Eischen" <eischen@vigrid.com>

Fix the current libc breakage in current:
o Back out the __std* stuff. Can't figure out how to do this right now,
so we'll save it for late.
o use _up as a pointer for extra fields that we need to access.
o back out the libc major version bump.

Submitted by: green
reviewed by: peter, imp, green, obrien (to varying degrees).

We'll fix the "how do we stop encoding sizeof(FILE) in binaries" part
later.

Remove freelists managed by Balloc/Bfree.
Change __dtoa to not free the string it allocated the previous time it was
called. The caller now frees the string after usage if appropiate.

PR: 15070
Reviewed by: deischen

Commit a libc fix going by the current state of the version numbering
bikeshed in -arch. It isn't quite over, but it has been well established
that this can be adjusted or refined. But we do seem to have consensis
on a major bump of some sort. After this, it should reasonably safe
to build world again.

This change is to get rid of __sF[] and use seperate __stdin/out/err
handles. This means we can pad on extra bits onto the end of FILE
at will without going through this all over again. __sF[] was evil
because it compiled the sizeof(FILE) into every stdio using program.

Asbestos suit on: check!
Peril sensitive sunglasses on: check!
*gulp!*

It sounded like a good idea at the time. The previous change breaks
FILE *buffer = stdout;
so back it out for now.

Take advantage of the current libc sizeof(FILE) breakage (__sF[]) and
try a hopefully more robust stdin/stdout/stderr. This costs an indirect
pointer fetch, but saves us from changes in 'FILE'. The __stdin stuff
is there to not pollute application name space if the application does
not use <stdio.h> and also in case something depended on the current
behavior where stdin etc was a #define.

Reviewed by: eischen, dillon

libc MT-safety, part 2.

Add a lock to FILE. flockfile and friends are now implemented
(for the most part) in libc. flockfile_debug is implemented in
libc_r; I suppose it's about time to kill it but will do it in
a future commit.

Fix a potential deadlock in _fwalk in a threaded environment.
A file flag (__SIGN) was added to stdio.h that, when set, tells
_fwalk to ignore it in its walk. This seemed to be needed in
refill.c because each file needs to be locked when flushing.

Add a stub for pthread_self in libc. This is needed by flockfile
which is allowed by POSIX to be recursive.

Make fgetpos() error return value (-1) match man page.

Remove recursive calls to locked functions (stdio); I think I've
got them all, but I may have missed a couple.

A few K&R -> ANSI conversions along with removal of a few instances
of "register".

$Id$ -> $FreeBSD$ in libc/stdio/rget.c

Not objected to: -arch, a few months ago

mdoc(7) police: mark LC_NUMERIC with .Dv.

Note that decimal point taken from locale (SUSv2)

Take decimal point from locale instead of hardcoded '.' (SUSv2)

Note that decimal point character taken from locale (SUSv2)

Use decimal point from localeconv() instead of hardcoded '.' (SUSv2)

Backout previous commit. Use of spinlocks was not approved.
PR: 15070

Protect freelists managed by Balloc/Bfree with a spinlock.
Change __dtoa to not free the string it allocated the previous time it was
called. The caller now frees the string after usage if appropiate.

PR: 15070

Fix a f^Hdamn typo, which prevented to fopen() more that 17 files at once.

Tested by: knu, sobomax and other #bsdcode'rs

mdoc(7) police: split punctuation characters + misc fixes.

Remove _THREAD_SAFE and make libc thread-safe by default by
adding (weak definitions to) stubs for some of the pthread
functions. If the threads library is linked in, the real
pthread functions will pulled in.

Use the following convention for system calls wrapped by the
threads library:
__sys_foo - actual system call
_foo - weak definition to __sys_foo
foo - weak definition to __sys_foo

Change all libc uses of system calls wrapped by the threads
library from foo to _foo. In order to define the prototypes
for _foo(), we introduce namespace.h and un-namespace.h
(suggested by bde). All files that need to reference these
system calls, should include namespace.h before any standard
includes, then include un-namespace.h after the standard
includes and before any local includes. <db.h> is an exception
and shouldn't be included in between namespace.h and
un-namespace.h namespace.h will define foo to _foo, and
un-namespace.h will undefine foo.

Try to eliminate some of the recursive calls to MT-safe
functions in libc/stdio in preparation for adding a mutex
to FILE. We have recursive mutexes, but would like to avoid
using them if possible.

Remove uneeded includes of <errno.h> from a few files.

Add $FreeBSD$ to a few files in order to pass commitprep.

Approved by: -arch

Fix bugs in the handling of > 8 positional arguments:
- The stack was getting smashed by __grow_type_table()
- reallocf() was being called with the wrong pointer
- The maximum argument number was being incorrectly computed

PR: misc/23521

Make it a bit clearer that asprintf doesn't actually "return" a pointer in
the normal sense of the word, but does it through one of its arguments which
is a pointer to a pointer.

PR: 23717
Submitted by: phk

Prepare for mdoc(7)NG.

mdoc(7) police: use canonical form of .Dd macro.

mdoc(7) police: use the new features of the Nm macro.

Update the documentation to describe the new mktemp() family behaviour.
Also notes that mkstemp() first appeared in 4.4BSD (change obtained
from OpenBSD)

Reviewed by: sheldonh

Use Fx macro wherever possible.

Increase the size of the mktemp() filename space by dropping the PID from
the encoding and using the character set [a-zA-Z0-9]. This gives a total
of 62^6 = 56800235584 possible temporary filenames for the usual default
invocation of 6 X's (compared to as few as 52 possibilities for the
previous algorithm where up to 5 characters were wasted by the PID).

Update some apparently bitrotten comments to reflect reality.

Audited by: eivind, freebsd-audit
Reviewed by: freebsd-current (a while ago)
Originally submitted by: Peter Jeremy <Peter.Jeremy@alcatel.com.au>

Avoid use of direct troff requests in mdoc(7) manual pages.

Better documentation of append mode. This should have gone in -current
directly. Now also .Xr fseek reference.

Prodded by: sheldonh

According to the susv2 man pages I have, remove(3) should act as
rmdir(2) on directories and unlink(2) otherwise. This modification,
and most of the man page update has been obtined from OpenBSD. This
was spotted by someone on a mailing lists a few months ago, but
I've lost their mail.

Reviewed by: sheldonh

Use `Er' variable to define first column width in ERRORS section.

Introduce ".Lb" macro to libc manpages.

More libraries manpages updates following.

Fix typo, reported by George Cox.
Fix hard sentence breaks.

Submitted by: George Cox <gjvc@sophos.com>

Remove more single-space hard sentence breaks.

Remove single-space hard sentence breaks. These degrade the quality
of the typeset output, tend to make diffs harder to read and provide
bad examples for new-comers to mdoc.

Simplify sytem call renaming. Instead of _foo() <-- _libc_foo <-- foo(),
just use _foo() <-- foo(). In the case of a libpthread that doesn't do
call conversion (such as linuxthreads and our upcoming libpthread), this
is adequate. In the case of libc_r, we still need three names, which are
now _thread_sys_foo() <-- _foo() <-- foo().

Convert all internal libc usage of: aio_suspend(), close(), fsync(), msync(),
nanosleep(), open(), fcntl(), read(), and write() to _foo() instead of foo().

Remove all internal libc usage of: creat(), pause(), sleep(), system(),
tcdrain(), wait(), and waitpid().

Make thread cancellation fully POSIX-compliant.

Suggested by: deischen

Improve the explanation on the (in)security of mktemp(3).

We no longer care about the VAX and Tahoe compilers :-)

Add three-tier symbol naming in support of POSIX thread cancellation
points. For library functions, the pattern is __sleep() <--
_libc_sleep() <-- sleep(). The arrows represent weak aliases. For
system calls, the pattern is _read() <-- _libc_read() <-- read().

Correct discrepancy between definition of argument to tempnam() and
the name by which it is referenced in the text.

While comparing this with OpenBSD (ie: trying to figure out what mkstemps()
is good for... :-)), I discovered that part of the change when mkstemps()
was brought in was missed - it was missing the termination case to make
sure it doesn't walk into the suffix. This isn't the same code OpenBSD
has, I think this is a little better as we terminate the loop in a better
spot.

Make __sfp() even more thread-safe.

Add (FILE *) locking.

Make __sfp() (FILE allocator) thread-safe: added locking like in malloc().

Fix HISTORY - the copyright header on the file of the GCC version was
misleading.

Submitted by: Theo de Raadt <deraadt@cvs.openbsd.org>

Add to the HISTORY.

Add unsigned char cast to isdigit

Remove UNSAFE_WARN ifdef for mktemp warning (never defined)
Use _mktemp internally

$Id$ -> $FreeBSD$

Add warnings, ala mktemp, to tempnam and tmpnam as a reminder that
these are inherently unsafe interfaces.

Do not allow TMPDIR to override path for setuid/setgid programs.

Treat an attempt to read from a write-only stream more consistently.

Submitted by: Anton Berezin <tobez@plab.ku.dk>
PR : 12852

Fixed missing "G" in the list item for the main description of %g and
%G formats.

asprintf() does use realloc() internally, but saying so in the manpage can be
misinterpreted to mean that the pointer passed to asprintf() must be suitable
for passing to realloc() as-is (ie. either a NULL pointer or a valid pointer).

Add $Id$, to make it simpler for members of the translation teams to
track.

The $Id$ line is normally at the bottom of the main comment block in the
man page, separated from the rest of the manpage by an empty comment,
like so;

.\" $Id$
.\"

If the immediately preceding comment is a @(#) format ID marker than the
the $Id$ will line up underneath it with no intervening blank lines.
Otherwise, an additional blank line is inserted.

Approved by: bde

Add a note that when a stream opened via fdopen() is closed via fclose(),
the underlying file descriptor is also closed. To me at least this wasn't
immediately obvious.

Various language and style concerns fixed.

Noted by: bde

Add mkstemps to the man page, and create a link for it.

Obtained from: OpenBSD
Poked in the eye about committing new functions without a manpage: obrien

Add mkstemps from OpenBSD. This has been in my tree for months and
hasn't caused any problems until the egcs import. This fix breaks the
world build, but my very next commit will remove mkstemps from the
egcs build.

Add support for long long modifier (e.g. %llx, %lld).

Reviewed by: bde

Fixed disordering and incoinsistent style in previous commit.

The pseudocode in the synopsis didn't come close to actually
compiling, since <stdio.h> correctly doesn't declare off_t although
the pseudo-prototypes for the new fseeko() and ftello() functions
use it. Handle this like the corresponding problem for va_list
versus the vprintf() family.

Fixed some English errors.

Decapitalize function name by prepending with word "The".

PR: docs/10247

Added functions fseeko() and ftello() (from susv2).
Fixed fgetpos() and fsetpos() for offsets > 2GB.

PR: 8637
Submitted by: Dan Nelson <dnelson@emsphone.com> (adjusted by me a little)

Note that dying on NULL is an implementation detail.

Check for a zero-length as well as a NULL string argument.

Revert last change. mkstemp() wasn't to blame, it's nvi. However,
mkstemp() is not behaving as documented.

Stop mk*temp() from being pathologically stupid in the face of a umask(0);
There are other ways to fix this than wrapping _gettemp(), but this was
the most convenient.

Discovered by: bde

Remove the description of EBADF (that's an implementation detail if I
ever saw one), and move the description of NULL behaviour out to a
'NOTES' section, with an extra note that programs should not rely up
on it.

Kinda-approve-by: bde (by not replying to the mail with the diff)

program written under FreeBSD -> programs written under FreeBSD

Noticed by: Alex Nash <nash@mcs.net>

Document that we will core-dump on getting a NULL pointer.

Apply patch to properly sscanf(3) when there is whitespace in the format
string. From the submitted patch:

Credit for patch: Chris Torek <torek@bsdi.com>
Tod Miller <millert@openbsd.org>

This makes us in line with SunOS 4.1.3_U1, Solaris 2.6, OpenBSD 2.3,
HP-UX 10.20, Irix 5.3. The previous behavior was in line with Ultrix 4.4.

PR: bin/7970
Submitted by: Niall Smart nialls@euristix.ie

Replace memory leaking instances of realloc with non-leaking reallocf.
In some cases replace if (a == null) a = malloc(x); else a =
realloc(a, x); with simple reallocf(a, x). Per ANSI-C, this is
guaranteed to be the same thing.

I've been running these on my system here w/o ill effects for some
time. However, the CTM-express is at part 6 of 34 for the CAM
changes, so I've not been able to do a build world with the CAM in the
tree with these changes. Shouldn't impact anything, but...

Replace my original asprintf() and vasprintf() hacks with something
more cleanly integrated with stdio. This should be faster and cleaner
since it doesn't memcpy() the data into a seperate buffer. This lets
stdio allocate and manage the buffer and then hand it over to the user.

Obtained from: Todd Miller <Todd.Miller@courtesan.com> via OpenBSD

Spelling corrections.

PR: 6868
Submitted by: Josh Gilliam <josh@quick.net>

Remote the NetBSD kludge for vfprintf.c

Don't assign the va_list variable 'ap' directly to the argtable because
va_list is not a pointer on alpha. Instead, use the va_arg() macro
to return the address that is stored in the argtable.

Remove leading underscores for the functions (weak symbols here) that
POSIX defines.

Fix a nasty flaw as a result of using the arc4random() pre-seeding of
leading XXX's. It could wrap an uppercase character through chars
like: [ \ ] ^ _ ` in between Z and a. The backslash and back tick
might be particularly nasty in a shell script context. Also, since
we've been using upper-case generated values for a while now, go with
the flow and use them in the pathname search rotation.

Add FILE locking stubs for libc.

Change the FILE locking to support kernel threads when linked with
libpthread (which you haven't see yet). This requires that libc become
thread-safe and thread-aware, testing __isthreaded before attempting
to do lock/unlock calls. The impact on non-threaded programs is minor.
This change works with libc_r, so it's the best compromise.

Fixed disordering and inconsistent style in previous commit.

Added #include <string.h> to get prototypes.

Fixed a few ancient typos, added a little missing stuff, and updated
references to abort() in light of POSIX mandated behaviour. I'm
still not 100% happy with much of the wording, but it's better
than it was.

Fixed uninitialized pointer in previous commit. mktemp() was broken.
I noticed cvs core dumps and uncleaned cvs temporary files in /tmp.

Fixed ANSIisms.

Many security improvements from OpenBSD:
implement mkdtemp
improve man page for mk*temp
use arc4random to seed extra XXX's randomly
Optionally warn of unsafe mktemp uses
From various commits by theo de raadt and Todd Miller.
Obtained from: OpenBSD

This should go into 2.2 after a testing period.

size_t -> unsigned
in arguments length INT_MAX overflow check
Suggested-by: bde

1. EOF was returned when the buffer size was larger than INT_MAX. This
case has very little to do with the output size being larger than
INT_MAX.
2. The new #include of <limits.h> was disordered.
3. The new declaration of `on' was disordered (integer types go together).
4. Testing an unsigned value for > 0 was fishy.

Submitted by: bde

Add overflow checks: if output size becomes bigger than INT_MAX,
just return EOF

Correct type of stored argument place (from previous fix)

1) Restore back comment about snprintf()
2) Optimize string buffer copy to call memcpy() and update pointers
only for count > 0, it makes snprintf(NULL, 0, ...) more efficient

Return back to BSD snprintf semantics which recent C9x standard adopts
instead of Singe Unix, thanx Bruce for explaining, I am not realize
standards war was there.

But now, fix n == 0 case to not return error and fix check for too
big n.

Things left to do: check for overflow in arguments.

1) Oops! Insert again if (n == 0) return 0.

Final word is Bruce's quote:

C9x specifies the BSD4.4-Lite behaviour:

[#3] ... Thus, the
null-terminated output has been completely written if and
only if the returned value is less than n.

It means that if we not have any null-terminated output as for n == 0
we can't return value less than n, so we forced to return value
equal to n i.e. 0

The next good thing is glibc compatibility, of course.

2) Do check for too big n in machine-independent way.
3) Minor optimization assuming EOF is < 0

Back out part related to "return 0 if n == 0" and return EOF as before.

The main argument is that it is impossible to determine if %n evaluated or not
when snprintf return 0, because it can happens for both n == 0 and n == 1.
Although EOF here is good indication of the end of process, if n is
decreased in the loop...
Since it is already supposed in many places that EOF *is* negative, f.e.
from Single Unix specs for snprintf
"return ... a negative value if an output error was encountered"
this not makes situation worse.

Fix snprintf(...%n...)
to pass not more than buffer size to %n agrument, old variant
always assume infinite buffer.
%n is for actually transmitted characters, not for planned ones.

Remove wrong comment about snprintf:
"return the number of bytes needed, rather the number used"

According to Single Unix specs:

Upon successful completion, these functions return the number of bytes
transmitted excluding the terminating null

snprintf return value fixes to conform Single Unix specs:

1) if buffer size is smaller than arguments size, return buffer
size, not arguments size as before.

2) if buffer size is 0, return 0, not EOF as before.
(now it is compatible with Linux and Apache implementations too).

NOTE: Single Unix specs says:

If the value of n {buffer size} is zero on a call to snprintf(), an
unspecified value less than 1 is returned.

It means we can't return EOF since EOF can take *any* value in general
not especially < 1. Better variant will be return -1 (it is less then
1 and different with n == 1 case) but -1 value is already occuped by
EOF in our implementation, so we can't distinguish true IO error
in that case. So 0 here is only possible case still conforming
to Single Unix specs.

Comment that long double is poorly implemented, not that it is unimplemented.

Fixed long double formats. They were mostly not implemented except
on systems where long doubles are just doubles. FreeBSD hasn't
been such a system since it started using gcc-2.5 many years ago.
The fix is of low quality. It loses precision.

scanf() of long doubles doesn't seem to be used much, but gdb-4.16
uses %Lg format in its expression parser if it thinks that the
system supports printf'ing of long doubles. The symptom was that
floating point literals were usually interpreted to be 0.0.

Sorted lists.

Removed the subdirectory paths from the definitions of MAN[1-9]. They
were a workaround for limitations in bsd.man.mk that were fixed about
2 years ago.

Rework previous commit.. I was confused by the number of diffs in the PR
and forgot what I was trying to do originally and accidently zapped
a feature. :-] The problem is that we are converting a counted buffer in
a malloc pool into a null terminated C-style string. I was calling realloc
originally to shrink the buffer to the desired size. If realloc failed, we
still returned the valid buffer - the only thing wrong was it was a tad
too large. The previous commit disabled this.

This commit now handles the three cases..
1: the buffer is exactly right for the null byte to terminate the
string (we don't call realloc).
2: it's got h.left = 0, so we must expand it to make room. If realloc
fails here, it's fatal.
3: if there's too much room, we realloc to shrink it - a failed realloc
is not fatal, we use the original buffer which is still valid.

Fix off-by-one error

PR: 3451
Submitted by: Tim Vanderhoek <ac199@hwcn.org>

Add 64 bit int support to scanf()
PR: 2080
Submitted by: David Dawes <dawes@rf900.physics.usyd.edu.au>

Changed all paths to be relative to src/lib instead of src/lib/libc
so that all these makefiles can be used to build libc_r too.

Added .if ${LIB} == "c" tests to restrict man page builds to libc
to avoid needlessly building them with libc_r too.

Split libc Makefile into Makefile and Makefile.inc to allow the
libc_r Makefile to include Makefile.inc too.

Fixed #include and/or prototype bugs in synopsis.

Fix race
Obtained from: Keith Bostic

Eliminate yet one function call when locale not used

Speedup in case locale not used

The w+ entry description was misformatted.

Pointed out by: bde

FIxed arg types (mostly missing consts) in synopsis.

Merge from Lite2

This commit was generated by cvs2svn to compensate for changes in r23658,
which included commits to RCS files with non-trunk default branches.

Fixed handling of input failure by the scanf family.
- 0 was returned instead of EOF when an input failure occured while
skipping white-space after 0 assignments. This fixes PR2606. The
diagnosis in PR2606 is wrong.
- EOF was returned instead of 0 when an input failure occurred after
zero assignments and nonzero suppressed assignments.
- EOF was spelled -1.

This should be in 2.2.

Revert $FreeBSD$ to $Id$

Add XXX comment describing potential memset non-portable issue

Nitpicked-by: joerg

Sort cross references.

Use collate for national [a-z]-like ranges

Should go in 2.2

The following patch to lib/libc/stdio implements positional arguments in
a manner consistent with other implementations. Its done in a way that
adds only a tiny amount of overhead when positional arguments are not used.
I also have a test program to go with this, but don't know where it belongs
in the tree.

Submitted-By: Bill Fenner <fenner@FreeBSD.ORG>

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

perror () does not prepend ": " for the non-NULL argument "". close PR 1492

Submitted by: Kent Vander Velden <graphix@iastate.edu>
Reviewed by:
Submitted by:
Obtained from:

.DV -> .Dv (FOPEN_MAX was invisible).

fwopen() argument type mis-described

Obtained from: NetBSD lib/2751 (der Mouse)

Correctly use .Fn instead of .Nm to reference function names
in a bunch of man pages.

Use the correct .Bx (BSD UNIX) or .At (AT&T UNIX) macros
instead of explicitly specifying the version in the text
in a bunch of man pages.