Remove svn:mergeinfo carried over from stable/9.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC rev. 265593:
(dd_out): Fix handling of all-zeroes block at end of input with
conv=sparse.

PR: bin/189174
PR: bin/189284
Reviewed by: kib

MFC r265229,r265239:

Add -J to filter by matching jail IDs and names.

date(1): Add "-R" flag to use RFC 2822 date and time output format [MFC]

As stated in the man page, this is equivalent to use "%a, %d %b %Y %T %z"
as the output format while LC_TIME is set to the "C" locale.

This is compatible with date(1) from the GNU core utilities.

This is an MFC of r264968 and r264970.

MFC r260246: sh(1): Discourage use of -e.

Also, do not say that ! before a pipeline is an operator, because it is
syntactically a keyword.

MFC r263847: sh: Fix memory leak with an assignment before a regular builtin

MFC r263846: sh: Fix memory leak when trying to set a read only variable.

MFC r263777: sh: Fix possible memory leaks and double frees with unexpected
SIGINT.

MFC all recent changes on -HEAD to dd(1).

MFC r238468: sh: Expand assignment-like words specially for
export/readonly/local.

Examples:
export x=~
now expands the tilde
local y=$1
is now safe, even if $1 contains IFS characters or metacharacters.

For a word to "look like an assignment", it must start with a name followed
by an equals sign, none of which may be quoted.

The special treatment applies when the first word (potentially after
"command") is "export", "readonly" or "local". There may be quoting
characters but no expansions. If "local" is overridden with a function there
is no special treatment ("export" and "readonly" cannot be overridden with a
function).

If things like
local arr=(1 2 3)
are ever allowed in the future, they cannot call a "local" function. This
would either be a run-time error or it would call the builtin.

This matches Austin Group bug #351, planned for the next issue of POSIX.1.

As for the MFC, it is easy to depend on this feature inadvertently, and
adding this fixes a regression from stable/8 that may be apparent in things
like
local x=${y+a @}.

PR: bin/166771
Relnotes: yes

MFC r263778:

Add a SUBDIR_PARALLEL option to bsd.subdir.mk, to allow make to process
all the SUBDIR entries in parallel, instead of serially. Apply this
option to a selected number of Makefiles, which can greatly speed up the
build on multi-core machines, when using make -j.

This can be extended to more Makefiles later on, whenever they are
verified to work correctly with parallel building.

I tested this on a 24-core machine, with make -j48 buildworld (N = 6):

before stddev after stddev
======= ====== ======= ======
real time 1741.1 16.5 959.8 2.7
user time 12468.7 16.4 14393.0 16.8
sys time 1825.0 54.8 2110.6 22.8

(user+sys)/real 8.2 17.1

E.g. the build was approximately 45% faster in real time. On machines
with less cores, or with lower -j settings, the speedup will not be as
impressive. But at least you can now almost max out a machine with
buildworld!

Submitted by: jilles

MFC r263833:

Enable parallel building for gnu/usr.bin and usr.bin/clang too.

MFH (r237568, r255422, r255460, r255766, r255767, r255774, r255829,
r256126, r257954, r261320, r261499, r263691, r263712): upgrade to
OpenSSH 6.6p1 via 6.3p1, 6.4p1 and 6.5p1.

Differences relative to head:

- No DNSSEC support since stable/9 does not have LDNS
- Sandboxing off by default, and uses rlimit instead of Capsicum
- ED25519 moved to the bottom of the order of preference to avoid
"new public key" warnings

MFC r262467: sh: Add -h option to SYNOPSIS

Reviewed by: jilles

MFC r260782:

-h and -H options backwards in manual page.

MFC r256050:

Remove useless check - ki_loginclass is an array; can't be NULL.

MFC r254134:

Add -c flag to pgrep(1) and pkill(1), to match login classes.

MFC r254335: sh: Allow a lone redirection before '|', ';;' or ';&'.

Example: </dev/null | :

PR: 181240

MFC r251211: test(1): Add information about replacing -nt/-ot.

MFC r248349: sh: Recognize "--" and explicitly reject options in wait
builtin.

If syntactically invalid job identifiers are to be taken as jobs that exited
with status 127, this should not apply to options, so that we can add
options later if need be.

MFC r251763: sh(1): A subshell environment has its own rlimits (ulimit).

This has always been the case and is intended (just like cd).

This matches Austin group issue #706.

MFC r248870 by joel: Minor mdoc fix in previous commit.

MFC r248692: sh(1): Mention possible ambiguities with $(( and ((.

In some other shells, things like $((a);(b)) are command substitutions.

Also, there are shells that have an extension ((ARITH)) that evaluates an
arithmetic expression and returns status 1 if the result is zero, 0
otherwise. This extension may lead to ambiguity with two subshells starting
in sequence.

MFC r251209: test(1): List non-standard primaries.

MFC r251078,r251433: sleep: Improve nanosleep() error handling:

* Work around kernel bugs that cause a spurious [EINTR] return if a
debugger (such as truss(1)) is attached.

* Write an error message if an error other than [EINTR] occurs.

PR: bin/178664

MFC: 249658, 249701

Update chio(1) and ch(4) to support reporting element designators.

This allows mapping a tape drive in a changer (as reported by
'chio status') to a sa(4) driver instance by comparing the
serial numbers.

The designators can be ASCII (which is printed out directly), binary
(which is printed in hex format) or UTF-8, which is printed in either
native UTF-8 format if the terminal can support it, or in %XX notation
for non-ASCII characters. Thanks to Hiroki Sato <hrs@> for the
explanation and example UTF-8 printing code.

chio.h: Modify the changer_element_status structure to add new
fields and definitions from the SMC3r16 spec.

Rename the original CHIOGSTATUS ioctl to OCHIOGTATUS and
define a new CHIOGSTATUS ioctl.

Clean up some tab/space issues.

chio.c: For the 'status' subcommand, print the designator field
if it is supplied by a device.

scsi_ch.h: Add new flags for DVCID and CURDATA to the READ
ELEMENT STATUS command structure.

Add a read_element_status_device_id structure
for the data fields in the new standard. Add new
unions, dt_or_obsolete and voltage_devid, to hold
and address data from either SCSI-2 or newer devices.

scsi_ch.c: Implement support for fetching device IDs with READ
ELEMENT STATUS data.

Add new arguments to scsi_read_element_status() to
allow the user to request the DVCID and CURDATA bits.
This isn't compiled into libcam (it's only an internal
kernel interface), so we don't need any special
handling for the API change.

If the user issues the new CHIOGSTATUS ioctl, copy all of
the available element status data out. If he issues the
OCHIOGSTATUS ioctl, we don't copy the new fields in the
structure.

Fix a bug in chopen() that would result in the peripheral
never getting unheld if chgetparams() failed.

Sponsored by: Spectra Logic
Submitted by: Po-Li Soong

MFC r250666:

Constify parameters, no functional or binary change.

MFC revisions 243049,243129,244134,245871,245912,249698,249924

MFC r249949:
Take some improvements from DragonFlyBSD:
- add const where appropriate
- add static where appropriate
- fix a whitespace issues

MFC r249811:
Literally follow POSIX:
If the bs= expr operand is specified and no conversions other than sync,
noerror, or notrunc are requested, the data returned from each input
block shall be written as a separate output block.

PR: standards/177742

MFC r249996:
Add missing static qualifiers

MFC r248972:

Mention that read_attributes, write_attributes, read_acl and write_acl
are always permitted for the file owner.

PR: kern/174948

MFC r230998,r233792: sh: Use vfork in a few common cases.

This uses vfork() for simple commands and command substitutions containing a
single simple command, invoking an external program under certain conditions
(no redirections or variable assignments, non-interactive shell, no job
control). These restrictions limit the amount of code executed in a vforked
child.

Various incarnations of this patch have been shown to bring performance
improvements:
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-January/037581.html

The use of vfork() can be disabled by setting a variable named
SH_DISABLE_VFORK.

MFC all features as of r248336:

The user visable feature is from r246083:

Add -l option to cat(1). This option causes cat(1) to use fcntl(2) to
set an exclusive advisory lock on stdout. This will be used to guarantee
orderly writing to METALOG.

Sponsored by: DARPA, AFRL
Obtained from: NetBSD (mason)

MFC 245610,245633,245635:
Include the thread name along with the command name when displaying the
command name of a thread from a multi-threaded process that doesn't have
an available argument list (such as kernel processes) and threads display
is enabled via -H.

MFC r246522: sh: Simplify mksyntax and make it fit for cross-compiling.

Now it outputs fixed files, which use constants provided by the C standard
library to determine appropriate values for the target machine.

Before, mksyntax inspected the host machine which resulted in subtle
breakage if e.g. char is signed on the host and unsigned on the target such
as when cross-compiling on x86 for ARM.

Submitted by: Christoph Mallon

MFC r246486:
Document P_PPTRACE.

MFC r246371: sh: Do not test for digit_contig in mksyntax.

ISO/IEC 9899:1999 (E) 5.2.1p3 guarantees that the values of the characters
0123456789 are contiguous.

The generated syntax.c and syntax.h remain the same.

Submitted by: Christoph Mallon

MFC r246507: sh: Fix a comment.

MFC r244734: test(1): Document == alias for =.

MFC r246557:

In the setfacl(1) manual page, make it clear that for NFSv4 ACLs,
one should really use -a and -x instead of -m.

MFC r246558:

Improve description of the "-m" option to setfacl(1).

MFC r246559:

Fix NFSv4 permission description in setfacl(1) manual page: the 'D'
means delete_child, not delete.

MFC r245535:
Remove useless variable 'Pflag':
-P is an alternative to -H and -L, and it is implemented using the Hflag and Lflag variables.

Approved by: cperciva (mentor, implicit)

MFC r245382: sh: Fix crash when parsing '{ } &'.

MFC r244791:
- Fix wording nit.

PR: 174787
Submitted by: Cody Rank

Revert r244108. yyparse() declaration isn't redundant in stable/9.

Reported by: tinderbox :)

MFC r235771:

Remove redundant declaration of yyparse.

Reported by: tinderbox

MFC to r242840:

Add y flag and environment variable LS_SAMESORT to specify the same
sorting order for time and name with the -t option. IEEE Std 1003.2
(POSIX.2) mandates that the -t option sort in descending order, and
that if two files have the same timestamp, they should be sorted in
ascending order of their names. The -r flag reverses both of these
sort orders, so they're never the same. This creates significant
problems for sequentially named files stored on FAT file systems,
where it can be impossible to list them in the order in which they
were created.

Add , (comma) option to print file sizes grouped and separated by
thousands using the non-monetary separator returned by localeconv(3),
typically a comma or period.

MFC r243072:
Follow the behavior as specified in POSIX:

if (exists AND (NOT f_option) AND
((not_writable AND input_is_terminal) OR i_option))
prompt

in particular, add the test for input_is_terminal

PR: bin/173039
Approved by: cperciva (implicit)

MFC r242766: sh: Fix two issues when an alias is redefined:

* The last character is not displayed.
* If the alias ends with itself (as a word), an infinite memory-eating loop
occurs.

If an alias is defined initially, a space is appended to avoid recursion but
this did not happen when an alias was later modified.

PR: bin/173418
Submitted by: Daniel F.

MFC 226961,239951:
Add a -h flag similar to the -h flag for ln to force mv(1) to treat a
symbolic link to a directory for the target as a symbolic link instead of
a directory. This makes it possible to atomically update a symbolic
link using rename().

MFC r240088:

Explicitly mention that "cp -p" copies ACLs.

PR: docs/168962

MFC r240087:

Fix an error in setfacl(1) that manifested like this:

# setfacl -m u:trasz:rwx x
# setfacl -m u:root:rwx x
# getfacl x
# file: x
# owner: root
# group: wheel
user::rw-
user:root:rwx
user:trasz:rwx
group::r--
mask::rwx
other::r--
# setfacl -m u:root:rwx x
setfacl: x: acl_calc_mask() failed: Invalid argument
setfacl: x: failed to set ACL mask

For NFSv4 ACLs, this sort of situation would result in duplicated
entries.

MFC r240085:

In getfacl(1) manual page, mention where to read about the ACL syntax.
PR submitter suggested adding it to acl(3) instead, but I don't think
pointing ordinary users at section 3 is a good idea.

PR: docs/162380

MFC r240084:

Make "setfacl -bd" an alias for "setfacl -k". Previously it would crash
on assert.

PR: bin/165807

MFC r240083:

Make setfacl(1) behave properly in situations like "setfacl -kd".

MFC r240645:
- Add 'dsiz' and 'ssiz' keywords to show data and stack size respectively.

MFC r240649:
- Bump date.

Approved by: kib (mentor)

MFC r240505: sh: Reduce code duplication: use setinputfile instead of
open/setinputfd combination.

MFC r240076:

Fix compact form of "synchronize" permission in setfacl(1) manual page - it
should be "s", not "S".

PR: docs/162380

MFC r238653: Use proper error message when fstat(2) fails on stdout.

PR: bin/159746

MFC r238652: style(9).

MFC 239883:

Avoid passing uninitialized stack to addelem() if called with an empty
arg.

PR: bin/171174

MFC r238377:

Merge libedit adjustment from NetBSD.

On recent versions of NetBSD's libedit, el_gets
now sets el_len to -1 on error so we can
distinguish between a NULL string and an error.

This fixes sh from exiting with newer versions
of libedit now allowing EINTR to return.

Obtained from: NetBSD
Reviewed by: jilles
MFC after: 3 weeks

MFH r236751: document sha256 / sha512 support
MFH r236892: remove mention of auth.conf from programs that don't use it
MFH r236963: remove dead code relating to auth.conf
MFH r236965 r236966 r236967 r237005 r237006 r237011: retire auth.conf

Approved by: re

MFC r237339:

Polish previous revision: if the fts_* routines have lstat()'ed the
directory entry then use the struct stat from that instead of doing
it again, and skip the rm_overwrite() call if fts_read() indicated
that the entry couldn't be a regular file.

Obtained from: OpenBSD

MFC r237284 (kevlo):

Fix potential symlink race condition in "rm -P" by adding a check
that the file we have opened is the one we expected. Also open in
non-blocking mode to avoid a potential hang with FIFOs.

Obtained from: NetBSD via OpenBSD

MFC r233648:
Remove trailing whitespace per mdoc lint warning

Approved by: cperciva (implicit)

MFC r235842

Mention the upper hard limit for -n option. Patch slightly modified
from PR version.

PR: 168255
Submitted by: Andy Kosela
Approved by: gjb (mentor)

MFC r235851:
Add 'cow' keyword to show per-process cow count.

MFC r235297, r235316:

Add a -v and -N option to kenv(1), so it can be more easily used in
scripts the way sysctl(8) is. The -N option, like in sysctl(8),
displays only the kenv names, not their values. The -v option prints an
individual kenv variable name with its value as name="value". This is
the inverse of sysctl(8)'s -n flag, since the default behaviour of
kenv(1) is already like sysctl(8) -n.

MFC r225868:

Make ps(1) automatically size its column widths.

MFC r232157, r232158:

r232157:
Fix various typos in manual pages.

Submitted by: amdmi3
PR: 165431

r232158:
Whitespace cleanup:
o Wrap sentences on to new lines
o Cleanup trailing whitespace

MFC r229201: sh: Make patmatch() non-recursive.

MFC r232466, r232469:
Direct users to swapinfo and pstat from df

PR: bin/165321
Approved by: cperciva

MFC r232694:

Document P_ORPHAN.

MFC r231990

Update to tcsh 6.18.01

MFC r229220: sh: Make various functions static.

MFC r229219: sh: Remove unused function scopyn().

MFC r230437: sh: Remove "kill" example function, which is superseded by the
kill builtin.

MFC r231001: sh: Fix swapped INTON/INTOFF.

A possible consequence of this bug was a memory leak if SIGINT arrived
during a 'set' command (listing variables).

MFH r226502,r226503: factor out the common code and fix two display bugs

Approved by: des (mentor)

MFC r230212:
sh: Fix execution of multiple statements in a trap when evalskip is set

Before this fix, only the first statement of the trap was executed if
evalskip was set. This is for example the case when:
o "-e" is set for this shell
o a trap is set on EXIT
o a function returns 1 and causes the script to abort

Reviewed by: jilles
Sponsored by: Yakaz (http://www.yakaz.com)

MFC r227840, r227846:

r227840:

No need in procfs(5).

r227846

Remove yet another outdated note about procfs(5) being required.

Spotted by: arundel

MFC r228937: sh: Do not force special builtins non-special in optimized
command substitition.

This is not necessary: errors are already caught in evalbackcmd() and
forcelocal handles changes to variables.

Note that this depends on r223024.

MFC r230095: sh: Properly show "Not a directory" error in cd builtin.

The errno message display added in r222292 did not take attempting to
cd to a non-directory or something that cannot be stat()ed into account.

PR: bin/164070

MFC r230117: sh: Fix out of bounds array access when trap is used with an
invalid signal.

MFC r228595:

Fix some obtuse fprintf tricks in bin/ed.

MFC r227122: sh(1): Improve documentation of field splitting.

This describes the POSIX-compliant splitting algorithm that first appeared
in 8.0.

MFC: r227006, r227281, r227282

Add a PCI front-end to esp(4) allowing it to support AMD Am53C974 and
replace amd(4) with the former in the amd64, i386 and pc98 GENERIC kernel
configuration files. Besides duplicating functionality, amd(4), which
previously also supported the AMD Am53C974, unlike esp(4) is no longer
maintained and has accumulated enough bit rot over time to always cause
a panic during boot as long as at least one target is attached to it
(see PR 124667).

PR: 124667
Approved by: re (kib)
Obtained from: NetBSD (based on)

MFC 225908, 225912, 225927:

225908:
- Reorder default ps(1) output according to reality.

225912:
- there's no reason the semantics of the -x flag are being explained in
the -a flag description
- be more precise regarding the relation between the -a flag and the
security.bsd.see_other_uids sysctl
- describe the format of the -t flag's argument
- 'con' no longer is a possible entry in the 'TT' column
- explain that the 'TT' column refers to pseudo-terminals
via mere numbers
- add a hint in the 'tt' keyword description that a
keyword 'tty' exists, which will give the full terminal pathname

225859:
- Correct a typo that was introduced in 225912

Approved by: re (kib)

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Fix memory leaks in error cases.

PR: 159011
Submitted by: Henning Petersen <henning.petersen@t-online.de>
Approved by: re (kib)
MFC after: 1 week

Rename ki_ocomm to ki_tdname and OCOMMLEN to TDNAMLEN.
Provide backward compatibility defines under BURN_BRIDGES.

Suggested by: jhb
Reviewed by: emaste
Sponsored by: Sandvine Incorporated
Approved by: re (kib)

sh(1): Extend documentation about subshells.

Because sh executes commands in subshell environments without forking in
more and more cases (particularly from 8.0 on), it makes sense to describe
subshell environments more precisely using ideas from POSIX, together with
some FreeBSD-specific items.

In particular, the hash and times builtins may not behave as if their state
is copied for a subshell environment while leaving the parent shell
environment unchanged.

Fix style, since this file has just been touched in a major way.
No actual code changes.

Some refactoring for easier maintenance of the code. This is a follow-up
to re-establishment of 64bit arithmetic, but is committed separately, to
not obscure that conversion. This commit does not change the observed
behaviour of expr in any way. Style will be fixed in a follow-up commit.

Make /bin/expr support 64bit numeric range and range checks by default,
again. This brings back the behaviour of expr in FreeBSD-4, which had been
reverted due to an assumed incompatbility with POSIX.1 for FreeBSD-5.

This issue has been discussed in the freebsd-standards list, and the
consensus was, that POSIX.1 is in fact not violated by this extension,
since it affects only cases of POSIX undefined behaviour (overflow of
signed long).

Other operating systems did upgrade their versions of expr to support
64bit range, after it had been initially brought to FreeBSD. They have
used it for a decade without problems, meanwhile.

The -e option is retained, but it will only select less strict checking
of numeric parameters (leading white-space, leading "+" are allowed and
skipped, an empty string is considered to represent 0 in numeric context.)
The call of check_utility_compat() as a means of establishing backwards
compatibility with FreeBSD-4 is considered obsolete, but preserved in
this commit. It is expected to be removed in a later revision of this
file.
Reviewed by: bde, das, jilles
MFC after: 2 month (those parts that do not violate POLA)

Bump date after the previous commit.

Update and sort the list of the available keywords.

sh: Include <limits.h> instead of non-standard <sys/limits.h>.

sh(1): Improve documentation of shell patterns:

* Shell patterns are also for ${var#pat} and the like.
* An '!' by itself will not trigger pathname generation so do not call it a
meta-character, even though it has a special meaning directly after an
'['.
* Character ranges are locale-dependent.
* A '^' will complement a character class like '!' but is non-standard.

MFC after: 1 week

sh(1): Document the case command better.

Suggested by: netchild
Reviewed by: gjb

Remove duplicated header files

Make ``realpath'' behave like ``realpath .''.

sh: Remove special support for background simple commands.

It expands the arguments in the parent shell process, which is incorrect.

sh: Add do-nothing -h option.

POSIX requires a -h option to sh and set, to locate and remember utilities
invoked by functions as they are defined. Given that this
locate-and-remember process is optional elsewhere, it seems safe enough to
make this option do nothing.

POSIX does not specify a long name for this option. Follow ksh in calling it
"trackall".

sh: Add case statement fallthrough (with ';&' instead of ';;').

Replacing ;; with the new control operator ;& will cause the next list to be
executed as well without checking its pattern, continuing until a list ends
with ;; or until the end of the case statement. This is like omitting
"break" in a C "switch" statement.

The sequence ;& was formerly invalid.

This feature is proposed for the next POSIX issue in Austin Group issue
#449.

sh: Skip variables with invalid names in "set", "export -p", "readonly -p".

This ensures the output of these commands is valid shell input.

sh: Reduce unnecessary forks with eval.

The eval special builtin now runs the code with EV_EXIT if it was run
with EV_EXIT itself.

In particular, this eliminates one fork when a command substitution contains
an eval command that ends with an external program or a subshell.

This is similar to what r220978 did for functions.

sh: Add support for named character classes in bracket expressions.

Example:
case x in [[:alpha:]]) echo yes ;; esac

Add "gid" and "group" keywords to display the effective group ID
and effective group name. Also, add "egid", "egroup" and "euid" aliases.

PR: bin/146331
Submitted by: Jeremie Le Hen <jeremie at le-hen dot org>

sh: Fix duplicate prototypes for builtins.

Have mkbuiltins write the prototypes for the *cmd functions to builtins.h
instead of builtins.c and include builtins.h in more .c files instead of
duplicating prototypes for *cmd functions in other headers.

sh: Save/restore changed variables in optimized command substitution.

In optimized command substitution, save and restore any variables changed by
expansions (${var=value} and $((var=assigned))), instead of trying to
determine if an expansion may cause such changes.

If $! is referenced in optimized command substitution, do not cause jobs to
be remembered longer.

This fixes $(jobs $!) again, simplifies the man page and shortens the code.

sh: Fix locale-dependent ranges in bracket expressions.

When I added UTF-8 support in r221646, the LC_COLLATE-based ordering broke
because of sign extension of char.

Because of libc restrictions, this does not work for UTF-8. For UTF-8
locales, ranges always use character code order.

sh: Read .profile from the home directory (or / if HOME is not set).

In most cases, login shells are started from the home directory, but not in
all, such as xterm -ls.

This commit depends on r222957 for read_profile() performing parameter
expansion.

PR: bin/50569

sh: Do parameter expansion on ENV before using it.

This is required by POSIX, and allows things like ENV=\$HOME/.shrc.

Note that tilde expansion is explicitly not performed.

sh: Do parameter expansion before printing PS4 (set -x).

The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.

All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).

Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).

Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.

sh: Fix $? in heredocs on simple commands.

PR: bin/41410

sh: Improve error message if the script cannot be opened.

Avoid "<nosuchfile>: cannot open <nosuchfile>: ...".

sh: Reduce more needless differences between error messages.

sh: Honour -n while processing -c string.

sh: Remove the "exp" builtin.

The "exp" builtin is undocumented, non-standard and not very useful.

If exp's return value is not used, something like
VAR=$(exp EXPRESSION)
is equivalent to
VAR=$((EXPRESSION))
except that errors in the expression are fatal and quoting special
characters is not needed in the latter case.

If exp's return value is used, something like
if exp EXPRESSION >/dev/null
can be replaced by
if [ $((EXPRESSION)) -ne 0 ]
with similar differences.

The exp-run showed that "let" is close enough to bash's and ksh's builtin
that removing it would break a few ports. Therefore, "let" remains in 9.x.

PR: bin/104432
Exp-run done by: pav (with some other sh(1) changes)

sh: Correct criterion for using CDPATH in cd.

CDPATH should be ignored not only for pathnames starting with '/' but also
for pathnames whose first component is '.' or '..'.

The man page already describes this behaviour.

sh: Various updates to the TOUR document.

sh: Fix unquoted $@/$* if IFS=''.

If IFS is null, unquoted $@/$* should still expand to separate words.
This differs from quoted $@ (which does not depend on IFS) in that pathname
generation is performed and empty words are removed.

sh: Show errno messages in cd.

sh: Remove obsolete token type TENDBQUOTE.

This token type was related to Almquist's original version of backquotes
that could not nest and fell into disuse fairly soon.

Fix some typos under bin/

Found by: codespell

Fix a bunch of typos and a couple of whitespace nits.

Helped by: codespell and vim's spellchecker

sh: Fix bss-based buffer overflow in . builtin.

If the length of a directory in PATH together with the given filename
exceeded FILENAME_MAX (which may happen even for pathnames that work), a
static buffer was overflown.

The static buffer is unnecessary, we can use the stalloc() stack.

Obtained from: NetBSD
MFC after: 1 week

sh: Expand aliases after assignments and redirections.

sh: Implement the cd -e flag proposed for the next POSIX issue.

This reflects failure to determine the pathname of the new directory in the
exit status (1). Normally, cd returns successfully if it did chdir() and the
call was successful.

In POSIX, -e only has meaning with -P; because our -L is not entirely
compliant and may fall back to -P mode, -e has some effect with -L as well.

sh: Allow terminating a heredoc with a terminator at EOF without a newline.

This is sometimes used with eval or old-style command substitution, and most
shells other than ash derivatives allow it.

It can also be used with scripts that violate POSIX's requirement on the
application that they end in a newline (scripts must be text files except
that line length is unlimited).

Example:
v=`cat <<EOF
foo
EOF`
echo $v

This commit does not add support for the similar construct with new-style
command substitution, like
v=$(cat <<EOF
foo
EOF)
This continues to require a newline after the terminator.

Spelling in P_HADTHREADS.

MFC after: 1 week

Fix the description of the "paddr" keyword. It shows a process
pointer just like a dublicating "uprocp".

MFC after: 1 week

sh: Minor optimization to output from ulimit/export/readonly.

No functional change is intended.

sh: Avoid close(-1) when evaluating a multi-command pipeline.

Valgrind complains about this.

Update sticky(7) cross references.

PR: docs/124468
X-MFC with: r218998

sh: Add \u/\U support (in $'...') for UTF-8.

Because we have no iconv in base, support for other charsets is not
possible.

Note that \u/\U are processed using the locale that was active when the
shell started. This is necessary to avoid behaviour that depends on the
parse/execute split (for example when placing braces around an entire
script). Therefore, UTF-8 encoding is implemented manually.

sh: Optimize variable code by storing the length of the name.

Obtained from: NetBSD

sh(1): Update BUGS section for UTF-8 support.

sh: Add UTF-8 support to pattern matching.

?, [...] patterns match codepoints instead of bytes. They do not match
invalid sequences. [...] patterns must not contain invalid sequences
otherwise they will not match anything. This is so that ${var#?} removes the
first codepoint, not the first byte, without putting UTF-8 knowledge into
the ${var#pattern} code. However, * continues to match any string and an
invalid sequence matches an identical invalid sequence. (This differs from
fnmatch(3).)

sh: Add UTF-8 support to ${#var}.

If the current locale uses UTF-8, ${#var} counts codepoints (more precisely,
bytes b with (b & 0xc0) != 0x80).

sh: Track if the current locale's charset is UTF-8 or not.

sh: Change the CTL* bytes to ones invalid in UTF-8.

This ensures that mbrtowc(3) can be used directly once it has been verified
that there is no CTL* byte. Dealing with a CTLESC byte within a multibyte
character would be complicated.

The new values do occur in iso-8859-* encodings. This decreases efficiency
slightly but should not affect correctness.

Caveat: Updating across this change and rebuilding without cleaning may
yield a subtly broken sh binary. By default, make buildworld will clean and
avoid problems.

sh: Add $'quoting' (C-style escape sequences).

A string between $' and ' may contain backslash escape sequences similar to
the ones in a C string constant (except that a single-quote must be escaped
and a double-quote need not be). Details are in the sh(1) man page.

This construct is useful to include unprintable characters, tabs and
newlines in strings; while this can be done with a command substitution
containing a printf command, that needs ugly workarounds if the result is to
end with a newline as command substitution removes all trailing newlines.

The construct may also be useful in future to describe unprintable
characters without needing to write those characters themselves in 'set -x',
'export -p' and the like.

The implementation attempts to comply to the proposal for the next issue of
the POSIX specification. Because this construct is not in POSIX.1-2008,
using it in scripts intended to be portable is unwise.

Matching the minimal locale support in the rest of sh, the \u and \U
sequences are currently not useful.

Exp-run done by: pav (with some other sh(1) changes)

sh: Apply set -u to variables in arithmetic.

Note that this only applies to variables that are actually used.
Things like (0 && unsetvar) do not cause an error.

Exp-run done by: pav (with some other sh(1) changes)

sh: Detect an error for ${#var<GARBAGE>}.

In particular, this makes things like ${#foo[0]} and ${#foo[@]} errors
rather than silent equivalents of ${#foo}.

PR: bin/151720
Submitted by: Mark Johnston
Exp-run done by: pav (with some other sh(1) changes)

Don't call -f option's argument "stdin".

MFC after: 3 days

sh: Set $? to 0 for background commands.

For backgrounded pipelines and subshells, the previous value of $? was being
preserved, which is incorrect.

For backgrounded simple commands containing a command substitution, the
status of the last command substitution was returned instead of 0.

If fork() fails, this is an error.

sh: Check setuid()/setgid() return values.

If the -p option is turned off, privileges from a setuid or setgid binary
are dropped. Make sure to check if this succeeds. If it fails, this is an
error which will cause the shell to abort except in interactive mode or if
'command' was used to make 'set' or an outer 'eval' or '.' non-special.

Note that taking advantage of this feature and writing setuid shell scripts
seems unwise.

MFC after: 1 week

sh: Remove duplicate code resetting uid/gid for set +p/+o privileged.

MFC after: 1 week

sh: Allow EV_EXIT through function calls, make {...} <redir more consistent.

If EV_EXIT causes an exit, use the exception mechanism to unwind
redirections and local variables. This way, if the final command is a
redirected command, an EXIT trap now executes without the redirections.

Because of these changes, EV_EXIT can now be inherited by the body of a
function, so do so. This means that a function no longer prevents a fork
before an exec being skipped, such as in
f() { head -1 /etc/passwd; }; echo $(f)

Wrapping a single builtin in a function may still cause an otherwise
unnecessary fork with command substitution, however.

An exit command or -e failure still invokes the EXIT trap with the
original redirections and local variables in place.

Note: this depends on SHELLPROC being gone. A SHELLPROC depended on
keeping the redirections and local variables and only cleaning up the
state to restore them.

sh: Do not word split "${#parameter}".

This is only a problem if IFS contains digits, which is unusual but valid.

Because of an incorrect fix for PR bin/12137, "${#parameter}" was treated
as ${#parameter}. The underlying problem was that "${#parameter}"
erroneously added CTLESC bytes before determining the length. This
was properly fixed for PR bin/56147 but the incorrect fix was not backed
out.

Reported by: Seeker on forums.freebsd.org
MFC after: 2 weeks

Document problems with -d/-w and the fact that -X is the default.

Suggested by: arundel@
Reviewed by: arundel@

Get rid of DSIZ; instead just call the sizing function if provided.

Make it possible to use permission sets (full_set, modify_set, read_set
and write_set) with setfacl(1).

PR: kern/154113
Submitted by: Shawn Webb <lattera at gmail dot com> (earlier version)
MFC after: 1 month

Add proper width calculation for time fields (time, cputime and usertime).
This fixes the ugly overflow in "ps aux" output for "[idle]".

Make "LOGIN" and "CLASS" columns width scale properly instead of wasting space.

sh(1): Describe subshell environment, command substitution more correctly.

POSIX does not require the shell to fork for a subshell environment, and we
use that possibility in various ways (command substitutions with a single
command and most subshells that are the final command of a shell process).
Therefore do not tie subshells to forking in the man page.

Command substitutions with expansions are a bit strange, causing a fork for
$(...$(($x))...) because $x might expand to y=2; they will probably be
changed later but this is how they work now.

Implement the usertime and systime keywords for ps, printing the
corresponding times reported by getrusage().

Submitted by: Dan Nelson <dnelson allantgroup com>
MFC after: 1 week

bin: Prefer strrchr() to rindex().

This removes the last index/rindex usage from /bin.

sh: Fix some parameter expansion variants ${#...}.

These already worked: $# ${#} ${##} ${#-} ${#?}
These now work as well: ${#+word} ${#-word} ${##word} ${#%word}

There is an ambiguity in the standard with ${#?}: it could be the length of
$? or it could be $# giving an error in the (impossible) case that it is not
set. We continue to use the former interpretation as it seems more useful.

Remove unnecessary cast.

Reviewed by: jilles

sh(1): Reduce excessive semicolon-separated sentences.

Reported by: Benjamin Kaduk

Export login class information via kinfo and make it possible to view
it using "ps -o class".

sh: Fix some warnings in code for arithmetic expressions.

Submitted by: eadler

kill: Note that this is used both as a normal program and a shell builtin.

Accept == as an alias of = which is a popular GNU extension.

This is intentionally undocumented for now since it's not part
of any standard.

MFC after: 1 month

When WITH_ICONV is set, use our in-tree iconv.

mdoc(7) markup.

Approved by: avg (mentor), kib (mentor)
MFC after: 3 days

Fix typos - remove duplicate "the".

PR: bin/154928
Submitted by: Eitan Adler <lists at eitanadler.com>
MFC after: 3 days

test: Note that this is used both as a normal program and a shell builtin.

MFC after: 1 week

sh: Detect dividing the smallest integer by -1.

This overflows and on some architectures such as amd64 it generates SIGFPE.
Generate an error on all architectures.

Fix typos.

PR: docs/131625
Submitted by: Andrew Wright <andrew at qemg.org>
MFC after: 1 month

sh(1): Update description of arithmetic.

sh: Import arithmetic expression code from dash.

New features:
* proper lazy evaluation of || and &&
* ?: ternary operator
* executable is considerably smaller (8K on i386) because lex and yacc are
no longer used

Differences from dash:
* arith_t instead of intmax_t
* imaxdiv() not used
* unset or null variables default to 0
* let/exp builtin (undocumented, will probably be removed later)

Obtained from: dash

sh: Fix two things about {(...)} <redir:

* In {(...) <redir1;} <redir2, do not drop redir1.
* Maintain the difference between (...) <redir and {(...)} <redir:
In (...) <redir, the redirection is performed in the child, while in
{(...)} <redir it should be performed in the parent (like {(...); :;}
<redir)

sh: Remove clearcmdentry()'s now unused argument.

sh: Forget all cached command locations on any PATH change.

POSIX requires this and it is simpler than the previous code that remembered
command locations when appending directories to PATH.

In particular,
PATH=$PATH
is no longer a no-op but discards all cached command locations.

sh: Do not try to execute binary files as scripts.

If execve() returns an [ENOEXEC] error, check if the file is binary before
trying to execute it using sh. A file is considered binary if at least one
of the first 256 bytes is '\0'.

In particular, trying to execute ELF binaries for the wrong architecture now
fails with an "Exec format error" message instead of syntax errors and
potentially strange results.

sh: Remove special code for shell scripts without magic number.

These are called "shell procedures" in the source.

If execve() failed with [ENOEXEC], the shell would reinitialize itself
and execute the program as a script. This requires a fair amount of code
which is not frequently used (most scripts have a #! magic number).
Therefore just execute a new instance of sh (_PATH_BSHELL) to run the
script.

Make sys_signame upper case.

This matches the constants from <signal.h> with 'SIG' removed, which POSIX
requires kill and trap to accept and 'kill -l' to write.

'kill -l', 'trap', 'trap -l' output is now upper case.

In Turkish locales, signal names with an upper case 'I' are now accepted,
while signal names with a lower case 'i' are no longer accepted, and the
output of 'killall -l' now contains proper capital 'I' without dot instead
of a dotted capital 'I'.

sh: Return only 126 or 127 for execve() failures.

Do not return 2 for errors other than [EACCES] or [ENOENT].

sh: Remove comment mentioning herefd, which is gone.

sh: Send messages about signals to stderr.

This is required by POSIX and seems to make more sense.

See also r217557.

sh: Clean up some old comments:
* There is no plan for an alternative to the command "set".
* Attempting to unset a readonly variable has not raised an error for quite
a while, so the order of unsetting a variable and a function with the same
name does not matter.

MFC after: 1 week

Document P_FOLLOWFORK.

MFC after: 2 weeks

sh: Fix signal messages being sent to the wrong file sometimes.

When a foreground job exits on a signal, a message is printed to stdout
about this. The buffer was not flushed after this which could result in the
message being written to the wrong file if the next command was a builtin
and had stdout redirected.

Example:
sh -c 'kill -9 $$'; : > foo; echo FOO:; cat foo

Reported by: gcooper
MFC after: 1 week

sh(1): Document changes to 'exit' from traps.

sh: If exit is used without args from a trap action, exit on the signal.

This is useful so that it is easier to exit on a signal than to reset the
trap to default and resend the signal. It matches ksh93. POSIX says that
'exit' without args from a trap action uses the exit status from the last
command before the trap, which is different from 'exit $?' and matches this
if the previous command is assumed to have exited on the signal.

If the signal is SIGSTOP, SIGTSTP, SIGTTIN or SIGTTOU, or if the default
action for the signal is to ignore it, a normal _exit(2) is done with exit
status 128+signal_number.

sh: Fix some things about -- in trap:
* Make 'trap --' do the same as 'trap' instead of nothing.
* Make '--' stop option processing (note that '-' action is not an option).

Side effect: The error message for an unknown option is different.

sh: Make 'trap -l' look like 'kill -l'.

sh: Follow-up to r216743, grabstackblock() can be replaced with stalloc().

grabstackblock() was used only once (but it is a very often executed piece
of code).

sh: Remove special %builtin PATH entry.
All builtins are now always found before a PATH search.

Most ash derivatives have an undocumented feature where the presence of an
entry "%builtin" in $PATH will cause builtins to be checked at that point of
the PATH search, rather than before looking at any directories as documented
in the man page (very old versions do document this feature).

I am removing this feature from sh, as it complicates the code, may violate
expectations (for example, /usr/bin/alias is very close to a forkbomb with
PATH=/usr/bin:%builtin, only /usr/bin/builtin not being another link saves
it) and appears to be unused (all the %builtin google code search finds is
in some sort of ash source code).

Note that aliases and functions took and take precedence above builtins.
Because aliases work on a lexical level they can only ever be overridden on
a lexical level (quoting or preceding 'builtin' or 'command'). Allowing
override of functions via PATH does not really fit in the model of sh and it
would work differently from %builtin if implemented.

Note: POSIX says special builtins are found before functions. We comply to
this because we do not allow functions with the same name as a special
builtin.

Silence from: freebsd-hackers@ (message sent 20101225)
Discussed with: dougb

Move repeated MAXSLP definition from machine/vmparam.h to sys/vmmeter.h.
Update the outdated comments describing MAXSLP and the process
selection algorithm for swap out.

Comments wording and reviewed by: alc

sh: Make exit without parameters from EXIT trap POSIX-compliant.

It should use the original exit status, just like falling off the
end of the trap handler.

Outside an EXIT trap, 'exit' is still equivalent to 'exit $?'.

sh: Do not call exitshell() from evalcommand() unless evalcommand() forked
itself.

This ensures that certain traps caused by builtins are executed.

Increase carried_error if we skip a file due to an error. This ensures
that setfacl(1) exits with proper exit status on failure.

PR: bin/149780
Submitted by: Ævar Arnfjörð Bjarmason (original version)
Reviewed by: trasz
MFC after: 3 weeks

sh: Check readonly status for assignments on regular builtins.

An error message is written, the builtin is not executed, nonzero exit
status is returned but the shell does not abort.

This was already checked for special builtins and external commands, with
the same consequences except that the shell aborts for special builtins.

Obtained from: NetBSD

sh: Check if dup2 for redirection from/to a file succeeds.

A failure (e.g. caused by ulimit -n being set very low) is a redirection
error.

Example:
ulimit -n 9; exec 9<.

sh: Avoid side effects from builtins in optimized command substitution.

Change the criterion for builtins to be safe to execute in the same process
in optimized command substitution from a blacklist of only cd, . and eval to
a whitelist.

This avoids clobbering the main shell environment such as by $(exit 4) and
$(set -x).

The builtins jobid, jobs, times and trap can still show information not
available in a child process; this is deliberately permitted. (Changing
traps is not.)

For some builtins, whether they are safe depends on the arguments passed to
them. Some of these are always considered unsafe to keep things simple; this
only harms efficiency a little in the rare case they are used alone in a
command substitution.

sh: Properly restore exception handler in fc.

If SIGINT arrived at exactly the right moment (unlikely), an exception
handler in a no longer active stack frame would be called.

Because the old handler was not used in the normal path, clang thought it
was a dead value and if an exception happened it would longjmp() to garbage.
This caused builtins/fc1.0 to fail if histedit.c was compiled with clang.

MFC after: 1 week

sh: Don't do optimized command substitution if expansions have side effects.

Before considering to execute a command substitution in the same process,
check if any of the expansions may have a side effect; if so, execute it in
a new process just like happens if it is not a single simple command.

Although the check happens at run time, it is a static check that does not
depend on current state. It is triggered by:
- expanding $! (which may cause the job to be remembered)
- ${var=value} default value assignment
- assignment operators in arithmetic
- parameter substitutions in arithmetic except ${#param}, $$, $# and $?
- command substitutions in arithmetic

This means that $((v+1)) does not prevent optimized command substitution,
whereas $(($v+1)) does, because $v might expand to something containing
assignment operators.

Scripts should not depend on these exact details for correctness. It is also
imaginable to have the shell fork if and when a side effect is encountered
or to create a new temporary namespace for variables.

Due to the $! change, the construct $(jobs $!) no longer works. The value of
$! should be stored in a variable outside command substitution first.

sh: Make expansion errors in optimized command substitution non-fatal.
Command substitutions consisting of a single simple command are executed in
the main shell process but this should be invisible apart from performance
and very few exceptions such as $(trap).

sh: Simplify "stack string" code slightly.

Maintain a pointer to the end of the stack string area instead of how much
space is left. This simplifies the macros in memalloc.h. The places where
the new variable must be updated are only where the memory area is created,
destroyed or resized.

sh: Fix integer overflow check, it checked an uninitialized variable.

sh: Allow arbitrary large numbers in CHECKSTRSPACE.
Reduce "stack string" API somewhat and simplify code.
Add a check for integer overflow of the "stack string" length (probably
incomplete).

sh(1): Explain why it is a bad idea to use aliases in scripts.

sh: Add kill builtin.

This allows specifying a %job (which is equivalent to the corresponding
process group).

Additionally, it improves reliability of kill from sh in high-load
situations and ensures "kill" finds the correct utility regardless of PATH,
as required by POSIX (unless the undocumented %builtin mechanism is used).

Side effect: fatal errors (any error other than kill(2) failure) now return
exit status 2 instead of 1. (This is consistent with other sh builtins, but
not in NetBSD.)

Code size increases about 1K on i386.

Obtained from: NetBSD

sh: Add a function to print warnings (with command name and newline).
This is like error() but without raising an exception.
It is particularly useful as a replacement for the warnx macro in
bltin/bltin.h.

sh: Make warnings in the printf builtin non-fatal, like in the program.

The #define for warnx now behaves much like the libc function (except that
it uses sh command name and output).

Also, it now uses C99 __VA_ARGS__ so there is no need for three different
macros for 0, 1 or 2 parameters.

sh: arith: Disallow decimal constants starting with 0 (containing 8 or 9).

Constants in arithmetic starting with 0 should be octal only.

This avoids the following highly puzzling result:
$ echo $((018-017))
3
by making it an error instead.

Remove dead code.

c is assigned 0 and *loc is pointing to NULL, so c!=0 cannot be true,
and dereferencing loc would be a bad idea anyway.

Coverity Prevent: CID 5113
Reviewed by: jilles

sh: Fix corruption of command substitutions with special chars after newline

The CTLESC byte to protect a special character was output before instead of
after a newline directly preceding the special character.

The special handling of newlines is because command substitutions discard
all trailing newlines.

Remove duplicate check, turning dead code into live code.

Coverity CID: 5114
Reviewed by: jilles

sh: Various simplifications to jobs.c:
* Prefer kill(-X) to killpg(X).
* Remove some dead code.
* No additional SIGINT is needed if int_pending() is already true.

No functional change is intended.

sh: Remove the herefd hack.

The herefd hack wrote out partial here documents while expanding them. It
seems unnecessary complication given that other expansions just allocate
memory. It causes bugs because the stack is also used for intermediate
results such as arithmetic expressions. Such places should disable herefd
for the duration but not all of them do, and I prefer removing the need for
disabling herefd to disabling it everywhere needed.

Here documents larger than 1024 bytes will use a bit more CPU time and
memory.

Additionally this allows a later change to expand here documents in the
current shell environment. (This is faster for small here documents but also
changes behaviour.)

Obtained from: dash

sh: Replace some macros and repeated code in expand.c with functions.

No functional change is intended, but the binary is about 1K smaller on
i386.

sh: Use vsnprintf() rather than crafting our own in fmtstr().
Add INTOFF/INTON as longjmp out of vsnprintf may cause memory leaks or
undefined behaviour.

sh: Improve internal-representation-to-text code to avoid binary output.

The code to translate the internal representation to text did not know about
various additions to the internal representation since the original ash and
therefore wrote binary stuff to the terminal.

The code is used in the jobs command and similar output.

Note that the output is far from complete and mostly serves for recognition
purposes.

sh: POSIX says there should not be a space between Done and (exitstatus).

(On the other hand, (core dumped) does need a space and so does [1] +.)

sh: Improve jobs output of pipelines.

If describing the status of a pipeline, write all elements of the pipeline
and show the status of the last process (which would also end up in $?).
Only write one report per job, not one for every process that exits.

To keep some earlier behaviour, if any process started by the shell in a
foreground job terminates because of a signal, write a message about the
signal (at most one message per job, however).

Also, do not write messages about signals in the wait builtin in
non-interactive shells. Only true foreground jobs now write such messages
(for example, "Terminated").

sh: Avoid marking a job as done before it is fully created.

In r208489, I added code to reap zombies when forking new processes, to
limit the amount of zombies. However, this can lead to marking a job as done
or stopped if it consists of multiple processes and the first process ends
very quickly. Fix this by only checking for zombies before forking the first
process of a job and not marking any jobs without processes as done or
stopped.

sh: jobs -p: Do not ask the kernel for the pgid.

The getpgid() call will fail if the first process in the job has already
terminated, resulting in output of "-1".

The pgid of a job is always the pid of the first process in the job and
other code already relies on this.

Move most of the remaining USD/PSD/SMM papers into share/doc

sh(1): Clean up documentation of built-in commands.

Make sure all built-in commands are in the subsection named such, except
exp, let and wordexp which are deliberately undocumented. The text said only
built-ins that really need to be a built-in were documented there but in
fact almost all of them were already documented.

sh(1): Document that command's -p option also works with -v/-V.

This was implemented in r201343.

sh: Code size optimizations to "stack string" memory allocation:
* Prefer one CHECKSTRSPACE with multiple USTPUTC to multiple STPUTC.
* Add STPUTS macro (based on function) and use it instead of loops that add
nul-terminated strings to the stack string.

No functional change is intended, but code size is about 1K less on i386.

sh: Pass multiple bytes at a time to lex.

This speeds up the expansion/arith6.0 test considerably.

sh: Fix confusing behaviour if chdir succeeded but getcwd failed in cd -P.

If getcwd fails, do not treat this as an error, but print a warning and
unset PWD. This is similar to the behaviour when starting the shell in a
directory whose name cannot be determined.

Fix some more warnings found by clang.

sh: Remove the check that alpha/name/in_name chars are not CTL* bytes.

Since is_alpha/is_name/is_in_name were made ASCII-only, this can no longer
happen.

Additionally, the check was wrong because it did not include the new
CTLQUOTEEND.

sh: Code size optimizations to buffered output.

This is mainly less use of the outc macro.

No functional change is intended, but code size is about 2K less on i386.

sh: Add printf builtin.

This was removed in 2001 but I think it is appropriate to add it back:
* I do not want to encourage people to write fragile and non-portable echo
commands by making printf much slower than echo.
* Recent versions of Autoconf use it a lot.
* Almost no software still wants to support systems that do not have
printf(1) at all.
* In many other shells printf is already a builtin.

Side effect: printf is now always the builtin version (which behaves
identically to /usr/bin/printf) and cannot be overridden via PATH (except
via the undocumented %builtin mechanism).

Code size increases about 5K on i386. Embedded folks might want to replace
/usr/bin/printf with a hard link to /usr/bin/alias.

sh: Add binary buffered output for use by the printf builtin.

sh: Update the suspend example for the change of the job control flag
from -j to -m, many years ago.

Due to r215266, this function now actually works.

sh: Do the additional actions if 'local -' restore changes -i/-m/-E/-V.

Example:
f() { local -; set +m; }; f
caused failure to execute external programs because the job control tty fd
was not opened.

echo(1): Clarify portability and mention literal "--" handling.

MFC after: 1 week

sh(1): Document r214304 (special builtin is illegal function name).

sh(1): Update for r214492. "${v+"hi}there"}".

The part hi}there is not a quoted string but nevertheless the closing brace
does not terminate the expansion.

sh: Remove unused man page for echo builtin.

The information in sh(1) about the echo builtin is equivalent, though less
extensive.

The echo(1) man page (bin/echo/echo.1) is different.

Unfortunately, sh's echo builtin and /bin/echo have gone out of sync and
this probably cannot be fixed any more.

Reported by: uqs (list of untouched files)
MFC after: 1 week

sh(1): Modernize the introduction a bit.

In particular, remove the text about ksh-like features, which are usually
taken for granted nowadays. The original Bourne shell is fading away and for
most users our /bin/sh is one of the most minimalistic they know.

test: Move tests to tools/regression/bin/test.
Convert the tests to the perl prove format.
Remove obsolete TEST.README (results of an old TEST.sh for some old Unices)
and TEST.csh (old tests without correct values, far less complete than
TEST.sh).

MFC after: 1 week

sh: Fix some issues with aliases and case, by importing dash checkkwd code.

This moves the function of the noaliases variable into the checkkwd
variable. This way it is properly reset on errors and aliases can be used
normally in the commands for each case (the case labels recognize the
keyword esac but no aliases).

The new code is clearer as well.

Obtained from: dash

Add a new libc function: cfmakesane(3).

I've noticed various terminal emulators that need to obtain a sane
default termios structure use very complex `hacks'. Even though POSIX
doesn't provide any functionality for this, extend our termios API with
cfmakesane(3), which is similar to the commonly supported cfmakeraw(3),
except that it fills the termios structure with sane defaults.

Change all code in our base system to use this function, instead of
depending on <sys/ttydefaults.h> to provide TTYDEF_*.

sh(1): Correct synopsis and make precise how $0 is set.

In particular, the extra argument to set $0 with -c was not documented.

MFC after: 1 week

Fix typo and grammar nit

Submitted by: arundel
MFC after: 7 days (or when the bikeshed has abated)

sh: Reindent evaltree().

sh: Use iteration instead of recursion to evaluate semicolon lists.
This reduces CPU and memory usage when executing long lists (such
as long functions).

Elaborate some more on the non-security implications of using -P

Submitted by: delphij
Discussion at: svn-src-all

sh: Tweak some string constants to reduce code size.

* Reduce some needless differences.
* Shorten some error messages that should not happen.

sh: Reject function names ending in one of !%*+-=?@}~

These do something else in ksh: name=(...) is an array or compound variable
assignment and the others are extended patterns.

This is the last patch of the ones tested in the exp run.

Exp-run done by: pav (with some other sh(1) changes)

sh: Detect various additional errors in the parser.

Apart from detecting breakage earlier or at all, this also fixes a segfault
in the testsuite. The "handling" of the breakage left an invalid internal
representation in some cases.

Examples:
echo a; do echo b
echo `) echo a`
echo `date; do do do`

Exp-run done by: pav (with some other sh(1) changes)

sh: Error out on various specials/keywords in the wrong place in backticks.

Example:
echo `date)`

Exp-run done by: pav (with some other sh(1) changes)
Obtained from: NetBSD (Christos Zoulas, NetBSD PR 11317)

sh: Fix some issues with CTL* bytes and ${var#pat}.

subevalvar() incorrectly assumed that CTLESC bytes were present iff the
expansion was quoted. However, they are present iff various processing such
as word splitting is to be done later on.

Example:
v=@$e@$e@$e@
y="${v##*"$e"}"
echo "$y"
failed if $e contained the magic CTLESC byte.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do IFS splitting on word in ${v+word} and ${v-word}.

The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.

Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.

Example:
for i in ${$+a b c}; do echo $i; done

Exp-run done by: pav (with some other sh(1) changes)

sh: Only accept a '}' inside ${v+-=?...} if double-quote state matches.
If double-quote state does not match, treat the '}' literally.

This ensures double-quote state remains the same before and after a
${v+-=?...} which helps with expand.c.

It makes things like
${foo+"\${bar}"}
which I have seen in the wild work as expected.

Exp-run done by: pav (with some other sh(1) changes)

sh: Make double-quotes quote a '}' inside ${v#...} and ${v%...}.

Exp-run done by: pav (with some other sh(1) changes)
PR: bin/57554

Language cleanup.

sh: Ignore double-quotes in arithmetic rather than treating them as quotes.

This provides similar behaviour, but allows a simpler parser.

This changes r206473.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do not allow overriding a special builtin with a function.
This is a syntax error.

POSIX does not say explicitly whether defining a function with the same name
as a special builtin is allowed, but it does say that it is impossible to
call such a function.

A special builtin can still be overridden with an alias.

This commit is part of a set of changes that will ensure that when
something looks like a special builtin to the parser, it is one. (Not the
other way around, as it remains possible to call a special builtin named
by a variable or other substitution.)

Exp-run done by: pav (with some other sh(1) changes)

sh: Make sure defined functions can actually be called.

Add some conservative checks on function names:
- Disallow expansions or quoting characters; these can only be called via
strange control characters
- Disallow '/'; these functions cannot be called anyway, as exec.c assumes
they are pathnames
- Make the CTL* bytes work properly in function names.

These are syntax errors.

POSIX does not require us to support more than names (letters, digits and
underscores, not starting with a digit), but I do not want to restrict it
that much at this time.

Exp-run done by: pav (with some other sh(1) changes)

sh: Check whether dup2 was successful for >&FD and <&FD.

A failure (usually caused by FD not being open) is a redirection error.

Exp-run done by: pav (with some other sh(1) changes)

sh: Change ! within a pipeline to start a new pipeline instead.

This is how ksh93 treats ! within a pipeline and makes the ! in
a | ! b | c
negate the exit status of the pipeline, as if it were
a | { ! b | c; }

Side effect: something like
f() ! a
is now a syntax error, because a function definition takes a command,
not a pipeline.

Exp-run done by: pav (with some other sh(1) changes)

sh(1): Clarify subshells/processes for pipelines.

For multi-command pipelines,
1. all commands are direct children of the shell (unlike the original
Bourne shell)
2. all commands are executed in a subshell (unlike the real Korn shell)

MFC after: 1 week

sh: Use <stddef.h> rather than <sys/stddef.h>.

<sys/stddef.h> is only for the kernel and conflicts with <stddef.h>.

We only need to look as far as '..' to find 'test/'.

Do not assume in growstackstr() that a "precious" character will be
immediately written into the stack after the call. Instead let the caller
manage the "space left".

Previously, growstackstr()'s assumption causes problems with STACKSTRNUL()
where we want to be able to turn a stack into a C string, and later
pretend the NUL is not there.

This fixes a bug in STACKSTRNUL() (that grew the stack) where:
1. STADJUST() called after a STACKSTRNUL() results in an improper adjust.
This can be seen in ${var%pattern} and ${var%%pattern} evaluation.
2. Memory leak in STPUTC() called after a STACKSTRNUL().

Reviewed by: jilles

In the spirit of r90111, depend on c89 and remove the "STATIC" macro
and its usage.

If one wishes to set breakpoints of static the functions here, they
cannot be inlined.

Submitted by: jhb

Make DEBUG traces 64-bit clean:
- Use %t to print ptrdiff_t values.
- Cast a ptrdiff_t value explicitly to int for a field width specifier.

While here, sort includes.

Submitted by: Garrett Cooper

Suggest that DEBUG_FLAGS be used to enable extra debugging rather than
frobbing CFLAGS directly. DEBUG_FLAGS is something that can be specified
on the make command line without having to edit the Makefile directly.

Submitted by: Garrett Cooper

Consistently use "STATIC" for all functions in order to be able to set
breakpoints with in a debugger. And use naked "static" for variables.

Noticed by: bde

If DEBUG is 3 or greater, disable STATICization of functions.
Also correct the documented location of the trace file.

Allow one to regression test 'sh' changes without having to install
a potentially bad /bin/sh first.

rm(1): clarify that -P works only when blocks are updated in-place

Suggested by: pjd, ivoras, arundel

mdoc: drop redundant .Pp and .LP calls

They have no effect when coming in pairs, or before .Bl/.Bd

Make a thread's address available via the kern proc sysctl, just like the
process address.

Add "tdaddr" keyword to ps(1) to display this thread address.

Distilled from Sandvine's patch set by Mark Johnston.

Clarify the combination effect of -P and -f to make it clear.

Submitted by: arundel
MFC after: 2 weeks

Correct sort order.

Merge from tbemd:
Add directory names directly and sort at the end.
Include bsd.arch.inc.mk so we can, in the future, more easily make arch
dependent changes in /bin (unlikely, but is needed for symmetry).

sh: Add __dead2 to two functions that do not return.

Apart from helping static analyzers, this also appears to reduce the size of
the binary slightly.

sh: Fix exit status if return is used within a loop condition.

sh: Apply variable assignments left-to-right in bltinlookup().

Example:
HOME=foo HOME=bar cd

Revert changes of 'assure' to 'ensure' made in r211936.

Approved by: rrs (mentor)

test(1): Fix markup, ( and ) must be separate arguments so leave spaces.

MFC after: 1 week

test(1): Clarify grammar ambiguity and -a/-o vs shell &&/||.

sh(1): Remove xrefs for expr(1) and getopt(1).

expr(1) should usually not be used as various forms of parameter expansion
and arithmetic expansion replicate most of its functionality in an easier
way.

getopt(1) should not be used at all in new code. Instead, getopts(1) or
entirely manual parsing should be used.

MFC after: 1 week

expr(1): Add sh(1) versions of examples, remove an incorrect example.

The three examples are better done using sh(1) itself these days.

The example
expr -- "$a" : ".*"
is incorrect in the general case, as "$a" may be an operator.

MFC after: 2 weeks

sh: Fix 'read' if all chars before the first IFS char are backslash-escaped.

Backslash-escaped characters did not set the flag for a non-IFS character.

MFC after: 2 weeks

sh: Improve comments in expand.c.

sh: Get rid of some magic numbers.

MFC after: 1 week

sh: Do not use locale for determining if something is a name.

This makes it impossible to use locale-specific characters in variable
names.

Names containing locale-specific characters make scripts only work with the
correct locale setting. Also, they did not even work in many practical cases
because multibyte character sets such as utf-8 are not supported.

This also avoids weirdness if LC_CTYPE is changed in the middle of a script.

Touch the man page date after updating the ustar limitations.

MFC after: 3 weeks

Correct an out-by-one error when earlying out ustar filenames that
are too long. Filenames escaping this test are caught later on,
so the bug doesn't cause any breakage.

Document the correct ustar limitations in pax. As I have no access
to the IEEE 1003.2 spec, I can only assume that the limitations
imposed are in fact correct.

Add regression tests for the filename limitations imposed by pax.

MFC after: 3 weeks

Fix incorrect usage of 'assure' and 'insure'.

Approved by: rrs (mentor)

Fix an off-by-one error where we try to split a path name that's
more than 100 characters long and the 101th last character is a
'/'.

MFC after: 3 weeks

sh: Remove remnants of '!!' to negate pattern.

This Almquist extension was disabled long ago.

In pathname generation, components starting with '!!' were treated as
containing wildcards, causing unnecessary readdir (which could fail, causing
pathname generation to fail while it should not).

sh(1): Add a brief summary of arithmetic expressions.

sh: Fix break/continue/return sometimes not skipping the rest of dot script.

In our implementation and most others, a break or continue in a dot script
can break or continue a loop outside the dot script. This should cause all
further commands in the dot script to be skipped. However, cmdloop() did not
know about this and continued to parse and execute commands from the dot
script.

As described in the man page, a return in a dot script in a function returns
from the function, not only from the dot script. There was a similar issue
as with break and continue. In various other shells, the return appears to
return from the dot script, but POSIX seems not very clear about this.

sh: Add a forgotten const.

sh: Fix shadowing of sigset.

sh: Fix heap-based buffer overflow in pathname generation.

The buffer for generated pathnames could be too small in some cases. It
happened to be always at least PATH_MAX long, so there was never an overflow
if the resulting pathnames would be usable.

This bug may be abused if a script subjects input from an untrusted source
to pathname generation, which a bad idea anyhow. Most shell scripts do not
work on untrusted data. secteam@ says no advisory is necessary.

PR: bin/148733
Reported by: Changming Sun snnn119 at gmail com
MFC after: 10 days

Remove unnecessary duplicate letters in mksyntax.c,
the table elements would just be overwritten twice.

sh: Return 0 from eval if no command was given.

This makes a difference if there is a command substitution.

To make this work, evalstring() has been changed to set exitstatus to 0 if
no command was executed (the string contained only whitespace).

Example:
eval $(false); echo $?
should print 0.

Document the new argument parser for sleep(1) and SIGINFO behaviour.
Remove higlight for the unportability notice.

MFC after: 3 weeks

Simplify argument parser by using sscanf(); simplify usage() by not
refusing to use stdio.
Reduce nesting level in the sleep loop by returning earlier for negative
timeouts.
Limit the maximum timeout to INT_MAX seconds.

Submitted by: bde
MFC after: 3 weeks

sh: Do not enter consecutive duplicates into the history.

This simply sets a flag in libedit. It has a shortcoming in that it does not
apply to multi-line commands.

Note that a configuration option for this is not going to happen, but always
having this seems better than not having it. NetBSD has done the same.

PR: bin/54683
Obtained from: NetBSD
MFC after: 1 month

Spelling fixes.

Deal with proper format for printing time_t.

Reported by: ache
MFC after: 3 weeks

Report the time left for the sleep on SIGINFO.
Be stricter in the checking of interval specification.

PR: bin/139345
MFC after: 3 weeks

kill: Stop processing if a syntactically invalid pid is encountered.

So a command like
kill _HUP 1
now fails without sending SIGTERM to init.

The behaviour when kill(2) fails remains unchanged: processing continues.
This matches other implementations and POSIX and is useful for killing
multiple processes at once when some of them may already be gone.

PR: bin/40282

Typo fix: s/ommited/omitted

MFC after: 4 days

sh: Fix crash due to uninitialized here-document.

If an ; or & token was followed by an EOF token, pending here-documents were
left uninitialized. Execution would crash, either in the main shell process
for literal here-documents or in a child process for expanded
here-documents. In the latter case the problem is hard to detect apart from
the core dumps and log messages.

Side effect: slightly different retries on inputs where EOF is not
persistent.

Note that tools/regression/bin/sh/parser/heredoc6.0 still causes a similar
crash in a child process. The text passed to eval is malformed and should be
rejected.

Fix alignment for the 'flags' label, and make more room for 'tdev'.

MFC after: 1 month

Add P_HASTHREADS flag description.

sh: Allow a background command consisting solely of redirections.

Example:
</dev/null &

MFC after: 2 weeks

sh: There cannot be a TNOT in simplecmd(), remove checks.

simplecmd() only handles simple commands and function definitions, neither
of which involves the ! keyword. The initial token on entry to simplecmd()
is one of the following: TSEMI, TAND, TOR, TNL, TEOF, TWORD, TRP.

Add -l to the synopsis

Submitted by: jhell at dataix dot net
MFC after: 3 days

sh: Use $PWD instead of getcwd() for the \w and \W prompt expansions.

This ensures that the logical working directory (which may include
symlinks) is shown and is similar to the default behaviour of the pwd
builtin.

sh: Forget about terminated background processes sooner.

Unless $! has been referenced for a particular job or $! still contains that
job's pid, forget about it after it has terminated. If $! has been
referenced, remember the job until the wait builtin has reported its
completion (either with the pid as parameter or without parameters).

In interactive mode, jobs are forgotten after termination has been reported,
which happens before primary prompts and through the jobs builtin. Even
then, though, remember a job if $! has been referenced.

This is similar to what is suggested by POSIX and should fix most memory
leaks (which also tend to cause sh to use more CPU time) with long running
scripts that start background jobs.

Caveats:
* Repeatedly referencing $! without ever doing 'wait', like
while :; do foo & echo started foo: $!; sleep 60; done
will still use a lot of memory and CPU time in the long run.
* The jobs and jobid builtins do not cause a job to be remembered for longer
like expanding $! does.

PR: bin/55346

Recognise the -l switch with pkill - list kill command(s) used.

PR: 143558
Submitted by: eitanadlerlist at gmail dot com
MFC after: 3 weeks

sh: Fix compilation with -DNO_HISTORY.

The LINENO code uses snprintf() and relied on "myhistedit.h" to pull in the
necessary <stdio.h>.

Compiling with -DNO_HISTORY disables all editing and history support and
allows linking without -ledit -ltermcap. This may be useful for embedded
systems.

MFC after: 2 weeks

sh: Add filename completion.

This uses the new libedit completion function with quoting support.

Unlike NetBSD, there is no 'set +o tabcomplete' option to disable
completion. I do not see any reason for such a special treatment, as
completion is rather useful and it is possible to do
bind ^I ed-insert
to disable completion and insert a tab character instead.

Submitted by: Guy Yur

More mdoc nitpicking to improve compatibility to mdocml

- .Nd in section NAME is not optional
- .Ed was missing
- "indent" is not a flag, but a literal argument for -offset
- stop switching font sizes for acronyms
- use .Brq instead of rolling our own

sh: Pass through SIGINT from a child if interactive and job control
is enabled.

This already worked if without job control.

In either case, this depends on it that a process that terminates due to
SIGINT exits on it (so not with status 1, or worse, 0).

Example:
sleep 5; echo continued
This does not print "continued" any more if sleep is aborted via ctrl+c.

MFC after: 1 month

Don't try to copy a socket after "xxx is a socket (not copied)." message.
Previously, it would either try to copy it anyway and fail (without -R),
or create fifo instead of the socket (with -R).

Found with: Coverity Prevent
CID: 5623
MFC after: 2 weeks

Explain that setting an absolute day value can fail and that the month should
usually be set first when using -v.

Adjust an example that sets the day to 30 before setting the month to 3 in
accordance with this approach as the example would always fail in February!

PR: 147354
MFC after: 2 weeks

sh: Pass TERM changes to libedit.

I have changed the patch slightly to ignore TERM changes in subshells.

PR: bin/146916
Submitted by: Guy Yur
Obtained from: NetBSD

Zero struct before reading from it

PR: 140384
Found by: clang static analyzer
MFC after: 3 weeks

sh: Fix a crash if a heredoc was not properly ended and parsing continued.

Example (in interactive mode):
cat <<EOF && )
The next command typed caused sh to segfault, because the state for the here
document was not reset.

Like parser_temp, this uses the fact that the parser is not re-entered.

sh: Change interaction of command substitution and here documents.

If a command substitution contains a newline token, this no longer starts
here documents of outer commands. This way, we follow POSIX's idea of the
command substitution being a separate script more closely. It also matches
other shells better and is consistent with newline characters in quotes not
starting here documents.

The extension tested in parser/heredoc3.0 ($(cat <<EOF)\ntext\nEOF\n)
continues to be supported.

In particular, this change allows things like
cat <<EOF && echo `pwd`
(a `` command substitution after a here document)
which formerly silently used an empty file as the here document, because the
EOF of the inner command "pwd" also forced an empty here document.

sh: Recognize "--" in . and exec.

Although "--" historically has not been required to be recognized for
certain special builtins that do not take options in POSIX, some other
implementations recognize options for them, requiring scripts to use "--" or
avoid operands starting with "-".

Operands starting with "-" can be avoided with eval by prepending a space,
and cannot occur with break, continue, exit, return and shift as they only
take numbers, nor with times as it does not take operands. With . and exec,
avoiding "-" is not so easy as it may require reimplementing the PATH
search; therefore the current proposal for POSIX is to require recognition
of "--" for them.

We continue to accept other strings starting with "-" as operands to . and
exec, and also "--" if it is alone to . (which would otherwise be invalid
anyway).

sh(1): Rework documentation of shell variables.

* Move the "environment variables" that do not need exporting to be
effective or that are set by the shell without exporting to a new section
"Special Variables".
* Add special variables LINENO and PPID.
* Add environment variables LANG, LC_* and PWD; also describe ENV under
environment variables.

sh(1): Improve wording of 'Special Parameters' section.

sh: Reap any zombies before forking for a background command.

This prevents accumulating huge amounts of zombies if a script executes
many background commands but no external commands or subshells.

Note that zombies will not be reaped during long calculations (within
the shell process) or read builtins, but those actions do not create
more zombies.

The terminated background commands will also still be remembered by the
shell.

PR: bin/55346

Fix back references in substitute command for pax(1)

pax(1) was trying to copy the back-referenced data from
the match pattern, not the matched data.

PR: bin/118132
Obtained from: Debian bug #451361
Reviewed by: jilles
MFC after: 3 weeks

Update authors and history.

MFC after: 1 week

sh: Fix pathname expansion with quoted slashes like *\/.

These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.

Because this is the first code I'm importing from dash to expand.c, add the
Herbert Xu copyright notice which is in dash's expand.c.

When pathname expanding *\/, the CTLESC representing the quoted state was
erroneously taken as part of the * pathname component. This CTLESC was then
seen by the pattern matching code as escaping the '\0' terminating the
string.

The code is slightly different because dash converts the CTLESC characters
to backslashes and removes all the other CTL* characters to allow
substituting glob(3).

The effect of the bug was also slightly different from dash (where nothing
matched at all). Because a CTLESC can escape a '\0' in some way, whether
files were included despite the bug depended on memory that should not be
read. In particular, on many machines /*\/ expanded to a strict subset of
what /*/ expanded to.

Example:
echo /*"/null"

This should print /dev/null, not /*/null.

PR: bin/146378
Obtained from: dash

sh(1): Fix "reserved word" vs "keyword" inconsistency.
Use "keyword" everywhere, like the output of the 'type' builtin, and only
mention "reserved word" once to say it is the same thing.

sh: Have only one copy of _PATH_STDPATH in the binary.

sh: Apply locale vars on builtins, recognize LC_MESSAGES as a locale var.

This allows doing things like LC_ALL=C some_builtin to run a builtin under a
different locale, just like is possible with external programs. The
immediate reason is that this allows making printf(1) a builtin without
breaking things like LC_NUMERIC=C printf '%f\n' 1.2

This change also affects special builtins, as even though the assignment is
persistent, the export is only to the builtin (unless the variable was
already exported).

Note: for this to work for builtins that also exist as external programs
such as /bin/test, the setlocale() call must be under #ifndef SHELL. The
shell will do the setlocale() calls which may not agree with the environment
variables.

sh: Use stalloc for arith variable names.

This is simpler than the custom memory tracker I added earlier, and is also
needed by the dash arith code I plan to import.

symlink(7): Add lpathconf(2) and *at system calls.

MFC after: 1 week

symlink(7): The ownership of symlinks is used by the system,
in at least three ways, so do not say it is ignored:
* who may delete/rename a symlink in a sticky directory
* who may do lchflags(2)/lchown(2)/lchmod(2)
* whose inode quota is charged

MFC after: 1 week

ln: Allow a trailing slash when creating a link to a directory.

In the 'ln source... directory' synopsis, the basename of each source
determines the name of the created link. Determine this using basename(3)
instead of strrchr(..., '/') which is incorrect if the pathname ends in a
slash.

The patch is somewhat changed to allow for basename(3) implementations that
change the passed pathname, and to fix the -w option's checking also.
The code to compare directory entries only applies to hard links, which
cannot be created to directories using ln.

Example:
ln -s /etc/defaults/ /tmp
This should create a symlink named defaults.

PR: 121568
Submitted by: Ighighi
MFC after: 1 week

ln: Refuse deleting a directory entry by hardlinking it to itself.

Two pathnames refer to the same directory entry iff the directories match
and the final components' names match.

Example: (assuming file1 is an existing file)
ln -f file1 file1
This now fails while leaving file1 intact. It used to delete file1 and then
complain it cannot be linked because it is gone.

With -i, this error is detected before the question is asked.

MFC after: 2 weeks

sh: On startup of the shell, use PWD from the environment if it is valid.
Unset PWD if it is incorrect and no value for it can be determined.
This preserves the logical current directory across shell invocations.

Example (assuming /home is a symlink):
$ cd
$ pwd
/home/foo
$ sh
$ pwd
/home/foo

Formerly the second pwd would show the physical path (symlinks resolved).

mdoc: order prologue macros consistently by Dd/Dt/Os

Although groff_mdoc(7) gives another impression, this is the ordering
most widely used and also required by mdocml/mandoc.

Reviewed by: ru
Approved by: philip, ed (mentors)

Update the list of the process flags for P_WKILLED.

MFC after: 4 weeks

sh: Partially revert r206146, allowing double-quotes in arithmetic.

These do pretty much nothing (except that parentheses are ignored), but
people seem to use them and allowing them does not hurt much.

Single-quotes seem not to be used and cause silently different behaviour
with ksh93 character constants.

sh: Automatically enable -o emacs in interactive shells with terminals.

This makes sh a bit more friendly in single user mode, make buildenv, chroot
and the like, and matches other shells.

The -o emacs can be overridden on the command line or in the ENV file.

sh: Document the expansion changes in the man page.

Note that the following sentence
> Enclosing the full parameter expansion string in double-quotes does not
> cause the following four varieties of pattern characters to be quoted,
> whereas quoting characters within the braces has this effect.
is now true, but used to be incorrect.

sh: Do tilde expansion in substitutions.

This applies to word in ${v-word}, ${v+word}, ${v=word}, ${v?word} (which
inherits quoting from the outside) and in ${v%word}, ${v%%word}, ${v#word},
${v##word} (which does not inherit any quoting).

In all cases tilde expansion is only attempted at the start of word, even if
word contains spaces. This agrees with POSIX and other shells.

This is the last part of the patch tested in the exp-run.

Exp-run done by: erwin (with some other sh(1) changes)

sh: Allow quoting pattern match characters in ${v%pat} and ${v#pat}.

Note that this depends on r206145 for allowing pattern match characters to
have their special meaning inside a double-quoted expansion like "${v%pat}".

PR: bin/117748
Exp-run done by: erwin (with some other sh(1) changes)

sh: Remove special handling for ' and " in arithmetic.
They will be treated like normal characters, resulting in a runtime
arithmetic expression error.

Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix various things about expansions:
* remove the backslash from \} inside double quotes inside +-=?
substitutions, e.g. "${$+\}a}"
* maintain separate double-quote state for ${v#...} and ${v%...};
single and double quotes are special inside, even in a double-quoted
string or here document
* keep track of correct order of substitutions and arithmetic

This is different from dash's approach, which does not track individual
double quotes in the parser, trying to fix this up during expansion.
This treats single quotes inside "${v#...}" incorrectly, however.

This is similar to NetBSD's approach (as submitted in PR bin/57554), but
recognizes the difference between +-=? and #% substitutions hinted at in
POSIX and is more refined for arithmetic expansion and here documents.

PR: bin/57554
Exp-run done by: erwin (with some other sh(1) changes)

sh: Treat unexpected newlines in substitutions as a syntax error.

The old approach was wrong because PS2 was not used and seems unlikely to
parse extensions (ksh93's ${ COMMAND} may well fail to parse).

Exp-run done by: erwin (with some other sh(1) changes)

Merge OpenBSD revisions 1.4 through 1.9, mostly style cleanups.

Obtained from: OpenBSD
MFC after: 3 month

Remove unused files.

Change all our own code to use st_*tim instead of st_*timespec.

Also remove some local patches to diff(1) which are now unneeded.

o) Add a keyword to displaying elapsed time in integer seconds, "etimes".
o) Give slightly better (i.e. any) documentation of the format of "etime".

Reviewed by: jilles

sh: Do not abort on a redirection error on a compound command.

Redirection errors on subshells already did not abort the shell because
the redirection is executed in the subshell.

Other shells seem to agree that these redirection errors should not abort
the shell.

Also ensure that the redirections will be cleaned up properly in cases like
command eval '{ shift x; } 2>/dev/null'

Example:
{ echo bad; } </var/empty/x; echo good

sh: Do not abort on a redirection error if there is no command word.

Although simple commands without a command word (only assignments and/or
redirections) are much like special builtins, POSIX and most shells seem to
agree that redirection errors should not abort the shell in this case. Of
course, the assignments persist and assignment errors are fatal.

To get the old behaviour portably, use the ':' special builtin.
To get the new behaviour portably, given that there are no assignments, use
the 'true' regular builtin.

sh: Fix longjmp clobber warnings in parser.c.

Make parsebackq a function instead of an emulated nested function.
This puts the setjmp usage in a smaller function where it is easier to avoid
bad optimizations.

Switch to our preferred license text.

Approved by: jedgar

sh: Make sure to popredir() even if a function caused an error.

sh: Make sure to popredir() even if a special builtin caused an error.

sh: Improve the command builtin:
* avoid unnecessary fork
* allow executing builtins via command
* executing a special builtin via command removes its special properties

Obtained from: NetBSD (parts)

Update the list of the process flags. Note that the lists of pending
signals for process and its threads are distinct.

Reviewed by: jilles
MFC after: 2 weeks

ed(1): make WARNS=6 clean

Although argc and argv are never read after the longjmp is complete,
gcc is not clever enough to see that and needlessly warns about it.
So add volatile to silence the compiler.

Approved by: ed (the co-mentor, not ed(1))

The NetBSD Foundation has granted permission to remove clause 3 and 4 from
their software.

Approved by: pjd
Obtained from: NetBSD

Do not restrict the allowed signals that can be specified by number
to the list of signals that has symbolic name. It was impossible to
send rt signals with kill(1) due to the check.

MFC after: 1 week

Fix expansion of \W in prompt strings when the working directory is "/".
The prompt string was truncated after \W when the working directory was "/".

PR: bin/89410
Submitted by: Dr Balwinder Singh Dheeman
MFC after: 1 week

Bump WARNS where possible.

Checked by: make universe
Approved by: ed (co-mentor)

- Implement -q option for pgrep(1).
- Add regression test to test -q option.

Initialize the execfile argument to NULL instead of _PATH_DEVNULL. This allows the -M option to be used without specifying -N.

PR: bin/138146
Approved by: rrs (mentor)
MFC after: 3 days

Make sure that FTS_COMFOLLOW is not set when the -P option is in effect.
Otherwise the -i option will show the inode number of the referenced file
for symbolic links given on the command line. Similarly, the file color
was printed according to the link target in colorized output.

PR: bin/102394
Reviewed by: jilles
MFC after: 2 weeks

Tell the compiler these structures are aligned to a byte boundary.
All the elements of these structs are char anyway, so it won't hurt
performance.

Bump warns back up to the default.

# we likely should have CTASSERTS to make sure they are the right size.
# but with libarchive based tar maybe we shouldn't bother.

sh: Do not stat() $MAIL/$MAILPATH in non-interactive shells.

These may be NFS mounted, and we should not touch them unless we are going
to do something useful with the information.

Arm doesn't seem to need such kit gloves. Bump the warnings to 3 here
as well.

Fixes for ls(1) long format (-l) output:

- Allow -h option to work if the listing contains at least one device
file.
- Align major and minor device numbers correctly to the size field.

PR: bin/125678
Approved by: trasz (mentor)
MFC after: 1 month

Print full path in the error message. It's possible that fts(3)
provides an empty fts_name and reporting the full path is more
appropriate especially with the -R option.

PR: bin/107515
Submitted by: bde
Approved by: trasz (mentor)
MFC after: 1 week

Remove stale references to utmp(5) and its corresponding filenames.

I removed utmp and its manpage, but not other manpages referring to it.

Raise WARNS for various tools where possible.

Submitted by: Marius Nünnerich <marius@nuenneri.ch>

Implement an "-x" option to cp(1), for compatibility with Linux and
feature parity with du(1) and similar: When set, cp(1) will not traverse
mount points.

Initial patch by: Graham J Lee leeg teaching.physics.ox.ac.uk

PR: bin/88056
Initial patch by: Graham J Lee leeg teaching.physics.ox.ac.uk
Approved by: ed (mentor)
MFC after: 1 month

Port everything in bin/ from utmp to utmpx.

date: use libc utmpx routines instead of the ones provided by libulog.
pax: don't depend on <utmp.h>

Let csh(1) use utmpx instead of utmp.

csh allows you to monitor the utmp(x) file to monitor certain user
logins. Unfortunately it needs to directly stat() this file. I don't
want to break this module , but eventually it shouldn't do that. The
idea of the getutxent(3) API is to hide file access.

Approved by: mp

Committed the wrong version in r201484. This time really fix the "-t"
functionality. Per the regression tests (pgrep-t.t & pkill-t.t), "-t"
should accept "v1", which means a plain number should be accepted for
UNIX98-style PTY's.

Fix the "-t" functionality. Per the regression tests (pgrep-t.t & pkill-t.t),
"-t" should accept "v1", which means a plain number should be accepted for
UNIX98-style PTY's.

sh: Send the "not found" message for builtin <cmd> to redirected fd 2.

sh: Fix some bugs with backquoted builtins:
- correctly handle error output in $(builtin 2>&1), clarify out1/out2 vs
output/errout in the code
- treat all builtins as regular builtins so errors do not abort the shell
and variable assignments do not persist
- respect the caller's INTOFF

Some bugs still exist:
- expansion errors may still abort the shell
- some side effects of expansions and builtins persist

sh(1): document ulimit -w (swapuse rlimit).

MFC after: 1 week

sh(1): Correct two places where "$@" lacked necessary quotes.

MFC after: 1 week

sh: Use PATH= assignment in type.

Example:
PATH=/var/empty; PATH=/bin type ls

sh: Allow command -pv and command -pV (lookup using _PATH_STDPATH).

sh: Ensure funcnest is decremented if there was an error in the function.

This will be important when things like 'command eval f' will be possible.
Currently, the funcnest = 0 assignment in RESET (called when returning to
the top level after an error in interactive mode) is really sufficient.

Fix memory leak when parsing backticks (``).

sh: arith: Return only 0 and 1 from && and ||.

This agrees with C, POSIX and other shells.

Add missing `void' for function without arguments.

Add missing `void' for function without arguments.

(S)LIST_HEAD_INITIALIZER takes a (S)LIST_HEAD as an argument.
Fix some wrong usages.
Note: this does not affect generated binaries as this argument is not used.

PR: 137213
Submitted by: Eygene Ryabinkin (initial version)
MFC after: 1 month

sh: Change varinit to use const better.

sh: Various warning fixes (from WARNS=6 NO_WERROR=1):
- const
- initializations to silence -Wuninitialized (it was safe anyway)
- remove nested extern declarations
- rename "index" locals to "idx"

Remove declaration of function that no longer exists.

Fix breakage introduced in last commit.

Improve ACL branding mismatch detection and reporting in some rare cases,
such as "setfacl -m ''".

sh: Do not run callers' exception handlers in subshells.

Reset the exception handler in the child to main's.

This avoids inappropriate double cleanups or shell duplication when the
exception is caught, such as 'fc' and future 'command eval' and 'command .'.

sh: Do not consider a tilde-prefix with expansions in it.

That is, do not do tilde expansion if any of the CTL* bytes (\201-\210), not
only CTLESC and CTLQUOTEMARK, are encountered. Such an expansion would look
up a user name with sh's internal representation.

The parser does not currently distinguish between backslashed and
unbackslashed \201-\210, so tilde expansion of user names with these bytes
in them is not so easy to fix.

sh: Add some __dead2 to indicate functions that do not return.

sh: Constify various strings.

Most of this is adding const keywords, but setvar() in var.c had to be
changed somewhat more.

sh: Remove setting variables from dotcmd/exportcmd.

It is already done by evalcommand(), unless special-ness has been removed,
in which case variable assignments should not persist. (These are currently
always special builtins, but this will change later: command builtin,
command substitution.)

This also fixes a memory leak when calling . with variable assignments.

Example:
valgrind --leak-check=full sh -c 'x=1 . /dev/null; x=2'

Plug a memory leak.

PR: bin/141835
Submitted by: Henning Petersen <henning.petersen t-online.de>
MFC after: 2 weeks

Let date(1) use utmpx instead of logwtmp().

utmpx also has OLD_TIME/NEW_TIME messages, which seem to be intended for
the same purpose as how we call logwtmp() here.

Fix some cases where file descriptors from redirections leak to programs.

- Redirecting fds that were not open before kept two copies of the
redirected file.
sh -c '{ :; } 7>/dev/null; fstat -p $$; true'
(both fd 7 and 10 remained open)
- File descriptors used to restore things after redirection were not
set close-on-exec, instead they were explicitly closed before executing
a program normally and before executing a shell procedure. The latter
must remain but the former is replaced by close-on-exec.
sh -c 'exec 7</; { exec fstat -p $$; } 7>/dev/null; true'
(fd 10 remained open)

The examples above are simpler than the testsuite because I do not want to
use fstat or procstat in the testsuite.

Fix various things about SIGINT handling:
* exception handlers are now run with interrupts disabled, which avoids
many race conditions
* fix some cases where SIGINT only aborts one command and continues the
script, in particular if a SIGINT causes an EINTR error which trumped the
interrupt.

Example:
sh -c 'echo < /some/fifo; echo This should not be printed'
The fifo should not have writers. When pressing ctrl+c to abort the open,
the shell used to continue with the next command.

Example:
sh -c '/bin/echo < /some/fifo; echo This should not be printed'
Similar. Note, however, that this particular case did not and does not work
in interactive mode with job control enabled.

sh: Ensure the same command input file is on top after executing a builtin.

This avoids weirdness when 'fc -e vi' or the like is done and there is a
syntax error in the file. Formerly an interactive shell tried to execute
stuff after the syntax error and exited.

This should also avoid similar issues with 'command eval' and 'command .'
when 'command' is implemented properly as in NetBSD sh.

Special builtins did not have this problem since errors in them cause the
shell to exit or to reset various state such as the current command input
file.

trap: do not consider a bad signal name a fatal error.
POSIX explicitly prescribes this.
Continue processing any other signals and return status 1.

Handle current work directories of arbitrary length. The argument to cd
continues to be limited by PATH_MAX (1024).

Obtained from: NetBSD
PR: 104456

sh: Some changes to stderr flushing:
* increase buffer size from 100 to 256 bytes
* remove implied flush from out2str(), in particular this avoids unnecessary
flushing in the middle of a -x tracing line
* rename dprintf() to out2fmt_flush(), make it flush out2 and use this
function in various places where flushing is desired after an error
message

Add pwait utility, which waits for any process to terminate.

This is similar to the Solaris utility of the same name.

Some use cases:
* rc.subr's wait_for_pids
* interactive use, e.g. to shut down the computer when some task is done
even if the task is already running

Discussed on: hackers@

Fix small resource leak (memory).

Reviewed by: gad
MFC after: 1 week

sh: Allow a newline before "in" in a for command, as required by POSIX.

sh: Use sigaction instead of signal/siginterrupt combination.

sh: Fix memory leak when using a variable in arithmetic like $((x)).

MFC after: 3 weeks

Increase width for %CPU, RSS and VSZ columns for now. Modern systems
tend to have larger memory, larger process, and more CPU.

Use our canonical .Dd format.

Submitted by: Ulrich Spoerlein

sh: Exempt $@ and $* from set -u

This seems more useful and will likely be in the next POSIX standard.

Also document more precisely in the man page what set -u does (note that
$@, $* and $! are the only special parameters that can ever be unset, all
the others are always set, although they may be empty).

Check error of dlfunc(3).

MFC after: 3 days

sh: Show more information about syntax errors in command substitution:
the line number where the command substitution started.
This applies to both the $() and `` forms but is most useful for ``
because the other line number is relative to the enclosed text there.
(For older versions, -v can be used as a workaround.)

Removed redundant WARNS setting.

Submitted by: Ulrich Spörlein

ls: Make -p not inhibit following symlinks.

According to the man page, when neither -H/-L nor -F/-d/-l are given, -H is
implied. This agrees with POSIX, GNU ls and Solaris ls. This means that -p,
although it is very similar to -F, does not prevent the implicit following
of symlinks.

PR: standards/128546

Clarify quoting of word in ${v=word} in sh(1).

sh: Send the "xyz: not found" message to redirected fd 2.
This also fixes that trying to execute a non-regular file with a command
name without '/' returns 127 instead of 126.
The fix is rather simplistic: treat CMDUNKNOWN as if the command were found
as an external program. The resulting fork is a bit wasteful but executing
unknown commands should not be very frequent.

PR: bin/137659