Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r245383,245392,247190,249220,251180,251797: New sh testcases.

These already work on stable/9.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

sh: Add test for r223282.

sh: Do parameter expansion before printing PS4 (set -x).

The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.

All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).

Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).

Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.

sh: Add simple tests for set -x and PS4.

sh: Honour -n while processing -c string.

sh: Add tests for -n flag. These already pass.

sh: Set $? to 0 for background commands.

For backgrounded pipelines and subshells, the previous value of $? was being
preserved, which is incorrect.

For backgrounded simple commands containing a command substitution, the
status of the last command substitution was returned instead of 0.

If fork() fails, this is an error.

sh: Allow EV_EXIT through function calls, make {...} <redir more consistent.

If EV_EXIT causes an exit, use the exception mechanism to unwind
redirections and local variables. This way, if the final command is a
redirected command, an EXIT trap now executes without the redirections.

Because of these changes, EV_EXIT can now be inherited by the body of a
function, so do so. This means that a function no longer prevents a fork
before an exec being skipped, such as in
f() { head -1 /etc/passwd; }; echo $(f)

Wrapping a single builtin in a function may still cause an otherwise
unnecessary fork with command substitution, however.

An exit command or -e failure still invokes the EXIT trap with the
original redirections and local variables in place.

Note: this depends on SHELLPROC being gone. A SHELLPROC depended on
keeping the redirections and local variables and only cleaning up the
state to restore them.

sh: Make execution/fork1.0 work even if the basename of ${SH} is not "sh".

sh: Fix two things about {(...)} <redir:

* In {(...) <redir1;} <redir2, do not drop redir1.
* Maintain the difference between (...) <redir and {(...)} <redir:
In (...) <redir, the redirection is performed in the child, while in
{(...)} <redir it should be performed in the parent (like {(...); :;}
<redir)

sh: Forget all cached command locations on any PATH change.

POSIX requires this and it is simpler than the previous code that remembered
command locations when appending directories to PATH.

In particular,
PATH=$PATH
is no longer a no-op but discards all cached command locations.

sh: Add test for shell script without '#!'.

sh: Send messages about signals to stderr.

This is required by POSIX and seems to make more sense.

See also r217557.

sh: Fix signal messages being sent to the wrong file sometimes.

When a foreground job exits on a signal, a message is printed to stdout
about this. The buffer was not flushed after this which could result in the
message being written to the wrong file if the next command was a builtin
and had stdout redirected.

Example:
sh -c 'kill -9 $$'; : > foo; echo FOO:; cat foo

Reported by: gcooper
MFC after: 1 week

sh: Remove special %builtin PATH entry.
All builtins are now always found before a PATH search.

Most ash derivatives have an undocumented feature where the presence of an
entry "%builtin" in $PATH will cause builtins to be checked at that point of
the PATH search, rather than before looking at any directories as documented
in the man page (very old versions do document this feature).

I am removing this feature from sh, as it complicates the code, may violate
expectations (for example, /usr/bin/alias is very close to a forkbomb with
PATH=/usr/bin:%builtin, only /usr/bin/builtin not being another link saves
it) and appears to be unused (all the %builtin google code search finds is
in some sort of ash source code).

Note that aliases and functions took and take precedence above builtins.
Because aliases work on a lexical level they can only ever be overridden on
a lexical level (quoting or preceding 'builtin' or 'command'). Allowing
override of functions via PATH does not really fit in the model of sh and it
would work differently from %builtin if implemented.

Note: POSIX says special builtins are found before functions. We comply to
this because we do not allow functions with the same name as a special
builtin.

Silence from: freebsd-hackers@ (message sent 20101225)
Discussed with: dougb

sh: Fix some tests that used sh instead of ${SH}
so they tested the wrong sh.

This was caused because these tests were committed after the sh -> ${SH}
change but were created before.

sh: Do not allow overriding a special builtin with a function.
This is a syntax error.

POSIX does not say explicitly whether defining a function with the same name
as a special builtin is allowed, but it does say that it is impossible to
call such a function.

A special builtin can still be overridden with an alias.

This commit is part of a set of changes that will ensure that when
something looks like a special builtin to the parser, it is one. (Not the
other way around, as it remains possible to call a special builtin named
by a variable or other substitution.)

Exp-run done by: pav (with some other sh(1) changes)

sh: Check whether dup2 was successful for >&FD and <&FD.

A failure (usually caused by FD not being open) is a redirection error.

Exp-run done by: pav (with some other sh(1) changes)

sh: Add a test trying to close a descriptor that is not open.

In stable/8 and older, this fails. Some of the redirection changes in head
have fixed it.

Allow one to regression test 'sh' changes without having to install
a potentially bad /bin/sh first.

sh: Apply variable assignments left-to-right in bltinlookup().

Example:
HOME=foo HOME=bar cd

sh: Reduce unnecessary testsuite failures with other shells.

sh: Get rid of unnecessary non-standard empty lists.

POSIX does not allow constructs like:
if cmd; then fi
{ }
Add a colon dummy command, except in a test that verifies that such empty
lists do not cause crashes when used as a function definition.

Fix some cases where file descriptors from redirections leak to programs.

- Redirecting fds that were not open before kept two copies of the
redirected file.
sh -c '{ :; } 7>/dev/null; fstat -p $$; true'
(both fd 7 and 10 remained open)
- File descriptors used to restore things after redirection were not
set close-on-exec, instead they were explicitly closed before executing
a program normally and before executing a shell procedure. The latter
must remain but the former is replaced by close-on-exec.
sh -c 'exec 7</; { exec fstat -p $$; } 7>/dev/null; true'
(fd 10 remained open)

The examples above are simpler than the testsuite because I do not want to
use fstat or procstat in the testsuite.

sh: Send the "xyz: not found" message to redirected fd 2.
This also fixes that trying to execute a non-regular file with a command
name without '/' returns 127 instead of 126.
The fix is rather simplistic: treat CMDUNKNOWN as if the command were found
as an external program. The resulting fork is a bit wasteful but executing
unknown commands should not be very frequent.

PR: bin/137659

sh: Fix crash with empty functions (f() { }) introduced in r196483

Empty pairs of braces are represented by a NULL node pointer, just like
empty lines at the top level.

Support for empty pairs of braces may be removed later. They make the code
more complex, have inconsistent behaviour (may or may not change $?), are
not specified by POSIX and are not allowed by some other shells like bash,
dash and ksh93.

Reported by: kan

sh: Fix crash when undefining or redefining a currently executing function.

Add a reference count to function definitions.
Memory may leak if multiple SIGINTs arrive in interactive mode,
this will be fixed later by changing SIGINT handling.

PR: bin/137640

Add tests for r194774.

Approved by: ed (mentor) (implicit)

Add tests for r194127 and r194128.

Approved by: ed (mentor)