Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r238468: sh: Expand assignment-like words specially for
export/readonly/local.

Examples:
export x=~
now expands the tilde
local y=$1
is now safe, even if $1 contains IFS characters or metacharacters.

For a word to "look like an assignment", it must start with a name followed
by an equals sign, none of which may be quoted.

The special treatment applies when the first word (potentially after
"command") is "export", "readonly" or "local". There may be quoting
characters but no expansions. If "local" is overridden with a function there
is no special treatment ("export" and "readonly" cannot be overridden with a
function).

If things like
local arr=(1 2 3)
are ever allowed in the future, they cannot call a "local" function. This
would either be a run-time error or it would call the builtin.

This matches Austin Group bug #351, planned for the next issue of POSIX.1.

As for the MFC, it is easy to depend on this feature inadvertently, and
adding this fixes a regression from stable/8 that may be apparent in things
like
local x=${y+a @}.

PR: bin/166771
Relnotes: yes

MFC r245383,245392,247190,249220,251180,251797: New sh testcases.

These already work on stable/9.

MFC r226892,r228007,r228873,r230121,r232839: sh: Various testcases that
already work.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

sh: Reduce unnecessary forks with eval.

The eval special builtin now runs the code with EV_EXIT if it was run
with EV_EXIT itself.

In particular, this eliminates one fork when a command substitution contains
an eval command that ends with an external program or a subshell.

This is similar to what r220978 did for functions.

sh: Fix $? in heredocs on simple commands.

PR: bin/41410

sh: Add already working testcases for $? in here-document.

If the here-document is attached to a compound command or subshell, $?
already works properly. This is both a workaround for bin/41410 and a
requirement for a true fix for bin/41410.

PR: bin/41410
MFC after: 1 week

sh: Fix unquoted $@/$* if IFS=''.

If IFS is null, unquoted $@/$* should still expand to separate words.
This differs from quoted $@ (which does not depend on IFS) in that pathname
generation is performed and empty words are removed.

sh: Add UTF-8 support to pattern matching.

?, [...] patterns match codepoints instead of bytes. They do not match
invalid sequences. [...] patterns must not contain invalid sequences
otherwise they will not match anything. This is so that ${var#?} removes the
first codepoint, not the first byte, without putting UTF-8 knowledge into
the ${var#pattern} code. However, * continues to match any string and an
invalid sequence matches an identical invalid sequence. (This differs from
fnmatch(3).)

sh: Add UTF-8 support to ${#var}.

If the current locale uses UTF-8, ${#var} counts codepoints (more precisely,
bytes b with (b & 0xc0) != 0x80).

sh: Apply set -u to variables in arithmetic.

Note that this only applies to variables that are actually used.
Things like (0 && unsetvar) do not cause an error.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do not word split "${#parameter}".

This is only a problem if IFS contains digits, which is unusual but valid.

Because of an incorrect fix for PR bin/12137, "${#parameter}" was treated
as ${#parameter}. The underlying problem was that "${#parameter}"
erroneously added CTLESC bytes before determining the length. This
was properly fixed for PR bin/56147 but the incorrect fix was not backed
out.

Reported by: Seeker on forums.freebsd.org
MFC after: 2 weeks

sh: Add test for bin/12137.

sh: Add test for obscure and ambiguous ${#?}.

sh: Fix some parameter expansion variants ${#...}.

These already worked: $# ${#} ${##} ${#-} ${#?}
These now work as well: ${#+word} ${#-word} ${##word} ${#%word}

There is an ambiguity in the standard with ${#?}: it could be the length of
$? or it could be $# giving an error in the (impossible) case that it is not
set. We continue to use the former interpretation as it seems more useful.

sh: Add some tests for ${#parameter}.

sh: Unset some more locale vars in two tests that may cause them to break.

sh: Detect dividing the smallest integer by -1.

This overflows and on some architectures such as amd64 it generates SIGFPE.
Generate an error on all architectures.

sh: Add tests for new features in arithmetic.

sh: Avoid side effects from builtins in optimized command substitution.

Change the criterion for builtins to be safe to execute in the same process
in optimized command substitution from a blacklist of only cd, . and eval to
a whitelist.

This avoids clobbering the main shell environment such as by $(exit 4) and
$(set -x).

The builtins jobid, jobs, times and trap can still show information not
available in a child process; this is deliberately permitted. (Changing
traps is not.)

For some builtins, whether they are safe depends on the arguments passed to
them. Some of these are always considered unsafe to keep things simple; this
only harms efficiency a little in the rare case they are used alone in a
command substitution.

sh: Add two tests for special cases in command substitution that already
work in stable/8.

sh: Don't do optimized command substitution if expansions have side effects.

Before considering to execute a command substitution in the same process,
check if any of the expansions may have a side effect; if so, execute it in
a new process just like happens if it is not a single simple command.

Although the check happens at run time, it is a static check that does not
depend on current state. It is triggered by:
- expanding $! (which may cause the job to be remembered)
- ${var=value} default value assignment
- assignment operators in arithmetic
- parameter substitutions in arithmetic except ${#param}, $$, $# and $?
- command substitutions in arithmetic

This means that $((v+1)) does not prevent optimized command substitution,
whereas $(($v+1)) does, because $v might expand to something containing
assignment operators.

Scripts should not depend on these exact details for correctness. It is also
imaginable to have the shell fork if and when a side effect is encountered
or to create a new temporary namespace for variables.

Due to the $! change, the construct $(jobs $!) no longer works. The value of
$! should be stored in a variable outside command substitution first.

sh: Add test for optimized command substitution.

This test verifies that certain expansions without side effects do not
cause the command substitution to be executed in a child process.

This is not a correctness requirement, but it involves a nontrivial amount
of code and it would be unfortunate if it stopped working.

sh: Make expansion errors in optimized command substitution non-fatal.
Command substitutions consisting of a single simple command are executed in
the main shell process but this should be invisible apart from performance
and very few exceptions such as $(trap).

sh: Add a testcase for cmdsubst errors that already works properly.
If a command substitution consists of one special builtin and there is a
redirection error, this should not abort the outer shell.
It was fixed in r201366 by ignoring special builtin properties for command
substitutions consisting of one builtin.

Restore two commented-out tests from plus-minus1.0 to a new file.

These two cases pass on -CURRENT but fail on stable/8.

Reviewed by: jilles

Remove commented-out test that's covered in plus-minus2.0 anyway.

Discussed with: jilles

sh: arith: Disallow decimal constants starting with 0 (containing 8 or 9).

Constants in arithmetic starting with 0 should be octal only.

This avoids the following highly puzzling result:
$ echo $((018-017))
3
by making it an error instead.

sh: Fix corruption of command substitutions with special chars after newline

The CTLESC byte to protect a special character was output before instead of
after a newline directly preceding the special character.

The special handling of newlines is because command substitutions discard
all trailing newlines.

sh: Add a test for r216387 (long arithmetic expression in here document).

sh: Add a test that manipulates various long strings.

It is quite effective at detecting mistakes in memalloc.c and code using it.

It is somewhat slow, but some of the patches in my queue improve it.

sh: Fix some issues with CTL* bytes and ${var#pat}.

subevalvar() incorrectly assumed that CTLESC bytes were present iff the
expansion was quoted. However, they are present iff various processing such
as word splitting is to be done later on.

Example:
v=@$e@$e@$e@
y="${v##*"$e"}"
echo "$y"
failed if $e contained the magic CTLESC byte.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do IFS splitting on word in ${v+word} and ${v-word}.

The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.

Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.

Example:
for i in ${$+a b c}; do echo $i; done

Exp-run done by: pav (with some other sh(1) changes)

sh: Only accept a '}' inside ${v+-=?...} if double-quote state matches.
If double-quote state does not match, treat the '}' literally.

This ensures double-quote state remains the same before and after a
${v+-=?...} which helps with expand.c.

It makes things like
${foo+"\${bar}"}
which I have seen in the wild work as expected.

Exp-run done by: pav (with some other sh(1) changes)

sh: Make double-quotes quote a '}' inside ${v#...} and ${v%...}.

Exp-run done by: pav (with some other sh(1) changes)
PR: bin/57554

Do not assume in growstackstr() that a "precious" character will be
immediately written into the stack after the call. Instead let the caller
manage the "space left".

Previously, growstackstr()'s assumption causes problems with STACKSTRNUL()
where we want to be able to turn a stack into a C string, and later
pretend the NUL is not there.

This fixes a bug in STACKSTRNUL() (that grew the stack) where:
1. STADJUST() called after a STACKSTRNUL() results in an improper adjust.
This can be seen in ${var%pattern} and ${var%%pattern} evaluation.
2. Memory leak in STPUTC() called after a STACKSTRNUL().

Reviewed by: jilles

Allow one to regression test 'sh' changes without having to install
a potentially bad /bin/sh first.

sh: Remove remnants of '!!' to negate pattern.

This Almquist extension was disabled long ago.

In pathname generation, components starting with '!!' were treated as
containing wildcards, causing unnecessary readdir (which could fail, causing
pathname generation to fail while it should not).

sh: Test that all bytes from 128 to 255 can be used in IFS.

To avoid multibyte issues, this test forces ISO8859-1 charset.

This also passes on stable/8.

sh: Test that all bytes from 1 to 127 can be used in IFS.

This also passes on stable/8.

sh: Fix heap-based buffer overflow in pathname generation.

The buffer for generated pathnames could be too small in some cases. It
happened to be always at least PATH_MAX long, so there was never an overflow
if the resulting pathnames would be usable.

This bug may be abused if a script subjects input from an untrusted source
to pathname generation, which a bad idea anyhow. Most shell scripts do not
work on untrusted data. secteam@ says no advisory is necessary.

PR: bin/148733
Reported by: Changming Sun snnn119 at gmail com
MFC after: 10 days

sh: Add more testcases for ${var:-word}.

Whether POSIX requires these is unclear.

They pass with 8-stable sh as well.

sh: Remove comment that the comma operator is missing in arithmetic
expansion.

The comma operator is not listed in POSIX.1-2008 XCU 1.1.2.1 Arithmetic
Precision and Operations (referenced by XCU 2.6.4 Arithmetic Expansion) and
is therefore not required.

sh: Fix pathname expansion with quoted slashes like *\/.

These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.

Because this is the first code I'm importing from dash to expand.c, add the
Herbert Xu copyright notice which is in dash's expand.c.

When pathname expanding *\/, the CTLESC representing the quoted state was
erroneously taken as part of the * pathname component. This CTLESC was then
seen by the pattern matching code as escaping the '\0' terminating the
string.

The code is slightly different because dash converts the CTLESC characters
to backslashes and removes all the other CTL* characters to allow
substituting glob(3).

The effect of the bug was also slightly different from dash (where nothing
matched at all). Because a CTLESC can escape a '\0' in some way, whether
files were included despite the bug depended on memory that should not be
read. In particular, on many machines /*\/ expanded to a strict subset of
what /*/ expanded to.

Example:
echo /*"/null"

This should print /dev/null, not /*/null.

PR: bin/146378
Obtained from: dash

sh: Add some simple testcases for pathname expansion.

sh: Add some more tests for ${v#...} and ${v%...}.

These pass on stable/8 as well.

sh: Add testcases for double-quotes within quoted ${var+-...} (non-POSIX).

POSIX leaves things like "${var+"word"}" undefined.
We follow traditional ash behaviour here.
Hence, these testcases also work on stable/8.

sh: Add test for nested arithmetic substitution.

Pre-r206145 sh changes nested arithmetic substitution to parentheses, which
is not always correct, as exercised by this test.

sh: Add test for some associativity in arithmetic.

sh: Do tilde expansion in substitutions.

This applies to word in ${v-word}, ${v+word}, ${v=word}, ${v?word} (which
inherits quoting from the outside) and in ${v%word}, ${v%%word}, ${v#word},
${v##word} (which does not inherit any quoting).

In all cases tilde expansion is only attempted at the start of word, even if
word contains spaces. This agrees with POSIX and other shells.

This is the last part of the patch tested in the exp-run.

Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix duplicate variable name in test.

sh: Add some testcases for the working parts of tilde expansion.

sh: Allow quoting pattern match characters in ${v%pat} and ${v#pat}.

Note that this depends on r206145 for allowing pattern match characters to
have their special meaning inside a double-quoted expansion like "${v%pat}".

PR: bin/117748
Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix various things about expansions:
* remove the backslash from \} inside double quotes inside +-=?
substitutions, e.g. "${$+\}a}"
* maintain separate double-quote state for ${v#...} and ${v%...};
single and double quotes are special inside, even in a double-quoted
string or here document
* keep track of correct order of substitutions and arithmetic

This is different from dash's approach, which does not track individual
double quotes in the parser, trying to fix this up during expansion.
This treats single quotes inside "${v#...}" incorrectly, however.

This is similar to NetBSD's approach (as submitted in PR bin/57554), but
recognizes the difference between +-=? and #% substitutions hinted at in
POSIX and is more refined for arithmetic expansion and here documents.

PR: bin/57554
Exp-run done by: erwin (with some other sh(1) changes)

sh: Add some testcases for the working parts of ${v%...} and ${v#...}.

sh: Add tests for " and $ inside `.

sh: Add some testcases for ${v=w}, ${v-w}, ${v+w}.

These expansions, which were already in the Bourne shell, work correctly for
the most part. The testcases are only about the parts that already work
correctly.

sh: arith: Add a test for a bug in the dash arith code,
which I plan to import at some point.
Our current code handles it fine and it should stay that way.

sh: arith: Test an octal constant as well.

sh: Add a regression test that tries out all arithmetic ops.

MFC after: 1 week

sh: Fix some bugs with backquoted builtins:
- correctly handle error output in $(builtin 2>&1), clarify out1/out2 vs
output/errout in the code
- treat all builtins as regular builtins so errors do not abort the shell
and variable assignments do not persist
- respect the caller's INTOFF

Some bugs still exist:
- expansion errors may still abort the shell
- some side effects of expansions and builtins persist

sh: arith: Return only 0 and 1 from && and ||.

This agrees with C, POSIX and other shells.

sh: Exempt $@ and $* from set -u

This seems more useful and will likely be in the next POSIX standard.

Also document more precisely in the man page what set -u does (note that
$@, $* and $! are the only special parameters that can ever be unset, all
the others are always set, although they may be empty).

Add some tests for ${var?} and set -u.

Add some tests for r194975 and r194977.

Approved by: ed (mentor) (implicit)