Remove svn:mergeinfo carried over from stable/9.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r238468: sh: Expand assignment-like words specially for
export/readonly/local.

Examples:
export x=~
now expands the tilde
local y=$1
is now safe, even if $1 contains IFS characters or metacharacters.

For a word to "look like an assignment", it must start with a name followed
by an equals sign, none of which may be quoted.

The special treatment applies when the first word (potentially after
"command") is "export", "readonly" or "local". There may be quoting
characters but no expansions. If "local" is overridden with a function there
is no special treatment ("export" and "readonly" cannot be overridden with a
function).

If things like
local arr=(1 2 3)
are ever allowed in the future, they cannot call a "local" function. This
would either be a run-time error or it would call the builtin.

This matches Austin Group bug #351, planned for the next issue of POSIX.1.

As for the MFC, it is easy to depend on this feature inadvertently, and
adding this fixes a regression from stable/8 that may be apparent in things
like
local x=${y+a @}.

PR: bin/166771
Relnotes: yes

MFC r254335: sh: Allow a lone redirection before '|', ';;' or ';&'.

Example: </dev/null | :

PR: 181240

MFC r248349: sh: Recognize "--" and explicitly reject options in wait
builtin.

If syntactically invalid job identifiers are to be taken as jobs that exited
with status 127, this should not apply to options, so that we can add
options later if need be.

MFC r245383,245392,247190,249220,251180,251797: New sh testcases.

These already work on stable/9.

MFC r245382: sh: Fix crash when parsing '{ } &'.

MFC r242767,r243252: sh: Add tests for modifying an alias (r242766/r243402).

MFC r226892,r228007,r228873,r230121,r232839: sh: Various testcases that
already work.

MFC r228109: test: Add more testcases.

The new testcases pass even on old stable/7, but some other implementations
manage to get them wrong.

Also remove a few duplicate testcases.

MFC r229742: sh: Avoid possible echo options in a testcase.

MFC r230212:
sh: Fix execution of multiple statements in a trap when evalskip is set

Before this fix, only the first statement of the trap was executed if
evalskip was set. This is for example the case when:
o "-e" is set for this shell
o a trap is set on EXIT
o a function returns 1 and causes the script to abort

Reviewed by: jilles
Sponsored by: Yakaz (http://www.yakaz.com)

MFC r230211:
sh: Test EXIT trap with multiple statements in it

Reviewed by: jilles

MFC r230095: sh: Properly show "Not a directory" error in cd builtin.

The errno message display added in r222292 did not take attempting to
cd to a non-directory or something that cannot be stat()ed into account.

PR: bin/164070

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

sh: Add a test for a fairly obscure case with aliases.

This also passes on stable/8.

sh: Test that '!' is literal if quoted and first char of bracket expression

This also works on stable/8.

sh: Add test for r223282.

sh: Add case statement fallthrough (with ';&' instead of ';;').

Replacing ;; with the new control operator ;& will cause the next list to be
executed as well without checking its pattern, continuing until a list ends
with ;; or until the end of the case statement. This is like omitting
"break" in a C "switch" statement.

The sequence ;& was formerly invalid.

This feature is proposed for the next POSIX issue in Austin Group issue
#449.

sh: Skip variables with invalid names in "set", "export -p", "readonly -p".

This ensures the output of these commands is valid shell input.

sh: Reduce unnecessary forks with eval.

The eval special builtin now runs the code with EV_EXIT if it was run
with EV_EXIT itself.

In particular, this eliminates one fork when a command substitution contains
an eval command that ends with an external program or a subshell.

This is similar to what r220978 did for functions.

sh: Add support for named character classes in bracket expressions.

Example:
case x in [[:alpha:]]) echo yes ;; esac

sh: Add test for LC_COLLATE-based character ranges in case.

sh: Add test for case pattern matching with iso-8859-1 charset.

This also passes on stable/8.

sh: Do parameter expansion on ENV before using it.

This is required by POSIX, and allows things like ENV=\$HOME/.shrc.

Note that tilde expansion is explicitly not performed.

sh: Do parameter expansion before printing PS4 (set -x).

The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.

All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).

Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).

Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.

sh: Add simple tests for set -x and PS4.

etire the cpumask_t type and replace it with cpuset_t usage.

This is intended to fix the bug where cpu mask objects are
capped to 32. MAXCPU, then, can now arbitrarely bumped to whatever
value. Anyway, as long as several structures in the kernel are
statically allocated and sized as MAXCPU, it is suggested to keep it
as low as possible for the time being.

Technical notes on this commit itself:
- More functions to handle with cpuset_t objects are introduced.
The most notable are cpusetobj_ffs() (which calculates a ffs(3)
for a cpuset_t object), cpusetobj_strprint() (which prepares a string
representing a cpuset_t object) and cpusetobj_strscan() (which
creates a valid cpuset_t starting from a string representation).
- pc_cpumask and pc_other_cpus are target to be removed soon.
With the moving from cpumask_t to cpuset_t they are now inefficient
and not really useful. Anyway, for the time being, please note that
access to pcpu datas is protected by sched_pin() in order to avoid
migrating the CPU while reading more than one (possible) word
- Please note that size of cpuset_t objects may differ between kernel
and userland. While this is not directly related to the patch itself,
it is good to understand that concept and possibly use the patch
as a reference on how to deal with cpuset_t objects in userland, when
accessing kernland members.
- KTR_CPUMASK is changed and now is represented through a string, to be
set as the example reported in NOTES.

Please additively note that no MAXCPU is bumped in this patch, but
private testing has been done until to MAXCPU=128 on a real 8x8x2(htt)
machine (amd64).

Please note that the FreeBSD version is not yet bumped because of
the upcoming pcpu changes. However, note that this patch is not
targeted for MFC.

People to thank for the time spent on this patch:
- sbruno, pluknet and Nicholas Esborn (nick AT desert DOT net) tested
several revision of the patches and really helped in improving
stability of this work.
- marius fixed several bugs in the sparc64 implementation and reviewed
patches related to ktr.
- jeff and jhb discussed the basic approach followed.
- kib and marcel made targeted review on some specific part of the
patch.
- marius, art, nwhitehorn and andreast reviewed MD specific part of
the patch.
- marius, andreast, gonzo, nwhitehorn and jceel tested MD specific
implementations of the patch.
- Other people have made contributions on other patches that have been
already committed and have been listed separately.

Companies that should be mentioned for having participated at several
degrees:
- Yahoo! for having offered the machines used for testing on big
count of CPUs.
- The FreeBSD Foundation for having sponsored my devsummit attendance,
which has been instrumental.
- Sandvine for having offered offices and infrastructure during
development.

(I really hope I didn't forget anyone, if it happened I apologize in
advance).

sh: Fix $? in heredocs on simple commands.

PR: bin/41410

sh: Add already working testcases for $? in here-document.

If the here-document is attached to a compound command or subshell, $?
already works properly. This is both a workaround for bin/41410 and a
requirement for a true fix for bin/41410.

PR: bin/41410
MFC after: 1 week

sh: Reduce more needless differences between error messages.

sh: Honour -n while processing -c string.

sh: Add tests for -n flag. These already pass.

sh: Add tests for some somewhat obscure aspects of function definitions.

sh: Add test for 'set +o'.

sh: Correct criterion for using CDPATH in cd.

CDPATH should be ignored not only for pathnames starting with '/' but also
for pathnames whose first component is '.' or '..'.

The man page already describes this behaviour.

sh: Add simple CDPATH test.

sh: Fix unquoted $@/$* if IFS=''.

If IFS is null, unquoted $@/$* should still expand to separate words.
This differs from quoted $@ (which does not depend on IFS) in that pathname
generation is performed and empty words are removed.

sh: Add test for r222173.

sh: Expand aliases after assignments and redirections.

sh: Add test for positional parameters with more than one digit.

This also passes on stable/8.

sh: Implement the cd -e flag proposed for the next POSIX issue.

This reflects failure to determine the pathname of the new directory in the
exit status (1). Normally, cd returns successfully if it did chdir() and the
call was successful.

In POSIX, -e only has meaning with -P; because our -L is not entirely
compliant and may fall back to -P mode, -e has some effect with -L as well.

sh: Allow terminating a heredoc with a terminator at EOF without a newline.

This is sometimes used with eval or old-style command substitution, and most
shells other than ash derivatives allow it.

It can also be used with scripts that violate POSIX's requirement on the
application that they end in a newline (scripts must be text files except
that line length is unlimited).

Example:
v=`cat <<EOF
foo
EOF`
echo $v

This commit does not add support for the similar construct with new-style
command substitution, like
v=$(cat <<EOF
foo
EOF)
This continues to require a newline after the terminator.

sh: Add tests for lines that look like heredoc delimiters but are not.

sh: Add \u/\U support (in $'...') for UTF-8.

Because we have no iconv in base, support for other charsets is not
possible.

Note that \u/\U are processed using the locale that was active when the
shell started. This is necessary to avoid behaviour that depends on the
parse/execute split (for example when placing braces around an entire
script). Therefore, UTF-8 encoding is implemented manually.

sh: Add UTF-8 support to pattern matching.

?, [...] patterns match codepoints instead of bytes. They do not match
invalid sequences. [...] patterns must not contain invalid sequences
otherwise they will not match anything. This is so that ${var#?} removes the
first codepoint, not the first byte, without putting UTF-8 knowledge into
the ${var#pattern} code. However, * continues to match any string and an
invalid sequence matches an identical invalid sequence. (This differs from
fnmatch(3).)

sh: Add UTF-8 support to ${#var}.

If the current locale uses UTF-8, ${#var} counts codepoints (more precisely,
bytes b with (b & 0xc0) != 0x80).

sh: Add $'quoting' (C-style escape sequences).

A string between $' and ' may contain backslash escape sequences similar to
the ones in a C string constant (except that a single-quote must be escaped
and a double-quote need not be). Details are in the sh(1) man page.

This construct is useful to include unprintable characters, tabs and
newlines in strings; while this can be done with a command substitution
containing a printf command, that needs ugly workarounds if the result is to
end with a newline as command substitution removes all trailing newlines.

The construct may also be useful in future to describe unprintable
characters without needing to write those characters themselves in 'set -x',
'export -p' and the like.

The implementation attempts to comply to the proposal for the next issue of
the POSIX specification. Because this construct is not in POSIX.1-2008,
using it in scripts intended to be portable is unwise.

Matching the minimal locale support in the rest of sh, the \u and \U
sequences are currently not useful.

Exp-run done by: pav (with some other sh(1) changes)

sh: Apply set -u to variables in arithmetic.

Note that this only applies to variables that are actually used.
Things like (0 && unsetvar) do not cause an error.

Exp-run done by: pav (with some other sh(1) changes)

sh: Detect an error for ${#var<GARBAGE>}.

In particular, this makes things like ${#foo[0]} and ${#foo[@]} errors
rather than silent equivalents of ${#foo}.

PR: bin/151720
Submitted by: Mark Johnston
Exp-run done by: pav (with some other sh(1) changes)

sh: Set $? to 0 for background commands.

For backgrounded pipelines and subshells, the previous value of $? was being
preserved, which is incorrect.

For backgrounded simple commands containing a command substitution, the
status of the last command substitution was returned instead of 0.

If fork() fails, this is an error.

sh: Allow EV_EXIT through function calls, make {...} <redir more consistent.

If EV_EXIT causes an exit, use the exception mechanism to unwind
redirections and local variables. This way, if the final command is a
redirected command, an EXIT trap now executes without the redirections.

Because of these changes, EV_EXIT can now be inherited by the body of a
function, so do so. This means that a function no longer prevents a fork
before an exec being skipped, such as in
f() { head -1 /etc/passwd; }; echo $(f)

Wrapping a single builtin in a function may still cause an otherwise
unnecessary fork with command substitution, however.

An exit command or -e failure still invokes the EXIT trap with the
original redirections and local variables in place.

Note: this depends on SHELLPROC being gone. A SHELLPROC depended on
keeping the redirections and local variables and only cleaning up the
state to restore them.

sh: Do not word split "${#parameter}".

This is only a problem if IFS contains digits, which is unusual but valid.

Because of an incorrect fix for PR bin/12137, "${#parameter}" was treated
as ${#parameter}. The underlying problem was that "${#parameter}"
erroneously added CTLESC bytes before determining the length. This
was properly fixed for PR bin/56147 but the incorrect fix was not backed
out.

Reported by: Seeker on forums.freebsd.org
MFC after: 2 weeks

sh: Add test for bin/12137.

sh: Add test for obscure and ambiguous ${#?}.

sh: Add test for bin/56147.

sh: Fix some parameter expansion variants ${#...}.

These already worked: $# ${#} ${##} ${#-} ${#?}
These now work as well: ${#+word} ${#-word} ${##word} ${#%word}

There is an ambiguity in the standard with ${#?}: it could be the length of
$? or it could be $# giving an error in the (impossible) case that it is not
set. We continue to use the former interpretation as it seems more useful.

sh: Add some tests for ${#parameter}.

sh: Test that . /dev/null returns exit status 0 and does not preserve $?.

Preserving $? may cause problems particularly if set -e is in effect.

It may be useful to preserve the old value of $? in the dot script but this
must not be implemented in such a way that it would break this test.

Accept == as an alias of = which is a popular GNU extension.

This is intentionally undocumented for now since it's not part
of any standard.

MFC after: 1 month

sh: Add some tests for omitting whitespace whereever possible.

sh: Split off some special behaviour into separate tests.

This allows some other shells to pass the tests for basic behaviour.

sh: Do not use "local" in the test runner as POSIX and ksh93 do not have it.

sh: Make execution/fork1.0 work even if the basename of ${SH} is not "sh".

sh: Test that the read builtin passes through all byte values
except NUL, newline and backslash.

This also passes on stable/8.

sh: Unset some more locale vars in two tests that may cause them to break.

sh: Detect dividing the smallest integer by -1.

This overflows and on some architectures such as amd64 it generates SIGFPE.
Generate an error on all architectures.

sh: Add tests for new features in arithmetic.

sh: Weaken some tests to allow /rescue/sh to pass everything.

/rescue/sh has a different _PATH_STDPATH which affects command -p.

sh: Fix two things about {(...)} <redir:

* In {(...) <redir1;} <redir2, do not drop redir1.
* Maintain the difference between (...) <redir and {(...)} <redir:
In (...) <redir, the redirection is performed in the child, while in
{(...)} <redir it should be performed in the parent (like {(...); :;}
<redir)

sh: Forget all cached command locations on any PATH change.

POSIX requires this and it is simpler than the previous code that remembered
command locations when appending directories to PATH.

In particular,
PATH=$PATH
is no longer a no-op but discards all cached command locations.

sh: Do not try to execute binary files as scripts.

If execve() returns an [ENOEXEC] error, check if the file is binary before
trying to execute it using sh. A file is considered binary if at least one
of the first 256 bytes is '\0'.

In particular, trying to execute ELF binaries for the wrong architecture now
fails with an "Exec format error" message instead of syntax errors and
potentially strange results.

sh: Add test for shell script without '#!'.

sh: Send messages about signals to stderr.

This is required by POSIX and seems to make more sense.

See also r217557.

sh: Add test for EXIT trap in command substitution.

This is not really realistic but is an opposition to $(trap).

sh: Fix signal messages being sent to the wrong file sometimes.

When a foreground job exits on a signal, a message is printed to stdout
about this. The buffer was not flushed after this which could result in the
message being written to the wrong file if the next command was a builtin
and had stdout redirected.

Example:
sh -c 'kill -9 $$'; : > foo; echo FOO:; cat foo

Reported by: gcooper
MFC after: 1 week

sh: If exit is used without args from a trap action, exit on the signal.

This is useful so that it is easier to exit on a signal than to reset the
trap to default and resend the signal. It matches ksh93. POSIX says that
'exit' without args from a trap action uses the exit status from the last
command before the trap, which is different from 'exit $?' and matches this
if the previous command is assumed to have exited on the signal.

If the signal is SIGSTOP, SIGTSTP, SIGTTIN or SIGTTOU, or if the default
action for the signal is to ignore it, a normal _exit(2) is done with exit
status 128+signal_number.

sh: Fix some things about -- in trap:
* Make 'trap --' do the same as 'trap' instead of nothing.
* Make '--' stop option processing (note that '-' action is not an option).

Side effect: The error message for an unknown option is different.

regression/date: unset all LC_xxx vars and set LANG/LC_ALL

When running with a custom locale setup, it's easy to confuse the
date regression tests and cause them to fail, e.g. when LANG='C'
but LC_ALL='el_GR.UTF-8'. Set LC_ALL to 'C', which overrides all
other LC_xxx options, to avoid this sort of problem.

Reviewed by: uqs, edwin

sh: Remove special %builtin PATH entry.
All builtins are now always found before a PATH search.

Most ash derivatives have an undocumented feature where the presence of an
entry "%builtin" in $PATH will cause builtins to be checked at that point of
the PATH search, rather than before looking at any directories as documented
in the man page (very old versions do document this feature).

I am removing this feature from sh, as it complicates the code, may violate
expectations (for example, /usr/bin/alias is very close to a forkbomb with
PATH=/usr/bin:%builtin, only /usr/bin/builtin not being another link saves
it) and appears to be unused (all the %builtin google code search finds is
in some sort of ash source code).

Note that aliases and functions took and take precedence above builtins.
Because aliases work on a lexical level they can only ever be overridden on
a lexical level (quoting or preceding 'builtin' or 'command'). Allowing
override of functions via PATH does not really fit in the model of sh and it
would work differently from %builtin if implemented.

Note: POSIX says special builtins are found before functions. We comply to
this because we do not allow functions with the same name as a special
builtin.

Silence from: freebsd-hackers@ (message sent 20101225)
Discussed with: dougb

sh: Make exit without parameters from EXIT trap POSIX-compliant.

It should use the original exit status, just like falling off the
end of the trap handler.

Outside an EXIT trap, 'exit' is still equivalent to 'exit $?'.

sh: Add simple test for 'exit' without parameters.

sh: Do not call exitshell() from evalcommand() unless evalcommand() forked
itself.

This ensures that certain traps caused by builtins are executed.

sh: Test that exit $? replaces the original exit status in an EXIT trap.

sh: Check readonly status for assignments on regular builtins.

An error message is written, the builtin is not executed, nonzero exit
status is returned but the shell does not abort.

This was already checked for special builtins and external commands, with
the same consequences except that the shell aborts for special builtins.

Obtained from: NetBSD

sh: Check if dup2 for redirection from/to a file succeeds.

A failure (e.g. caused by ulimit -n being set very low) is a redirection
error.

Example:
ulimit -n 9; exec 9<.

sh: Avoid side effects from builtins in optimized command substitution.

Change the criterion for builtins to be safe to execute in the same process
in optimized command substitution from a blacklist of only cd, . and eval to
a whitelist.

This avoids clobbering the main shell environment such as by $(exit 4) and
$(set -x).

The builtins jobid, jobs, times and trap can still show information not
available in a child process; this is deliberately permitted. (Changing
traps is not.)

For some builtins, whether they are safe depends on the arguments passed to
them. Some of these are always considered unsafe to keep things simple; this
only harms efficiency a little in the rare case they are used alone in a
command substitution.

sh: Add two tests for special cases in command substitution that already
work in stable/8.

sh: Don't do optimized command substitution if expansions have side effects.

Before considering to execute a command substitution in the same process,
check if any of the expansions may have a side effect; if so, execute it in
a new process just like happens if it is not a single simple command.

Although the check happens at run time, it is a static check that does not
depend on current state. It is triggered by:
- expanding $! (which may cause the job to be remembered)
- ${var=value} default value assignment
- assignment operators in arithmetic
- parameter substitutions in arithmetic except ${#param}, $$, $# and $?
- command substitutions in arithmetic

This means that $((v+1)) does not prevent optimized command substitution,
whereas $(($v+1)) does, because $v might expand to something containing
assignment operators.

Scripts should not depend on these exact details for correctness. It is also
imaginable to have the shell fork if and when a side effect is encountered
or to create a new temporary namespace for variables.

Due to the $! change, the construct $(jobs $!) no longer works. The value of
$! should be stored in a variable outside command substitution first.

sh: Add test for optimized command substitution.

This test verifies that certain expansions without side effects do not
cause the command substitution to be executed in a child process.

This is not a correctness requirement, but it involves a nontrivial amount
of code and it would be unfortunate if it stopped working.

sh: Make expansion errors in optimized command substitution non-fatal.
Command substitutions consisting of a single simple command are executed in
the main shell process but this should be invisible apart from performance
and very few exceptions such as $(trap).

sh: Add a testcase for cmdsubst errors that already works properly.
If a command substitution consists of one special builtin and there is a
redirection error, this should not abort the outer shell.
It was fixed in r201366 by ignoring special builtin properties for command
substitutions consisting of one builtin.

Restore two commented-out tests from plus-minus1.0 to a new file.

These two cases pass on -CURRENT but fail on stable/8.

Reviewed by: jilles

Remove commented-out test that's covered in plus-minus2.0 anyway.

Discussed with: jilles

sh: Make warnings in the printf builtin non-fatal, like in the program.

The #define for warnx now behaves much like the libc function (except that
it uses sh command name and output).

Also, it now uses C99 __VA_ARGS__ so there is no need for three different
macros for 0, 1 or 2 parameters.

sh: arith: Disallow decimal constants starting with 0 (containing 8 or 9).

Constants in arithmetic starting with 0 should be octal only.

This avoids the following highly puzzling result:
$ echo $((018-017))
3
by making it an error instead.

sh: Fix corruption of command substitutions with special chars after newline

The CTLESC byte to protect a special character was output before instead of
after a newline directly preceding the special character.

The special handling of newlines is because command substitutions discard
all trailing newlines.

sh: Fix some tests that used sh instead of ${SH}
so they tested the wrong sh.

This was caused because these tests were committed after the sh -> ${SH}
change but were created before.

sh: Add a test for r216387 (long arithmetic expression in here document).

sh: Make the test for cd/pwd with long pathnames more useful:
* Use $(getconf PATH_MAX /) to make sure we actually exercise the hard part
* Delete our test area even if the test fails

sh: Add a test that manipulates various long strings.

It is quite effective at detecting mistakes in memalloc.c and code using it.

It is somewhat slow, but some of the patches in my queue improve it.

sh: Add another simple test for the wait builtin.

test: Move tests to tools/regression/bin/test.
Convert the tests to the perl prove format.
Remove obsolete TEST.README (results of an old TEST.sh for some old Unices)
and TEST.csh (old tests without correct values, far less complete than
TEST.sh).

MFC after: 1 week

sh: Add simple tests for printf.

These are not meant as a replacement for tools/regression/usr.bin/printf/*
but to detect errors specific to making it a shell builtin.

sh: Fix some issues with aliases and case, by importing dash checkkwd code.

This moves the function of the noaliases variable into the checkkwd
variable. This way it is properly reset on errors and aliases can be used
normally in the commands for each case (the case labels recognize the
keyword esac but no aliases).

The new code is clearer as well.

Obtained from: dash

sh: Detect various additional errors in the parser.

Apart from detecting breakage earlier or at all, this also fixes a segfault
in the testsuite. The "handling" of the breakage left an invalid internal
representation in some cases.

Examples:
echo a; do echo b
echo `) echo a`
echo `date; do do do`

Exp-run done by: pav (with some other sh(1) changes)

sh: Fix some issues with CTL* bytes and ${var#pat}.

subevalvar() incorrectly assumed that CTLESC bytes were present iff the
expansion was quoted. However, they are present iff various processing such
as word splitting is to be done later on.

Example:
v=@$e@$e@$e@
y="${v##*"$e"}"
echo "$y"
failed if $e contained the magic CTLESC byte.

Exp-run done by: pav (with some other sh(1) changes)

sh: Do IFS splitting on word in ${v+word} and ${v-word}.

The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.

Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.

Example:
for i in ${$+a b c}; do echo $i; done

Exp-run done by: pav (with some other sh(1) changes)

sh: Only accept a '}' inside ${v+-=?...} if double-quote state matches.
If double-quote state does not match, treat the '}' literally.

This ensures double-quote state remains the same before and after a
${v+-=?...} which helps with expand.c.

It makes things like
${foo+"\${bar}"}
which I have seen in the wild work as expected.

Exp-run done by: pav (with some other sh(1) changes)

sh: Make double-quotes quote a '}' inside ${v#...} and ${v%...}.

Exp-run done by: pav (with some other sh(1) changes)
PR: bin/57554

sh: Do not allow overriding a special builtin with a function.
This is a syntax error.

POSIX does not say explicitly whether defining a function with the same name
as a special builtin is allowed, but it does say that it is impossible to
call such a function.

A special builtin can still be overridden with an alias.

This commit is part of a set of changes that will ensure that when
something looks like a special builtin to the parser, it is one. (Not the
other way around, as it remains possible to call a special builtin named
by a variable or other substitution.)

Exp-run done by: pav (with some other sh(1) changes)

sh: Make sure defined functions can actually be called.

Add some conservative checks on function names:
- Disallow expansions or quoting characters; these can only be called via
strange control characters
- Disallow '/'; these functions cannot be called anyway, as exec.c assumes
they are pathnames
- Make the CTL* bytes work properly in function names.

These are syntax errors.

POSIX does not require us to support more than names (letters, digits and
underscores, not starting with a digit), but I do not want to restrict it
that much at this time.

Exp-run done by: pav (with some other sh(1) changes)

sh: Check whether dup2 was successful for >&FD and <&FD.

A failure (usually caused by FD not being open) is a redirection error.

Exp-run done by: pav (with some other sh(1) changes)

sh: Add a test trying to close a descriptor that is not open.

In stable/8 and older, this fails. Some of the redirection changes in head
have fixed it.

sh: Change ! within a pipeline to start a new pipeline instead.

This is how ksh93 treats ! within a pipeline and makes the ! in
a | ! b | c
negate the exit status of the pipeline, as if it were
a | { ! b | c; }

Side effect: something like
f() ! a
is now a syntax error, because a function definition takes a command,
not a pipeline.

Exp-run done by: pav (with some other sh(1) changes)

sh: Add some testcases for alias expansion.

sh: Allow running 'prove' from tools/regression/bin/sh again
without needing to set special environment variables, testing the 'sh' from
PATH.

Embellish this testcase a little bit to be more clear what the output is
and why. The first case is correct usage which has but one correct output.
The 2nd and 3rd cases are incorrect usage in which the exact output is
not standardized and various shells give various allowable output.

Do not assume in growstackstr() that a "precious" character will be
immediately written into the stack after the call. Instead let the caller
manage the "space left".

Previously, growstackstr()'s assumption causes problems with STACKSTRNUL()
where we want to be able to turn a stack into a C string, and later
pretend the NUL is not there.

This fixes a bug in STACKSTRNUL() (that grew the stack) where:
1. STADJUST() called after a STACKSTRNUL() results in an improper adjust.
This can be seen in ${var%pattern} and ${var%%pattern} evaluation.
2. Memory leak in STPUTC() called after a STACKSTRNUL().

Reviewed by: jilles

Allow one to regression test 'sh' changes without having to install
a potentially bad /bin/sh first.

Correct regression test to not show a false positive when run as root.

sh: Fix exit status if return is used within a loop condition.

sh: Apply variable assignments left-to-right in bltinlookup().

Example:
HOME=foo HOME=bar cd

sh: Fix 'read' if all chars before the first IFS char are backslash-escaped.

Backslash-escaped characters did not set the flag for a non-IFS character.

MFC after: 2 weeks

sh: Add simple tests for backslashes in the read builtin.

sh: Add a test that 'read' leaves the file pointer at the correct place.

Naive buffering would break the common while read x... construct, which did
not appear to be tested yet.

sh: Weaken builtins/command4 test to only require a nonzero exit status.

This matches what is in POSIX; various other shells use different exit
statuses.

Note that it is still required that there be no output.

Add a Makefile for tools/regression/bin and support 'all' in
pax/Makefile.

MFC after: 3 weeks

Correct an out-by-one error when earlying out ustar filenames that
are too long. Filenames escaping this test are caught later on,
so the bug doesn't cause any breakage.

Document the correct ustar limitations in pax. As I have no access
to the IEEE 1003.2 spec, I can only assume that the limitations
imposed are in fact correct.

Add regression tests for the filename limitations imposed by pax.

MFC after: 3 weeks

sh: Remove remnants of '!!' to negate pattern.

This Almquist extension was disabled long ago.

In pathname generation, components starting with '!!' were treated as
containing wildcards, causing unnecessary readdir (which could fail, causing
pathname generation to fail while it should not).

sh: Test that all bytes from 128 to 255 can be used in IFS.

To avoid multibyte issues, this test forces ISO8859-1 charset.

This also passes on stable/8.

Fix keyword expansion properties.

sh: Add a test for breaking from a loop outside the current function.

It is unwise to rely on this but I'd like to know if this would break.

sh: Add a test for break from a trap action.

sh: Reduce unnecessary testsuite failures with other shells.

sh: Split off a more dubious test from parser/heredoc2.0.

sh: Get rid of unnecessary non-standard empty lists.

POSIX does not allow constructs like:
if cmd; then fi
{ }
Add a colon dummy command, except in a test that verifies that such empty
lists do not cause crashes when used as a function definition.

sh: Fix break/continue/return sometimes not skipping the rest of dot script.

In our implementation and most others, a break or continue in a dot script
can break or continue a loop outside the dot script. This should cause all
further commands in the dot script to be skipped. However, cmdloop() did not
know about this and continued to parse and execute commands from the dot
script.

As described in the man page, a return in a dot script in a function returns
from the function, not only from the dot script. There was a similar issue
as with break and continue. In various other shells, the return appears to
return from the dot script, but POSIX seems not very clear about this.

sh: Test that all bytes from 1 to 127 can be used in IFS.

This also passes on stable/8.

sh: Fix heap-based buffer overflow in pathname generation.

The buffer for generated pathnames could be too small in some cases. It
happened to be always at least PATH_MAX long, so there was never an overflow
if the resulting pathnames would be usable.

This bug may be abused if a script subjects input from an untrusted source
to pathname generation, which a bad idea anyhow. Most shell scripts do not
work on untrusted data. secteam@ says no advisory is necessary.

PR: bin/148733
Reported by: Changming Sun snnn119 at gmail com
MFC after: 10 days

sh: Add more testcases for ${var:-word}.

Whether POSIX requires these is unclear.

They pass with 8-stable sh as well.

sh: Return 0 from eval if no command was given.

This makes a difference if there is a command substitution.

To make this work, evalstring() has been changed to set exitstatus to 0 if
no command was executed (the string contained only whitespace).

Example:
eval $(false); echo $?
should print 0.

sh: Add a test for a corner case in eval that already works correctly.

sh: Fix crash due to uninitialized here-document.

If an ; or & token was followed by an EOF token, pending here-documents were
left uninitialized. Execution would crash, either in the main shell process
for literal here-documents or in a child process for expanded
here-documents. In the latter case the problem is hard to detect apart from
the core dumps and log messages.

Side effect: slightly different retries on inputs where EOF is not
persistent.

Note that tools/regression/bin/sh/parser/heredoc6.0 still causes a similar
crash in a child process. The text passed to eval is malformed and should be
rejected.

sh: Allow a background command consisting solely of redirections.

Example:
</dev/null &

MFC after: 2 weeks

sh: Remove comment that the comma operator is missing in arithmetic
expansion.

The comma operator is not listed in POSIX.1-2008 XCU 1.1.2.1 Arithmetic
Precision and Operations (referenced by XCU 2.6.4 Arithmetic Expansion) and
is therefore not required.

sh: Fix a crash if a heredoc was not properly ended and parsing continued.

Example (in interactive mode):
cat <<EOF && )
The next command typed caused sh to segfault, because the state for the here
document was not reset.

Like parser_temp, this uses the fact that the parser is not re-entered.

sh: Change interaction of command substitution and here documents.

If a command substitution contains a newline token, this no longer starts
here documents of outer commands. This way, we follow POSIX's idea of the
command substitution being a separate script more closely. It also matches
other shells better and is consistent with newline characters in quotes not
starting here documents.

The extension tested in parser/heredoc3.0 ($(cat <<EOF)\ntext\nEOF\n)
continues to be supported.

In particular, this change allows things like
cat <<EOF && echo `pwd`
(a `` command substitution after a here document)
which formerly silently used an empty file as the here document, because the
EOF of the inner command "pwd" also forced an empty here document.

sh: Recognize "--" in . and exec.

Although "--" historically has not been required to be recognized for
certain special builtins that do not take options in POSIX, some other
implementations recognize options for them, requiring scripts to use "--" or
avoid operands starting with "-".

Operands starting with "-" can be avoided with eval by prepending a space,
and cannot occur with break, continue, exit, return and shift as they only
take numbers, nor with times as it does not take operands. With . and exec,
avoiding "-" is not so easy as it may require reimplementing the PATH
search; therefore the current proposal for POSIX is to require recognition
of "--" for them.

We continue to accept other strings starting with "-" as operands to . and
exec, and also "--" if it is alone to . (which would otherwise be invalid
anyway).

sh: Add some simple tests for ., exec and return from . script.

sh: Add some simplistic tests for the wait builtin.

sh: Fix pathname expansion with quoted slashes like *\/.

These are git commits 36f0fa8fcbc8c7b2b194addd29100fb40e73e4e9 and
d6d06ff5c2ea0fa44becc5ef4340e5f2f15073e4 in dash.

Because this is the first code I'm importing from dash to expand.c, add the
Herbert Xu copyright notice which is in dash's expand.c.

When pathname expanding *\/, the CTLESC representing the quoted state was
erroneously taken as part of the * pathname component. This CTLESC was then
seen by the pattern matching code as escaping the '\0' terminating the
string.

The code is slightly different because dash converts the CTLESC characters
to backslashes and removes all the other CTL* characters to allow
substituting glob(3).

The effect of the bug was also slightly different from dash (where nothing
matched at all). Because a CTLESC can escape a '\0' in some way, whether
files were included despite the bug depended on memory that should not be
read. In particular, on many machines /*\/ expanded to a strict subset of
what /*/ expanded to.

Example:
echo /*"/null"

This should print /dev/null, not /*/null.

PR: bin/146378
Obtained from: dash

sh: Add some simple testcases for pathname expansion.

Fix error in comment.

sh: Add some parser tests.

case1.0 tests POSIX requirements and one more for keywords in case
statements. The others test very special cases of command substitution.

These also work on stable/8.

Generate some tests for sh's case command from the fnmatch tests.

I'm committing the generated files because I don't like a build dependency
for the sh(1) tests, and they are small and will not change much.

sh: Fix bug in assignment error test.

The test failed if the command returned nonzero exit status, and it really
should return that.

sh: Apply locale vars on builtins, recognize LC_MESSAGES as a locale var.

This allows doing things like LC_ALL=C some_builtin to run a builtin under a
different locale, just like is possible with external programs. The
immediate reason is that this allows making printf(1) a builtin without
breaking things like LC_NUMERIC=C printf '%f\n' 1.2

This change also affects special builtins, as even though the assignment is
persistent, the export is only to the builtin (unless the variable was
already exported).

Note: for this to work for builtins that also exist as external programs
such as /bin/test, the setlocale() call must be under #ifndef SHELL. The
shell will do the setlocale() calls which may not agree with the environment
variables.

sh: Add some more tests for ${v#...} and ${v%...}.

These pass on stable/8 as well.

sh: Add testcases for double-quotes within quoted ${var+-...} (non-POSIX).

POSIX leaves things like "${var+"word"}" undefined.
We follow traditional ash behaviour here.
Hence, these testcases also work on stable/8.

sh: On startup of the shell, use PWD from the environment if it is valid.
Unset PWD if it is incorrect and no value for it can be determined.
This preserves the logical current directory across shell invocations.

Example (assuming /home is a symlink):
$ cd
$ pwd
/home/foo
$ sh
$ pwd
/home/foo

Formerly the second pwd would show the physical path (symlinks resolved).

sh: Test that bogus values of PWD are not imported from the environment.

Current versions pass this test trivially by never importing PWD, but I plan
to change sh to import PWD if it is an absolute pathname for the current
directory, possibly containing symlinks.

sh: Add test for nested arithmetic substitution.

Pre-r206145 sh changes nested arithmetic substitution to parentheses, which
is not always correct, as exercised by this test.

sh: Add test for some associativity in arithmetic.

sh: Do tilde expansion in substitutions.

This applies to word in ${v-word}, ${v+word}, ${v=word}, ${v?word} (which
inherits quoting from the outside) and in ${v%word}, ${v%%word}, ${v#word},
${v##word} (which does not inherit any quoting).

In all cases tilde expansion is only attempted at the start of word, even if
word contains spaces. This agrees with POSIX and other shells.

This is the last part of the patch tested in the exp-run.

Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix duplicate variable name in test.

sh: Add some testcases for the working parts of tilde expansion.

sh: Allow quoting pattern match characters in ${v%pat} and ${v#pat}.

Note that this depends on r206145 for allowing pattern match characters to
have their special meaning inside a double-quoted expansion like "${v%pat}".

PR: bin/117748
Exp-run done by: erwin (with some other sh(1) changes)

sh: Fix various things about expansions:
* remove the backslash from \} inside double quotes inside +-=?
substitutions, e.g. "${$+\}a}"
* maintain separate double-quote state for ${v#...} and ${v%...};
single and double quotes are special inside, even in a double-quoted
string or here document
* keep track of correct order of substitutions and arithmetic

This is different from dash's approach, which does not track individual
double quotes in the parser, trying to fix this up during expansion.
This treats single quotes inside "${v#...}" incorrectly, however.

This is similar to NetBSD's approach (as submitted in PR bin/57554), but
recognizes the difference between +-=? and #% substitutions hinted at in
POSIX and is more refined for arithmetic expansion and here documents.

PR: bin/57554
Exp-run done by: erwin (with some other sh(1) changes)

sh: Add some testcases for the working parts of ${v%...} and ${v#...}.

sh: Do not abort on a redirection error on a compound command.

Redirection errors on subshells already did not abort the shell because
the redirection is executed in the subshell.

Other shells seem to agree that these redirection errors should not abort
the shell.

Also ensure that the redirections will be cleaned up properly in cases like
command eval '{ shift x; } 2>/dev/null'

Example:
{ echo bad; } </var/empty/x; echo good

sh: Add test for redirection error on subshell (should not abort).

sh: Do not abort on a redirection error if there is no command word.

Although simple commands without a command word (only assignments and/or
redirections) are much like special builtins, POSIX and most shells seem to
agree that redirection errors should not abort the shell in this case. Of
course, the assignments persist and assignment errors are fatal.

To get the old behaviour portably, use the ':' special builtin.
To get the new behaviour portably, given that there are no assignments, use
the 'true' regular builtin.

sh: Add test for assignment errors (e.g. trying to change a readonly var).
We currently ignore readonly status for assignments before regular builtins
and external programs (these assignments are not persistent anyway), so just
check that the readonly variable really is not changed.
The test depends on the command builtin changes for 'command :'.

sh: Add tests for " and $ inside `.

sh: Add some testcases for ${v=w}, ${v-w}, ${v+w}.

These expansions, which were already in the Bourne shell, work correctly for
the most part. The testcases are only about the parts that already work
correctly.

sh: Add various testcases for here documents.

They are mainly about expansions in here documents but because all the
testcases are in $() command substitution, we also test that $() command
substitution is recursively parsed (or very close to it).

sh: Make sure to popredir() even if a function caused an error.

sh: Make sure to popredir() even if a special builtin caused an error.

sh: Improve the command builtin:
* avoid unnecessary fork
* allow executing builtins via command
* executing a special builtin via command removes its special properties

Obtained from: NetBSD (parts)

sh: arith: Add a test for a bug in the dash arith code,
which I plan to import at some point.
Our current code handles it fine and it should stay that way.

sh: arith: Test an octal constant as well.

sh: Make sure the mail2.0 test can actually fail if $MAIL is not touched.

sh: Do not stat() $MAIL/$MAILPATH in non-interactive shells.

These may be NFS mounted, and we should not touch them unless we are going
to do something useful with the information.

sh: Send the "not found" message for builtin <cmd> to redirected fd 2.

sh: Add a regression test that tries out all arithmetic ops.

MFC after: 1 week

Use t1 & t2 as more symbolic than 'a' and 'b' for the two times.

Hide expected error output so the 'prove -r' results are easy to read.

Allow to be driven by 'prove -r'.

sh: Fix some bugs with backquoted builtins:
- correctly handle error output in $(builtin 2>&1), clarify out1/out2 vs
output/errout in the code
- treat all builtins as regular builtins so errors do not abort the shell
and variable assignments do not persist
- respect the caller's INTOFF

Some bugs still exist:
- expansion errors may still abort the shell
- some side effects of expansions and builtins persist

sh: Use PATH= assignment in type.

Example:
PATH=/var/empty; PATH=/bin type ls

sh: Allow command -pv and command -pV (lookup using _PATH_STDPATH).

sh: arith: Return only 0 and 1 from && and ||.

This agrees with C, POSIX and other shells.

sh: Do not run callers' exception handlers in subshells.

Reset the exception handler in the child to main's.

This avoids inappropriate double cleanups or shell duplication when the
exception is caught, such as 'fc' and future 'command eval' and 'command .'.

sh: Test ;<newline> as well as ; in the 'for' parser test.

Disable job control when running 'sh -i' in the testsuite.

Job control tty manipulations sometimes cause the tests to stop (SIGTTOU
and the like) when run from the 'prove' tool.

Fix some cases where file descriptors from redirections leak to programs.

- Redirecting fds that were not open before kept two copies of the
redirected file.
sh -c '{ :; } 7>/dev/null; fstat -p $$; true'
(both fd 7 and 10 remained open)
- File descriptors used to restore things after redirection were not
set close-on-exec, instead they were explicitly closed before executing
a program normally and before executing a shell procedure. The latter
must remain but the former is replaced by close-on-exec.
sh -c 'exec 7</; { exec fstat -p $$; } 7>/dev/null; true'
(fd 10 remained open)

The examples above are simpler than the testsuite because I do not want to
use fstat or procstat in the testsuite.

sh: Ensure the same command input file is on top after executing a builtin.

This avoids weirdness when 'fc -e vi' or the like is done and there is a
syntax error in the file. Formerly an interactive shell tried to execute
stuff after the syntax error and exited.

This should also avoid similar issues with 'command eval' and 'command .'
when 'command' is implemented properly as in NetBSD sh.

Special builtins did not have this problem since errors in them cause the
shell to exit or to reset various state such as the current command input
file.

trap: do not consider a bad signal name a fatal error.
POSIX explicitly prescribes this.
Continue processing any other signals and return status 1.

Add a test for r199631.

Add a few very basic tests for cd -{L,P} and pwd -{L,P}.

sh: Allow a newline before "in" in a for command, as required by POSIX.

sh: Exempt $@ and $* from set -u

This seems more useful and will likely be in the next POSIX standard.

Also document more precisely in the man page what set -u does (note that
$@, $* and $! are the only special parameters that can ever be unset, all
the others are always set, although they may be empty).

Add some tests for ${var?} and set -u.

sh: Send the "xyz: not found" message to redirected fd 2.
This also fixes that trying to execute a non-regular file with a command
name without '/' returns 127 instead of 126.
The fix is rather simplistic: treat CMDUNKNOWN as if the command were found
as an external program. The resulting fork is a bit wasteful but executing
unknown commands should not be very frequent.

PR: bin/137659

sh: Disallow mismatched quotes in backticks (`...`).

Due to the amount of code removed by this, it seems that allowing unmatched
quotes was a deliberate imitation of System V sh and real ksh. Most other
shells do not allow unmatched quotes (e.g. bash, zsh, pdksh, NetBSD /bin/sh,
dash).

PR: bin/137657

sh: Fix crash with empty functions (f() { }) introduced in r196483

Empty pairs of braces are represented by a NULL node pointer, just like
empty lines at the top level.

Support for empty pairs of braces may be removed later. They make the code
more complex, have inconsistent behaviour (may or may not change $?), are
not specified by POSIX and are not allowed by some other shells like bash,
dash and ksh93.

Reported by: kan

Add some tests for a fixed bug in an uncommitted patch.
(Trying to get syntax errors for sh -c ':; do' and `:; do`.)

sh: Fix crash when undefining or redefining a currently executing function.

Add a reference count to function definitions.
Memory may leak if multiple SIGINTs arrive in interactive mode,
this will be fixed later by changing SIGINT handling.

PR: bin/137640

Add some tests for r194975 and r194977.

Approved by: ed (mentor) (implicit)

Designate special builtins as such in command -V and type.
Also document various properties of special builtins that we implement.

Approved by: ed (mentor) (implicit)

Add test for r190698.

Submitted by: Eygene Ryabinkin
Approved by: ed (mentor) (implicit)

Add tests for r194774.

Approved by: ed (mentor) (implicit)

Add tests for r194406 and r194516.

Approved by: ed (mentor)

Add tests for r194127 and r194128.

Approved by: ed (mentor)

Add tests for r193169.

Add regression tests for the date(1) + argument for user-defined
format strings.

PR: bin/127514
Submitted by: edwin@
MFC after: 1 week

Add a test for r191009.

Test the r190298 change.

Add a few tests for the read built-in.

Update test for r190284.

Update and extend the tests for alias.

Bootstrapping merge history for resolver.

Update for 185401, errors now go to stderr.

Add a test for r185231.

Add a test for r182300.

Add a regression test for r181017.

Submitted by: simon

Set magic fbsd:nokeywords property that allows files to bypass
keyword expansion. (file-specific replacement for CVSROOT/exclude)

Test the expansion of $LINENO.

Add a test case for options.c revision 1.26.

Add a regression test for cd.c's revision 1.35.

Update the number of reported tests to keep prove(1) happy.

Correct the "move file from directory to existing directory" test for
cross-device moves.

Portability improvements to investigate behavior of other OSs.
Now works under Solaris and Linux.

Add more tests. All rename(2)-based tests now succeed.
The performance of the cross-device equivalents is under investigation.

Verify that the moved source is no longer there.

Regression tests for upcoming makeup of mv.
Case 20 corresponds to PR bin/118367.

The exit status of a case statement where none of the patterns is matched
is supposed to be 0, not the status of the previous command.

Reported by: Eygene Ryabinkin
PR: 116559
Approved by: re (gnn)

Add regression tests for exec.c rev 1.30.

Add regression tests for PR 105078.

Test that variable assignments preceding special built-ins affect the current
shell environment, eg after "VAR=val eval" VAR remains set. Currently fails
but will be implemented soon.

Add test cases that check utility syntax errors and redirection errors. For
special built-in utilities they must terminate the shell, for other
utilities only a error message shall be written. We currently fail both
tests.

Add test cases for the command built-in, including its -v and -V options which
I'll commit real soon.

Count the number of found test cases instead of hard-coding them. Allow an
arbitrary exit status.

- Add tests for hash and return and more tests for set -e.
- Simplify regress.sh, sort tests.

Simplify and add PR 77067 which addresses the same bug.

Fix syntax error that crept in somehow.

Add a couple of regression tests for /bin/sh.