Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r248349: sh: Recognize "--" and explicitly reject options in wait
builtin.

If syntactically invalid job identifiers are to be taken as jobs that exited
with status 127, this should not apply to options, so that we can add
options later if need be.

MFC r245383,245392,247190,249220,251180,251797: New sh testcases.

These already work on stable/9.

MFC r242767,r243252: sh: Add tests for modifying an alias (r242766/r243402).

MFC r226892,r228007,r228873,r230121,r232839: sh: Various testcases that
already work.

MFC r229742: sh: Avoid possible echo options in a testcase.

MFC r230212:
sh: Fix execution of multiple statements in a trap when evalskip is set

Before this fix, only the first statement of the trap was executed if
evalskip was set. This is for example the case when:
o "-e" is set for this shell
o a trap is set on EXIT
o a function returns 1 and causes the script to abort

Reviewed by: jilles
Sponsored by: Yakaz (http://www.yakaz.com)

MFC r230211:
sh: Test EXIT trap with multiple statements in it

Reviewed by: jilles

MFC r230095: sh: Properly show "Not a directory" error in cd builtin.

The errno message display added in r222292 did not take attempting to
cd to a non-directory or something that cannot be stat()ed into account.

PR: bin/164070

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

sh: Test that '!' is literal if quoted and first char of bracket expression

This also works on stable/8.

sh: Add case statement fallthrough (with ';&' instead of ';;').

Replacing ;; with the new control operator ;& will cause the next list to be
executed as well without checking its pattern, continuing until a list ends
with ;; or until the end of the case statement. This is like omitting
"break" in a C "switch" statement.

The sequence ;& was formerly invalid.

This feature is proposed for the next POSIX issue in Austin Group issue
#449.

sh: Skip variables with invalid names in "set", "export -p", "readonly -p".

This ensures the output of these commands is valid shell input.

sh: Add support for named character classes in bracket expressions.

Example:
case x in [[:alpha:]]) echo yes ;; esac

sh: Add test for LC_COLLATE-based character ranges in case.

sh: Add test for case pattern matching with iso-8859-1 charset.

This also passes on stable/8.

etire the cpumask_t type and replace it with cpuset_t usage.

This is intended to fix the bug where cpu mask objects are
capped to 32. MAXCPU, then, can now arbitrarely bumped to whatever
value. Anyway, as long as several structures in the kernel are
statically allocated and sized as MAXCPU, it is suggested to keep it
as low as possible for the time being.

Technical notes on this commit itself:
- More functions to handle with cpuset_t objects are introduced.
The most notable are cpusetobj_ffs() (which calculates a ffs(3)
for a cpuset_t object), cpusetobj_strprint() (which prepares a string
representing a cpuset_t object) and cpusetobj_strscan() (which
creates a valid cpuset_t starting from a string representation).
- pc_cpumask and pc_other_cpus are target to be removed soon.
With the moving from cpumask_t to cpuset_t they are now inefficient
and not really useful. Anyway, for the time being, please note that
access to pcpu datas is protected by sched_pin() in order to avoid
migrating the CPU while reading more than one (possible) word
- Please note that size of cpuset_t objects may differ between kernel
and userland. While this is not directly related to the patch itself,
it is good to understand that concept and possibly use the patch
as a reference on how to deal with cpuset_t objects in userland, when
accessing kernland members.
- KTR_CPUMASK is changed and now is represented through a string, to be
set as the example reported in NOTES.

Please additively note that no MAXCPU is bumped in this patch, but
private testing has been done until to MAXCPU=128 on a real 8x8x2(htt)
machine (amd64).

Please note that the FreeBSD version is not yet bumped because of
the upcoming pcpu changes. However, note that this patch is not
targeted for MFC.

People to thank for the time spent on this patch:
- sbruno, pluknet and Nicholas Esborn (nick AT desert DOT net) tested
several revision of the patches and really helped in improving
stability of this work.
- marius fixed several bugs in the sparc64 implementation and reviewed
patches related to ktr.
- jeff and jhb discussed the basic approach followed.
- kib and marcel made targeted review on some specific part of the
patch.
- marius, art, nwhitehorn and andreast reviewed MD specific part of
the patch.
- marius, andreast, gonzo, nwhitehorn and jceel tested MD specific
implementations of the patch.
- Other people have made contributions on other patches that have been
already committed and have been listed separately.

Companies that should be mentioned for having participated at several
degrees:
- Yahoo! for having offered the machines used for testing on big
count of CPUs.
- The FreeBSD Foundation for having sponsored my devsummit attendance,
which has been instrumental.
- Sandvine for having offered offices and infrastructure during
development.

(I really hope I didn't forget anyone, if it happened I apologize in
advance).

sh: Reduce more needless differences between error messages.

sh: Add test for 'set +o'.

sh: Correct criterion for using CDPATH in cd.

CDPATH should be ignored not only for pathnames starting with '/' but also
for pathnames whose first component is '.' or '..'.

The man page already describes this behaviour.

sh: Add simple CDPATH test.

sh: Add test for r222173.

sh: Implement the cd -e flag proposed for the next POSIX issue.

This reflects failure to determine the pathname of the new directory in the
exit status (1). Normally, cd returns successfully if it did chdir() and the
call was successful.

In POSIX, -e only has meaning with -P; because our -L is not entirely
compliant and may fall back to -P mode, -e has some effect with -L as well.

sh: Add UTF-8 support to pattern matching.

?, [...] patterns match codepoints instead of bytes. They do not match
invalid sequences. [...] patterns must not contain invalid sequences
otherwise they will not match anything. This is so that ${var#?} removes the
first codepoint, not the first byte, without putting UTF-8 knowledge into
the ${var#pattern} code. However, * continues to match any string and an
invalid sequence matches an identical invalid sequence. (This differs from
fnmatch(3).)

sh: Add test for bin/56147.

sh: Test that . /dev/null returns exit status 0 and does not preserve $?.

Preserving $? may cause problems particularly if set -e is in effect.

It may be useful to preserve the old value of $? in the dot script but this
must not be implemented in such a way that it would break this test.

sh: Split off some special behaviour into separate tests.

This allows some other shells to pass the tests for basic behaviour.

sh: Test that the read builtin passes through all byte values
except NUL, newline and backslash.

This also passes on stable/8.

sh: Unset some more locale vars in two tests that may cause them to break.

sh: Weaken some tests to allow /rescue/sh to pass everything.

/rescue/sh has a different _PATH_STDPATH which affects command -p.

sh: Add test for EXIT trap in command substitution.

This is not really realistic but is an opposition to $(trap).

sh: If exit is used without args from a trap action, exit on the signal.

This is useful so that it is easier to exit on a signal than to reset the
trap to default and resend the signal. It matches ksh93. POSIX says that
'exit' without args from a trap action uses the exit status from the last
command before the trap, which is different from 'exit $?' and matches this
if the previous command is assumed to have exited on the signal.

If the signal is SIGSTOP, SIGTSTP, SIGTTIN or SIGTTOU, or if the default
action for the signal is to ignore it, a normal _exit(2) is done with exit
status 128+signal_number.

sh: Fix some things about -- in trap:
* Make 'trap --' do the same as 'trap' instead of nothing.
* Make '--' stop option processing (note that '-' action is not an option).

Side effect: The error message for an unknown option is different.

sh: Make exit without parameters from EXIT trap POSIX-compliant.

It should use the original exit status, just like falling off the
end of the trap handler.

Outside an EXIT trap, 'exit' is still equivalent to 'exit $?'.

sh: Add simple test for 'exit' without parameters.

sh: Do not call exitshell() from evalcommand() unless evalcommand() forked
itself.

This ensures that certain traps caused by builtins are executed.

sh: Test that exit $? replaces the original exit status in an EXIT trap.

sh: Make warnings in the printf builtin non-fatal, like in the program.

The #define for warnx now behaves much like the libc function (except that
it uses sh command name and output).

Also, it now uses C99 __VA_ARGS__ so there is no need for three different
macros for 0, 1 or 2 parameters.

sh: Make the test for cd/pwd with long pathnames more useful:
* Use $(getconf PATH_MAX /) to make sure we actually exercise the hard part
* Delete our test area even if the test fails

sh: Add another simple test for the wait builtin.

sh: Add simple tests for printf.

These are not meant as a replacement for tools/regression/usr.bin/printf/*
but to detect errors specific to making it a shell builtin.

Embellish this testcase a little bit to be more clear what the output is
and why. The first case is correct usage which has but one correct output.
The 2nd and 3rd cases are incorrect usage in which the exact output is
not standardized and various shells give various allowable output.

Allow one to regression test 'sh' changes without having to install
a potentially bad /bin/sh first.

Correct regression test to not show a false positive when run as root.

sh: Fix exit status if return is used within a loop condition.

sh: Fix 'read' if all chars before the first IFS char are backslash-escaped.

Backslash-escaped characters did not set the flag for a non-IFS character.

MFC after: 2 weeks

sh: Add simple tests for backslashes in the read builtin.

sh: Add a test that 'read' leaves the file pointer at the correct place.

Naive buffering would break the common while read x... construct, which did
not appear to be tested yet.

sh: Weaken builtins/command4 test to only require a nonzero exit status.

This matches what is in POSIX; various other shells use different exit
statuses.

Note that it is still required that there be no output.

Fix keyword expansion properties.

sh: Add a test for breaking from a loop outside the current function.

It is unwise to rely on this but I'd like to know if this would break.

sh: Add a test for break from a trap action.

sh: Reduce unnecessary testsuite failures with other shells.

sh: Get rid of unnecessary non-standard empty lists.

POSIX does not allow constructs like:
if cmd; then fi
{ }
Add a colon dummy command, except in a test that verifies that such empty
lists do not cause crashes when used as a function definition.

sh: Fix break/continue/return sometimes not skipping the rest of dot script.

In our implementation and most others, a break or continue in a dot script
can break or continue a loop outside the dot script. This should cause all
further commands in the dot script to be skipped. However, cmdloop() did not
know about this and continued to parse and execute commands from the dot
script.

As described in the man page, a return in a dot script in a function returns
from the function, not only from the dot script. There was a similar issue
as with break and continue. In various other shells, the return appears to
return from the dot script, but POSIX seems not very clear about this.

sh: Return 0 from eval if no command was given.

This makes a difference if there is a command substitution.

To make this work, evalstring() has been changed to set exitstatus to 0 if
no command was executed (the string contained only whitespace).

Example:
eval $(false); echo $?
should print 0.

sh: Add a test for a corner case in eval that already works correctly.

sh: Recognize "--" in . and exec.

Although "--" historically has not been required to be recognized for
certain special builtins that do not take options in POSIX, some other
implementations recognize options for them, requiring scripts to use "--" or
avoid operands starting with "-".

Operands starting with "-" can be avoided with eval by prepending a space,
and cannot occur with break, continue, exit, return and shift as they only
take numbers, nor with times as it does not take operands. With . and exec,
avoiding "-" is not so easy as it may require reimplementing the PATH
search; therefore the current proposal for POSIX is to require recognition
of "--" for them.

We continue to accept other strings starting with "-" as operands to . and
exec, and also "--" if it is alone to . (which would otherwise be invalid
anyway).

sh: Add some simple tests for ., exec and return from . script.

sh: Add some simplistic tests for the wait builtin.

Fix error in comment.

Generate some tests for sh's case command from the fnmatch tests.

I'm committing the generated files because I don't like a build dependency
for the sh(1) tests, and they are small and will not change much.

sh: Apply locale vars on builtins, recognize LC_MESSAGES as a locale var.

This allows doing things like LC_ALL=C some_builtin to run a builtin under a
different locale, just like is possible with external programs. The
immediate reason is that this allows making printf(1) a builtin without
breaking things like LC_NUMERIC=C printf '%f\n' 1.2

This change also affects special builtins, as even though the assignment is
persistent, the export is only to the builtin (unless the variable was
already exported).

Note: for this to work for builtins that also exist as external programs
such as /bin/test, the setlocale() call must be under #ifndef SHELL. The
shell will do the setlocale() calls which may not agree with the environment
variables.

sh: Do not abort on a redirection error on a compound command.

Redirection errors on subshells already did not abort the shell because
the redirection is executed in the subshell.

Other shells seem to agree that these redirection errors should not abort
the shell.

Also ensure that the redirections will be cleaned up properly in cases like
command eval '{ shift x; } 2>/dev/null'

Example:
{ echo bad; } </var/empty/x; echo good

sh: Make sure to popredir() even if a function caused an error.

sh: Make sure to popredir() even if a special builtin caused an error.

sh: Improve the command builtin:
* avoid unnecessary fork
* allow executing builtins via command
* executing a special builtin via command removes its special properties

Obtained from: NetBSD (parts)

sh: Send the "not found" message for builtin <cmd> to redirected fd 2.

sh: Use PATH= assignment in type.

Example:
PATH=/var/empty; PATH=/bin type ls

sh: Allow command -pv and command -pV (lookup using _PATH_STDPATH).

sh: Do not run callers' exception handlers in subshells.

Reset the exception handler in the child to main's.

This avoids inappropriate double cleanups or shell duplication when the
exception is caught, such as 'fc' and future 'command eval' and 'command .'.

Disable job control when running 'sh -i' in the testsuite.

Job control tty manipulations sometimes cause the tests to stop (SIGTTOU
and the like) when run from the 'prove' tool.

sh: Ensure the same command input file is on top after executing a builtin.

This avoids weirdness when 'fc -e vi' or the like is done and there is a
syntax error in the file. Formerly an interactive shell tried to execute
stuff after the syntax error and exited.

This should also avoid similar issues with 'command eval' and 'command .'
when 'command' is implemented properly as in NetBSD sh.

Special builtins did not have this problem since errors in them cause the
shell to exit or to reset various state such as the current command input
file.

trap: do not consider a bad signal name a fatal error.
POSIX explicitly prescribes this.
Continue processing any other signals and return status 1.

Add a test for r199631.

Add a few very basic tests for cd -{L,P} and pwd -{L,P}.

Add some tests for a fixed bug in an uncommitted patch.
(Trying to get syntax errors for sh -c ':; do' and `:; do`.)

Designate special builtins as such in command -V and type.
Also document various properties of special builtins that we implement.

Approved by: ed (mentor) (implicit)

Add test for r190698.

Submitted by: Eygene Ryabinkin
Approved by: ed (mentor) (implicit)

Add tests for r194406 and r194516.

Approved by: ed (mentor)

Add tests for r194127 and r194128.

Approved by: ed (mentor)

Add tests for r193169.

Test the r190298 change.

Add a few tests for the read built-in.

Update test for r190284.

Update and extend the tests for alias.

Bootstrapping merge history for resolver.

Update for 185401, errors now go to stderr.

Add a test for r182300.

Set magic fbsd:nokeywords property that allows files to bypass
keyword expansion. (file-specific replacement for CVSROOT/exclude)

Test the expansion of $LINENO.

Add a test case for options.c revision 1.26.

Add a regression test for cd.c's revision 1.35.

The exit status of a case statement where none of the patterns is matched
is supposed to be 0, not the status of the previous command.

Reported by: Eygene Ryabinkin
PR: 116559
Approved by: re (gnn)

Add regression tests for exec.c rev 1.30.

Test that variable assignments preceding special built-ins affect the current
shell environment, eg after "VAR=val eval" VAR remains set. Currently fails
but will be implemented soon.

Add test cases for the command built-in, including its -v and -V options which
I'll commit real soon.

Count the number of found test cases instead of hard-coding them. Allow an
arbitrary exit status.

- Add tests for hash and return and more tests for set -e.
- Simplify regress.sh, sort tests.

Add a couple of regression tests for /bin/sh.