Copy stable/11@r303970 to releng/11.0 as part of the 11.0-RELEASE
cycle.

Prune svn:mergeinfo from the new branch, and rename it to RC1.

Update __FreeBSD_version.

Use the quarterly branch for the default FreeBSD.conf pkg(8) repo and
the dvd1.iso packages population.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC 303406,303501: Fix panic when using aio_fsync().

303406:
Adjust tests in fsync job scheduling loop to reduce indentation.

303501:
Fix locking issues with aio_fsync().

- Use correct lock in aio_cancel_sync when dequeueing job.
- Add _locked variants of aio_set/clear_cancel_function and use those
to avoid lock recursion when adding and removing fsync jobs to the
per-process sync queue.
- While here, add a basic test for aio_fsync().

PR: 211390
Approved by: re (kib)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix bitstring allocation on 32-bit platforms

sys/sys/bitstring.h
Fix a rounding calculation that could undersize a bitstring on
32-bit platforms.

tests/sys/sys/bitstring_test.h
Add a test for bitstr_size

PR: 210260
Reported by: Mark Millard
Reviewed by: gibbs
Approved by: re (marius)
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6848

Account for AIO socket operations in thread/process resource usage.

File and disk-backed I/O requests store counts of read/written disk
blocks in each AIO job so that they can be charged to the thread that
completes an AIO request via aio_return() or aio_waitcomplete(). This
change extends AIO jobs to store counts of received/sent messages and
updates socket backends to set these counts accordingly. Note that
the socket backends are careful to only charge a single messages for
each AIO request even though a single request on a blocking socket might
invoke sosend or soreceive multiple times. This is to mimic the
resource accounting of synchronous read/write.

Adjust the UNIX socketpair AIO test to verify that the message resource
usage counts update accordingly for aio_read and aio_write.

Approved by: re (hrs)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D6911

Skip sys/acl tests on systems lacking perl

tests/sys/acl/Makefile
add perl to the required_programs for all tests in this directory

Reviewed by: ngie
Approved by: re (gjb)
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6870

Always create loopback routes on every fib

Always create loopback routes on every fib, for both IPv4 and IPv6

etc/rc.d/routing
Create loopback IPv4 and IPv6 routes on every fib at boot. Revert
278302; now that all FIBs have IPv6 loopback routes, the
"route add -reject" commands won't fail.

tests/etc/rc.d/routing_test.sh
Greatly simplify static_ipv6_loopback_route_for_each_fib. It was
written under the assumption that loopback routes would be added to
a given fib by the kernel as soon as an interface is configured on
that fib. However, the logic can be much simpler now that we simply
add loopback routes to all fibs at boot. This also removes the need
to run the test as root, removes the restriction that
net.add_addr_allfibs=0, and removes the need to configure fibs in
kyua.conf.

Also, add a test case for IPv4 loopback routes

Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6582

Return the correct status when a partially completed request is cancelled.

After the previous changes to fix requests on blocking sockets to complete
across multiple operations, an edge case exists where a request can be
cancelled after it has partially completed. POSIX doesn't appear to
dictate exactly how to handle this case, but in general I feel that
aio_cancel() should arrange to cancel any request it can, but that any
partially completed requests should return a partial completion rather
than ECANCELED. To that end, fix the socket AIO cancellation routine to
return a short read/write if a partially completed request is cancelled
rather than ECANCELED.

Sponsored by: Chelsio Communications

Don't prematurely return short completions on blocking sockets.

Always requeue an AIO job at the head of the socket buffer's queue if
sosend() or soreceive() returns EWOULDBLOCK on a blocking socket.
Previously, requests were only requeued if they returned EWOULDBLOCK
and completed no data. Now after a partial completion on a blocking
socket the request is queued and the remaining request is retried when
the socket is ready. This allows writes larger than the currently
available space on a blocking socket to fully complete. Reads on a
blocking socket that satifsy the low watermark can still return a short
read (same as read()).

In order to track previously completed data, the internal 'status'
field of the AIO job is used to store the amount of previously
computed data.

Non-blocking sockets continue to return short completions for both
reads and writes.

Add a test for a "large" AIO write on a blocking socket that writes
twice the socket buffer size to a UNIX domain socket.

Sponsored by: Chelsio Communications

Add bit_count to the bitstring(3) api

Add a bit_count function, which efficiently counts the number of bits set in
a bitstring.

sys/sys/bitstring.h
tests/sys/sys/bitstring_test.c
share/man/man3/bitstring.3
Add bit_alloc

sys/kern/subr_unit.c
Use bit_count instead of a naive counting loop in check_unrhdr, used
when INVARIANTS are enabled. The userland test runs about 6x faster
in a generic build, or 8.5x faster when built for Nehalem, which has
the POPCNT instruction.

sys/sys/param.h
Bump __FreeBSD_version due to the addition of bit_alloc

UPDATING
Add a note about the ABI incompatibility of the bitstring(3)
changes, as suggested by lidl.

Suggested by: gibbs
Reviewed by: gibbs, ngie
MFC after: 9 days
X-MFC-With: 299090, 300538
Relnotes: yes
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6255

Commit a missing change from 299090

tests/sys/kern/Makefile
Reenable a disabled compiler warning, the need for which was
eliminated by r299090.

Reviewed by: ngie
MFC after: 4 weeks
X-MFC-With: 299090
Sponsored by: Spectra Logic Corp

kern_descrip_test: Fix trivial buffer overrun with readlink(2)

Reported by: Coverity
CID: 1229965, 1229972
Sponsored by: EMC / Isilon Storage Division

Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division

Improve performance and functionality of the bitstring(3) api

Two new functions are provided, bit_ffs_at() and bit_ffc_at(), which allow
for efficient searching of set or cleared bits starting from any bit offset
within the bit string.

Performance is improved by operating on longs instead of bytes and using
ffsl() for searches within a long. ffsl() is a compiler builtin in both
clang and gcc for most architectures, converting what was a brute force
while loop search into a couple of instructions.

All of the bitstring(3) API continues to be contained in the header file.
Some of the functions are large enough that perhaps they should be uninlined
and moved to a library, but that is beyond the scope of this commit.

sys/sys/bitstring.h:
Convert the majority of the existing bit string implementation from
macros to inline functions.

Properly protect the implementation from inadvertant macro expansion
when included in a user's program by prefixing all private
macros/functions and local variables with '_'.

Add bit_ffs_at() and bit_ffc_at(). Implement bit_ffs() and
bit_ffc() in terms of their "at" counterparts.

Provide a kernel implementation of bit_alloc(), making the full API
usable in the kernel.

Improve code documenation.

share/man/man3/bitstring.3:
Add pre-exisiting API bit_ffc() to the synopsis.

Document new APIs.

Document the initialization state of the bit strings
allocated/declared by bit_alloc() and bit_decl().

Correct documentation for bitstr_size(). The original code comments
indicate the size is in bytes, not "elements of bitstr_t". The new
implementation follows this lead. Only hastd assumed "elements"
rather than bytes and it has been corrected.

etc/mtree/BSD.tests.dist:
tests/sys/Makefile:
tests/sys/sys/Makefile:
tests/sys/sys/bitstring.c:
Add tests for all existing and new functionality.

include/bitstring.h
Include all headers needed by sys/bitstring.h

lib/libbluetooth/bluetooth.h:
usr.sbin/bluetooth/hccontrol/le.c:
Include bitstring.h instead of sys/bitstring.h.

sbin/hastd/activemap.c:
Correct usage of bitstr_size().

sys/dev/xen/blkback/blkback.c
Use new bit_alloc.

sys/kern/subr_unit.c:
Remove hard-coded assumption that sizeof(bitstr_t) is 1. Get rid of
unrb.busy, which caches the number of bits set in unrb.map. When
INVARIANTS are disabled, nothing needs to know that information.
callapse_unr can be adapted to use bit_ffs and bit_ffc instead.
Eliminating unrb.busy saves memory, simplifies the code, and
provides a slight speedup when INVARIANTS are disabled.

sys/net/flowtable.c:
Use the new kernel implementation of bit-alloc, instead of hacking
the old libc-dependent macro.

sys/sys/param.h
Update __FreeBSD_version to indicate availability of new API

Submitted by: gibbs, asomers
Reviewed by: gibbs, ngie
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6004

Automate the subr_unit test.

Build and install the subr_unit test program originally written by phk, and
run it with the other ATF tests.

tests/sys/kern/Makefile
* Build and install the subr_unit test as a plain test

sys/kern/subr_unit.c
* Reduce the default number of repetitions from 100 to 1, and add a
command-line parser to override it.
* Don't be so noisy by default
* Fix an include problem for the test build

Reviewed by: ngie
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6038

Fix including Kyuafile in packaged base system.

Fix a related typo while here.

Note, this change results in the Kyuafile inclusion in the runtime
package, which needs to be fixed, however addresses the PR as far
as I can tell in my tests.

PR: 209114
Submitted by: ngie
Sponsored by: The FreeBSD Foundation

Fix issues identified by Coverity

- Always munmap memory regions after mmap'ing them.
- Make sure getpagesize() returns a value greater than 0 and use a
cached value instead of always calling getpagesize(3).
- Remove intermediate variable for assigning from $TMPDIR if set in the
environment to eliminate warnings about pointer conversions with "/tmp",
and to mute an invalid buffer overflow concern from Coverity
(snprintf and tacking on a NUL terminator was alleviating that concern
before).
- Remove useless self-test of psize before it's initialized.
- Check the return values of getrlimit/setrlimit.

Cosmetic changes:
- Replace a `(void*)0` with NULL.
- Do some minor whitespace clean up.
- Remove an unnecessary cast to mmap.
- Make all munmap calls use ATF_REQUIRE_MSG instead of using the:

> if (munmap(..) == -1)
> atf_tc_fail(..)

idiom. Employ the new idiom consistently when calling munmap.

CID: 1331351, 1331382-1331386, 1331513, 1331514, 1331565, 1331583, 1331694
Differential Revision: https://reviews.freebsd.org/D6012
MFC after: 2 weeks
Reported by: Coverity
Reviewed by: markj
Sponsored by: EMC / Isilon Storage Division

Fix leaks and test for getpagesize() returning == -1

- close file descriptors after use.
- Always munmap memory regions after mmap'ing them.
- Make sure getpagesize() returns a value greater than 0 and use a
cached value instead of always calling getpagesize(3).

CID: 1331374-1331377, 1331653-1331662
Differential Revision: https://reviews.freebsd.org/D6011
MFC after: 2 weeks
Reported by: Coverity
Reviewed by: cem
Sponsored by: EMC / Isilon Storage Division

Minor cosmetic cleanup

- Remove spurious trailing whitespace in licensing header
- Remove unnecessary semi-colon after comment [*]

MFC after: 3 days
Submitted by: pfg [*]
Sponsored by: EMC / Isilon Storage Division

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

Add a test for cancelling an active AIO request on a socket.

The older AIO code awakened all pending AIO requests on a socket
when any data arrived. This could result in AIO daemons blocking on
an empty socket buffer. These requests could not be cancelled
which led to a deadlock during process exit. This test reproduces
this case. The newer AIO code is able to cancel the pending AIO
request correctly.

Reviewed by: ngie (-ish)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D4363

Set test_argv to NULL, not 0, if not executing a specific test

MFC after: 1 week
Submitted by: pfg
Sponsored by: EMC / Isilon Storage Division

Correct error messages in recently added test.

The large read test uses an empty file created via mkstemp() rather than
/dev/null as character devices are subject to two different clamping
sysctls. However, I forgot to update some of the error messages after
changing to mkstemp() that were still referring to /dev/null.

Use a SKIP testplan instead of bailing out if/when the tester isn't
root, or the geom class can't be loaded cleanly [*]

This makes sure that scenarios that are easy to hit aren't counted
as false positives with kyua test

MFC after: 1 week
PR: 208101
Sponsored by: EMC / Isilon Storage Division

Fully handle size_t lengths in AIO requests.

First, update the return types of aio_return() and aio_waitcomplete() to
ssize_t.

POSIX requires aio_return() to return a ssize_t so that it can represent
all return values from read() and write(). aio_waitcomplete() should use
ssize_t for the same reason.

aio_return() has used ssize_t in <aio.h> since r31620 but the manpage and
system call entry were not updated. aio_waitcomplete() has always
returned int.

Note that this does not require new system call stubs as this is
effectively only an API change in how the compiler interprets the return
value.

Second, allow aio_nbytes values up to IOSIZE_MAX instead of just INT_MAX.

aio_read/write should now honor the same length limits as normal read/write.

Third, use longs instead of ints in the aio_return() and aio_waitcomplete()
system call functions so that the 64-bit size_t in the in-kernel aiocb
isn't truncated to 32-bits before being copied out to userland or
being returned.

Finally, a simple test has been added to verify the bounds checking on the
maximum read size from a file.

DIRDEPS_BUILD: Connect MK_TESTS.

Sponsored by: EMC / Isilon Storage Division

Refactor the AIO subsystem to permit file-type-specific handling and
improve cancellation robustness.

Introduce a new file operation, fo_aio_queue, which is responsible for
queueing and completing an asynchronous I/O request for a given file.
The AIO subystem now exports library of routines to manipulate AIO
requests as well as the ability to run a handler function in the
"default" pool of AIO daemons to service a request.

A default implementation for file types which do not include an
fo_aio_queue method queues requests to the "default" pool invoking the
fo_read or fo_write methods as before.

The AIO subsystem permits file types to install a private "cancel"
routine when a request is queued to permit safe dequeueing and cleanup
of cancelled requests.

Sockets now use their own pool of AIO daemons and service per-socket
requests in FIFO order. Socket requests will not block indefinitely
permitting timely cancellation of all requests.

Due to the now-tight coupling of the AIO subsystem with file types,
the AIO subsystem is now a standard part of all kernels. The VFS_AIO
kernel option and aio.ko module are gone.

Many file types may block indefinitely in their fo_read or fo_write
callbacks resulting in a hung AIO daemon. This can result in hung
user processes (when processes attempt to cancel all outstanding
requests during exit) or a hung system. To protect against this, AIO
requests are only permitted for known "safe" files by default. AIO
requests for all file types can be enabled by setting the new
vfs.aio.enable_usafe sysctl to a non-zero value. The AIO tests have
been updated to skip operations on unsafe file types if the sysctl is
zero.

Currently, AIO requests on sockets and raw disks are considered safe
and are enabled by default. aio_mlock() is also enabled by default.

Reviewed by: cem, jilles
Discussed with: kib (earlier version)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D5289

Ensure that we test the event condition when a disabled kevent is enabled.

r274560 modified kqueue_register() to only test the event condition if the
corresponding knote is not disabled. However, this check takes place before
the EV_ENABLE flag is used to clear the KN_DISABLED flag on the knote, so
enabling a previously-disabled kevent would not result in a notification for
a triggered event. This change fixes the problem by testing for EV_ENABLED
before possibly checking the event condition.

This change also updates a kqueue regression test to exercise this case.

PR: 206368
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5307

Test directories can build in parallel fine.

Sponsored by: EMC / Isilon Storage Division

Similar to r295116, add an additional 1 second sleep after calling ggatel
before calling dd to defeat a race when writing out to the geom_gate(4)
device

MFC after: 1 month
Reported by: Jenkins
Sponsored by: EMC / Isilon Storage Division

Use basenames for getfacl, setfacl, and zpool to work around the
fact that Jenkins hardcodes image sizes to 2GB with the FreeBSD_HEAD
job

This is to stop the unnecessary failure emails because we've gone
over the 2GB limit

MFC after: 1 week
X-MFC with: r295341
Sponsored by: EMC / Isilon Storage Division

Require /bin/getfacl and /bin/setfacl when running the acl tests

For cases where these utilities aren't installed, the tests would fail today
in a non-intuitive manner on sub-testcase #3 in each of the test scripts

MFC after: 1 week
Reviewed by: markj
Sponsored by: EMC / Isilon Storage Division

Use the pidfile support added to ggated(8) in r294973 to ensure that the
ggated(8) daemon used by the tests is the instance specifically invoked by
the tests instead of one or more daemon instances running on the system

MFC after: 1 month
Sponsored by: EMC / Isilon Storage Division

Add an additional 1 second sleep to after calling ggatec before calling
dd to defeat a race when writing out to the geom_gate(4) device

This will quell the Jenkins failure emails until I come up with a better
solution

MFC after: 1 month
Reported by: Jenkins
Sponsored by: EMC / Isilon Storage Division

kqueue EVFILT_PROC: avoid collision between NOTE_CHILD and NOTE_EXIT

NOTE_CHILD and NOTE_EXIT return something in kevent.data: the parent
pid (ppid) for NOTE_CHILD and the exit status for NOTE_EXIT.
Do not let the two events be combined, since one would overwrite
the other's data.

PR: 180385
Submitted by: David A. Bright <david_a_bright@dell.com>
Reviewed by: jhb
MFC after: 1 month
Sponsored by: Dell Inc.
Differential Revision: https://reviews.freebsd.org/D4900

PID file support hasn't been committed for ggated(8) yet. Unbreak running
the testcase more than once by restoring the "killall ggated"

MFC after: 15 days
Sponsored by: EMC / Isilon Storage Division

Remove unnecessary kldload logic added to geom_subr.sh in r293028

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Integrate
tools/regression/geom_{concat,eli,gate,mirror,nop,raid3,shsec,stripe,uzip}
in to the FreeBSD test suite as
tests/sys/geom/class/{concat,eli,gate,mirror,nop,raid3,shsec,stripe,uzip}

The tools/regression/geom and tools/regression/geom_part testcases are being
left alone because both test sets are both currently broken.

The majority of this work was done on ^/user/ngie/more-tests2 . The differences
are as follows:
- tests/sys/geom/class/Makefile.inc is not present; it was
inlined into the class's Makefiles for explicitness.
- The testcases officially require root via kyua
- The geom_gate(4) tests don't use the pidfile changes proposed in
https://reviews.freebsd.org/D4836 .

MFC after: 1 month
Sponsored by: EMC / Isilon Storage Division

Rename `recvfd` and `sendfd` variables in recvfd/sendfd functions to avoid
-Wshadow issues with gcc

MFC after: 1 week
Reported by: bz, jenkins
Sponsored by: EMC / Isilon Storage Division

Integrate tools/regression/sockets/unix_passfd into the FreeBSD test
suite as tests/sys/kern/unix_passfd_test

- Convert testcases to ATF
- Fix an alignment issues
- Mark rights_creds_payload(..) as an expected failure (see PR # 181741)

Based [in part] on the following Differential Revision:
https://reviews.freebsd.org/D689

MFC after: 1 week
Submitted by: markj
Sponsored by: EMC / Isilon Storage Division

Add ptrace(2) reporting for LWP events.

Add two new LWPINFO flags: PL_FLAG_BORN and PL_FLAG_EXITED for reporting
thread creation and destruction. Newly created threads will stop to report
PL_FLAG_BORN before returning to userland and exiting threads will stop to
report PL_FLAG_EXIT before exiting completely. Both of these events are
only enabled and reported if PT_LWP_EVENTS is enabled on a process.

Remove retval to fix a -Wunused-but-set-variable warning from gcc 4.9

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

- Remove unused but set ssize in shutdown_send_sigpipe
- Add #ifdef TEST_SEQ_PACKET_SOURCE_ADDRESS` for untestable code
because FreeBSD doesn't have a means to map source addresses for
SEQ_PACKET AF_UNIX sockets (paraphrased). Put pathname variable
under the #ifdef to mute another unused but set variable warning

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Clean trailing whitespace

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

- Fix an improperly sized buffer for `pathname` [1]
- Fix a -Wunused-but-set-variable warning [2]

MFC after: 1 week
Reported by: cppcheck [1], gcc 4.9 [2]
Sponsored by: EMC / Isilon Storage Division

Fix style(9) a bit and ensure that error from initializing kqueue(2) is
sane

- Push the kqueue(2) initialization down so the errno will correspond with
the failure instead of potentially being stomped on by functions called
by `PLAIN_REQUIRE_KERNEL_MODULE`
- Delete trailing whitespace
- Add spaces between braces for conditional and control blocks (for/if)
- Use err/errx instead of perror+printf+exit/printf+exit.
- Remove braces for single-line conditionals

Tested with and without -DDEBUG

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Place cancel and error under #ifdef DEBUG to mute
-Wunused-but-set-variable warnings reported by gcc 4.9

Remove some trailing whitespace as well

Tested with and without -DDEBUG

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Delete the comment about running `test_libugidfw_strings` before testing
`mac_is_present` so it doesn't accidentally confuse people

MFC after: 3 days
X-MFC with: r292650
Sponsored by: EMC / Isilon Storage Division

Move mac_bsdextended check up before running the test_libugidfw_strings testcases

I realize that these tests could be run before mac_bsdextended is loaded, but
it would overcomplicate things to special case handle the testcases before doing
the mac_bsdextended(4) feature check

The testcases will be split up so they can be run separately in the near future

MFC after: 3 days
Sponsored by: EMC / Isilon Storage Division

Dump out the output from flock_helper on failure so failures with the
test app can be debugged

MFC after: 1 week
Obtained from: Isilon OneFS (^/onefs/head@r511419)
Sponsored by: EMC / Isilon Storage Division

Integrate tools/regression/mac/mac_bsdextended and
tools/regression/mac/mac_portacl into the FreeBSD test suite as
tests/sys/mac/bsdextended and tests/sys/mac/portacl, respectively

MFC after: 1 month
Sponsored by: EMC / Isilon Storage Division

Add ATF_REQUIRE_FEATURE and PLAIN_REQUIRE_FEATURE macros for
testing for kernel features via the feature_present(3) libcall

The semantics are similar to the other macros in the header (skip
testcase with ATF macro; exit with appropriate exit code with the
PLAIN macro)

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Fix LDADD/DPADD that should be LIBADD.

Sponsored by: EMC / Isilon Storage Division

Disable mqueue test # 3 and # 4 until __mq_oshandle is properly publicized

This will unbreak the test script and make things green again after r291440

Sponsored by: EMC / Isilon Storage Division

Disable a couple of tests, perhaps temporarily, since they use private
symbols that are not exported from librt.

Avoid requiring 'make depend' here.

Really this should not be a DPSRCS. The acct_test.c should not #include
convert.c, but just link it in as a normal SRCS.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Integrate contrib/netbsd-tests/kernel/t_mqueue into the FreeBSD test
suite as tests/sys/kern/mqueue_test

MFC after: 1 week

Fix up convert.c generation

- Use a temporary file for convert.c to reduce likelihood of an interrupted
build resulting in bad code being written to convert.c
- Truncate the file instead of appending to it to ensure that the file being
touched will not result in duplicate declarations/definitions from
kern_acct.c if/when kern_acct.c changes.

MFC after: 1 week

Copy README into /usr/tests

Add a few sentences describing how to run the tests.

Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D4224

Integrate tools/regression/pipe in to the FreeBSD test suite as
tests/sys/kern/pipe

- Fix style(9) bugs
- Fix compiler warnings
- Use `nitems(x)` instead of `sizeof(x) / sizeof(*x)` pattern

The testcases will be converted over to ATF eventually, but for now will be
integrated in as plain C tests

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Integrate contrib/netbsd-tests/kernel/t_lockf.c into the FreeBSD test suite as
tests/sys/kern/lockf_test

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Integrate acct(2) testcase in as tests/sys/kern/acct/acct_test

The :encode_tv_random_million testcase fails the epsilon tests a few thousand
times out of one million, so expect the testcase to fail

MFC after: 1 week
Submitted by: keramida
Sponsored by: EMC / Isilon Storage Division

Add tests for the copyin(9) handling of illegal buffers.

Reviewed by: emaste, ngie
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D3925

Integrate tools/regression/acltools into the FreeBSD test suite as tests/sys/acl

- Make the requirements more complete for the testcases
- Detect prerequisites so the tests won't fail (zfs.ko is loaded, zpool(1)
is available, ACL support is enabled with UFS, etc).
- Work with temporary files/directories/mountpoints that work with atf/kyua
- Limit the testcases to work on temporary filesystems to reduce tainting the
test host

MFC after: 2 weeks
Reviewed by: trasz (earlier version)
Differential Revision: https://reviews.freebsd.org/D3810

Integrate tools/test/posixshm and tools/regression/posixshm into the FreeBSD
test suite as tests/sys/posixshm

Some other highlights:
- Convert the testcases over to ATF
- Don't use hardcoded paths to /tmp (which violate the ATF/kyua samdbox); use
mkstemp to generate temporary paths for non-SHM_ANON shm objects.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Add more SUBDIR_PARALLEL.

MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division

Integrate tools/regression/vfs into the FreeBSD test suite as tests/sys/vfs

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Refactor the test/ Makefiles after recent changes to bsd.test.mk (r289158) and
netbsd-tests.test.mk (r289151)

- Eliminate explicit OBJTOP/SRCTOP setting
- Convert all ad hoc NetBSD test integration over to netbsd-tests.test.mk
- Remove unnecessary TESTSDIR setting
- Use SRCTOP where possible for clarity

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Divison

Tweak: use 'mainlwp' instead of 'mainpid' since this is a thread (LWP)
identifier, not a pid.

Fix build with older GCC which, doesn't like 'main' being a variable name.

Fix various edge cases related to system call tracing.
- Always set td_dbg_sc_* when P_TRACED is set on system call entry
even if the debugger is not tracing system call entries. This
ensures the fields are valid when reporting other stops that
occur at system call boundaries such as for PT_FOLLOW_FORKS or
when only tracing system call exits.
- Set TDB_SCX when reporting the stop for a new child process in
fork_return(). This causes the event to be reported as a system
call exit.
- Report a system call exit event in fork_return() for new threads in
a traced process.
- Copy td_dbg_sc_* to new threads instead of zeroing. This ensures
that td_dbg_sc_code in particular will report the system call that
created the new thread or process when it reports a system call
exit event in fork_return().
- Add new ptrace tests to verify that new child processes and threads
report system call exit events with a valid pl_syscall_code via
PT_LWPINFO.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D3822

Revert r288682

I meant to do this on ^/user/ngie/more-tests

Pointyhat to: ngie (use svn info next time...)

Remove some paths preparing for a re-copy from head

Use _exit() instead of exit() in child processes created during tests.

Suggested by: kib

Add a test to verify that a traced process sees its original parent via
getppid() after a debugger process that is not the parent has attached.

Reviewed by: kib (earlier version)
Differential Revision: https://reviews.freebsd.org/D3615

Properly size the children[] arrays in the follow fork tests.

Add more mmap tests related to character devices.
- Add cdev-related tests for bad args.
- Add two simple tests cases for mapping /dev/zero that test for
MAP_ANON-like behavior.

Reviewed by: alc, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D3323

Add various tests to ensure that invalid arguments passed to mmap()
trigger failures.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D3269

Convert the map_at_zero test case to ATF. In particular, this will
facilitate adding more mmap() tests.

MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D3268

Clear P_TRACED before reparenting a detached process back to its
original parent. Otherwise the debugee will be set as an orphan of
the debugger.

Add tests for tracing forks via PT_FOLLOW_FORK.

Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D2809

Fixed shutdown(2) unix(4) tests for SOCK_SEQPACKET after r285910 (by ed).

Pass unsigned long argument to ioctl().

Eliminates "ioctl sign-extension" warnings.

PR: 200896

The bug caught by flock's test 16 was fixed by r268384.

MFC after: 3 days

Make cleanup routines idempotent

cleanup routines can be executed at any point during the execution of the
body, including even before the body has done any real work. In those
cases, cleanup routines should be careful to not raise spurious errors so
as to not "override" the actual result of the test case.

This is just general good coding style but is not a problem in practice
for these specific tests. (The way I discovered the issue, though, was
due to a regression I introduced in Kyua itself while refactoring some
internals.)

MFC after: 1 week

Misc fixes from projects/bmake

Differential Revision: D2748
Reviewed by: brooks imp

Add a CHILD_REQUIRE macro similar to ATF_REQUIRE for use in child processes
of the main test process.

Differential Revision: https://reviews.freebsd.org/D2664
Reviewed by: ngie (previous version)

Append to SUBDIR, not set it

Pointyhat to: ngie

Consistently only use one end of the pipe in the parent and debugger
processes and do not rely on EOF due to a close() in the debugger.

PR: 200489
Differential Revision: https://reviews.freebsd.org/D2674
Reviewed by: kib, ngie, rodrigc

Tweak the description of when waitpid() doesn't return any status for a
non-blocking wait to avoid the word "empty".

Requested by: ngie

Do not allow a process to reap an orphan (a child currently being
traced by another process such as a debugger). The parent process does
need to check for matching orphan pids to avoid returning ECHILD if an
orphan has exited, but it should not return the exited status for the
child until after the debugger has detached from the orphan process
either explicitly or implicitly via wait().

Add two tests for for this case: one where the debugger is the direct
child (thus the parent has a non-empty children list) and one where
the debugger is not a direct child (so the only "child" of the parent
is the orphan).

Differential Revision: https://reviews.freebsd.org/D2644
Reviewed by: kib
MFC after: 2 weeks

Only reparent a traced process to its old parent if the tracing process is
not the old parent. Otherwise, proc_reap() will leave the zombie in place
resulting in the process' status being returned twice to its parent.

Add test cases for PT_TRACE_ME and PT_ATTACH which are fixed by
this change.

Differential Revision: https://reviews.freebsd.org/D2594
Reviewed by: kib
MFC after: 2 weeks

Move all test integration pieces for etc/ from etc/ to tests/

This is being done to fix breakage with make distribution with read-only
source trees as make distribution doesn't use make obj like building
tests/ does in all cases

Reported by: Wolfgang Zenker <wolfgang@lyxys.ka.sub.org>
Suggested by: jhb
X-MFC with: r282059
MFC after: 1 week

Fix typo. It should have been atf_tc_skip, not atf_skip

Reported by: many, Jenkins
Pointyhat to: ngie
MFC after: 4 days

ATF_REQUIRE_KERNEL_MODULE: use atf_skip, not ATF_REQUIRE_MSG so the testcase
no longer bombs out
PLAIN_REQUIRE_KERNEL_MODULE: use printf + _exit, no err so the testcase no
longer bombs out if it prints to stderr

MFC after: 5 days

Adjust CFLAGS to find freebsd_test_suite/macros.h

MFC after: 6 days

Use PLAIN_REQUIRE_KERNEL_MODULE to require "mqueuefs"

MFC after: 6 days

- Use ATF_REQUIRE_KERNEL_MDOULE to require aio(4)
- Don't use /tmp as a basis for temporary files as it's outside of the ATF
sandbox
- Don't override MAX macro in sys/param.h

MFC after: 6 days

Use ATF_REQUIRE_KERNEL_MODULE instead of aio_available function

MFC after: 6 days

Add initial (unpolished) macros for interfacing with the FreeBSD test suite

This is very rough, but will be replaced/redesigned some time soon after I fix
the Jenkins breakage I introduced

MFC after: 6 days

Fill in the copyright boilerplate for the test program

MFC after: 6 days

Move tests/sys/kern/mmap_test to tests/sys/vm/mmap_test

As jhb noted, the actual mmap(2) implementation is under sys/vm, not
sys/kern/, so the correct logical place is tests/sys/vm/, not
tests/sys/kern/

X-MFC with: r282076
MFC after: 6 days

Integrate tools/regression/mmap into the FreeBSD test suite as
tests/sys/kern/mmap_test

MFC after: 1 week

Integrate tools/regression/aio/aiotest and tools/regression/aio/kqueue into the
FreeBSD test suite as tests/sys/aio

MFC after: 1 week

Integrate tools/regression/mqueue into the FreeBSD test suite as
tests/sys/mqueue

MFC after: 1 week

Integrate tools/regression/fifo into the FreeBSD test suite as tests/sys/fifo
and tools/regression/file into the FreeBSD test suite as tests/sys/file

MFC after: 1 week

Integrate tools/regression/kqueue into the FreeBSD test suite as
tests/sys/kqueue

MFC after: 1 week

Integrate tools/regression/execve into the FreeBSD test suite as
tests/sys/kern/execve

MFC after: 1 week

Add some new modes to OpenCrypto. These modes are AES-ICM (can be used
for counter mode), and AES-GCM. Both of these modes have been added to
the aesni module.

Included is a set of tests to validate that the software and aesni
module calculate the correct values. These use the NIST KAT test
vectors. To run the test, you will need to install a soon to be
committed port, nist-kat that will install the vectors. Using a port
is necessary as the test vectors are around 25MB.

All the man pages were updated. I have added a new man page, crypto.7,
which includes a description of how to use each mode. All the new modes
and some other AES modes are present. It would be good for someone
else to go through and document the other modes.

A new ioctl was added to support AEAD modes which AES-GCM is one of them.
Without this ioctl, it is not possible to test AEAD modes from userland.

Add a timing safe bcmp for use to compare MACs. Previously we were using
bcmp which could leak timing info and result in the ability to forge
messages.

Add a minor optimization to the aesni module so that single segment
mbufs don't get copied and instead are updated in place. The aesni
module needs to be updated to support blocked IO so segmented mbufs
don't have to be copied.

We require that the IV be specified for all calls for both GCM and ICM.
This is to ensure proper use of these functions.

Obtained from: p4: //depot/projects/opencrypto
Relnotes: yes
Sponsored by: FreeBSD Foundation
Sponsored by: NetGate

Use PROGS instead of PROG and remove unnecessary SRCS?= assignment

Using PROG instead of PROGS will in cases of high -j with -DNO_ROOT cause
the PROG to show up more than once as it's handling the SCRIPTS install case
in a recursive manner, separate from the non-recursive case

After the recent batch of commits to bsd.progs.mk to fix behavior with how
variables are defaulted to, explicitly setting SRCS for a PROG is no longer
required

MFC after: 1 week
Reviewed by: asomers
Phabric: D1130
Sponsored by: EMC / Isilon Storage Division

Remove unused tests/lib directory

MFC after: 1 week

Remove explicit KYUAFILE=yes from tests/sys/Makefile

This causes the directory traversing Kyuafile to be installed, which in turn
causes tests/sys/pjdfstest to always be run from /usr/tests/sys

Let KYUAFILE default to auto, so the file generated by suite.test.mk gets
installed instead

MFC after: 1 week

Integrate pjdfstest test suite execution into kyua

pjdfstest execution is opt-in and must be done as root due to some of the
assumptions made by the test suite and lack of error checking in the non-root
case

A description of how to execute pjdfstest with kyua is provided in
share/pjdfstest/README

Phabric: D824 (an earlier prototype patch)
MFC after: 1 month
Relnotes: yes
Sponsored by: EMC / Isilon Storage Division

Fix source address selection on unbound sockets in the presence of multiple
fibs. Use the mbuf's or the socket's fib instead of RT_ALL_FIBS. Fixes PR
187553. Also fixes netperf's UDP_STREAM test on a nondefault fib.

sys/netinet/ip_output.c
In ip_output, lookup the source address using the mbuf's fib instead
of RT_ALL_FIBS.

sys/netinet/in_pcb.c
in in_pcbladdr, lookup the source address using the socket's fib,
because we don't seem to have the mbuf fib. They should be the same,
though.

tests/sys/net/fibs_test.sh
Clear the expected failure on udp_dontroute.

PR: 187553
CR: https://reviews.freebsd.org/D772
MFC after: 3 weeks
Sponsored by: Spectra Logic

Abort the create_socket test if socket creation fails.

MFC after: 1 week
Reported by: Coverity
CID: 1232756

Numerous small fixes, mostly suggested by Coverity.

tests/sys/kern/unix_seqpacket_test.c
* Remove a duplicate error check in mk_pair_of_sockets
* Always close sockets in the success path of ATF test cases. Don't
bother with the error paths, because those are mostly assertions
anyway. Most of these socket leaks were reported by Coverity.
All of them are harmless, because each ATF test case runs in its
own process.
* Fix the len argument to send in shutdown_send and
shutdown_send_sigpipe. The old version was using sizeof a pointer
instead of sizeof the char array. Reported by Coverity.
* Change a few ATF_CHECK to ATF_REQUIRE if the test can't reasonably
continue past a failure.

Reported by: Coverity Scan
CID: 1229995, 1229991, 1229988, 1229994, 1229989, 1229992
CID: 1229993, 1229990, 1229984, 1229967, 1230005, 1229977
CID: 1229966, 1230004, 1229976
MFC after: 1 week
Sponsored by: Spectra Logic

Fix unintended KBI change from r264905. Add _fib versions of
ifa_ifwithnet() and ifa_ifwithdstaddr() The legacy functions will call the
_fib() versions with RT_ALL_FIBS, preserving legacy behavior.

sys/net/if_var.h
sys/net/if.c
Add legacy-compatible functions as described above. Ensure legacy
behavior when RT_ALL_FIBS is passed as fibnum.

sys/netinet/in_pcb.c
sys/netinet/ip_output.c
sys/netinet/ip_options.c
sys/net/route.c
sys/net/rtsock.c
sys/netinet6/nd6.c
Call with _fib() functions if we must use a specific fib, or the
legacy functions otherwise.

tests/sys/netinet/fibs_test.sh
tests/sys/netinet/udp_dontroute.c
Improve the udp_dontroute test. The bug that this test exercises is
that ifa_ifwithnet() will return the wrong address, if multiple
interfaces have addresses on the same subnet but with different
fibs. The previous version of the test only considered one possible
failure mode: that ifa_ifwithnet_fib() might fail to find any
suitable address at all. The new version also checks whether
ifa_ifwithnet_fib() finds the correct address by checking where the
ARP request goes.

Reported by: bz, hrs
Reviewed by: hrs
MFC after: 1 week
X-MFC-with: 264905
Sponsored by: Spectra Logic

Remove last two NO_MAN= in the tree. In both of these cases, MAN= is
what is needed.

Add regression test for PR kern/189088.

PR: kern/189088
MFC after: 3 weeks
Sponsored by: Spectra Logic

Fix a panic when removing an IP address from an interface, if the same address
exists on another interface. The panic was introduced by change 264887, which
changed the fibnum parameter in the call to rtalloc1_fib() in
ifa_switch_loopback_route() from RT_DEFAULT_FIB to RT_ALL_FIBS. The solution
is to use the interface fib in that call. For the majority of users, that will
be equivalent to the legacy behavior.

PR: kern/189089
Reported by: neel
Reviewed by: neel
MFC after: 3 weeks
X-MFC with: 264887
Sponsored by: Spectra Logic

Style fixes, mostly trailing whitespace elimination. No functional change.

Reported by: pho
MFC after: 3 weeks

Fix subnet and default routes on different FIBs on the same subnet.

These two bugs are closely related. The root cause is that ifa_ifwithnet
does not consider FIBs when searching for an interface address.

sys/net/if_var.h
sys/net/if.c
Add a fib argument to ifa_ifwithnet and ifa_ifwithdstadddr. Those
functions will only return an address whose interface fib equals the
argument.

sys/net/route.c
Update calls to ifa_ifwithnet and ifa_ifwithdstaddr with fib
arguments.

sys/netinet/in.c
Update in_addprefix to consider the interface fib when adding
prefixes. This will prevent it from not adding a subnet route when
one already exists on a different fib.

sys/net/rtsock.c
sys/netinet/in_pcb.c
sys/netinet/ip_output.c
sys/netinet/ip_options.c
sys/netinet6/nd6.c
Add RT_DEFAULT_FIB arguments to ifa_ifwithdstaddr and ifa_ifwithnet.
In some cases it there wasn't a clear specific fib number to use.
In others, I was unable to test those functions so I chose
RT_DEFAULT_FIB to minimize divergence from current behavior. I will
fix some of the latter changes along with PR kern/187553.

tests/sys/netinet/fibs_test.sh
tests/sys/netinet/udp_dontroute.c
tests/sys/netinet/Makefile
Revert r263738. The udp_dontroute test was right all along.
However, bugs kern/187550 and kern/187553 cancelled each other out
when it came to this test. Because of kern/187553, ifa_ifwithnet
searched the default fib instead of the requested one, but because
of kern/187550, there was an applicable subnet route on the default
fib. The new test added in r263738 doesn't work right, however. I
can verify with dtrace that ifa_ifwithnet returned the wrong address
before I applied this commit, but route(8) miraculously found the
correct interface to use anyway. I don't know how.

Clear expected failure messages for kern/187550 and kern/187552.

PR: kern/187550
PR: kern/187552
Reviewed by: melifaro
MFC after: 3 weeks
Sponsored by: Spectra Logic

Fix host and network routes for new interfaces when net.add_addr_allfibs=0

sys/net/route.c
In rtinit1, use the interface fib instead of the process fib. The
latter wasn't very useful because ifconfig(8) is usually invoked
with the default process fib. Changing ifconfig(8) to use setfib(2)
would be redundant, because it already sets the interface fib.

tests/sys/netinet/fibs_test.sh
Clear the expected ATF failure

sys/net/if.c
Pass the interface fib in calls to rtrequest1_fib and rtalloc1_fib

sys/netinet/in.c
sys/net/if_var.h
Add a fibnum argument to ifa_switch_loopback_route, a subroutine of
in_scrubprefix. Pass it the interface fib.

PR: kern/187549
Reviewed by: melifaro
MFC after: 3 weeks
Sponsored by: Spectra Logic Corporation

Fix variable type to avoid printf formatter warning.

This fixes the build under powerpc64 where gcc complains about a mismatch
between a %zd printf formatter and an int variable passed to it.

Correct ARP update handling when the routes for network interfaces are
restricted to a single FIB in a multifib system.

Restricting an interface's routes to the FIB to which it is assigned (by
setting net.add_addr_allfibs=0) causes ARP updates to fail with "arpresolve:
can't allocate llinfo for x.x.x.x". This is due to the ARP update code hard
coding it's lookup for existing routing entries to FIB 0.

sys/netinet/in.c:
When dealing with RTM_ADD (add route) requests for an interface, use
the interface's assigned FIB instead of the default (FIB 0).

sys/netinet/if_ether.c:
In arpresolve(), enhance error message generated when an
lla_lookup() fails so that the interface causing the error is
visible in logs.

tests/sys/netinet/fibs_test.sh
Clear ATF expected error.

PR: kern/167947
Submitted by: Nikolay Denev <ndenev@gmail.com> (previous version)
Reviewed by: melifaro
MFC after: 3 weeks
Sponsored by: Spectra Logic Corporation

tests/sys/netinet/Makefile
tests/sys/netinet/fibs.sh
Replace fibs:udp_dontroute with fibs:src_addr_selection_by_subnet.
The original test was poorly written; it was actually testing
kern/167947 instead of the desired kern/187553. The root cause of the
bug is that ifa_ifwithnet did not have a fib argument. The new test
more directly targets that behavior.

tests/sys/netinet/udp_dontroute.c
Delete the auxilliary binary used by the old test

PR: kern/187553
MFC after: 3 weeks
Sponsored by: Spectra Logic Corporation

Add several ATF tests that deal with multiple fibs. They're described in
several different PRs, but the tests share some common code, so I'm
committing them together.

sbin/ifconfig/tests
sbin/ifconfig/tests/fibs_test.sh
sbin/ifconfig/tests/Makefile
sbin/ifconfig/Makefile
Add fibs_test.sh, which regresses bin/187551

tests/sys/netinet
tests/sys/netinet/fibs_test.sh
tests/sys/netinet/udp_dontroute.c
tests/sys/netinet/Makefile
tests/sys/Makefile
Add fibs_test.sh, which regresses kern/167947, kern/187552
kern/187549, kern/187550, and kern/187553

etc/mtree/BSD.tests.dist
Add newly created directories

PR: bin/187551
PR: kern/167947
PR: kern/187552
PR: kern/187549
PR: kern/187550
PR: kern/187553
Discussed with: melifaro
MFC after: 3 weeks
Sponsored by: Spectra Logic Corporation

Added sysctl kern.maxfiles increase test, do not use /etc/passwd for tests
and use volatile sig_atomic_t for signal handler variable.

Reviewed by: asomers (previous version)
Sponsored by: EMC / Isilon storage division

Make bsd.test.mk the only public mk fragment for the building of tests.

Change {atf,plain,tap}.test.mk to be internal implementation details of
bsd.test.mk. Makefiles that build tests should now only include bsd.test.mk
and declaratively specify what they want to build, without worrying about
the internal implementation of the mk files.

The reason for this change is to permit building test programs of different
interfaces from a single directory, which is something I had a need for
while porting tests over from src/tools/regression/.

Additionally, this change makes it possible to perform some other requested
changes to bsd.test.mk in an easier manner. Coming soon.

Replace 4.4BSD Lite's unix domain socket backpressure hack with a cleaner
mechanism, based on the new SB_STOP sockbuf flag. The old hack dynamically
changed the sending sockbuf's high water mark whenever adding or removing
data from the receiving sockbuf. It worked for stream sockets, but it never
worked for SOCK_SEQPACKET sockets because of their atomic nature. If the
sockbuf was partially full, it might return EMSGSIZE instead of blocking.

The new solution is based on DragonFlyBSD's fix from commit
3a6117bbe0ed6a87605c1e43e12a1438d8844380 on 2008-05-27. It adds an SB_STOP
flag to sockbufs. Whenever uipc_send surpasses the socket's size limit, it
sets SB_STOP on the sending sockbuf. sbspace() will then return 0 for that
sockbuf, causing sosend_generic and friends to block. uipc_rcvd will
likewise clear SB_STOP. There are two fringe benefits: uipc_{send,rcvd} no
longer need to call chgsbsize() on every send and receive because they don't
change the sockbuf's high water mark. Also, uipc_sense no longer needs to
acquire the UIPC linkage lock, because it's simpler to compute the
st_blksizes.

There is one drawback: since sbspace() will only ever return 0 or the
maximum, sosend_generic will allow the sockbuf to exceed its nominal maximum
size by at most one packet of size less than the max. I don't think that's
a serious problem. In fact, I'm not even positive that FreeBSD guarantees a
socket will always stay within its nominal size limit.

sys/sys/sockbuf.h
Add the SB_STOP flag and adjust sbspace()

sys/sys/unpcb.h
Delete the obsolete unp_cc and unp_mbcnt fields from struct unpcb.

sys/kern/uipc_usrreq.c
Adjust uipc_rcvd, uipc_send, and uipc_sense to use the SB_STOP
backpressure mechanism. Removing obsolete unpcb fields from
db_show_unpcb.

tests/sys/kern/unix_seqpacket_test.c
Clear expected failures from ATF.

Obtained from: DragonFly BSD
PR: kern/185812
Reviewed by: silence from freebsd-net@ and rwatson@
MFC after: 3 weeks
Sponsored by: Spectra Logic Corporation

kern/Makefile
Set WARNS=5 for all files in this directory

kern/unix_seqpacket_test.c
Fix compiler warnings. Most were benign, but rcvbuf_oversized
wasn't working as intended because I forgot to set the buffer sizes.

MFC after: 2 weeks
Reported by: pho
Sponsored by: Spectra Logic Corporation

Only use -lpthread for unix_seqpacket_test, not for all test programs.

Reported by: Garrett Cooper
MFC after: 3 weeks
Sponsored by: Spectra Logic Corporation

Fix PR kern/185813 "SOCK_SEQPACKET AF_UNIX sockets with asymmetrical
buffers drop packets". It was caused by a check for the space available
in a sockbuf, but it was checking the wrong sockbuf.

sys/sys/sockbuf.h
sys/kern/uipc_sockbuf.c
Add sbappendaddr_nospacecheck_locked(), which is just like
sbappendaddr_locked but doesn't validate the receiving socket's
space. Factor out common code into sbappendaddr_locked_internal().
We shouldn't simply make sbappendaddr_locked check the space and
then call sbappendaddr_nospacecheck_locked, because that would cause
the O(n) function m_length to be called twice.

sys/kern/uipc_usrreq.c
Use sbappendaddr_nospacecheck_locked for SOCK_SEQPACKET sockets,
because the receiving sockbuf's size limit is irrelevant.

tests/sys/kern/unix_seqpacket_test.c
Now that 185813 is fixed, pipe_128k_8k fails intermittently due to
185812. Make it fail every time by adding a usleep after starting
the writer thread and before starting the reader thread in
test_pipe. That gives the writer time to fill up its send buffer.
Also, clear the expected failure message due to 185813. It actually
said "185812", but that was a typo.

PR: kern/185813
Reviewed by: silence from freebsd-net@ and rwatson@
MFC after: 3 weeks
Sponsored by: Spectra Logic Corporation

Preserve naming consistency for test cases.

Pointed out by: jmmv
Sponsored by: EMC / Isilon storage division

Changed name of test case to a more descriptive one and moved
comment to the "descr" property.

Suggested by: jmmv
Sponsored by: EMC / Isilon storage division

Added a regression test for r234131.

Discussed with: asomers
Sponsored by: EMC / Isilon storage division

test_eagain_*_* should've been using nonblocking sockets instead of
blocking sockets. The error was not exposed as long as the kernel
suffered from PR kern/185812. Now corrected, these tests pass on
DragonFlyBSD 3.6.0.

PR: kern/185812
Sponsored by: Spectra Logic Corporation
MFC after: 2 weeks

Use DESTDIR for the installation of the /usr/tests/local symlink.

MFC after: 5 days

Install a symlink from /usr/tests/local to /usr/local/tests.

This is to let Kyua descend into any tests that may have been installed by
ports under /usr/local/tests when running the test suite from /usr/tests.

Some ports (namely those that build Kyua) already install test programs
into /usr/local/tests. Just make sure to select the TEST option while
building them.

MFC after: 3 days

tests/sys/Makefile
use TESTS_SUBDIRS for kern instead of SUBDIRS. I don't think it
makes a difference in this case, but TESTS_SUBDIRS is generally
correct for subdirectories that contain tests.

Sponsored by: Spectra Logic
MFC after: 5 days
X-MFC-With: r261133

Fix the Makefiles so that the tests I submitted in r261081 will actually
get built.

Sponsored by: Spectra Logic
MFC after: 13 days
X-MFC-With: r261081

Replace the old unix_seqpacket and unix_seqpacket_exercise tests, which
were a little broken and not automatable, with unix_seqpacket_test.
It's coverage is a superset of the old tests and it uses ATF. It
includes test cases for bugs kern/185813 and kern/185812.

PR: kern/185812
PR: kern/185813
Sponsored by: Spectra Logic
MFC after: 2 weeks

Install category Kyuafiles from their category directories.

Move the installation of /usr/tests/lib/Kyuafile from src/tests/lib/
to src/lib/. This is to keep the src/tests/ hierarchy unaware of the
rest of the tree, which makes things clearer in general. In particular:

1) Everything related to the construction of /usr/tests/lib/ is kept
in src/lib/. There is no need to think about different directories
and how they relate to each other. (The same applies for libexec,
usr.bin, etc. but these are not yet handled.)

2) src/tests becomes the place to keep cross-functional test programs
and nothing else, which also helps in simplifying things.

Reviewed by: freebsd-testing
Approved by: rpaulo (mentor)

Set up the /usr/tests hierarchy.

Populate /usr/tests with the only test programs that currently live
in the tree (those in lib/libcrypt/tests/) and add all the build
machinery to accompany this change.

In particular:

- Add a WITHOUT_TESTS variable that users can define to request that
no tests be put in /usr/tests.
- Add a top-level Kyuafile for /usr/tests and a way to create similar
Kyuafiles in top-level subdirectories.
- Add a BSD.tests.dist file to define the directory layout of
/usr/tests.

Submitted by: Julio Merino jmmv google.com
Reviewed by: sjg
MFC after: 2 weeks