MFC r311954, r311996, r312077, r312080:

Rework tty_drain() to poll the hardware for completion, and restore
drain timeout handling to historical freebsd behavior.

The primary reason for these changes is the need to have tty_drain() call
ttydevsw_busy() at some reasonable sub-second rate, to poll hardware that
doesn't signal an interrupt when the transmit shift register becomes empty
(which includes virtually all USB serial hardware). Such hardware hangs
in a ttyout wait, because it never gets an opportunity to trigger a wakeup
from the sleep in tty_drain() by calling ttydisc_getc() again, after
handing the last of the buffered data to the hardware.

Restructure the tty_drain loop so that device-busy is checked one more time
after tty_timedwait() returns an error only if the error is EWOULDBLOCK;
other errors cause an immediate return. This fixes the case of the tty
disappearing while in tty_drain().

Check tty_gone() after allocating IO buffers. The tty lock has to be
dropped then reacquired due to using M_WAITOK, which opens a window in
which the tty device can disappear. Check for this and return ENXIO
back up the call chain so that callers can cope.

Correct the comments about how much buffer is allocated.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Use strchr() and strrchr().

It seems strchr() and strrchr() are used more often than index() and
rindex(). Therefore, simply migrate all kernel code to use it.

For the XFS code, remove an empty line to make the code identical to
the code in the Linux kernel.

Fix whitespace inconsistencies in the TTY layer and its drivers owned by me.

Remove statistics from the TTY queues.

I added counters to see how often fast copying to userspace was actually
performed, which was only useful during development. Remove these
statistics now we know it to be effective.

Remove a dead initialization.

Spotted by: scan-build (uqs)

Enable secure TTY input buffer flushing by default.

I'm leaving the sysctl there. If people really notice a slowdown, they
can revert to the old behaviour.

Discussed with: kib

Add a new sysctl: kern.tty_inq_flush_secure.

When enabled all TTY input queue buffers are zeroed when flushing or
closing the TTY. Because TTY input queues are also used to store filled
in passwords, this may be an interesting switch to enable for security
minded people.

Use unsigned longs for the TTY's sysctl stats.

Spotted by: clang

Slightly improve the design of the TTY buffer.

The TTY buffers used the standard <sys/queue.h> lists. Unfortunately
they have a big shortcoming. If you want to have a double linked list,
but no tail pointer, it's still not possible to obtain the previous
element in the list. Inside the buffers we don't need them. This is why
I switched to custom linked list macros. The macros will also keep track
of the amount of items in the list. Because it doesn't use a sentinel,
we can just initialize the queues with zero.

In its simplest form (the output queue), we will only keep two
references to blocks in the queue, namely the head of the list and the
last block in use. All free blocks are stored behind the last block in
use.

I noticed there was a very subtle bug in the previous code: in a very
uncommon corner case, it would uma_zfree() a block in the queue before
calling memcpy() to extract the data from the block.

Fix some edge cases in the TTY queues:

- In the current design, when a TTY decreases its baud rate, it tries to
shrink the queues. This may not always be possible, because it will
not free any blocks that are still filled with data.

Change the TTY queues to store a `quota' value as well, which means it
will not free any blocks when changing the baud rate, but when placing
blocks back into the queue. When the amount of blocks exceeds the
quota, they get freed.

It also fixes some edge cases, where TIOCSETA during read()/
write()-calls could actually make the queue a tiny bit bigger than in
normal cases.

- Don't leak blocks of memory when calling TIOCSETA when the device
driver abandons the TTY while allocating memory.

- Create ttyoutq_init() and ttyinq_init() to initialize the queues,
instead of initializing them by hand. The new TTY snoop driver also
creates an outq, so it's good to have a proper interface to do this.

Obtained from: //depot/projects/mpsafetty/...

Integrate the new MPSAFE TTY layer to the FreeBSD operating system.

The last half year I've been working on a replacement TTY layer for the
FreeBSD kernel. The new TTY layer was designed to improve the following:

- Improved driver model:

The old TTY layer has a driver model that is not abstract enough to
make it friendly to use. A good example is the output path, where the
device drivers directly access the output buffers. This means that an
in-kernel PPP implementation must always convert network buffers into
TTY buffers.

If a PPP implementation would be built on top of the new TTY layer
(still needs a hooks layer, though), it would allow the PPP
implementation to directly hand the data to the TTY driver.

- Improved hotplugging:

With the old TTY layer, it isn't entirely safe to destroy TTY's from
the system. This implementation has a two-step destructing design,
where the driver first abandons the TTY. After all threads have left
the TTY, the TTY layer calls a routine in the driver, which can be
used to free resources (unit numbers, etc).

The pts(4) driver also implements this feature, which means
posix_openpt() will now return PTY's that are created on the fly.

- Improved performance:

One of the major improvements is the per-TTY mutex, which is expected
to improve scalability when compared to the old Giant locking.
Another change is the unbuffered copying to userspace, which is both
used on TTY device nodes and PTY masters.

Upgrading should be quite straightforward. Unlike previous versions,
existing kernel configuration files do not need to be changed, except
when they reference device drivers that are listed in UPDATING.

Obtained from: //depot/projects/mpsafetty/...
Approved by: philip (ex-mentor)
Discussed: on the lists, at BSDCan, at the DevSummit
Sponsored by: Snow B.V., the Netherlands
dcons(4) fixed by: kan