MFC r315418,r315480,r316019:

Add teken_256to16() to convert xterm-256 256-color codes to xterm 16-color
codes. This will be used to fix bright colors.

Improve teken_256to8(). Use a lookup table instead of calculations. The
calculations were inaccurate since they used indexes into the xterm-256
6x6x6 color map instead of actual xterm colors. Also, change the threshold
for converting to a primary color: require the primary's component to be
2 or more higher instead of just higher. This affects about 1/5 of the
table entries and gives uniformly distributed colors in the 6x6x6 submap
except for greys (35 entries each for red, green, blue, cyan, brown and
magenta, instead of approx. only 15 each for the mixed colors). Even
more mixed colors would be better for matching colors, but uniform
distribution is best for preserving contrast.

For teken_256to16(), bright colors are just the ones with luminosity >=
60%. These are actually light colors (more white instead of more
saturation), while xterm bright colors except for white itself are
actually saturated with no white, so have luminosity only 50%.

These functions are layering violations. teken cannot do correct
conversions since it shouldn't know the color maps of anything except
xterm. Translating through xterm-16 colors loses information. This
gives bugs like xterm-256 near-brown -> xterm-16 red -> VGA red.

------------------------------------------------------------------------
r315480 | bde | 2017-03-18 11:13:54 +0000 (Sat, 18 Mar 2017) | 51 lines

Fix bright colors for syscons, and make them work for the first time
for vt. Restore syscons' rendering of background (bg) brightness as
foreground (fg) blinking and vice versa, and add rendering of blinking
as background brightness to vt.

Bright/saturated is conflated with light/white in the implementation
and in this description.

Bright colors were broken in all cases, but appeared to work in the
only case shown by "vidcontrol show". A boldness hack was applied
only in 1 layering-violation place (for some syscons sequences) where
it made some cases seem to work but was undone by clearing bold using
ANSI sequences, and more seriously was not undone when setting
ANSI/xterm dark colors so left them bright. Move this hack to drivers.

The boldness hack is only for fg brightness. Restore/add a similar hack
for bg brightness rendered as fg blinking and vice versa. This works
even better for vt, since vt changes the default text mode to give the
more useful bg brightness instead of fg blinking.

The brightness bit in colors was unnecessarily removed by the boldness
hack. In other cases, it was lost later by teken_256to8(). Use
teken_256to16() to not lose it. teken_256to8() was intended to be
used for bg colors to allow finer or bg-specific control for the more
difficult reduction to 8; however, since 16 bg colors actually work
on VGA except in syscons text mode and the conversion isn't subtle
enough to significantly in that mode, teken_256to8() is not used now.

There are still bugs, especially in vidcontrol, if bright/blinking
background colors are set.

Restore XOR logic for bold/bright fg in syscons (don't change OR
logic for vt). Remove broken ifdef on FG_UNDERLINE and its wrong
or missing bit and restore the correct hard-coded bit. FG_UNDERLINE
is only for mono mode which is not really supported.

Restore XOR logic for blinking/bright bg in syscons (in vt, add
OR logic and render as bright bg). Remove related broken ifdef
on BG_BLINKING and its missing bit and restore the correct
hard-coded bit. The same bit means blinking or bright bg depending
on the mode, and we want to ignore the difference everywhere.

Simplify conversions of attributes in syscons. Don't pretend to
support bold fonts. Don't support unusual encodings of brightness.
It is as good as possible to map 16 VGA colors to 16 xterm-16
colors. E.g., VGA brown -> xterm-16 Olive will be converted back
to VGA brown, so we don't need to convert to xterm-256 Brown. Teken
cons25 compatibility code already does the same, and duplicates some
small tables. This is mostly for the sc -> te direction. The other
direction uses teken_256to16() which is too generic.

------------------------------------------------------------------------
r316019 | bde | 2017-03-27 10:48:28 +0000 (Mon, 27 Mar 2017) | 30 lines

Oops, my fix for bright colors broke bright black some more (in cases
that used to work via the bold hack).

Fix the table entry for bright black. Fix spelling of plain black in
nearby table entries (use the macro for black everywhere everywhere).
Fix the currently-unused non-bright color table to not have bright
colors in entries 9-15.

Improve nearby comments. Start converting to the xterm terminology
and default rendering of "bright" instead of "light" for bright
colors.

Syscons wasn't affected by the bug since I optimized it a little by
converting colors 0-15 directly. This also fixes the layering of
the conversion for these colors.

Apply the same optimization to vt (actually the layer above it). This
also moves the conversion 1 closer to the correct layer for colors
0-15.

The optimization of just avoiding 2 calls to a trivial function is worth
about 10% for simple output to the virtual buffer with occasional
rendering. The optimization is so large because the 2 calls are done
on every character, so although there are too many other calls and
other instructions per character, there are only about 10 times as
many. Old versions of syscons were about 10 times faster for simple
output, by using a fast path with about 12 instructions per character.
Rendering to even slow hardware takes relatively little time provided
it is rarely actually done.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Partially implement the mysterious cons25 \e[x escape sequence.

It seems the terminfo library on some systems (OS X, Linux) may emit the
sequence \e[x to reset to default attributes. Apart from using the
zero-command, this escape sequence allows many more operations, such as
setting ANSI colors. I don't see this used anywhere, so this should be
sufficient for now.

This deficiency was spotted by the Debian GNU/kFreeBSD. They have their
own patch, which is slightly flawed in my opinion. I don't know why they
never reported this issue to us.

MFC after: 1 week

Add 256 color support.

It is quite inconvenient that if an application for xterm uses 256 color
mode, text suddenly starts to blink (because of ;5; in the middle).
We'd better just implement 256 color mode and add a conversion routine
from 256 to 8 color mode, which doesn't seem to be too bad in practice.

Remapping colors is done quite simple. If one of the channels is most
actively represented, primary colors are used. If two channels are most
actively represented, secondary colors are used. If all three channels
are equal (gray), it picks between black and white.

Reported by: Paul B. Mahol <onemda gmail com>

Add a new escape sequence to switch between cons25 and xterm.

Just run this to switch to xterm:

printf '\e[=T'

If you get bored and want to switch back to cons25, run this:

printf '\e[=1T'

I can now send an email to the lists, asking whether people are
interested in trying the xterm emulator.

Commit all local modifications I have to libteken:

- Make xterm/cons25 support runtime configurable. This allows me to
share libteken between syscons and my new vt driver.
- Add a fix to print blanks after printing a double width character to
prevent rendering artifacts.
- Add some more utility functions that I use in the vt driver.

Move libteken out of the syscons directory.

I initially committed libteken to sys/dev/syscons/teken, but now that
I'm working on a console driver myself, I noticed this was not a good
decision. Move it to sys/teken to make it easier for other drivers to
use a terminal emulator.

Also list teken.c in sys/conf/files, instead of listing it in all the
files.arch files separately.

Restore support for bell pitch/duration.

Because we only support a single argument to tf_param, use 16 bits for
the pitch and 16 bits for the duration. While there, make the argument
unsigned. There isn't a single param call that needs a signed integer.

Submitted by: danfe (modified)

Don't forget to mark the color translation array as const.

Spotted by: Christoph Mallon <christoph mallon gmx de>

Fix for my previous commit: color mapping is not 1:1.

Cons25 doesn't seem to use a straight 1:1 mapping to the ANSI colors,
but uses the same color numbers as at least used by syscons on i386. I
suspect if you change the definitions on a different architecture,
things may break? Not sure.

Add a small array to convert syscons-style color codes to ANSI
equivalents, which are used by libteken internally. I didn't notice this
bug, because I only tested my code with black, white and green, all of
them shared the same numbers.

Make vidcontrol's color setting work again.

It turns out I forgot to implement two escape sequences that allows the
user to change the default foreground and background colors. I thought
they were implemented by syscons itself, but vidcontrol just generates
some escape sequences, which get interpreted by the terminal emulator.

Reported by: mgp (forums)

Replace syscons terminal renderer by a new renderer that uses libteken.

Some time ago I started working on a library called libteken, which is
terminal emulator. It does not buffer any screen contents, but only
keeps terminal state, such as cursor position, attributes, etc. It
should implement all escape sequences that are implemented by the
cons25 terminal emulator, but also a fair amount of sequences that are
present in VT100 and xterm.

A lot of random notes, which could be of interest to users/developers:

- Even though I'm leaving the terminal type set to `cons25', users can
do experiments with placing `xterm-color' in /etc/ttys. Because we
only implement a subset of features of xterm, this may cause
artifacts. We should consider extending libteken, because in my
opinion xterm is the way to go. Some missing features:

- Keypad application mode (DECKPAM)
- Character sets (SCS)

- libteken is filled with a fair amount of assertions, but unfortunately
we cannot go into the debugger anymore if we fail them. I've done
development of this library almost entirely in userspace. In
sys/dev/syscons/teken there are two applications that can be helpful
when debugging the code:

- teken_demo: a terminal emulator that can be started from a regular
xterm that emulates a terminal using libteken. This application can
be very useful to debug any rendering issues.

- teken_stress: a stress testing application that emulates random
terminal output. libteken has literally survived multiple terabytes
of random input.

- libteken also includes support for UTF-8, but unfortunately our input
layer and font renderer don't support this. If users want to
experiment with UTF-8 support, they can enable `TEKEN_UTF8' in
teken.h. If you recompile your kernel or the teken_demo application,
you can hold some nice experiments.

- I've left PC98 the way it is right now. The PC98 platform has a custom
syscons renderer, which supports some form of localised input. Maybe
we should port PC98 to libteken by the time syscons supports UTF-8?

- I've removed the `dumb' terminal emulator. It has been broken for
years. It hasn't survived the `struct proc' -> `struct thread'
conversion.

- To prevent confusion among people that want to hack on libteken:
unlike syscons, the state machines that parse the escape sequences are
machine generated. This means that if you want to add new escape
sequences, you have to add an entry to the `sequences' file. This will
cause new entries to be added to `teken_state.h'.

- Any rendering artifacts that didn't occur prior to this commit are by
accident. They should be reported to me, so I can fix them.

Discussed on: current@, hackers@
Discussed with: philip (at 25C3)