- Copy stable/10 (r259064) to releng/10.0 as part of the
10.0-RELEASE cycle.
- Update __FreeBSD_version [1]
- Set branch name to -RC1

[1] 10.0-CURRENT __FreeBSD_version value ended at '55', so
start releng/10.0 at '100' so the branch is started with
a value ending in zero.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Link the demo application against ncursesw to make Unicode work.

Properly use the cursor to bound the position for CUP.

We must take the origin region into account when clamping the cursor
position.

MFC after: 3 days

Fix whitespace inconsistencies found in homegrown Symbol.maps.

Simply let teken_stress use arc4random.

This makes it run quite a bit faster, since it makes system calls less
often.

Tab should not blank cells.

It seems I was under the impression that a tab differs from a single
forward tabulation, namely that it blanks the underlying cells. This
seems not to be the case. They are identical.

This should fix applications like jove(1) that use tabs instead of
explicit cursor position setting.

Reported by: Brett Glass <brett lariat net>
MFC after: 3 days, after it's tested

Fix various whitespace inconsistencies in sys/teken.

mdoc: reorder sections consistently

Add proper build infrastructure for teken.

I'm not sure whether we should install teken as a library on any stock
FreeBSD installation, but I can imagine people want to tinker around
with it now and then. Create a /sys/teken/libteken, which holds a
Makefile to install a shared library version of the terminal emulator,
complete with a manpage.

Also add Makefiles for the demo/stress applications, to build it against
the shared library.

Use proper bounds checking on VPA.

We must check against tp->t_cursor.tp_row, not row, to figure out
whether we must clamp the cursor position.

Submitted by: luigi
MFC after: 3 weeks

Partially implement the mysterious cons25 \e[x escape sequence.

It seems the terminfo library on some systems (OS X, Linux) may emit the
sequence \e[x to reset to default attributes. Apart from using the
zero-command, this escape sequence allows many more operations, such as
setting ANSI colors. I don't see this used anywhere, so this should be
sufficient for now.

This deficiency was spotted by the Debian GNU/kFreeBSD. They have their
own patch, which is slightly flawed in my opinion. I don't know why they
never reported this issue to us.

MFC after: 1 week

Make ^L with cons25 and origin mode bit more sane.

Even though cons25 normally doesn't support origin regions, this
emulator does allow you to do it. It makes more sense to blank only the
origin region when emitting ^L instead of blanking the entire screen.
Apart from that, we should always place the cursor inside the origin
region, which doesn't happen right now.

Just use <stdint.h> instead of <inttypes.h>. We don't need it here.

Fall back to ASCII codepoints for box drawing.

Even though the default VGA font provides box drawing fonts, there is no
guarantee any font will provide these as well (i.e. ISO-8859-*, KOI8-R).
Just use ASCII characters for box drawing.

PR: kern/141633

Place home and end before insert and delete.

These keys have different sequences when using cursorkeys, while insert
and delete stay the same. If they are placed like this, libteken will
return NULL instead of a proper sequence for these characters.

Allow Syscons terminal emulators to provide function key strings.

xterm and cons25 have some incompatibilities when it comes to escape
sequences for special keys, such as F1 to F12, home, end, etc. Add a new
te_fkeystr() that can be used to override the strings.

scterm-sck won't do anything with this, but scterm-teken will use
teken_get_sequences() to obtain the proper sequence.

Always home the cursor when changing the scrolling region.

I thought this only had to be done when in origin mode, to ensure that
the cursor is not placed outside the origin, but it seems this is also
done when not in origin mode.

This fixes some artifacts when pressing ^L while running irssi in tmux.
(Almost) nobody noticed this, because cons25 doesn't have scrolling
regions.

Discard Device Control Strings and Operating System Commands.

These strings often contain things like:

- Window titles.
- Extended key map functionality.
- Color palette switching.

We could look at these features in the future (if people consider them
to be important enough), but we'd better discard them now. This fixes
some artifacts people reported when using TERM=xterm.

Reported by: des@, Paul B. Mahol

Add support for VT200-style mouse input.

Right now if applications want to use the mouse on the command line,
they use sysmouse(4) and install a signal handler in the kernel to
deliver signals when mouse events arrive. This conflicts with my plan to
change to TERM=xterm, so implement proper VT200-style mouse input.

Because mouse input is now streamed through the TTY, it means you can
now SSH to another system on the console and use the mouse there as
well. The disadvantage of the VT200 mouse protocol, is that it doesn't
seem to generate events when moving the cursor. Only when pressing and
releasing mouse buttons.

There are different protocols as well, but this one seems to be most
commonly supported.

Reported by: Paul B. Mahol <onemda gmail com>
Tested with: vim(1)

Add 256 color support.

It is quite inconvenient that if an application for xterm uses 256 color
mode, text suddenly starts to blink (because of ;5; in the middle).
We'd better just implement 256 color mode and add a conversion routine
from 256 to 8 color mode, which doesn't seem to be too bad in practice.

Remapping colors is done quite simple. If one of the channels is most
actively represented, primary colors are used. If two channels are most
actively represented, secondary colors are used. If all three channels
are equal (gray), it picks between black and white.

Reported by: Paul B. Mahol <onemda gmail com>

Properly get out of origin mode if the cursor has to move outside of it.

In some cases events may occur that move the cursor outside the
scrolling region while in origin mode, which is normally not possible.
Events like these include:

- Alignment test.
- Restore cursor.

Properly switch off origin mode in these cases.

MFC after: 1 month

Get rid of now deprecated SCS wrappers.

We always build SCS, even when processing 8-bit data. There is no reason
why we should be able to disable it now.

Make the fuzzer a bit more useful by forcing 7-bit data into it.

Getting valid UTF-8 sequences is quite unlikely, so we'd better just
convert data to 7 bits and make it extra likely for escape sequences to
occur.

Add a new escape sequence to switch between cons25 and xterm.

Just run this to switch to xterm:

printf '\e[=T'

If you get bored and want to switch back to cons25, run this:

printf '\e[=1T'

I can now send an email to the lists, asking whether people are
interested in trying the xterm emulator.

Conformance: ignore {delete,insert} line while outside the scrolling region.

I noticed a small inconsistency in delete and insert line between xterm
and libteken. libteken allows these actions to happen while the cursor
is placed outside the scrolling region, while xterm does not.

This behaviour seems to be VT100-like. Confirmation:

http://www.vt100.net/docs/vt102-ug/chapter5.html
"This sequence is ignored when cursor is outside scrolling region."

MFC after: 1 month

Add __unused.

It turns out my previous commit does survive a buildkernel, but not
compilation of the individual test tools that use WARNS=6.

Make SCS work in 8-bit mode.

This means we can finally do things like VT100 box drawing when using
Syscons (8-bit characters). As far as I know, the only remaining issue
is the absense of proper escape sequences for special keyboard
characters (cursor, F1 to F12, etc) and xterm emulation should be ready
for general use.

Enabling xterm would have the following advantages:

- Easier possible migration to Unicode. cons25 termcap entries are very
8-bit centric. They use things like CP437 characters for box drawing,
etc.

- Better support for SSH'ing to other operating systems/devices. Most
switches use VT100-style admin interfaces.

- Reduced bandwidth, because applications can now use things like
scrolling regions.

- You can finally use applications like dtach(1) on both the console and
inside an xterm.

Commit all local modifications I have to libteken:

- Make xterm/cons25 support runtime configurable. This allows me to
share libteken between syscons and my new vt driver.
- Add a fix to print blanks after printing a double width character to
prevent rendering artifacts.
- Add some more utility functions that I use in the vt driver.

Small style(9) bug introduced in the previous commit.

Make 8-bit support run-time configurable.

Now to do the same for xterm support. This means people can eventually
toy around with xterm+UTF-8 without recompiling their kernel.

Make resizing of teken terminals a bit more safe.

Just perform a full reset when resizing the terminal. This means the
cursor, scrolling region, etc. are never positioned outside the
terminal.

Expose the TF_REVERSE flag to the console driver.

Right now libteken processes TF_REVERSE internally and returns the
toggled colors to the console driver. This isn't entirely correct. This
means that the bold flag is always processed by the foreground color,
while reversing should be done after the foreground color has been set
to a brighter version by the bold flag.

This is no problem with the syscons driver, because with VGA it only
supports 16 foreground and 8 background colors. My WIP console driver
reconfigures the graphics hardware to disable the blink functionality
and uses 16 foreground and 16 background colors. This means that this
driver will handle the TF_REVERSE flag a little different from what
syscons does right now.

Move libteken out of the syscons directory.

I initially committed libteken to sys/dev/syscons/teken, but now that
I'm working on a console driver myself, I noticed this was not a good
decision. Move it to sys/teken to make it easier for other drivers to
use a terminal emulator.

Also list teken.c in sys/conf/files, instead of listing it in all the
files.arch files separately.

Small fixes to Unicode handling:

- Add more mappings for Greek characters and the Euro sign.
- Print UTF-8 characters in the log file as hexadecimal.

Make the demo/stress tools build again, after my changes to param().

Restore support for bell pitch/duration.

Because we only support a single argument to tf_param, use 16 bits for
the pitch and 16 bits for the duration. While there, make the argument
unsigned. There isn't a single param call that needs a signed integer.

Submitted by: danfe (modified)

Small cleanup to previous commit.

Just use % 8, instead of & 0x7. This makes it easier to understand.

Just use default behaviour on tabstops when using too many columns.

It seems I didn't fix this issue before committing teken to the tree. My
initial idea was to somehow add an error mechanism to instruct the video
driver author to increase T_NUMCOL when using very big terminals. It
turns out we have platforms where we have gigantic consoles on systems
like the Apple PowerMac G5, which means we crash there right now.

Just ignore tabstops placed beyond column 160. Just force tabs to be
placed on each 8 columns.

Reported by: nwhitehorn

Make a 1:1 mapping between syscons stats and terminal emulators.

After I imported libteken into the source tree, I noticed syscons didn't
store the cursor position inside the terminal emulator, but inside the
virtual terminal stat. This is not very useful, because when you
implement more complex forms of line wrapping, you need to keep track of
more state than just the cursor position.

Because the kernel messages didn't share the same terminal emulator as
ttyv0, this caused a lot of strange things, like kernel messages being
misplaced and a missing notification to resize the terminal emulator for
kernel messages never to be resized when using vidcontrol.

This patch just removes kernel_console_ts and adds a special parameter
to te_puts to determine whether messages should be printed using regular
colors or the ones for kernel messages.

Reported by: ache
Tested by: nyan, garga (older version)

Properly implement GIO_ATTR and CONS_GETINFO.

It seems I didn't implement these two ioctl()'s properly, which meant
vidcontrol couldn't properly obtain certain terminal parameters.

Add Unicode rendering to the teken demo application.

Some time ago I tried adding Unicode rendering to the teken demo
application, but I didn't get it working. It seems I forgot to call
setlocale(). Polish this code and make sure it doesn't get lost.

Also a small fix for my previous commit: all Unicode characters in
teken_boxdrawing are below 0x10000, so store them as 16-bit values.

Properly implement the VT100 SCS sequences in xterm-mode.

Even though VT100-like devices can display non-ASCII characters, they do
not use an 8-bit character set. Special escape sequences allow the VT100
to switch character maps. The special graphics character set stores the
box drawing characters, starting at 0x60, ending at 0x7e. This means
we now pass the character map tests in vttest, even the save/restore
cursor test, combined with character maps. dialog(1) also works a lot
better now.

This commit also includes some other minor fixes:

- Default to 24 lines in teken_demo when using xterm emulation.
- Make white foreground and background work in teken_demo.

Don't forget to mark the color translation array as const.

Spotted by: Christoph Mallon <christoph mallon gmx de>

Fix for my previous commit: color mapping is not 1:1.

Cons25 doesn't seem to use a straight 1:1 mapping to the ANSI colors,
but uses the same color numbers as at least used by syscons on i386. I
suspect if you change the definitions on a different architecture,
things may break? Not sure.

Add a small array to convert syscons-style color codes to ANSI
equivalents, which are used by libteken internally. I didn't notice this
bug, because I only tested my code with black, white and green, all of
them shared the same numbers.

Make vidcontrol's color setting work again.

It turns out I forgot to implement two escape sequences that allows the
user to change the default foreground and background colors. I thought
they were implemented by syscons itself, but vidcontrol just generates
some escape sequences, which get interpreted by the terminal emulator.

Reported by: mgp (forums)

Allow experimental libteken features to be tested without changing code.

The teken library already supports UTF-8 handling and xterm emulation,
but we have reasons to disable this right now. Because we should make it
easy and interesting for people to experiment with these features, allow
them to be set in kernel configuration files.

Before this commit we had a flag called `TEKEN_CONS25' to enable
cons25-style emulation. I'm calling it the opposite now, `TEKEN_XTERM',
because we want to enable it in kernel configuration files explicitly.

Requested by: kib

Import yet some more small fixes to libteken sources:

- Implement NP (ASCII 12, Form Feed). When used with cons25, it should
clear the screen and place the cursor at the top of the screen. When
used with xterm, it should just simulate a newline.

- When we want to use xterm emulation, make teken_demo set TERM to
xterm.

Spotted by: Paul B. Mahol <onemda@gmail.com>

Fix rendering glitch in cons25 emulation.

Because we now have cons25-style linewrapping, we must also use cons25-
style reverse linewrapping. This means that a ^H on column 0 will move
the cursor one line up.

Also fix a small regression: if the user invokes a RIS (Reset to Initial
State), we must show the cursor again.

Spotted by: Paul B. Mahol <onemda gmail com>

Print control characters, even though they are normally not visible.

With cons25, there are printable characters below 0x1B. This is not the
case with ASCII, UTF-8, etc. but in this case we just have to.

Also don't set LC_CTYPE to UTF-8 when libteken is compiled without UTF-8
in the demo-application.

Resolve some regressions related to tabs and linewrap handling.

It turns out I was looking too much at mimicing xterm, that I didn't
take the differences of cons25 into account. There are some differences
between xterm and cons25 that are important. Create a new #define called
TEKEN_CONS25 that can be toggled to switch between cons25 and xterm
mode.

- Don't forget to redraw the cursor after processing a forward/backward
tabulation.

- Implement cons25-style (WYSE?) autowrapping. This form of autowrapping
isn't that nice. It wraps the cursor when printing something on column
80. xterm wraps when printing the first character that doesn't fit.

- In cons25, a \t shouldn't overwrite previous contents, while xterm
does.

Reported by: Garrett Cooper <yanefbsd gmail com>

Remove an unneeded assertion in libteken.

The cursor is only inside the scrolling region when we are in origin
mode. In that case, it should use originreg instead of scrollreg. It is
completely valid to place the cursor outside the scrolling region.

Replace syscons terminal renderer by a new renderer that uses libteken.

Some time ago I started working on a library called libteken, which is
terminal emulator. It does not buffer any screen contents, but only
keeps terminal state, such as cursor position, attributes, etc. It
should implement all escape sequences that are implemented by the
cons25 terminal emulator, but also a fair amount of sequences that are
present in VT100 and xterm.

A lot of random notes, which could be of interest to users/developers:

- Even though I'm leaving the terminal type set to `cons25', users can
do experiments with placing `xterm-color' in /etc/ttys. Because we
only implement a subset of features of xterm, this may cause
artifacts. We should consider extending libteken, because in my
opinion xterm is the way to go. Some missing features:

- Keypad application mode (DECKPAM)
- Character sets (SCS)

- libteken is filled with a fair amount of assertions, but unfortunately
we cannot go into the debugger anymore if we fail them. I've done
development of this library almost entirely in userspace. In
sys/dev/syscons/teken there are two applications that can be helpful
when debugging the code:

- teken_demo: a terminal emulator that can be started from a regular
xterm that emulates a terminal using libteken. This application can
be very useful to debug any rendering issues.

- teken_stress: a stress testing application that emulates random
terminal output. libteken has literally survived multiple terabytes
of random input.

- libteken also includes support for UTF-8, but unfortunately our input
layer and font renderer don't support this. If users want to
experiment with UTF-8 support, they can enable `TEKEN_UTF8' in
teken.h. If you recompile your kernel or the teken_demo application,
you can hold some nice experiments.

- I've left PC98 the way it is right now. The PC98 platform has a custom
syscons renderer, which supports some form of localised input. Maybe
we should port PC98 to libteken by the time syscons supports UTF-8?

- I've removed the `dumb' terminal emulator. It has been broken for
years. It hasn't survived the `struct proc' -> `struct thread'
conversion.

- To prevent confusion among people that want to hack on libteken:
unlike syscons, the state machines that parse the escape sequences are
machine generated. This means that if you want to add new escape
sequences, you have to add an entry to the `sequences' file. This will
cause new entries to be added to `teken_state.h'.

- Any rendering artifacts that didn't occur prior to this commit are by
accident. They should be reported to me, so I can fix them.

Discussed on: current@, hackers@
Discussed with: philip (at 25C3)