XTerm

Jul 28, 2016   #Command Line Tools 

Introduction

This post is written for “System Under Test” blog and is a research on how XTerm project is tested.

Disclaimer: the author of this post is not involved into a development of XTerm project. All of the following is merely a high-level report about what a curious developer can see from looking at XTerm’s source code. The author didn’t go very deep into the topic, could overlook something etc. Having that said, we always appreciate feedback from our colleagues who have advanced experience with the projects we write about, especially from the developers and maintainers.

What is the project about?

The project’s home page is located at: http://invisible-island.net/xterm/, here is the description from Wikipedia:

xterm is the standard terminal emulator for the X Window System.

xterm originated prior to the X Window System. It was originally written as a stand-alone terminal emulator for the VAXStation 100 (VS100) by Mark Vandevoorde, a student of Jim Gettys, in the summer of 1984, when work on X started. It rapidly became clear that it would be more useful as part of X than as a standalone program, so it was retargeted to X.

After many years as part of the X reference implementation, around 1996 the main line of development then shifted to XFree86 (which itself forked from X11R6.3), and it is now actively maintained by Thomas Dickey.

Many xterm variants are also available. Most terminal emulators for X started as variations on xterm.

https://en.wikipedia.org/wiki/Xterm

We will look at three projects: XTerm, Vttest tool and so-called ncurses test programs. The maintainer of all of these projects is Thomas E. Dickey.

What is tested in XTerm?

Using frontend-backend terminology, we may think of a terminal emulator (like XTerm or OS X Terminal) as a frontend that communicates with a backend, which is operating system, with a middleware in between: shell (like bash or zsh).

Historically, the protocols for this communication between terminal’s frontend and backend evolved around usage of ANSI escape codes, and were eventually standardized in ECMA-48/ISO 6429/ANSI X3.64 standards. The first popular video terminal to support these codes was the Digital VT100, introduced in 1978 (https://en.wikipedia.org/wiki/ANSI_escape_code).

Ability to correctly recognize and handle sufficiently large number of escape codes is a crucial part of implementation of any terminal. For example, to make Vim, man or command-line editing work correctly, one has to implement escape controls that are required for these programs, otherwise terminal emulator will produce malformed output (or crash).

The following example demonstrates what terminal emulator (frontend) receives from operating system (backend), after you type vi and then press “Enter” in OS X Terminal. Terminal emulator has to interpret these codes and react on them accordingly.

How Vim is rendered
In a real world, ESC character (ASCII decimal 27/hex 0x1B/octal 033) is not printable. It is manually substituted in the log trace (left image) to be human-readable as "ESC".

One of the key features of XTerm is that it supports most of the control sequences ever available for terminal emulators:

XTerm provides DEC VT102/VT220 and selected features from higher-level terminals such as VT320/VT420/VT520 (VTxxx). It also provides Tektronix 4014 emulation for programs that cannot use the window system directly.

http://invisible-island.net/xterm/xterm.faq.html#what_is_it

According to Comparing versions, by counting controls, XTerm implements at least twice as many of escape codes as any of well-known terminals: rxvt, putty, konsole, vte and some others, and at this point one may really wonder how all of this functionality is tested by XTerm’s developer.

Getting started

Getting XTerm

If you want to run XTerm on Mac OS X, you have to have XQuartz installed.

The source code is available on project’s page, Download section:

wget ftp://invisible-island.net/xterm/xterm.tar.gz
tar xzvf xterm.tar.gz
cd xterm-325
./configure --prefix=$(pwd)/Build/ --enable-trace
make
make install

to run xterm:

./Build/bin/xterm

Passing --enable-trace enables tracing functions: when you run xterm two files are created: Trace-child.out and Trace-parent.out, where xterm writes debug information. Enabling this flag is not relevant for this post, but can be interesting for someone who wants to look behind the scenes and see what information is relevant to a developer of XTerm.

Getting Vttest

Vttest’s source code is located at: http://invisible-island.net/vttest/:

wget ftp://invisible-island.net/vttest/vttest.tar.gz
tar xzvf vttest.tar.gz
cd vttest-20140305
./configure --prefix=$(pwd)/Build
make
make install

to run:

./Build/bin/vttest

Getting ncurses test programs

From http://invisible-island.net/ncurses/ncurses-examples.html:

wget ftp://invisible-island.net/ncurses-examples/current/ncurses-examples.tar.gz
tar xzvf ncurses-examples.tar.gz
cd ncurses-examples-20160709
./configure
make

3 groups of tests

Exploration of XTerm, vttest and ncurses test programs reveals three corresponding groups of tests:

  1. vttests/* in XTerm’s source tree: test scripts written in Perl and Bash
  2. vttest program
  3. ncurses test programs

These groups are independent from each other however they all share the same approach:

  • All three groups are visual and manual tests. This means that to verify them, developer has to run these tests (including tests inside vttest program) manually and look for discrepancies between expected visual output and real visual output produced by terminal emulator that is under test.
  • All of these tests have to be run from inside terminal emulator that is being tested.

XTerm/vttests/*

$ ls -l
-rwxr-xr-x@ 1 Stanislaw  staff  3434 Dec 11  2011 16colors.sh
-rwxr-xr-x@ 1 Stanislaw  staff  2083 Feb 26  2014 256colors.pl
-rwxr-xr-x@ 1 Stanislaw  staff  5043 Feb 26  2014 256colors2.pl
-rwxr-xr-x@ 1 Stanislaw  staff  1947 Feb 26  2014 88colors.pl
-rwxr-xr-x@ 1 Stanislaw  staff  5105 Feb 26  2014 88colors2.pl
-rwxr-xr-x@ 1 Stanislaw  staff  3118 Dec 11  2011 8colors.sh
-rwxr-xr-x@ 1 Stanislaw  staff  2942 Dec 11  2011 acolors.sh
-rwxr-xr-x@ 1 Stanislaw  staff  3429 Dec 11  2011 doublechars.sh
-rwxr-xr-x@ 1 Stanislaw  staff  3412 Feb 26  2014 dynamic.pl
-rwxr-xr-x@ 1 Stanislaw  staff  2711 Dec 11  2011 dynamic.sh
-rwxr-xr-x@ 1 Stanislaw  staff  3359 Dec 11  2011 dynamic2.sh
-rwxr-xr-x@ 1 Stanislaw  staff  2661 Aug 10  2015 fonts.sh
-rwxr-xr-x@ 1 Stanislaw  staff  4841 Dec 28  2014 paste64.pl
-rwxr-xr-x@ 1 Stanislaw  staff  4313 Feb 26  2014 query-color.pl
-rwxr-xr-x@ 1 Stanislaw  staff  4713 Feb 26  2014 query-fonts.pl
-rwxr-xr-x@ 1 Stanislaw  staff  3669 Oct  7  2014 resize.pl
-rwxr-xr-x@ 1 Stanislaw  staff  3203 Dec 11  2011 resize.sh
-rwxr-xr-x@ 1 Stanislaw  staff  8263 Feb 26  2014 tcapquery.pl
-rwxr-xr-x@ 1 Stanislaw  staff  2850 Dec 11  2011 title.sh

Example: ./vttests/colors16.sh

Example: ./vttests/dynamic2.sh

Example: ./vttests/resize.sh

Vttest

Vttest tests the compatibility (demonstrates the non-compatibility) of so-called “VT100-compatible” terminals.

http://invisible-island.net/vttest/vttest.html

Vttest is an application which is used to demonstrate features of VT100 and related terminals, or emulations thereof, such as xterm. The program was originally written in 1986 by Per Lindberg. It has been maintained and extended since 1996 by Thomas Dickey, to test and demonstrate features of xterm. https://en.wikipedia.org/wiki/Vttest

This program should be run inside a terminal program under test.

I have run vttest in three terminals: XTerm, Terminal.app (default terminal of OS X) and iTerm2 (well-known open-source terminal emulator for OS X).

On the following screenshots XTerm has white background - it always demonstrates correct expected behavior in vttest. Terminal.app and iTerm2 have black background and sometimes diverge from expected by the test cases, they are hidden by default and can be expanded.

Example: cursor movements

Example: autowrap, mixing control and print characters.

Example: WRAP AROUND mode setting

Example: origin mode test

ncurses test programs

ncurses test programs are part of ncurses source tree. However, they have a separate project page: ncurses test programs and can be also obtained in a separate tar.

The ncurses test programs are used both for testing/demonstrating features of ncurses, as well as for occasional comparisons with other curses implementations.

To understand what is tested here is definition of ncurses:

ncurses (new curses) is a programming library providing an application programming interface (API) that allows the programmer to write text-based user interfaces in a terminal-independent manner. It is a toolkit for developing “GUI-like” application software that runs under a terminal emulator. It also optimizes screen changes, in order to reduce the latency experienced when using remote shells.

Strictly speaking, System Under Test for these programs is ncurses, not XTerm, however running these programs from a terminal-emulator-under-test can give a large amount of additional testing information for XTerm especially because the maintainer of both ncurses and XTerm is the same person.

Excerpt from README:

The programs in this directory are used to test and demonstrate ncurses.
Some are interesting in themselves, while others merely show how some of
the library calls are used.  Some date back to the initial releases of
ncurses during the mid-1990s, such as:

    blue       - Blue Moon, a nifty solitaire (uses color)
    bs.c       - the game of Battleships (uses color)
    firework.c - multi-colored fireworks (uses color)
    gdc.c      - Great Digital Clock (uses color)
    hanoi.c    - the game of hanoi (uses color essentially)
    knight.c   - the game of Knight's Tour (uses color)
    lrtest.c   - test of access to the lower-right corner
    ncurses.c  - multi-test program (uses color)
    newdemo.c  - another test from PDCurses (uses color)
    rain.c     - rain drops keep falling on my head...
    tclock.c   - analog/digital clock
    testcurs.c - a test from the PDCurses people (uses color)
    tracemunch - Perl script to make trace scripts easier to read
    worm.c     - worms run all over your screen (uses color)
    xmas.c     - Xmas greeting card

Example: Worm

Example: Great digital clock

Example: Xmas greeting card

Conclusion

XTerm has 3 groups of tests:

  • Perl/bash test scripts in XTerm’s tree
  • Vttest program
  • ncurses test programs

Vttest and ncurses test programs can be used to test any terminal emulator, however their main system-under-test is XTerm.

All of 3 test groups are manual, visual tests that have to be run from terminal emulator which is under test. Developer has to run these tests manually to compare expected visual output with actual visual output and look for discrepancies.

More about testing in System Under Test blog

If you want to see some tool or project covered in this blog, or if you want to contribute a post about some tool or project - let us know: [email protected].

More articles

Wine

Sep 23, 2014   #Interview  by @ligurio 

LLVM

Mar 24, 2016   #Compilers  by @AlexDenisov 

FreeBSD

Mar 31, 2016   #Operating Systems  by @AlexDenisov 

GNU Make

Jun 12, 2016   #Build Tools  by @StanislavPankevich