Introduction
This post is written for “System Under Test” blog and is a research on how XTerm project is tested.
Disclaimer: the author of this post is not involved into a development of XTerm project. All of the following is merely a high-level report about what a curious developer can see from looking at XTerm’s source code. The author didn’t go very deep into the topic, could overlook something etc. Having that said, we always appreciate feedback from our colleagues who have advanced experience with the projects we write about, especially from the developers and maintainers.
What is the project about?
The project’s home page is located at: http://invisible-island.net/xterm/, here is the description from Wikipedia:
xterm is the standard terminal emulator for the X Window System.
xterm originated prior to the X Window System. It was originally written as a stand-alone terminal emulator for the VAXStation 100 (VS100) by Mark Vandevoorde, a student of Jim Gettys, in the summer of 1984, when work on X started. It rapidly became clear that it would be more useful as part of X than as a standalone program, so it was retargeted to X.
After many years as part of the X reference implementation, around 1996 the main line of development then shifted to XFree86 (which itself forked from X11R6.3), and it is now actively maintained by Thomas Dickey.
Many xterm variants are also available. Most terminal emulators for X started as variations on xterm.
We will look at three projects: XTerm, Vttest tool and so-called ncurses test programs. The maintainer of all of these projects is Thomas E. Dickey.
What is tested in XTerm?
Using frontend-backend terminology, we may think of a terminal emulator (like XTerm or OS X Terminal) as a frontend that communicates with a backend, which is operating system, with a middleware in between: shell (like bash or zsh).
Historically, the protocols for this communication between terminal’s frontend and backend evolved around usage of ANSI escape codes, and were eventually standardized in ECMA-48/ISO 6429/ANSI X3.64 standards. The first popular video terminal to support these codes was the Digital VT100, introduced in 1978 (https://en.wikipedia.org/wiki/ANSI_escape_code).
Ability to correctly recognize and handle sufficiently large number of escape codes is a crucial part of implementation of any terminal. For example, to make Vim, man or command-line editing work correctly, one has to implement escape controls that are required for these programs, otherwise terminal emulator will produce malformed output (or crash).
The following example demonstrates what terminal emulator (frontend) receives from operating system (backend), after you type vi and then press “Enter” in OS X Terminal. Terminal emulator has to interpret these codes and react on them accordingly.
In a real world, ESC character (ASCII decimal 27/hex 0x1B/octal 033) is not printable. It is manually substituted in the log trace (left image) to be human-readable as "ESC".
One of the key features of XTerm is that it supports most of the control sequences ever available for terminal emulators:
XTerm provides DEC VT102/VT220 and selected features from higher-level terminals such as VT320/VT420/VT520 (VTxxx). It also provides Tektronix 4014 emulation for programs that cannot use the window system directly.
According to Comparing versions, by counting controls, XTerm implements at least twice as many of escape codes as any of well-known terminals: rxvt, putty, konsole, vte and some others, and at this point one may really wonder how all of this functionality is tested by XTerm’s developer.
Getting started
Getting XTerm
If you want to run XTerm on Mac OS X, you have to have XQuartz installed.
The source code is available on project’s page, Download section:
wget ftp://invisible-island.net/xterm/xterm.tar.gz
tar xzvf xterm.tar.gz
cd xterm-325
./configure --prefix=$(pwd)/Build/ --enable-trace
make
make install
to run xterm:
./Build/bin/xterm
Passing --enable-trace enables tracing functions: when you run xterm two files are created: Trace-child.out and Trace-parent.out, where xterm writes debug information. Enabling this flag is not relevant for this post, but can be interesting for someone who wants to look behind the scenes and see what information is relevant to a developer of XTerm.
Getting Vttest
Vttest’s source code is located at: http://invisible-island.net/vttest/:
wget ftp://invisible-island.net/vttest/vttest.tar.gz
tar xzvf vttest.tar.gz
cd vttest-20140305
./configure --prefix=$(pwd)/Build
make
make install
to run:
./Build/bin/vttest
Getting ncurses test programs
From http://invisible-island.net/ncurses/ncurses-examples.html:
wget ftp://invisible-island.net/ncurses-examples/current/ncurses-examples.tar.gz
tar xzvf ncurses-examples.tar.gz
cd ncurses-examples-20160709
./configure
make
3 groups of tests
Exploration of XTerm, vttest and ncurses test programs reveals three corresponding groups of tests:
vttests/*in XTerm’s source tree: test scripts written in Perl and Bashvttestprogramncurses test programs
These groups are independent from each other however they all share the same approach:
- All three groups are visual and manual tests. This means that to verify them, developer has to run these tests (including tests inside
vttestprogram) manually and look for discrepancies between expected visual output and real visual output produced by terminal emulator that is under test. - All of these tests have to be run from inside terminal emulator that is being tested.
XTerm/vttests/*
$ ls -l
[email protected] 1 Stanislaw staff 3434 Dec 11 2011 16colors.sh
[email protected] 1 Stanislaw staff 2083 Feb 26 2014 256colors.pl
[email protected] 1 Stanislaw staff 5043 Feb 26 2014 256colors2.pl
[email protected] 1 Stanislaw staff 1947 Feb 26 2014 88colors.pl
[email protected] 1 Stanislaw staff 5105 Feb 26 2014 88colors2.pl
[email protected] 1 Stanislaw staff 3118 Dec 11 2011 8colors.sh
[email protected] 1 Stanislaw staff 2942 Dec 11 2011 acolors.sh
[email protected] 1 Stanislaw staff 3429 Dec 11 2011 doublechars.sh
[email protected] 1 Stanislaw staff 3412 Feb 26 2014 dynamic.pl
[email protected] 1 Stanislaw staff 2711 Dec 11 2011 dynamic.sh
[email protected] 1 Stanislaw staff 3359 Dec 11 2011 dynamic2.sh
[email protected] 1 Stanislaw staff 2661 Aug 10 2015 fonts.sh
[email protected] 1 Stanislaw staff 4841 Dec 28 2014 paste64.pl
[email protected] 1 Stanislaw staff 4313 Feb 26 2014 query-color.pl
[email protected] 1 Stanislaw staff 4713 Feb 26 2014 query-fonts.pl
[email protected] 1 Stanislaw staff 3669 Oct 7 2014 resize.pl
[email protected] 1 Stanislaw staff 3203 Dec 11 2011 resize.sh
[email protected] 1 Stanislaw staff 8263 Feb 26 2014 tcapquery.pl
[email protected] 1 Stanislaw staff 2850 Dec 11 2011 title.sh
Example: ./vttests/colors16.sh
Example: ./vttests/dynamic2.sh
Example: ./vttests/resize.sh
Vttest
Vttest tests the compatibility (demonstrates the non-compatibility) of so-called “VT100-compatible” terminals.
http://invisible-island.net/vttest/vttest.html
Vttest is an application which is used to demonstrate features of VT100 and related terminals, or emulations thereof, such as xterm. The program was originally written in 1986 by Per Lindberg. It has been maintained and extended since 1996 by Thomas Dickey, to test and demonstrate features of xterm. https://en.wikipedia.org/wiki/Vttest
This program should be run inside a terminal program under test.
I have run vttest in three terminals: XTerm, Terminal.app (default terminal of OS X) and iTerm2 (well-known open-source terminal emulator for OS X).
On the following screenshots XTerm has white background - it always demonstrates correct expected behavior in vttest. Terminal.app and iTerm2 have black background and sometimes diverge from expected by the test cases, they are hidden by default and can be expanded.
Example: cursor movements
Example: autowrap, mixing control and print characters.
Example: WRAP AROUND mode setting
Example: origin mode test
ncurses test programs
ncurses test programs are part of ncurses source tree. However, they have a separate project page: ncurses test programs and can be also obtained in a separate tar.
The ncurses test programs are used both for testing/demonstrating features of ncurses, as well as for occasional comparisons with other curses implementations.
To understand what is tested here is definition of ncurses:
ncurses (new curses) is a programming library providing an application programming interface (API) that allows the programmer to write text-based user interfaces in a terminal-independent manner. It is a toolkit for developing “GUI-like” application software that runs under a terminal emulator. It also optimizes screen changes, in order to reduce the latency experienced when using remote shells.
Strictly speaking, System Under Test for these programs is ncurses, not XTerm, however running these programs from a terminal-emulator-under-test can give a large amount of additional testing information for XTerm especially because the maintainer of both ncurses and XTerm is the same person.
Excerpt from README:
The programs in this directory are used to test and demonstrate ncurses.
Some are interesting in themselves, while others merely show how some of
the library calls are used. Some date back to the initial releases of
ncurses during the mid-1990s, such as:
blue - Blue Moon, a nifty solitaire (uses color)
bs.c - the game of Battleships (uses color)
firework.c - multi-colored fireworks (uses color)
gdc.c - Great Digital Clock (uses color)
hanoi.c - the game of hanoi (uses color essentially)
knight.c - the game of Knight's Tour (uses color)
lrtest.c - test of access to the lower-right corner
ncurses.c - multi-test program (uses color)
newdemo.c - another test from PDCurses (uses color)
rain.c - rain drops keep falling on my head...
tclock.c - analog/digital clock
testcurs.c - a test from the PDCurses people (uses color)
tracemunch - Perl script to make trace scripts easier to read
worm.c - worms run all over your screen (uses color)
xmas.c - Xmas greeting card
Example: Worm
Example: Great digital clock
Example: Xmas greeting card
Conclusion
XTerm has 3 groups of tests:
- Perl/bash test scripts in XTerm’s tree
Vttestprogramncurses test programs
Vttest and ncurses test programs can be used to test any terminal emulator, however their main system-under-test is XTerm.
All of 3 test groups are manual, visual tests that have to be run from terminal emulator which is under test. Developer has to run these tests manually to compare expected visual output with actual visual output and look for discrepancies.
More about testing in System Under Test blog
If you want to see some tool or project covered in this blog, or if you want to contribute a post about some tool or project - let us know: [email protected].