Project Updates - Tickit window scrolling, Circle scroll indicators

Some progress on a few of my projects:

Tickit::Window now supports scroll and scrollrect even if the window partly covered by other floating windows. With the expose-after-scroll behaviour turned on (soon to be default in a later version once I know it works nicely), it always returns true, and tries to scroll as many regions of the screen as it can, queueing expose operations for the areas that needed it.

A good example of this working can be seen in the newly-updated Tickit::Widget::Scroller, which now supports a scroll position indicator, a small floating window in the top or bottom right corner.

Another example can be seen as the latest feature in Circle::FE::Term. Scrolling a channel window gives an indicator in the bottom right corner, which also includes a count of new lines of content added since scrolling, if there are any.

The Past, The Present and The Future - LPW2012

For those who are interested, here are the slides of the talk I gave about Futures at LPW2012 this year:

The Past, The Present and The Future

Pangoterm - config and scrollback

I've been working more on pangoterm, and it now has two important features that bring it much closer to being a real usable terminal.

The first is that it now supports a lot more configuration by command-line, and also the same things by a configuration file. Support for a config file isn't something I personally have directly needed so far, mostly because the compiled-in defaults are already what I'd like. But it's something other people have been asking for, and this helps support other users too, so it's nice to have.

I also found quite a neat way to implement it too, requiring just one line of macro to declare the existence of a configuration setting and give its default value and help description. This then creates a normal variable that can be read in the usual way:

CONF_INT(scrollback_size, 0, 1000, "Scrollback size", "LINES");
...
pt->scroll_size = CONF_scrollback_size;

The other main feature that's now in place is scrollback buffer. Now, whenever text disappears off the top of the screen it is saved, and can be recalled again by scrolling the mouse wheel or using the Shift-PageUp and Shift-PageDown keys. A small custom-drawn indicator shows the scroll position without needing to consume any screen space in the normal state of not being scrolled.

I also found and fixed an almost embarrassingly-bad bug around the redrawing code. When erasing cells, it used to make one cairo call for every cell to be erased, leading to many calls whenever an entire line had to be cleared. Now it composes multiple blank cells with the same pen, in the same way as it composes multiple cells of text; and is faster when scrolling partial lines, and so on.

While I've already been using it quite a lot lately to run the vim I use to edit its own source code, for some tasks I've still been using xterm. But not any more - with the addition of scrollback to pangoterm, I find I'm now using it for everything. It's become my default terminal both for home and work use.

libvterm / pangoterm performance improvements

I recently made a couple of small but drastic improvements to libvterm and pangoterm's performance, when scrolling long output. Using my not-so-scientific test of taking a ~500KB text file and running:

$ time cat file

This technically measures the amount of time it takes cat to write the data into the TTY, but since the buffer is a fixed 4KiB in size, it also measures the time that pangoterm takes to read all but the final 4KiB of the file, which is under 1% of the file. I managed to obtain the following timings. Before I started:

real    0m4.206s
user    0m0.000s
sys     0m0.036s

By optimising libvterm's moverect buffer operation with memmove() (revision -r511):

real    0m2.035s
user    0m0.000s
sys     0m0.048s

With pangoterm deferred updates, that delay re-rendering of the screen until all the PTY data has been read, or every 20 msec (revision -r482):

real    0m0.358s
user    0m0.004s
sys     0m0.016s

It's now well over 10 times faster than it used to be. Ohh.. and it beats xterm. By quite a bit.

real    0m2.294s
user    0m0.000s
sys     0m0.036s

Dear Perl users on Windows...

I am close to giving up trying to support my Perl modules on Windows. Personally, I have nothing against it, despite all the odd quirky "not-quite-UNIX"es about it. I'm sure the same could be said in reverse, of all the "not-quite-Windows"es about the UNIXes, coming from a Windows developer.

However, I am continually annoyed at the almost complete lack of any smoke-test results ever coming from Cygwin or MSWin32 machines. I can upload a new module to CPAN and within literally hours have a dozen or so smoke-test reports from Linux and FreeBSD machines. Within a few days, a flood more results from these, and also Darwin, OpenBSD, NetBSD, Solaris. But quite often I can sit and wait weeks, if not months before I see the first result from Cygwin or MSWin32.

It's not that I'll ever actually stop looking after code - if someone were to point out a problem or send me a patch, I'm sure I could include it. It's just very hard for me as a developer without a Windows box to know if any of my code actually works on Windows, or if not, to be able to fix it. While I don't have any BSD, Solaris, or OS X boxes, I do at least have the confidence of the smoke-test reports to claim my code works there.

I must therefore conclude one of two things:

• Nobody is using Perl, or at least my Perl modules, on Windows; or
• Nobody using Perl on Windows cares about testing and code quality

I would really dislike having to declare this, because I feel sure there ought to be Windows users around who care. So perhaps someone somewhere can look into actually installing the test reporter module. It isn't hard, really...

Tickit, floating windows and menus

The latest release of Tickit, version 0.18, adds support for floating windows. These are windows that rather than divide space of their parent window, instead float above it by obscuring content below, creating a Z-ordering. Drawing operations on windows are aware of clipping regions caused by floating windows above them, and are accounted for automatically without individual widgets needing to be aware of it.

A special kind of floating window called a popup window can be created. These are always created as children of the root window, and take first priority at input events. These can be used to create things like popup menus, such as this early prototype of Tickit::Widget::Menu.

This is still somewhat of a work-in-progress, and by the time it's released it should support proper Unicode linedrawing rather than the ASCII variant. I'm also planning to create a way to make pop-up windows of arbitrary widgets, to create things like modal dialog boxes.

Don't be too lazy

Today I fixed a bug in my C library rewrite of Tickit.

The manifestation of the bug was that Tickit::Console didn't work properly via the C/XS version of Tickit, but worked fine on the Perl version. It wasn't receiving keyboard input at all. Yet I know basic keyboard input works fine on all the simple demos with the C library version, so something more subtle was up.

On close inspection it turned out that libtickit was being too lazy with constructing its libtermkey instance for handling keyboard input. The previous code arrangement was that constructing the instance was deferred as late as possible, until one of the input-handling methods was actually called. This meant that setting the filehandle could be done as a normal accessor and not as a constructor argument. (C functions lack the neatness of Perl's named argument style, so my usual style is minimal constructors and lots of mutation accessors).

This worked fine for simple cases, because after setup the demo programs all wait on a call to tickit_term_input_wait(), and the first thing that does is create the TermKey instance, which sets up the terminal for non-canonical input mode and disables local echo. This ensures input arrives a key at a time.

However, since the Tickit::Console example runs via Tickit::Async, the way it runs is to wait in a poll() loop waiting on readability on STDIN. When STDIN becomes readable does it call tickit_term_input_readable(), and only then does it create the TermKey instance that actually sets up the terminal.

The upshot here is that until that entire first line of input is received, the terminal isn't actually set up to the correct mode in the first place.

Fixing this bug was a simple matter of making the underlying tickit_term_set_input_handle() mutator eagerly allocate the TermKey instance immediately, so that the terminal is already set up in the correct mode once the Tickit::Term constructor returns. That way it works correctly in both synchronous and asynchronous code.

In summary - when creating lazy functionality in a program, make sure that your laziness doesn't cause you to neglect to set something up that someone else was relying on. Be lazy, but don't be too lazy.

Wide mouse support in libvterm / libtermkey

(If you don't want to read the boring history you can skip to the end...)

Mouse in terminals. It's often been a mess. Traditionally terminals just send some ASCII-like representation of the keys you press, and mouse doesn't easily fit that. Therefore, when DEC first started adding mouse ability to terminals, they had to find a way to encode events into the byte stream.

They decided on a CSI sequence. Despite that CSI sequences can include numeric parameters, they decided the encoding should be a simple CSI M with no arguments, followed by three plain bytes which encode the buttons and position (for reasons that I suspect are lost in the mists of time). These three bytes encode, in order, a bitmask containing the button event and modifier keys, then the line and column number. In order to avoid sending C0 bytes within the stream for these, each byte is offset by adding 0x20 to ensure it's not a C0 control byte. This has the downside of leaving atmost 224 (256 - 32) columns or lines that can be encoded. On your traditional 80x25 glass teletype of course that was not a problem, but with today's 30inch widescreen monitors, it's easily possible to make a virtual terminal wider than this limit.

To solve this, a few extended encoding schemes have been invented. The first was the somewhat bizarre encoding simply named "Extended Encoding". This encodes the three values not as single bytes, but as UTF-8 encoded characters. This has a number of downsides, most notably the fact that it cannot be recognised unambiguously as compared to the other ones, so it's best to avoid it. Next are two largely-similar schemes that both use the parameters of CSI sequences to encode the parameters of the mouse event. rxvt encoding simply uses CSI M with the three values as CSI parameters, whereas the nicest of the encodings, so-called SGR encoding, uses either CSI < M or CSI < m to encode button press vs. button release. This fixes the other shortcoming of all the other encodings, in that SGR encoding can encode which button was released on a release event, whereas all the others can only encode that it was a release.

I recently got around to implementing these in my two main terminal-related projects; libvterm and libtermkey. libvterm (demonstrated here via pangoterm) can issue mouse events in any of these encodings if requested by the application. libtermkey will automatically recognise basic, rxvt and SGR encodings automatically, but won't handle the "extended" UTF-8 encoding, because it can't be unambiguously identified from basic.

And now without further ado, here is a rather uneventful screenshot that shows an enormously wide pangoterm demonstrating mouse events past the 224th column.

And in case the text is a little small to read here, it looks like this:

$ ./demo -m -p 1006
Mouse mode active
<MousePress(1) @ (72,33)>
<MouseRelease(1) @ (72,33)>
<MousePress(1) @ (148,25)>
<MouseRelease(1) @ (149,25)>
<MousePress(1) @ (219,26)>
<MouseRelease(1) @ (219,26)>
<MousePress(1) @ (288,28)>
<MouseRelease(1) @ (288,28)>
<MousePress(1) @ (297,68)>
<MouseRelease(1) @ (297,68)>
<C-c>
Mouse mode deactivated

Should work on a recent xterm etc.. as well.

Pangoterm - coloured icons

Lately I've been working more on pangoterm again. A few days ago I was feeling oddly artistic, and decided to draw an icon for it, to distinguish it in window lists, etc..

I use a variety of differently-coloured terminals to represent different scenarios (mostly SSH terminals, picking colours to represent different machines), by changing the background colour.

I thought to myself "wouldn't it be great if the window icon could change background colour to set the actual background colour used by that terminal window".

So I did that.

By a little bit of custom CSS hackery, it loads the SVG via a stylesheet that overrides the fill colour of the background colour element, and sets it to the colour chosen by that terminal. The upshot here is that the icon used in the workspace switcher, window list, and other places, looks like the terminal window.

ANSI vs DEC, arbitrary scrolling in terminals

Lately I've been looking at trying to finish the effectively half-finished effort that DECSTBM, IL and DL give us, by extending it with column awareness. I notice that VT420 gives us these things; namely DECSLRM (set left/right margin) an analogy to DECSTM, and DECIC/DECDC, to insert or delete a column. Using this combination, you can define an arbitrary scrolling rectangle using DECSTBM+DECSLRM, then scroll it in any of the four directions by placing the cursor in the topleft corner and issuing IL/DL/DECIC/DECDC as required. The VT420 manual says that the terminal defaults to a mode wherein "Left and right margins cannot be changed." (DECVSSM; vertical split-screen mode). You have to enable it first by issuing CSI ? 69 h.

This scheme allows us all sorts of new abilities, such as:

• Up/downward scrolling of a vertically-split window by setting top/bottom/left/right margins and scrolling as normal with SU/SD.
• Bounded ICH/DCH that doesn't extend all the way to the righthand edge by setting right margin, allowing us to implement insert/delete in a readline-alike whose entry area is not the entire width of the terminal.

(Both of these abilities I would verymuch like ;) )

However in actual practice it's not quite as simple as this. DECSLRM is encoded as CSI s, to follow DECSTBM at CSI r. This creates a problem because ANSI.SYS defines CSI s as SAVE, an operation identical to DECSC.

So what to do about this problem? My initial thought was to say "ignore ANSI.SYS", and support just the DEC scheme. I feel that arbitrary scrolling is too important a feature to be lacking for the sake of this legacy compatibility.

I can therefore see a number of possible ways out of this:

0. Ignore DECSLRM and any column-based scrolling ability (the status quo). I dislike this because it means I don't get to use scrolling rectangles.
1. Entirely remove ANSI SAVE/RESTORE ability on CSI s/CSI t and repurpose CSI s to mean DECSLRM. I'd have no problem with this, but there may be the odd legacy application or two that somehow expects ANSI SAVE/RESTORE to actually work (even though it is semantically identical to the one-byte-shorter DECSC/DECRC).
2. Try to multiplex both meanings onto one sequence by using some sort of heuristic to determine which meaning might be meant. Such thoughts as:
   1. Use the value of DECVSSM to decide which meaning to apply to CSI s - if DECVSSM is enabled then CSI s means DECSLRM; if disabled it means ANSI SAVE. This feels to me the most preferrable solution.
   2. Use the value of mode ?1049 or similar, because most(all?) applications wanting to use scrolling rectangles will be using alternate buffer anyway. This would make it impossible to use DECSLRM in main-screen applications, such as any scrolling readline-alike console.
   3. Use the setting of DECSTBM to decide - if top/bottom margins are set it's likely that the application will want to set left/right and that it's an application that knows DEC-like things and won't want to use ANSI SAVE anyway. This does make it impossible to just use DECSLRM on its own though, to give you bounded ICH/DCH.
3. Ignore entirely what VT420 does, and extend this behaviour in some other way. Perhaps for example, allow DECSTBM to take up to four parameters, defining the left and right margins in the third and fourth. This has the notable advantage that any application currently sending just the two-parameter form simply supplies default values for the other two positions, implying the full column width.
4. Make it user-configurable at runtime by some config option, commandline flag, etc...

My votes in preference are 3, 2a, 1; and thereafter the rest all sound horrible. But I'd be interested in hearing what ideas anyone else has...

Patches^W Unit Tests Welcome

There's a saying in Open Source communities "Patches Welcome". However, I find that some sorts of patches are more welcome than others.

Recently I've been working on putting strptime into POSIX. In response of one of my patches on the perl5-porters@ mailing list, Nicholas Clark wrote me a reply, attached to which was a new set of unit tests, most of which the code did not currently pass, but ought to.

As patches go, this is one of the most useful patches I have ever received. He didn't try (too much) to fix the code, didn't spend that long explaining the problems. He just sent a unit test. Now let me explain why this is useful.

With a unit test script, I can have it run against any version of the code I write. I can run it every minute until it works. I can run it at 3 in the morning during one of my late-night hacking sessions. I can run it on the underground train on my commute to work. It is close, immediate, convenient.

A discussion over a patch to the code alone would be at least an order of magnitude slower than this. People keep different time schedules. People aren't always available online. People don't always get what each other means the first time, leading to more back-and-forth discussion until an agreement is reached.

Without wishing to upset too many members of the Open Source community that might disagree with me, I would say "Patches (to code) Welcome, but unit tests, examples, or documentation would be preferred." I can easily hack up some code to pass the tests you supply, whereas if you just supply some code it's much harder for me to work out what you actually wanted.

libtermkey 0.12 + Term::TermKey 0.11

Last week I updated libtermkey to version 0.12, and associated with it, the Perl module wrapper Term::TermKey to version 0.11.

Primary changes are the fact it will now support an abstract mode, where an instance can be constructed not associated with a TTY filehandle. Instead of being read from the filehandle, bytes provided to the library directly by the application.

While I originally wrote this to support someone using the library where the application is already reading in bytes from the TTY by some other mechanism, I did lately find a new use for it. By applying a small string parser to a readline inputrc file, its raw byte sequences can be turned into libtermkey representations instead. I don't quite have an example of this robust enough yet to demonstrate, but hopefully by the next version of Term::TermKey I'll get around to writing one. I have a plan to write a Tickit::Widget::ReadlineAlike, which would go as far as parsing the user's inputrc file to discover all the keybindings to apply.

Also included in 0.12 are plenty of manpage updates, including also an online copy of them rendered into HTML on my website here.