Don't be too lazy

Today I fixed a bug in my C library rewrite of Tickit.

The manifestation of the bug was that Tickit::Console didn't work properly via the C/XS version of Tickit, but worked fine on the Perl version. It wasn't receiving keyboard input at all. Yet I know basic keyboard input works fine on all the simple demos with the C library version, so something more subtle was up.

On close inspection it turned out that libtickit was being too lazy with constructing its libtermkey instance for handling keyboard input. The previous code arrangement was that constructing the instance was deferred as late as possible, until one of the input-handling methods was actually called. This meant that setting the filehandle could be done as a normal accessor and not as a constructor argument. (C functions lack the neatness of Perl's named argument style, so my usual style is minimal constructors and lots of mutation accessors).

This worked fine for simple cases, because after setup the demo programs all wait on a call to tickit_term_input_wait(), and the first thing that does is create the TermKey instance, which sets up the terminal for non-canonical input mode and disables local echo. This ensures input arrives a key at a time.

However, since the Tickit::Console example runs via Tickit::Async, the way it runs is to wait in a poll() loop waiting on readability on STDIN. When STDIN becomes readable does it call tickit_term_input_readable(), and only then does it create the TermKey instance that actually sets up the terminal.

The upshot here is that until that entire first line of input is received, the terminal isn't actually set up to the correct mode in the first place.

Fixing this bug was a simple matter of making the underlying tickit_term_set_input_handle() mutator eagerly allocate the TermKey instance immediately, so that the terminal is already set up in the correct mode once the Tickit::Term constructor returns. That way it works correctly in both synchronous and asynchronous code.

In summary - when creating lazy functionality in a program, make sure that your laziness doesn't cause you to neglect to set something up that someone else was relying on. Be lazy, but don't be too lazy.

Wide mouse support in libvterm / libtermkey

(If you don't want to read the boring history you can skip to the end...)

Mouse in terminals. It's often been a mess. Traditionally terminals just send some ASCII-like representation of the keys you press, and mouse doesn't easily fit that. Therefore, when DEC first started adding mouse ability to terminals, they had to find a way to encode events into the byte stream.

They decided on a CSI sequence. Despite that CSI sequences can include numeric parameters, they decided the encoding should be a simple CSI M with no arguments, followed by three plain bytes which encode the buttons and position (for reasons that I suspect are lost in the mists of time). These three bytes encode, in order, a bitmask containing the button event and modifier keys, then the line and column number. In order to avoid sending C0 bytes within the stream for these, each byte is offset by adding 0x20 to ensure it's not a C0 control byte. This has the downside of leaving atmost 224 (256 - 32) columns or lines that can be encoded. On your traditional 80x25 glass teletype of course that was not a problem, but with today's 30inch widescreen monitors, it's easily possible to make a virtual terminal wider than this limit.

To solve this, a few extended encoding schemes have been invented. The first was the somewhat bizarre encoding simply named "Extended Encoding". This encodes the three values not as single bytes, but as UTF-8 encoded characters. This has a number of downsides, most notably the fact that it cannot be recognised unambiguously as compared to the other ones, so it's best to avoid it. Next are two largely-similar schemes that both use the parameters of CSI sequences to encode the parameters of the mouse event. rxvt encoding simply uses CSI M with the three values as CSI parameters, whereas the nicest of the encodings, so-called SGR encoding, uses either CSI < M or CSI < m to encode button press vs. button release. This fixes the other shortcoming of all the other encodings, in that SGR encoding can encode which button was released on a release event, whereas all the others can only encode that it was a release.

I recently got around to implementing these in my two main terminal-related projects; libvterm and libtermkey. libvterm (demonstrated here via pangoterm) can issue mouse events in any of these encodings if requested by the application. libtermkey will automatically recognise basic, rxvt and SGR encodings automatically, but won't handle the "extended" UTF-8 encoding, because it can't be unambiguously identified from basic.

And now without further ado, here is a rather uneventful screenshot that shows an enormously wide pangoterm demonstrating mouse events past the 224th column.

And in case the text is a little small to read here, it looks like this:

$ ./demo -m -p 1006
Mouse mode active
<MousePress(1) @ (72,33)>
<MouseRelease(1) @ (72,33)>
<MousePress(1) @ (148,25)>
<MouseRelease(1) @ (149,25)>
<MousePress(1) @ (219,26)>
<MouseRelease(1) @ (219,26)>
<MousePress(1) @ (288,28)>
<MouseRelease(1) @ (288,28)>
<MousePress(1) @ (297,68)>
<MouseRelease(1) @ (297,68)>
<C-c>
Mouse mode deactivated

Should work on a recent xterm etc.. as well.

Pangoterm - coloured icons

Lately I've been working more on pangoterm again. A few days ago I was feeling oddly artistic, and decided to draw an icon for it, to distinguish it in window lists, etc..

I use a variety of differently-coloured terminals to represent different scenarios (mostly SSH terminals, picking colours to represent different machines), by changing the background colour.

I thought to myself "wouldn't it be great if the window icon could change background colour to set the actual background colour used by that terminal window".

So I did that.

By a little bit of custom CSS hackery, it loads the SVG via a stylesheet that overrides the fill colour of the background colour element, and sets it to the colour chosen by that terminal. The upshot here is that the icon used in the workspace switcher, window list, and other places, looks like the terminal window.

ANSI vs DEC, arbitrary scrolling in terminals

Lately I've been looking at trying to finish the effectively half-finished effort that DECSTBM, IL and DL give us, by extending it with column awareness. I notice that VT420 gives us these things; namely DECSLRM (set left/right margin) an analogy to DECSTM, and DECIC/DECDC, to insert or delete a column. Using this combination, you can define an arbitrary scrolling rectangle using DECSTBM+DECSLRM, then scroll it in any of the four directions by placing the cursor in the topleft corner and issuing IL/DL/DECIC/DECDC as required. The VT420 manual says that the terminal defaults to a mode wherein "Left and right margins cannot be changed." (DECVSSM; vertical split-screen mode). You have to enable it first by issuing CSI ? 69 h.

This scheme allows us all sorts of new abilities, such as:

• Up/downward scrolling of a vertically-split window by setting top/bottom/left/right margins and scrolling as normal with SU/SD.
• Bounded ICH/DCH that doesn't extend all the way to the righthand edge by setting right margin, allowing us to implement insert/delete in a readline-alike whose entry area is not the entire width of the terminal.

(Both of these abilities I would verymuch like ;) )

However in actual practice it's not quite as simple as this. DECSLRM is encoded as CSI s, to follow DECSTBM at CSI r. This creates a problem because ANSI.SYS defines CSI s as SAVE, an operation identical to DECSC.

So what to do about this problem? My initial thought was to say "ignore ANSI.SYS", and support just the DEC scheme. I feel that arbitrary scrolling is too important a feature to be lacking for the sake of this legacy compatibility.

I can therefore see a number of possible ways out of this:

0. Ignore DECSLRM and any column-based scrolling ability (the status quo). I dislike this because it means I don't get to use scrolling rectangles.
1. Entirely remove ANSI SAVE/RESTORE ability on CSI s/CSI t and repurpose CSI s to mean DECSLRM. I'd have no problem with this, but there may be the odd legacy application or two that somehow expects ANSI SAVE/RESTORE to actually work (even though it is semantically identical to the one-byte-shorter DECSC/DECRC).
2. Try to multiplex both meanings onto one sequence by using some sort of heuristic to determine which meaning might be meant. Such thoughts as:
   1. Use the value of DECVSSM to decide which meaning to apply to CSI s - if DECVSSM is enabled then CSI s means DECSLRM; if disabled it means ANSI SAVE. This feels to me the most preferrable solution.
   2. Use the value of mode ?1049 or similar, because most(all?) applications wanting to use scrolling rectangles will be using alternate buffer anyway. This would make it impossible to use DECSLRM in main-screen applications, such as any scrolling readline-alike console.
   3. Use the setting of DECSTBM to decide - if top/bottom margins are set it's likely that the application will want to set left/right and that it's an application that knows DEC-like things and won't want to use ANSI SAVE anyway. This does make it impossible to just use DECSLRM on its own though, to give you bounded ICH/DCH.
3. Ignore entirely what VT420 does, and extend this behaviour in some other way. Perhaps for example, allow DECSTBM to take up to four parameters, defining the left and right margins in the third and fourth. This has the notable advantage that any application currently sending just the two-parameter form simply supplies default values for the other two positions, implying the full column width.
4. Make it user-configurable at runtime by some config option, commandline flag, etc...

My votes in preference are 3, 2a, 1; and thereafter the rest all sound horrible. But I'd be interested in hearing what ideas anyone else has...

Patches^W Unit Tests Welcome

There's a saying in Open Source communities "Patches Welcome". However, I find that some sorts of patches are more welcome than others.

Recently I've been working on putting strptime into POSIX. In response of one of my patches on the perl5-porters@ mailing list, Nicholas Clark wrote me a reply, attached to which was a new set of unit tests, most of which the code did not currently pass, but ought to.

As patches go, this is one of the most useful patches I have ever received. He didn't try (too much) to fix the code, didn't spend that long explaining the problems. He just sent a unit test. Now let me explain why this is useful.

With a unit test script, I can have it run against any version of the code I write. I can run it every minute until it works. I can run it at 3 in the morning during one of my late-night hacking sessions. I can run it on the underground train on my commute to work. It is close, immediate, convenient.

A discussion over a patch to the code alone would be at least an order of magnitude slower than this. People keep different time schedules. People aren't always available online. People don't always get what each other means the first time, leading to more back-and-forth discussion until an agreement is reached.

Without wishing to upset too many members of the Open Source community that might disagree with me, I would say "Patches (to code) Welcome, but unit tests, examples, or documentation would be preferred." I can easily hack up some code to pass the tests you supply, whereas if you just supply some code it's much harder for me to work out what you actually wanted.

libtermkey 0.12 + Term::TermKey 0.11

Last week I updated libtermkey to version 0.12, and associated with it, the Perl module wrapper Term::TermKey to version 0.11.

Primary changes are the fact it will now support an abstract mode, where an instance can be constructed not associated with a TTY filehandle. Instead of being read from the filehandle, bytes provided to the library directly by the application.

While I originally wrote this to support someone using the library where the application is already reading in bytes from the TTY by some other mechanism, I did lately find a new use for it. By applying a small string parser to a readline inputrc file, its raw byte sequences can be turned into libtermkey representations instead. I don't quite have an example of this robust enough yet to demonstrate, but hopefully by the next version of Term::TermKey I'll get around to writing one. I have a plan to write a Tickit::Widget::ReadlineAlike, which would go as far as parsing the user's inputrc file to discover all the keybindings to apply.

Also included in 0.12 are plenty of manpage updates, including also an online copy of them rendered into HTML on my website here.

Perl - more Socket

Some of you may have noticed that I'm now maintaining Socket dual-life on CPAN. Recently uploaded is version 1.96, which is an extraction of what was in bleadperl, updated to support building out-of-core on versions of perl back to at least 5.10.0, and some nicely rewritten documentation.

My plans for 1.97 are to neaten up the build system a bit more. Currently it's a rather hastily-written set of support code to handle the dual-life nature of it, so it can build on CPAN outside of core. Once that is in place, it will be much easier to add support for new features.

At this point I'll be starting to take more ideas from around CPAN. What constants or structure handling functions need adding. What socket options are being used in practice? And new protocol families it doesn't yet support - e.g. PF_RFCOMM (Bluetooth)? If anyone has any feature requests, now would be an excellent time to get them to me. :)