Alt modifier - high-bit or escape?

I've wanted to write this up for quite a while. Haven't quite got around to it mostly for not knowing how to start. So instead of agonising over an introduction, I'll just paste IRC logs instead.

#vim on Freenode, on a subject of whether terminals should encode the Alt modifier key by high-bit mask, or Escape prefix:

You -always- want alt-sends-escape. Anything about high-bits just breaks UTF-8 and is always broken.

alt==highbit => it is impossible to distinguish UTF-8 input from Alt-modified input. Cannot be done. All input is broken. Fail. Go home
alt==esc => it is possible to distinguish Escape prefix from Alt-modified input by using timing information. Not perfect, sometimes false hits, but generally works.

False positives happen if you type too quickly on a possibly-laggy connection, and multiple 1-byte writes coaless into a single 2-byte read buffer. False negatives ought not happen if your terminal is working nicely, but could be possible in buggy terminals (hello ConnectBot + Hackers' Keyboard), or if e.g. network MTU issues start cutting your buffers in weeeeird places.

I recommend 20msec.

20 is good because the human eye/brain will fudge over intervals shorter than 20 msec.. if two things happen within 20msec, or something that you expect to happen, happens within 20msec after, you'll perceive it as instant.

So a 20msec escape timeout means your brain doesn't notice that 20msec delay after you press the Escape key, before it hits back to Normal mode again. But 20msec is looooads of time for vim to wait for the trailing letter that might just be coming after the ESC byte if it was in fact an Alt- modified key.

Incidentally, this is the style used by my libvterm terminal emulator library, and libtermkey terminal key event input library.

Wide mouse support in libvterm / libtermkey

(If you don't want to read the boring history you can skip to the end...)

Mouse in terminals. It's often been a mess. Traditionally terminals just send some ASCII-like representation of the keys you press, and mouse doesn't easily fit that. Therefore, when DEC first started adding mouse ability to terminals, they had to find a way to encode events into the byte stream.

They decided on a CSI sequence. Despite that CSI sequences can include numeric parameters, they decided the encoding should be a simple CSI M with no arguments, followed by three plain bytes which encode the buttons and position (for reasons that I suspect are lost in the mists of time). These three bytes encode, in order, a bitmask containing the button event and modifier keys, then the line and column number. In order to avoid sending C0 bytes within the stream for these, each byte is offset by adding 0x20 to ensure it's not a C0 control byte. This has the downside of leaving atmost 224 (256 - 32) columns or lines that can be encoded. On your traditional 80x25 glass teletype of course that was not a problem, but with today's 30inch widescreen monitors, it's easily possible to make a virtual terminal wider than this limit.

To solve this, a few extended encoding schemes have been invented. The first was the somewhat bizarre encoding simply named "Extended Encoding". This encodes the three values not as single bytes, but as UTF-8 encoded characters. This has a number of downsides, most notably the fact that it cannot be recognised unambiguously as compared to the other ones, so it's best to avoid it. Next are two largely-similar schemes that both use the parameters of CSI sequences to encode the parameters of the mouse event. rxvt encoding simply uses CSI M with the three values as CSI parameters, whereas the nicest of the encodings, so-called SGR encoding, uses either CSI < M or CSI < m to encode button press vs. button release. This fixes the other shortcoming of all the other encodings, in that SGR encoding can encode which button was released on a release event, whereas all the others can only encode that it was a release.

I recently got around to implementing these in my two main terminal-related projects; libvterm and libtermkey. libvterm (demonstrated here via pangoterm) can issue mouse events in any of these encodings if requested by the application. libtermkey will automatically recognise basic, rxvt and SGR encodings automatically, but won't handle the "extended" UTF-8 encoding, because it can't be unambiguously identified from basic.

And now without further ado, here is a rather uneventful screenshot that shows an enormously wide pangoterm demonstrating mouse events past the 224th column.

And in case the text is a little small to read here, it looks like this:

$ ./demo -m -p 1006
Mouse mode active
<MousePress(1) @ (72,33)>
<MouseRelease(1) @ (72,33)>
<MousePress(1) @ (148,25)>
<MouseRelease(1) @ (149,25)>
<MousePress(1) @ (219,26)>
<MouseRelease(1) @ (219,26)>
<MousePress(1) @ (288,28)>
<MouseRelease(1) @ (288,28)>
<MousePress(1) @ (297,68)>
<MouseRelease(1) @ (297,68)>
<C-c>
Mouse mode deactivated

Should work on a recent xterm etc.. as well.

Pangoterm - coloured icons

Lately I've been working more on pangoterm again. A few days ago I was feeling oddly artistic, and decided to draw an icon for it, to distinguish it in window lists, etc..

I use a variety of differently-coloured terminals to represent different scenarios (mostly SSH terminals, picking colours to represent different machines), by changing the background colour.

I thought to myself "wouldn't it be great if the window icon could change background colour to set the actual background colour used by that terminal window".

So I did that.

By a little bit of custom CSS hackery, it loads the SVG via a stylesheet that overrides the fill colour of the background colour element, and sets it to the colour chosen by that terminal. The upshot here is that the icon used in the workspace switcher, window list, and other places, looks like the terminal window.

ANSI vs DEC, arbitrary scrolling in terminals

Lately I've been looking at trying to finish the effectively half-finished effort that DECSTBM, IL and DL give us, by extending it with column awareness. I notice that VT420 gives us these things; namely DECSLRM (set left/right margin) an analogy to DECSTM, and DECIC/DECDC, to insert or delete a column. Using this combination, you can define an arbitrary scrolling rectangle using DECSTBM+DECSLRM, then scroll it in any of the four directions by placing the cursor in the topleft corner and issuing IL/DL/DECIC/DECDC as required. The VT420 manual says that the terminal defaults to a mode wherein "Left and right margins cannot be changed." (DECVSSM; vertical split-screen mode). You have to enable it first by issuing CSI ? 69 h.

This scheme allows us all sorts of new abilities, such as:

• Up/downward scrolling of a vertically-split window by setting top/bottom/left/right margins and scrolling as normal with SU/SD.
• Bounded ICH/DCH that doesn't extend all the way to the righthand edge by setting right margin, allowing us to implement insert/delete in a readline-alike whose entry area is not the entire width of the terminal.

(Both of these abilities I would verymuch like ;) )

However in actual practice it's not quite as simple as this. DECSLRM is encoded as CSI s, to follow DECSTBM at CSI r. This creates a problem because ANSI.SYS defines CSI s as SAVE, an operation identical to DECSC.

So what to do about this problem? My initial thought was to say "ignore ANSI.SYS", and support just the DEC scheme. I feel that arbitrary scrolling is too important a feature to be lacking for the sake of this legacy compatibility.

I can therefore see a number of possible ways out of this:

0. Ignore DECSLRM and any column-based scrolling ability (the status quo). I dislike this because it means I don't get to use scrolling rectangles.
1. Entirely remove ANSI SAVE/RESTORE ability on CSI s/CSI t and repurpose CSI s to mean DECSLRM. I'd have no problem with this, but there may be the odd legacy application or two that somehow expects ANSI SAVE/RESTORE to actually work (even though it is semantically identical to the one-byte-shorter DECSC/DECRC).
2. Try to multiplex both meanings onto one sequence by using some sort of heuristic to determine which meaning might be meant. Such thoughts as:
   1. Use the value of DECVSSM to decide which meaning to apply to CSI s - if DECVSSM is enabled then CSI s means DECSLRM; if disabled it means ANSI SAVE. This feels to me the most preferrable solution.
   2. Use the value of mode ?1049 or similar, because most(all?) applications wanting to use scrolling rectangles will be using alternate buffer anyway. This would make it impossible to use DECSLRM in main-screen applications, such as any scrolling readline-alike console.
   3. Use the setting of DECSTBM to decide - if top/bottom margins are set it's likely that the application will want to set left/right and that it's an application that knows DEC-like things and won't want to use ANSI SAVE anyway. This does make it impossible to just use DECSLRM on its own though, to give you bounded ICH/DCH.
3. Ignore entirely what VT420 does, and extend this behaviour in some other way. Perhaps for example, allow DECSTBM to take up to four parameters, defining the left and right margins in the third and fourth. This has the notable advantage that any application currently sending just the two-parameter form simply supplies default values for the other two positions, implying the full column width.
4. Make it user-configurable at runtime by some config option, commandline flag, etc...

My votes in preference are 3, 2a, 1; and thereafter the rest all sound horrible. But I'd be interested in hearing what ideas anyone else has...

libvterm/pangoterm and Tickit

Lately I've written a bit about my terminal emulator library, libvterm, and briefly mentioned Tickit, my terminal UI module for Perl. I'll write about each in more detail soon, but I thought since I hit an important milestone recently, I'd write a little something more about libvterm and pangoterm.

libvterm is a purely abstract C99 library that implements the bulk of the logic of being a terminal emulator. Bytes from the PTY master are fed into it by the containing program, and it maintains the abstract state of the terminal; the position of the cursor, the state of the pen, what charcters are where with what attributes, and so on. It calls callback functions registered by the containing program, to inform it of damaged screen regions that need repainting. Two of the main selling points of the library are

• It is purely abstract C99, doesn't rely on POSIX or any particular rendering/UI system


• During normal operation of just feeding it bytes and processing events, it does not use the malloc system.

These properties make it ideal for a number of situations, ranging from desktop applications, to small embedded systems or operating system kernels.

pangoterm is a GTK/Pango-driven embedding of this libary, in a simple single-.c-file application, mostly for me to develop and test it. It is currently maintained in the libvterm source tree.

For a while now this combination has been complete enough to drive vim sufficient to edit its own source code - pangoterm and libvterm are now self-hosting. A couple of weeks ago I finally managed to fix the last of a number of small issues making it not quite perfect. Last week I also managed to get pangoterm to completely correctly render a Tickit-based program; the final missing piece being some of the mouse tracking modes.

I now have a bit of extra configuration in my .vimrc to take advantage of a few of pangoterm's abilities, such as support for italics.


As well as italics, it also supports strikethrough and alternative fonts, although so far I've only managed to find one alternative font that actually looks at all decent alongside DejaVu Sans Mono. These are all shown off quite well by Tickit's demo-pen.pl example script here.


And finally here, a demo of the xterm-like 256 colour handling.


These screenshots briefly show Tickit working nicely with pangoterm. Sometime soon I shall get around to writing about Tickit in more detail, and also explaining some of my further plans for the whole Tickit+libtermkey vs libvterm combination.

Perl - Term::Terminfo - version 0.06

I've just uploaded a new version of Term::Terminfo. In brief; this is a small wrapper around the terminfo database, and can be used to enquire about properties of a given terminal. For annoying historic reasons, each of these properties is known by at least two names; its short "capname", a two or three letter code often found in the actual file on disk, and a longer "varname", the name of a variable in the curses C library, which stores its value for the current terminal.

This latest version, 0.06, adds a whole duplicate set of methods - varname accessors. Prior to 0.06, the properties were only accessible using their short capnames.

It's probably best to use the longer varnames anyway. They are more self-documenting, and a little more future-proof against the admittedly-remote possibility of new variables being added in the future.

I'm also planning to support unibilium in a later version. This is a standalone terminfo-parsing library, which is useful for reading terminfo data without linking against the full curses library. This is especially useful when trying to build a replacement for curses...

Extended colour support, terminals, and ECMA-48

tl;dr summary: Terminal authors - please accept CSI 38:5:123 m as well as anything else, to set extended colours. NOTE THE COLON. Many terminals these days are starting to support extended colour modes; specifically things like 256 colours. They're all doing it wrong.

CSI 38;5;123 m

No no no no no. That is NOT what you think. It does not select colour 123 from palette 5. CSI arguments are separated by semicolons. This sets three unrelated SGRs; 38, 5, and 123. 38 sets a foreground colour to .. er.. nothing in particular. 5 is blinking mode, 123 has no defined meaning to my knowledge. All the other following are exactly identical:

CSI 5;38;123 m
CSI 38 m CSI 5 m CSI 123 m

ECMA-48 already defines a perfectly good way for parameters to take sub-parameters. It's the colon. The correct way to encode this concept is

CSI 38:5:123 m

Why does this matter? It matters for parsers not to have to understand what is going on. Consider

CSI 3;38:9:5:4:3:2:1;11;1

This is equivalent to

CSI 3 CSI 38:9:5:4:3:2:1 CSI 11; CSI 1

I.e. I have no idea what palette 9 is, but I can definitely parse out the contents of this SGR from the others, knowing exactly where it ends. I don't have to "just know" that palette 9 happens to take 5 parameters. The CSI encodes this. My own terminal emulator library, libvterm, already understands these colon-based arguments. It parses them correctly. This is important for other palettes that don't take just one value. For example, the RGB, RGBA, CMY, and CMYK palettes. It's vital to be able to parse out these sub-arguments from the single SGR 38 or 48. Standards exist for a reason, people. Please use them.

---

Edit 2012/12/20: actually, I have recently learned that xterm now supports this in its correct form as well, since xterm patch 282.