Since it's been a while since I wrote anything, and I've lately been doing some work on both pangoterm and the underlying libvterm I thought I'd write a little post summarising the recent updates.
Two new features now supported in pangoterm are bracketed paste mode, and configurable basic and high-brightness palettes.
Bracketed paste mode is a feature that applications can enable, which causes text pasted into the terminal via the clipboard to be surrounded by special markers indicating this fact. Applications like text editors can make use of these markers to interpret the keystrokes that arrive as literal text, overriding whatever other meaning they might wish to apply during that time.
There's also a few bugfixes; interpretation of numeric keypad events and IME handling (written by FireFly on Freenode), and a fix for cells with the maximally-supported set of combining chars applied (such as so-called Zalgotext).
On the libvterm front, there is now a second set of callbacks at the State and Screen layers, to allow embedding applications to at least inspect CSI, DCS or OSC strings that libvterm itself doesn't understand, thus allowing some method of extension by applications. I'm not terribly happy with the interface shape of these yet - it will likely change in future - but for now it serves its purpose to satisfy Neovim.
There's a fix for an awkward bug I've been trying to isolate for a while, which involved the Screen layer in its most aggressive optimisation setting, where there is a pending scrollrect and pending damage, and another scrollrect arrives. This illusive bug was the cause of occasional rendering failures during scroll after a quick line edit in command shells, and similar situations.
I've been keeping a list of terminal sequences unrecognised by libvterm and pangoterm. For a while I've kept the DEC double-width line and double-height line sequences in there, because I wasn't sure how they would interact with the arbitrary scrolling rectangles defined by VT4xx. The VT400 manual doesn't really mention how to handle this, but by chance I happened to be reading a VT500 manual instead, which does. It explains that the double-width or -height sequences aren't recognised in SLRM (Set Left/Right Mode; newly renamed from the identical but differently-named Vertical Split-Screen Mode in VT400); and that enabling SLRM will revert all lines back to single-width, single-height.
This neatly handles the problem that would otherwise occur, in that these line modes apply to entire lines. If a partial scrolling operation were to affect such a line, what would happen to the double-width characters in it? By linking it with SLRM this is avoided - either the terminal is in a mode where double-width can happen, or it's an a mode where partial lines can be scrolled - but never both.
With this neatly resolved, I finally got around to implementing DECDHL and DECDWL in libvterm and pangoterm. Here's a screenshot showing the ever-useful vttest.
With this out of the way, the list of unsupported VT1xx features is rapidly diminishing. About all that's left asides real-hardware things like interlace mode (which will be impossible to do), is 80/132 column mode. That's another one I'm still not sure how to implement...
I've been working more on pangoterm, and it now has two important features that bring it much closer to being a real usable terminal.
The first is that it now supports a lot more configuration by command-line, and also the same things by a configuration file. Support for a config file isn't something I personally have directly needed so far, mostly because the compiled-in defaults are already what I'd like. But it's something other people have been asking for, and this helps support other users too, so it's nice to have.
I also found quite a neat way to implement it too, requiring just one line of macro to declare the existence of a configuration setting and give its default value and help description. This then creates a normal variable that can be read in the usual way:
CONF_INT(scrollback_size, 0, 1000, "Scrollback size", "LINES"); ... pt->scroll_size = CONF_scrollback_size;
The other main feature that's now in place is scrollback buffer. Now, whenever text disappears off the top of the screen it is saved, and can be recalled again by scrolling the mouse wheel or using the Shift-PageUp and Shift-PageDown keys. A small custom-drawn indicator shows the scroll position without needing to consume any screen space in the normal state of not being scrolled.
I also found and fixed an almost embarrassingly-bad bug around the redrawing code. When erasing cells, it used to make one cairo call for every cell to be erased, leading to many calls whenever an entire line had to be cleared. Now it composes multiple blank cells with the same pen, in the same way as it composes multiple cells of text; and is faster when scrolling partial lines, and so on.
While I've already been using it quite a lot lately to run the vim I use to edit its own source code, for some tasks I've still been using xterm. But not any more - with the addition of scrollback to pangoterm, I find I'm now using it for everything. It's become my default terminal both for home and work use.
$ time cat fileThis technically measures the amount of time it takes cat to write the data into the TTY, but since the buffer is a fixed 4KiB in size, it also measures the time that pangoterm takes to read all but the final 4KiB of the file, which is under 1% of the file. I managed to obtain the following timings. Before I started:
real 0m4.206s user 0m0.000s sys 0m0.036sBy optimising libvterm's moverect buffer operation with memmove() (revision -r511):
real 0m2.035s user 0m0.000s sys 0m0.048sWith pangoterm deferred updates, that delay re-rendering of the screen until all the PTY data has been read, or every 20 msec (revision -r482):
real 0m0.358s user 0m0.004s sys 0m0.016sIt's now well over 10 times faster than it used to be. Ohh.. and it beats xterm. By quite a bit.
real 0m2.294s user 0m0.000s sys 0m0.036s
(If you don't want to read the boring history you can skip to the end...)
Mouse in terminals. It's often been a mess. Traditionally terminals just send some ASCII-like representation of the keys you press, and mouse doesn't easily fit that. Therefore, when DEC first started adding mouse ability to terminals, they had to find a way to encode events into the byte stream.
They decided on a CSI sequence. Despite that CSI sequences can include numeric parameters, they decided the encoding should be a simple CSI M with no arguments, followed by three plain bytes which encode the buttons and position (for reasons that I suspect are lost in the mists of time). These three bytes encode, in order, a bitmask containing the button event and modifier keys, then the line and column number. In order to avoid sending C0 bytes within the stream for these, each byte is offset by adding 0x20 to ensure it's not a C0 control byte. This has the downside of leaving atmost 224 (256 - 32) columns or lines that can be encoded. On your traditional 80x25 glass teletype of course that was not a problem, but with today's 30inch widescreen monitors, it's easily possible to make a virtual terminal wider than this limit.
To solve this, a few extended encoding schemes have been invented. The first was the somewhat bizarre encoding simply named "Extended Encoding". This encodes the three values not as single bytes, but as UTF-8 encoded characters. This has a number of downsides, most notably the fact that it cannot be recognised unambiguously as compared to the other ones, so it's best to avoid it. Next are two largely-similar schemes that both use the parameters of CSI sequences to encode the parameters of the mouse event. rxvt encoding simply uses CSI M with the three values as CSI parameters, whereas the nicest of the encodings, so-called SGR encoding, uses either CSI < M or CSI < m to encode button press vs. button release. This fixes the other shortcoming of all the other encodings, in that SGR encoding can encode which button was released on a release event, whereas all the others can only encode that it was a release.
I recently got around to implementing these in my two main terminal-related projects; libvterm and libtermkey. libvterm (demonstrated here via pangoterm) can issue mouse events in any of these encodings if requested by the application. libtermkey will automatically recognise basic, rxvt and SGR encodings automatically, but won't handle the "extended" UTF-8 encoding, because it can't be unambiguously identified from basic.
And now without further ado, here is a rather uneventful screenshot that shows an enormously wide pangoterm demonstrating mouse events past the 224th column.
And in case the text is a little small to read here, it looks like this:
$ ./demo -m -p 1006 Mouse mode active <MousePress(1) @ (72,33)> <MouseRelease(1) @ (72,33)> <MousePress(1) @ (148,25)> <MouseRelease(1) @ (149,25)> <MousePress(1) @ (219,26)> <MouseRelease(1) @ (219,26)> <MousePress(1) @ (288,28)> <MouseRelease(1) @ (288,28)> <MousePress(1) @ (297,68)> <MouseRelease(1) @ (297,68)> <C-c> Mouse mode deactivated
Should work on a recent xterm etc.. as well.
