Released: Tickit 0.47

New up in Tickit 0.47, some fairly small and incremental updates:

• Support the 'blink' terminal attribute

  Both in libtickit C library and Tickit perl module now support the blink attribute, much to my hesitation. ;)

  I'm not sure I want to encourage this sort of thing, but the Neovim project said they wanted this, so I've reluctantly added support for it all the same.

• Bugfix for renderbuffer 'get*' methods

  When offset and clipping are applied, previously the get* methods didn't pay attention to this, fetching content relative to the toplevel, or segfaulting if requested out of bounds. This has now been fixed.

• Tickit::Widget::HBox and ::VBox have now been moved to the Tickit-Widgets distribution.

  This dist is now linked to explicitly from the documentation. This supports the longterm goal of turning 'Tickit' into purely the window-layer downwards, and having all the widget support live in its own distribution, backed eventually by its own C library.

• Nicer handling of fallback terminfo attributes for definitions missing them.

  Certain distributions of terminfo databases seem to be lacking certain essential attributes for some termtypes. Such examples as the upstream 'screen' terminfo is lacking erase_chars. libtickit now includes some fallback "likely to work" strings to handle these cases. This turns it from an instant failure on startup, to an at-worst wrong output, but hopefully most terminals should understand these standard strings. At least, if they don't I suspect libtickit is far from the only place that's broken, if they don't supply a more correct string in their terminfo.

Perl - Tickit - 0.40

Latest Tickit version (0.40) is now up on CPAN. Recent changes include:

• Mouse drag-and-drop events (0.32)

  Windows now create more interesting events to represent mouse drag-and-drop operations. Starting, moving, and ending a mouse-drag all create events that can help widgets render more interesting behaviours.

• Added Tickit::RenderBuffer (0.33)

  RenderBuffer is an in-memory buffer to store content that will eventually be rendered to the terminal. In effect it stores a double-buffer of content, allowing widgets to draw in whatever order is most convenient for them, before efficiently flushing it in a top-to-bottom manner.

  Being implemented in C/XS instead of Perl allows it to operate more efficiently than the previous-generation direct rendering to Windows. As it stores the content before rendering it also allows better handling of Unicode line-drawing characters; allowing for characters to be merged together out of multiple line segments, creating the ability for complex line-drawing shapes to be easily rendered.

• Added timer support to core event loop (0.34)

  Widgets can now react to timed events, allowing for animation effects and other behaviours.

• Focus management (0.34 and 0.35)

  Container widgets now provide management of the focus-chain order of their children, allowing the whole widget tree to maintain the "next" and "previous" direction of focusing. The base code handles the <Tab> and <Shift-Tab> key events to cycle input focus around the widget tree automatically.

• Use RenderBuffer in favour of direct Window rendering (0.40)

  Now that RenderBuffer is in the core distribution, all widgets should be using it rather the previous render method to render directly on the window. This is a stepping-stone change to allow for further improvements.

• New sizing model (0.40)

  New methods are provided by the base Tickit::Widget class that cache the requested size of the widget, and only inform the parent container when this size actually changes. This allows for more efficient reshaping and redrawing operations when widgets change their size requirements, without needing to recalculate the whole widget tree.

These final two changes help support a couple of interesting planned improvements:

• Widget minimal/maximal size handling

  Because the base widget class now handles size information more directly, it can implement bounds on minimal and maximal widget size. These will be derived from Tickit::Style. It may also be possible to consider padding and margin controls in the base widget class and thus automatically apply to every widget.

• Whole-tree rendering via RenderBuffer

  By adding area masks to RenderBuffer it should then support being used as a single buffer object to render the entire window hierarchy. Because the masking code will be implemented in C code, it will much more efficient than the current pure-perl Window-based solution involving visibility testing per character cell. This will allow for much more better redrawing performance.

  This will also allow RenderBuffer to move out of the Tickit.xs file and into libtickit itself, where it can be useful to non-Perl code (such as native C programs or other language bindings).

Tickit version 0.31

(mostly a copy of the mail to the tickit-dev mailing list)

A lot of stuff happening lately. And also I haven't written one of these for ages. I won't go into every detail, but here's a rundown of the most interesting parts:

• Rect/RectSet are now C library based (0.26)

  Fairly simple, no surprises here. C implementation means it's available in C and other languages, and probably a bit faster in actual use.

• New Term event binding API (0.26)

  Rather than a single on_key/on_mouse/etc..., there is now just a list of possible event handlers. Event handling subs don't have to be restricted to a single event; each is registered with a bitmask. This is done to more closely match the C API.

  The existing Perl API of having a single on_* handler for each event type is still supported, by wrapping the newer API.

• All windows are now FLOAT windows (0.28)

  As was first suggested in 0.23, all the windows now use the new float logic. The previous environment variable has now been removed. This hopefully shouldn't actually affect anything as it's been the default for a while now, but does simplify the code internals.

• $win->close and no more weak references (0.28)

  Using weak references and relying on DESTROY works OK in some circumstances in Perl, but won't scale to C and other languages, and still makes for tricky logic. To this end, I've removed all the weaken()ing and replaced it with an explicit ->close method to remove a window. This also makes it much more robust in nontrivial cases.

  This change is mostly of interest to container widget developers, or in more dynamic long-lived programs.

• Tickit::Style (0.29)

  This one's the big main one of the list; in fact so bit I'll probably write another mail. In summary; we now have something of a first attempt at being able to separate out style from widget implementations, in a way that's easy to add to application- or user-specific style files. More on this later.

• Tickit::Pen now comes in mutable and immutable forms (0.30)

  Since most pens don't get mutated, and Tickit::Style performs better with cached pens, I've split the idea of a Pen into mutable and immutable types. Tickit::Style returns immutable pens, so widgets shouldn't attempt to mutate them.

  Tickit::Pen->new itself still returns a mutable pen for now, but in the future this may change; code that specifically wants a mutable or immutable pen should use the appropriate subclass.

• Tickit::Pen changes to support upcoming Tickit::RenderContext (0.31)

  A few small changes that allow the new Tickit::RenderContext to work better.

Also some changes in the underlying libtickit C library:

• Generic string/integer value termctl operations

  Primarily provided to let the xterm driver set the window title, etc..., but the general idea is something similar to ioctl(), so we don't have to extend the API a thousand times just to add lots of little options for specific terminals.

• Split xterm/TI-based driver model

  To support more specific options in future, and also to give a better (or more accurate) terminfo-based driver. There are now two drivers, selected by the $TERM environment variable, so the xterm-specific things can be done nicely, and still arrange for the generic terminfo driver to work.

  This also allows for other terminal-specific drivers in future, in case we find those useful. Perhaps a Win32 console one too.

Project Updates - Tickit window scrolling, Circle scroll indicators

Some progress on a few of my projects:

Tickit::Window now supports scroll and scrollrect even if the window partly covered by other floating windows. With the expose-after-scroll behaviour turned on (soon to be default in a later version once I know it works nicely), it always returns true, and tries to scroll as many regions of the screen as it can, queueing expose operations for the areas that needed it.

A good example of this working can be seen in the newly-updated Tickit::Widget::Scroller, which now supports a scroll position indicator, a small floating window in the top or bottom right corner.

Another example can be seen as the latest feature in Circle::FE::Term. Scrolling a channel window gives an indicator in the bottom right corner, which also includes a count of new lines of content added since scrolling, if there are any.

Tickit, floating windows and menus

The latest release of Tickit, version 0.18, adds support for floating windows. These are windows that rather than divide space of their parent window, instead float above it by obscuring content below, creating a Z-ordering. Drawing operations on windows are aware of clipping regions caused by floating windows above them, and are accounted for automatically without individual widgets needing to be aware of it.

A special kind of floating window called a popup window can be created. These are always created as children of the root window, and take first priority at input events. These can be used to create things like popup menus, such as this early prototype of Tickit::Widget::Menu.

This is still somewhat of a work-in-progress, and by the time it's released it should support proper Unicode linedrawing rather than the ASCII variant. I'm also planning to create a way to make pop-up windows of arbitrary widgets, to create things like modal dialog boxes.

Don't be too lazy

Today I fixed a bug in my C library rewrite of Tickit.

The manifestation of the bug was that Tickit::Console didn't work properly via the C/XS version of Tickit, but worked fine on the Perl version. It wasn't receiving keyboard input at all. Yet I know basic keyboard input works fine on all the simple demos with the C library version, so something more subtle was up.

On close inspection it turned out that libtickit was being too lazy with constructing its libtermkey instance for handling keyboard input. The previous code arrangement was that constructing the instance was deferred as late as possible, until one of the input-handling methods was actually called. This meant that setting the filehandle could be done as a normal accessor and not as a constructor argument. (C functions lack the neatness of Perl's named argument style, so my usual style is minimal constructors and lots of mutation accessors).

This worked fine for simple cases, because after setup the demo programs all wait on a call to tickit_term_input_wait(), and the first thing that does is create the TermKey instance, which sets up the terminal for non-canonical input mode and disables local echo. This ensures input arrives a key at a time.

However, since the Tickit::Console example runs via Tickit::Async, the way it runs is to wait in a poll() loop waiting on readability on STDIN. When STDIN becomes readable does it call tickit_term_input_readable(), and only then does it create the TermKey instance that actually sets up the terminal.

The upshot here is that until that entire first line of input is received, the terminal isn't actually set up to the correct mode in the first place.

Fixing this bug was a simple matter of making the underlying tickit_term_set_input_handle() mutator eagerly allocate the TermKey instance immediately, so that the terminal is already set up in the correct mode once the Tickit::Term constructor returns. That way it works correctly in both synchronous and asynchronous code.

In summary - when creating lazy functionality in a program, make sure that your laziness doesn't cause you to neglect to set something up that someone else was relying on. Be lazy, but don't be too lazy.

libvterm/pangoterm and Tickit

Lately I've written a bit about my terminal emulator library, libvterm, and briefly mentioned Tickit, my terminal UI module for Perl. I'll write about each in more detail soon, but I thought since I hit an important milestone recently, I'd write a little something more about libvterm and pangoterm.

libvterm is a purely abstract C99 library that implements the bulk of the logic of being a terminal emulator. Bytes from the PTY master are fed into it by the containing program, and it maintains the abstract state of the terminal; the position of the cursor, the state of the pen, what charcters are where with what attributes, and so on. It calls callback functions registered by the containing program, to inform it of damaged screen regions that need repainting. Two of the main selling points of the library are

• It is purely abstract C99, doesn't rely on POSIX or any particular rendering/UI system


• During normal operation of just feeding it bytes and processing events, it does not use the malloc system.

These properties make it ideal for a number of situations, ranging from desktop applications, to small embedded systems or operating system kernels.

pangoterm is a GTK/Pango-driven embedding of this libary, in a simple single-.c-file application, mostly for me to develop and test it. It is currently maintained in the libvterm source tree.

For a while now this combination has been complete enough to drive vim sufficient to edit its own source code - pangoterm and libvterm are now self-hosting. A couple of weeks ago I finally managed to fix the last of a number of small issues making it not quite perfect. Last week I also managed to get pangoterm to completely correctly render a Tickit-based program; the final missing piece being some of the mouse tracking modes.

I now have a bit of extra configuration in my .vimrc to take advantage of a few of pangoterm's abilities, such as support for italics.


As well as italics, it also supports strikethrough and alternative fonts, although so far I've only managed to find one alternative font that actually looks at all decent alongside DejaVu Sans Mono. These are all shown off quite well by Tickit's demo-pen.pl example script here.


And finally here, a demo of the xterm-like 256 colour handling.


These screenshots briefly show Tickit working nicely with pangoterm. Sometime soon I shall get around to writing about Tickit in more detail, and also explaining some of my further plans for the whole Tickit+libtermkey vs libvterm combination.

Perl - Term::TermKey - version 0.09

Last week saw a new version of Term::TermKey (0.09), and also the underlying libtermkey (0.9). This contains a fairly small new feature, giving control of the way that EINTR is handled.

Version 0.8 added graceful handling of EINTR to restart IO operations rather than fail with an error. This had unfortunate knock-on effects for the Perl-level wrapping of it, because of the deferred nature of Perl's safe signals. On a signal (such as the not-so-unlikely SIGWINCH) a flag would be set, but the termkey_waitkey(3) operation would be restarted, not returning back to Perl's control until a keypress event was actually received. This upset programs that wish to respond to SIGWINCH and redraw the terminal to the new size.

Version 0.9 of libtermkey now therefore has a new flag, TERMKEY_FLAG_EINTR whose presence makes the blocking IO operations (termkey_waitkey(3) and termkey_advisereadable(3)) to return a new result code, TERMKEY_RES_ERROR after which the caller can inspect the value of errno, to observe an EINTR.

Again because of Perl's safe signal handling, the Perl wrapping of libtermkey has to always enable this flag, so it can invoke the $SIG{WINCH} signal handler, for example. The Term::TermKey module therefore now always sets TERMKEY_FLAG_EINTR on the underlying TermKey instance, and emulates the presence or absence of this flag at the Perl level, by optionally restarting its IO operation, or itself returning TERMKEY_RES_ERROR.

An unfortunate bug here in the emulation and hiding of this flag from the Perl level means that Term::TermKey 0.09 fails to correctly read the TermKey flags back out of the underlying object. In particular it fails to be able to check on the presence of TERMKEY_FLAG_UTF8 that libtermkey itself may have enabled, after detecting a UTF-8 locale. This causes Tickit's unit tests to break with the familiar "Wide character in syswrite at ..." error.

This bug has now been fixed in the source code repository, and will be present in the next version, 0.10. Tickit 0.10 also has an independent fix for the same bug, by using Perl's ${^UTF8LOCALE} instead of reading the TermKey flags back out again.

Also upcoming in libtermkey 0.10, will be some Solaris portability fixes, and a new canonicalisation flag, which turns a TERMKEY_SYM_DEL key into TERMKEY_SYM_BACKSPACE, for those terminals that send DEL on Backspace.

XS beats Pure Perl

Someone reported some test failures trying to install Tickit, which seemed to be related to shortcomings in Text::CharWidth. The latter seems to have very poor unit test coverage on itself, so the failures didn't appear during its installation, only when Tickit::Utils was tested against it. On initial inspection I wondered if Text::CharWidth simply wasn't using wcswidth(3) correctly, and whether I should get around to my plan of rewriting bits of Tickit::Utils in XS instead for performance, as well as work around this bug.

This turned out to be quite a good idea. Implementing cols2chars() and chars2cols() in XS instead of Perl makes them at least 10 times faster. I tested it on four strings; two ASCII and two Unicode; a long and a short of each:

Calls/sec		PP	XS	Ratio
chars2cols	along	48685	406504	834.97%
chars2cols	ashort	72674	704225	969.02%
chars2cols	ulong	37341	387596	1037.99%
chars2cols	ushort	52966	714285	1348.57%
cols2chars	along	16350	403255	2466.39%
cols2chars	ashort	58685	649350	1106.50%
cols2chars	ulong	13561	362318	2671.76%
cols2chars	ushort	50556	632911	1251.90%

In fact, some cases it turns out to be 24 times faster.

I haven't looked into too much detail on why, but I suspect a large amount of the reason is to do with the way the XS functions primarily walk along the internal UTF-8 representation of the strings, counting bytes, characters, and columns as they go, and returning the appropriate count(s) when the required. The pureperl implementation doesn't have direct access to the byte offsets, so only has character numbers to work to. The frequent character-to-byte or byte-to-character conversions at all the boundaries between the functions result in multiple UTF-8 byte skip counting steps along the string each time a function is entered or left, generally slowing it down.

As to the original test failure, it turned out to be entirely unrelated lack of locale support in the platform's libc. The XS implementations fail there in the same way. But having implemented the above improvements, I decided to leave them in anyway.

XS faster than Pure Perl; who'd have thought it?