LeoNerd's programming thoughts

libtermkey 0.12 + Term::TermKey 0.11

2012-01-29T17:16:00.000Z

Last week I updated libtermkey to version 0.12, and associated with it, the Perl module wrapper Term::TermKey to version 0.11.

Primary changes are the fact it will now support an abstract mode, where an instance can be constructed not associated with a TTY filehandle. Instead of being read from the filehandle, bytes provided to the library directly by the application.

While I originally wrote this to support someone using the library where the application is already reading in bytes from the TTY by some other mechanism, I did lately find a new use for it. By applying a small string parser to a readline inputrc file, its raw byte sequences can be turned into libtermkey representations instead. I don't quite have an example of this robust enough yet to demonstrate, but hopefully by the next version of Term::TermKey I'll get around to writing one. I have a plan to write a Tickit::Widget::ReadlineAlike, which would go as far as parsing the user's inputrc file to discover all the keybindings to apply.

Also included in 0.12 are plenty of manpage updates, including also an online copy of them rendered into HTML on my website here.

Perl - more Socket

2011-12-13T16:22:00.001Z

Some of you may have noticed that I'm now maintaining Socket dual-life on CPAN. Recently uploaded is version 1.96, which is an extraction of what was in bleadperl, updated to support building out-of-core on versions of perl back to at least 5.10.0, and some nicely rewritten documentation.

My plans for 1.97 are to neaten up the build system a bit more. Currently it's a rather hastily-written set of support code to handle the dual-life nature of it, so it can build on CPAN outside of core. Once that is in place, it will be much easier to add support for new features.

At this point I'll be starting to take more ideas from around CPAN. What constants or structure handling functions need adding. What socket options are being used in practice? And new protocol families it doesn't yet support - e.g. PF_RFCOMM (Bluetooth)? If anyone has any feature requests, now would be an excellent time to get them to me. :)

No longer thinking about Perl 5.8

2011-11-24T20:11:00.000Z

This week I got around to rebuilding all my local Perl module packages for the Perl 5.14.1 now in debian testing. Because that replaced the 5.12.4 that was previously running, I've rebuilt that into my homedir with perlbrew.

Added to the 5.10.1 I previously had, that's still one system and two custom installed perls I have. I try to remember to test everything before releasing it to CPAN on all these three.

At this point, I can't really be bothered to look after 5.8 as well, so effectively I'm no longer testing anything on 5.8. I'll still keep an eye on smoke-test results and see if they're passing or failing, and if I see a failure that looks easy to fix I might have a go at it. But no promises now.

If I manage to break anything on 5.8 in a module of mine and anyone actually cares about it, feel free to raise me a bug on rt.cpan.org. But apart from that, I hope I don't have to think too much about 5.8 any more.

LPW2011 Talk Slides

2011-11-15T19:00:00.001Z

If anyone wants to see the slides of the talks I co-presented with Tom this year at LPW, they're at

Though, at least the latter talk was mostly a series of code demos, so the slides alone won't make much sense.

Perl - Tiny lightweight structures module

2011-11-03T19:11:00.000Z

$ cat struct.pm 
package struct;

use strict;
use warnings;

sub import
{
   shift;
   my ( $name, $fields ) = @_;
   my $caller = caller;

   my %subs;
   foreach ( 0 .. $#$fields ) {
      my $idx = $_;
      $subs{$fields->[$idx]} = sub :lvalue { shift->[$idx] };
   }

   my $pkg = "struct::impl::$name";

   no strict 'refs';
   *{$pkg."::$_"} = $subs{$_} for keys %subs;
   *{$caller."::$name"} = sub { bless [ @_ ], $pkg };
}

1;

$ perl
use struct Point => [qw( x y )];
my $p = Point(10,20);
printf "Point is at (%d,%d)\n", $p->x, $p->y;
$p->x = 30;
printf "Point is now at (%d,%d)\n", $p->x, $p->y;
$p->z = 40;

__END__
Point is at (10,20)
Point is now at (30,20)
Can't locate object method "z" via package "struct::impl::Point" at - line 6.

It's specifically and intentionally not an object class. You cannot subclass it. You cannot provide methods. You cannot apply roles or mixins or metaclasses or traits or antlers or whatever else is in fashion this week.

On the other hand, it is tiny, single-file, creates cheap lightweight array-backed structures, uses nothing outside of core. And I defy anyone to even measure its startup overhead with a repeatable benchmark.

It's intended simply to be a slightly nicer way to store internal data structures, where otherwise you might be tempted to abuse a hash, complete with the risk of typoing key names.

Would anyone use this, if it were available?

Perl - Dual-life Socket

2011-10-26T14:51:00.000+01:00

I've been working on how to dual-life the Socket module, so it can go on CPAN and give older Perl versions the benefits of recent additions; namely getaddrinfo(3)/getnameinfo(3) wrappings and other IPv6 support.

I have what's currently marked as an unauthorized release now on CPAN, though I'm hoping to get co-maint on it to release officially. In the meantime, it'd be useful to get some smoketest reports on what works and what doesn't. I've tested on Linux at perl 5.10.1, 5.12.4 and 5.14.1. Other OSes, especially MSWin32, would be most appreciated.

http://search.cpan.org/~pevans/Socket-1.94_03/

libvterm/pangoterm and Tickit

2011-09-30T15:18:00.011+01:00

Lately I've written a bit about my terminal emulator library, libvterm, and briefly mentioned Tickit, my terminal UI module for Perl. I'll write about each in more detail soon, but I thought since I hit an important milestone recently, I'd write a little something more about libvterm and pangoterm.

libvterm is a purely abstract C99 library that implements the bulk of the logic of being a terminal emulator. Bytes from the PTY master are fed into it by the containing program, and it maintains the abstract state of the terminal; the position of the cursor, the state of the pen, what charcters are where with what attributes, and so on. It calls callback functions registered by the containing program, to inform it of damaged screen regions that need repainting. Two of the main selling points of the library are

It is purely abstract C99, doesn't rely on POSIX or any particular rendering/UI system

During normal operation of just feeding it bytes and processing events, it does not use the malloc system.

These properties make it ideal for a number of situations, ranging from desktop applications, to small embedded systems or operating system kernels.

pangoterm is a GTK/Pango-driven embedding of this libary, in a simple single-.c-file application, mostly for me to develop and test it. It is currently maintained in the libvterm source tree.

For a while now this combination has been complete enough to drive vim sufficient to edit its own source code - pangoterm and libvterm are now self-hosting. A couple of weeks ago I finally managed to fix the last of a number of small issues making it not quite perfect. Last week I also managed to get pangoterm to completely correctly render a Tickit-based program; the final missing piece being some of the mouse tracking modes.

I now have a bit of extra configuration in my .vimrc to take advantage of a few of pangoterm's abilities, such as support for italics.

As well as italics, it also supports strikethrough and alternative fonts, although so far I've only managed to find one alternative font that actually looks at all decent alongside DejaVu Sans Mono. These are all shown off quite well by Tickit's demo-pen.pl example script here.

And finally here, a demo of the xterm-like 256 colour handling.

These screenshots briefly show Tickit working nicely with pangoterm. Sometime soon I shall get around to writing about Tickit in more detail, and also explaining some of my further plans for the whole Tickit+libtermkey vs libvterm combination.

Perl - Sentinel - version 0.02

2011-09-28T15:16:00.008+01:00

I have just uploaded the first (non-development) release of Sentinel; version 0.02.

It provides a little helper function that creates lvalues, suitable for lvalue typed accessors that want to run real code on assignment (such as for type checking or coercion, or update triggers), rather than just update a scalar.

use Sentinel;
 
sub foo :lvalue
{
   my $self = shift;
   sentinel get => sub { return $self->get_foo },
            set => sub { $self->set_foo( $_[0] ) };
}

It makes the following two lines equivalent

my $obj = Some::Object->new;

$obj->set_foo( 100 );
$obj->foo = 100;

This is rare among my modules, more or less the first thing I've written mostly as a result of ranting about it on #perl, rather than because I actually wanted it. My argument kept being that if anyone does want an lvalue accessor, it's trivially easy to write one given some function sentinel as in this example, and that actually implementing the sentinel function itself isn't hard; it's a small piece of obvious XS magic.

So here it is.

It has to cheat somewhat on versions of Perl before 5.14, because of the way lvalue context isn't properly as powerful as it now is. Long story short - there may be unit-test failures on some older versions of Perl; but I can't tell yet because the CPAN Testers web frontend is still down, so I can't see the smokers. If anyone sees any version-related failures, please do let me know.

Edit 2011/11/29: By the power of crowdsourced smoke testing, it appears this does work stably across many Perl versions - thanks all who've informed me.

Perl - Term::TermKey - version 0.09

2011-08-30T19:27:00.005+01:00

Last week saw a new version of Term::TermKey (0.09), and also the underlying libtermkey (0.9). This contains a fairly small new feature, giving control of the way that EINTR is handled.

Version 0.8 added graceful handling of EINTR to restart IO operations rather than fail with an error. This had unfortunate knock-on effects for the Perl-level wrapping of it, because of the deferred nature of Perl's safe signals. On a signal (such as the not-so-unlikely SIGWINCH) a flag would be set, but the termkey_waitkey(3) operation would be restarted, not returning back to Perl's control until a keypress event was actually received. This upset programs that wish to respond to SIGWINCH and redraw the terminal to the new size.

Version 0.9 of libtermkey now therefore has a new flag, TERMKEY_FLAG_EINTR whose presence makes the blocking IO operations (termkey_waitkey(3) and termkey_advisereadable(3)) to return a new result code, TERMKEY_RES_ERROR after which the caller can inspect the value of errno, to observe an EINTR.

Again because of Perl's safe signal handling, the Perl wrapping of libtermkey has to always enable this flag, so it can invoke the $SIG{WINCH} signal handler, for example. The Term::TermKey module therefore now always sets TERMKEY_FLAG_EINTR on the underlying TermKey instance, and emulates the presence or absence of this flag at the Perl level, by optionally restarting its IO operation, or itself returning TERMKEY_RES_ERROR.

An unfortunate bug here in the emulation and hiding of this flag from the Perl level means that Term::TermKey 0.09 fails to correctly read the TermKey flags back out of the underlying object. In particular it fails to be able to check on the presence of TERMKEY_FLAG_UTF8 that libtermkey itself may have enabled, after detecting a UTF-8 locale. This causes Tickit's unit tests to break with the familiar "Wide character in syswrite at ..." error.

This bug has now been fixed in the source code repository, and will be present in the next version, 0.10. Tickit 0.10 also has an independent fix for the same bug, by using Perl's ${^UTF8LOCALE} instead of reading the TermKey flags back out again.

Also upcoming in libtermkey 0.10, will be some Solaris portability fixes, and a new canonicalisation flag, which turns a TERMKEY_SYM_DEL key into TERMKEY_SYM_BACKSPACE, for those terminals that send DEL on Backspace.

IO::Async and AnyEvent

2011-08-27T14:31:00.007+01:00

I've recently been working on a new IO::Async loop implementation, IO::Async::Loop::AnyEvent. Like the Glib and POE loops, this one uses another event system as its underlying implementation; AnyEvent.

What makes this one a little different and noteworthy, is that AnyEvent claims not to be an event system as such, but rather a compatibility layer on top of event systems. I have so far resisted writing this particular loop implementation on the grounds that, since AnyEvent just applies a cross-compatibility layer on top of some other existing event system anyway, IO::Async might as well use that underlying event system directly. In practice however this doesn't quite seem to work out all the time; existing code exists and is already written. Sometimes that existing code already works using AnyEvent directly, making it harder to drop a small section of IO::Async-based code inside it.

In that situation, an individual module/function/etc.. can construct itself an IO::Async::Loop::AnyEvent, to allow that component to use IO::Async-based functionallity, while still interacting correctly with the rest of the AnyEvent-based program.

It is not primarily intended that this module be used as the basis of an entire program; mostly because a neater solution for mixed IO::Async+AnyEvent exists in the form of AnyEvent::Impl::IOAsync.

There are still some minor issues to deal with currently; most notably that because each of IO::Async and AnyEvent could use the other as an event source, there are some circularity problems when AnyEvent picks AnyEvent::Impl::IOAsync to use, when IO::Async::Loop::AnyEvent loads it.

Deep recursion on subroutine "AnyEvent::Impl::IOAsync::io" at /home/leo/src/perl/IO-Async-Loop-AnyEvent/blib/lib/IO/Async/Loop/AnyEvent.pm line 103.

Hmmmmm.... A little work there still needed :)

Perl - Term::Terminfo - version 0.06

2011-07-31T16:25:00.001+01:00

I've just uploaded a new version of Term::Terminfo. In brief; this is a small wrapper around the terminfo database, and can be used to enquire about properties of a given terminal. For annoying historic reasons, each of these properties is known by at least two names; its short "capname", a two or three letter code often found in the actual file on disk, and a longer "varname", the name of a variable in the curses C library, which stores its value for the current terminal.

This latest version, 0.06, adds a whole duplicate set of methods - varname accessors. Prior to 0.06, the properties were only accessible using their short capnames.

It's probably best to use the longer varnames anyway. They are more self-documenting, and a little more future-proof against the admittedly-remote possibility of new variables being added in the future.

I'm also planning to support unibilium in a later version. This is a standalone terminfo-parsing library, which is useful for reading terminfo data without linking against the full curses library. This is especially useful when trying to build a replacement for curses...

XS beats Pure Perl

2011-07-15T13:47:00.004+01:00

Someone reported some test failures trying to install Tickit, which seemed to be related to shortcomings in Text::CharWidth. The latter seems to have very poor unit test coverage on itself, so the failures didn't appear during its installation, only when Tickit::Utils was tested against it. On initial inspection I wondered if Text::CharWidth simply wasn't using wcswidth(3) correctly, and whether I should get around to my plan of rewriting bits of Tickit::Utils in XS instead for performance, as well as work around this bug.

This turned out to be quite a good idea. Implementing cols2chars() and chars2cols() in XS instead of Perl makes them at least 10 times faster. I tested it on four strings; two ASCII and two Unicode; a long and a short of each:

Calls/sec		PP	XS	Ratio
chars2cols	along	48685	406504	834.97%
chars2cols	ashort	72674	704225	969.02%
chars2cols	ulong	37341	387596	1037.99%
chars2cols	ushort	52966	714285	1348.57%
cols2chars	along	16350	403255	2466.39%
cols2chars	ashort	58685	649350	1106.50%
cols2chars	ulong	13561	362318	2671.76%
cols2chars	ushort	50556	632911	1251.90%

In fact, some cases it turns out to be 24 times faster.

I haven't looked into too much detail on why, but I suspect a large amount of the reason is to do with the way the XS functions primarily walk along the internal UTF-8 representation of the strings, counting bytes, characters, and columns as they go, and returning the appropriate count(s) when the required. The pureperl implementation doesn't have direct access to the byte offsets, so only has character numbers to work to. The frequent character-to-byte or byte-to-character conversions at all the boundaries between the functions result in multiple UTF-8 byte skip counting steps along the string each time a function is entered or left, generally slowing it down.

As to the original test failure, it turned out to be entirely unrelated lack of locale support in the platform's libc. The XS implementations fail there in the same way. But having implemented the above improvements, I decided to leave them in anyway.

XS faster than Pure Perl; who'd have thought it?

Perl - IO::Async - version 0.41

2011-06-20T00:07:00.000+01:00

Wow; seems I haven't been writing these posts as much as I should be. I last wrote about version 0.34, and just now I've uploaded version 0.41 to CPAN. Quite a bit changed between then and now, here's a rundown of the most important bits:

Added IO::Async::FileStream. This behaves like a read-only IO::Async::Stream, but reads its data from a regular file on the filesystem. Like tail -f, or File::Tail, it watches the file for changes in size, and follows appended data.

Added IO::Async::Function. This is a true IO::Async::Notifier subclass to represent an asynchronous block of code; which was previously handled by DetachedCode. Being a true Notifier subclass gives it many advantages, and should provide a better base to build other things on. IO::Async::Resolver is now a subclass of Function, not DetachedCode.

Added IO::Async::Process. This is another IO::Async::Notifier subclass, this time to represent an external process.

Support encoding parameter in IO::Async::Stream. This allows the Stream to handle text in some encoding, rather than simply raw bytes.

Support first_interval parameter to IO::Async::Timer::Periodic. If supplied, this will be the first wait time when the timer is started. In particular, if it is zero the timer's first invocation will happen immediately.

Allow Loop->listen to be extended via extensions, similar to ->connect. Such extensions as IO::Async::SSL are already set up to make use of this.

Distribution now uses Test::Fatal rather than Test::Exception. The former is smaller and simpler, whereas the latter relies on clever tricks to hack on caller(), which sometimes breaks some setups.

Added convenient method in IO::Async::Loop for handling socket addresses; extract_addrinfo. This recognises strings for common networking constants; 'inet', 'inet6' and 'unix' as socket families, 'stream', 'dgram' and 'raw' as socket types. These convenient forms are also recognised by the ->connect and ->listen methods.

Now prefers to use the IPv6 support functions found in Perl 5.14's Socket module; only falling back to using Socket::GetAddrInfo in earlier perls.

Sleep Sorting with IO::Async and CPS

2011-06-10T11:53:00.004+01:00

There's a bit of a silly meme going around lately; an implementation of a sorting algorithm that works by many parallel sleeps. So I thought I'd have a go at it from an IO::Async perspective.

use CPS qw( kpareach );
use IO::Async::Loop;

my $loop = IO::Async::Loop->new;

kpareach( \@ARGV,
   sub {
      my ( $val, $k ) = @_;
      $loop->enqueue_timer(
         delay => $val,
         code  => sub { print "$val\n"; goto $k },
      );
   },
   sub { $loop->loop_stop },
);

$loop->loop_forever;

This produces such output as:

$ perl sleepsort.pl 3 8 4 2 6 5 7 1
1
2
3
4
5
6
7
8

It's a little more verbose than I'd like it though. I have been pondering creating a CPS::Async module, which would provide some more simple CPS-like versions of the usual blocking primatives we're used to, like sleep and sysread. Using this new hypothetical module would lead to something looking like:

use CPS qw( kpareach );
use CPS::Async qw( ksleep );

kpareach( \@ARGV,
   sub {
      my ( $val, $k ) = @_;
      ksleep( $val, sub {
         print "$val\n";
         goto $k;
      } );
   },
   sub { CPS::Async::stop },
);

CPS::Async::run;

This would be a small wrapper around IO::Async, providing CPS-style functions that turn into method calls on some implied IO::Async::Loop object; similar to for example, the AnyEvent::Impl::IOAsync, which exposes an AnyEvent API using IO::Async.

I have plenty of other projects to keep me amused currently, but I'll sit it on the back burner in case something interesting happens that way...

libtermkey - read keypresses from terminals

2011-05-24T22:04:00.000+01:00

I have just released version 0.8 of libtermkey. It contains a small set of bugfixes on the previous version (0.7), relating to handling the signal-raising keys (Ctrl-C, Ctrl-\, Ctrl-Z), gracefully handles EINTR from read(2) calls, and actually gets around to implementing the CSI u modified Unicode encoding scheme I documented a long time ago.

libtermkey is a library for reading keypress events (and in fact mouse events too) from a terminal, in a terminal-based application. I won't list all its points and features, but as a brief overview:

It presents events to the application in a structure, containing basic key information and a bitfield of modifiers in effect, rather than a single flat enumeration integer, as curses tries to do. In this way, it can easily represent the various modifier-combinations on cursor keys, like arrows or PageUp/PageDown, and can easily represent any Unicode character vs. any special key.

It supplies a pair of functions (termkey_strfkey and termkey_strpkey) for converting between these structural notations and plain-text human-readable strings, such as "Ctrl-PageUp". These assist with easy reporting of keypresses in applications, or for config file parsing.

Perl bindings exist, primarily in the form of Term::TermKey.

Wearing Two Hats

2011-05-19T17:15:00.000+01:00

A while ago, I wrote libtermkey, a C library for reading keypress events from a terminal. I wrote a Perl binding for it, Term::TermKey. On a separate note, I also maintain IO::Async, an event framework for Perl. Naturally, the combination of these two lead me to write an IO::Async module to handle libtermkey, Term::TermKey::Async. These all work together quite well.

However, these two things are separate considerations. There's nothing specific to Perl, in libtermkey, nor anything specific to IO::Async in Term::TermKey. After some thought, and discussions on #perl, I decided in the end, that I had to realise these were two separate concerns, two different problem domains, that neither should be allowed to influence the other. In short, I had to wear two hats.

With my libtermkey hat on, I went to join #poe and #anyevent on irc.perl.org, and talked my way around designing two new modules, which are now happily sitting on CPAN as well: POE::Wheel::TermKey and AnyEvent::TermKey.

With so many CPAN modules effectively being glue between two others (such as in the case of Term::TermKey::Async being the glue between Term::TermKey and IO::Async), it's often the case that at least one if not both sides of the module are written by the same author. It is important to recognise these cases, and to consider whether the community as a whole would be better served by taking a look around to see if other modules and other use cases need attention as well.

When wearing hats, it is important to remember which hat you are wearing, and only try to wear one hat at once.

Deterministic Random Testing

2011-04-26T21:21:00.000+01:00

I've just written an implementation of a weighted random shuffle. Because the function is random, it won't return the same results every time. This makes it hard to unit-test; I can't just run it on some given input and assert it returns some given output. Indeed, by its nature, any ordering of the results is potentially valid. One way to solve this is to run it a large number of times, counting the frequency of various output results, and asserting that roughly the right proportion of results are returned. Of course, exactly how close "roughly right" is, is hard to determine.

This isn't a very satisfactory testing method, however. Get the tolerance too tight, and spurious random differences cause tests to fail unnecessarily. Too loose, and you might miss a subtle logic bug that skews the probabilities of some case.

The weighted random shuffle algorithm calls int rand $limit a number of times, each time passing in a small integer. This effectively uses the RNG like a dice roll, randomly selecting some integer 0 .. $limit - 1. Because each call takes a small integer, and because the algorithm is entirely deterministic for any given set of random results, this suggests a better testing method. If we could instead enumerate all possible returns of random numbers, each one exactly once, we can generate all possible results from the shuffle algorithm in their ideal proportions. Because this will be an exact deterministic count, free from randomness, we can assert exact values for the results.

So this is what I did. I've created a module, for now simply called Unrandom, which exports one function, unrandomly. It is used like the following:

use Unrandom 'unrandomly';

unrandomly {
  my $da = 1 + int rand 6;
  my $db = 1 + int rand 6;
  say "Roll 2d6: $da + $db = " . ($da+$db);
};

The unrandomly function has the effect of replacing the rand function with one under its control while it runs the block of code. It runs the block of code a number of times, enumerating the entire tree of possible return values, in a given deterministic order:

Roll 2d6: 1 + 1 = 2
Roll 2d6: 1 + 2 = 3
Roll 2d6: 1 + 3 = 4
Roll 2d6: 1 + 4 = 5
Roll 2d6: 1 + 5 = 6
Roll 2d6: 1 + 6 = 7
Roll 2d6: 2 + 1 = 3
Roll 2d6: 2 + 2 = 4
...
Roll 2d6: 6 + 5 = 11
Roll 2d6: 6 + 6 = 12

Because each possible combination is returned exactly once, we can unit test that this dice-rolling algorithm does in fact give us the right distribution of results; there will be exactly one 2, two 3s, etc... We don't have to run it a few million times, and check that we got "roughly" the right number; we can be exact.

While I'm using this code in a unit-test, there's nothing directly test-related in the code. It could be useful anywhere that statistical modelling is used, or other problems involving random integer generation.

I'm now just looking for a good name to call it, so I can extract it from the unit tests and give it a life of its own on CPAN.

Suggestions anyone?

Extended colour support, terminals, and ECMA-48

2011-04-09T12:30:00.006+01:00

tl;dr summary: Terminal authors - please accept CSI 38:5:123 m as well as anything else, to set extended colours. NOTE THE COLON.

Many terminals these days are starting to support extended colour modes; specifically things like 256 colours. They're all doing it wrong.

CSI 38;5;123 m

No no no no no.

That is NOT what you think. It does not select colour 123 from palette 5. CSI arguments are separated by semicolons. This sets three unrelated SGRs; 38, 5, and 123. 38 sets a foreground colour to .. er.. nothing in particular. 5 is blinking mode, 123 has no defined meaning to my knowledge. All the other following are exactly identical:

CSI 5;38;123 m
CSI 38 m CSI 5 m CSI 123 m

ECMA-48 already defines a perfectly good way for parameters to take sub-parameters. It's the colon. The correct way to encode this concept is

CSI 38:5:123 m

Why does this matter? It matters for parsers not to have to understand what is going on. Consider

CSI 3;38:9:5:4:3:2:1;11;1

This is equivalent to

CSI 3 CSI 38:9:5:4:3:2:1 CSI 11; CSI 1

I.e. I have no idea what palette 9 is, but I can definitely parse out the contents of this SGR from the others, knowing exactly where it ends. I don't have to "just know" that palette 9 happens to take 5 parameters. The CSI encodes this.

My own terminal emulator library, libvterm, already understands these colon-based arguments. It parses them correctly. This is important for other palettes that don't take just one value. For example, the RGB, RGBA, CMY, and CMYK palettes. It's vital to be able to parse out these sub-arguments from the single SGR 38 or 48.

Standards exist for a reason, people. Please use them.

When failure isn't failure

2011-03-29T23:54:00.001+01:00

Lately I have been looking at two different problems, with a common theme.

My first problem concerns Parser::MGC, and its ability to read input lazily as needed, rather than needing to slurp an entire file all at once. This ability is provided by the from_reader method, which takes a CODE reference to a reader function.

As the documentation points out, this is only supported for reading input that's broken across skippable whitespace. This is because it's implemented by calling the reader function to look for more input if the current input buffer is completely exhausted. It cannot work in general, for splitting the input stream arbitrarily, because Perl's regular expression engine does not give sufficient feedback. It is not possible to ask, after a match attempt, whether the engine reached the end of the stream, For example, when looking for a match for m/food/, an input of "fool" definitely fails, whereas an input of "foo" is not yet a failure, because it might be that reading more input from the stream can complete the match. If the regular expression engine gave such feedback, then the reader function could be invoked again to provide more input that may help to resolve the parse.

My second problem concerns how to handle UTF-8 encoded data in nonblocking reads. An IO::Async::Stream object wraps a bytestream, such as a TCP socket or pipe. If the underlying stream contains UTF-8 encoded Unicode text, then the Unicode characters need to be decoded from these bytes, by using the Encode module.

The trouble here is that Encode does not provide a way to do this sanely. It is quite likely that a multibyte UTF-8 sequence gets split across multiple read calls. To cope with such a case, Encode has a mode where it will stop on the first error it encounters (called FB_QUIET), returning the prefix it has decoded so far, and deleting the bytes so consumed from the input. The intention here is that another call supplies more bytes, and it continues from there. Problem is, it returns on any failure, whether that's running out of input bytes or encountering an invalid byte. Without the ability to distinguish these two different conditions, it is impossible to handle nonblocking or stream-based UTF-8 decoding while still having sensible error handling.

The common theme of these two problems is that neither considers the nature of a failure, treating various reasons the same. Both cases have two kinds of failure: one a failure because something has been received that is not correct; the other a failure because something that would be correct has simply not yet been received.

Sometimes, failure is not really failure at all. Sometimes it is simply deferred success that is yet to happen.

Carp from somewhere else

2011-03-04T16:12:00.000Z

Carp provides two main functions, carp and croak, as replacements for core Perl's warn and die. The functions from Carp report the file and line number slightly differently from core Perl; walking up the callstack looking for a different package name. The idea being these are used to report errors in values passed in to the function, typically bad arguments from the caller.

These functions use the dynamic callstack (as provided by caller()) at the time the message is warned (or thrown) to create their context. One scenario where this does not lead to sensible results is when the called function is internally implemented using a closure via some other package again.

Having thought about this one a bit, it seems what's required is to split collecting the callstack information, from creating the message. Collect the information in one function call, and pass it into the other.

This would be useful in CPS for example. Because the closures used in CPS::kseq or kpar aren't necessarily invoked during the dynamic scope of the function that lexically contains the code, a call to croak may not be able to infer the calling context. Even if they are, the presence of stack frames in the CPS package would confuse croak's scanning of the callstack. Instead, it would be better to capture the calling context using whence, and pass it into whoops if required for generating a message.

For example, this from IO::Async::Connector:

sub connect
{
   my ( %params ) = @_;
   ...

   my $where = whence;

   kpar(
      sub {
         my ( $k ) = @_;
         if( exists $params{host} and exists $params{service} ) { 
            my $on_resolve_error = $params{on_resolve_error} or whoops $where, "Expected 'on_resolve_error' callback";
            ...
}

These functions would be a fairly simple replacement of carp and croak; capture the callsite information at entry to a function, and pass it to the message warning function.

It does occur to me though, the code will be small and self-contained, and not specific to CPS. I think it ought to live in its own module somewhere - any ideas on a name?

IPv6 in Perl

2011-01-22T14:39:00.002Z

A lot of people are talking about IPv6 lately. And perhaps with good reason - it's been years in coming, but finally we're really starting to run out of IPv4 addresses. Tools like the IPv4 Address Report may put a certain amount of panic on things with their realtime Flash-based countdown widget, but the problem is real and does need sorting.

The latest development release of Perl; perl-5.13.9, now has full support for IPv6, and the new address handling functions specified in RFC 2553. The new functions all live in Socket.

As well as the low-level AF_INET6 constant, and the pack_sockaddr_in6 and unpack_sockaddr_in6 structure functions (already in place in 5.13.8 in fact), there is now the full set of getaddrinfo, getnameinfo and associated constants. These allow fully protocol-agnostic handling of connections and addresses. There is now enough in core to support IO::Socket::IP. This is a fully protocol-agnostic replacement of IO::Socket::INET.

use IO::Socket::IP;

my $sock = IO::Socket::IP->new(
   PeerHost    => "www.google.com",
   PeerService => "www",
) or die "Cannot construct socket - $@";

printf "Now connected to %s:%s\n", $sock->peerhost_service;

...

What could be simpler?

Perhaps now is the time to make a case for putting IO::Socket::IP itself in the core distribution, such that when perl-5.14.0 comes out, it will be properly ready for the next 30 years of the Internet?

IO::Socket::IP's API is designed to be a drop-in replacement for the IPv4-only IO::Socket::INET. Any program currently using INET ought to work exactly the same, by simply substituting IP instead, and now will also work on IPv6.

If you maintain a distribution that uses IO::Socket::INET, please try out IO::Socket::IP instead. I'd be very keen to hear from anyone who finds it doesn't JustWork in their situation.

See also my earlier post on the subject; Perl - IO::Socket::IP

Perl - IO::Async - version 0.34

2010-12-30T18:17:00.006Z

There's been four releases of IO::Async since I last wrote about version 0.30. Here's a rough summary of the more important changes and additions between then and version 0.34:

New Notifier class IO::Async::Timer::Absolute, to invoke events at a fixed point in the future.

New Notifier class IO::Async::PID, to watch a child process for exit(2).

New Notifier class IO::Async::Protocol::LineStream, to implement stream protocols that use lines of plain text.

New method on IO::Async::Protocol that wraps connect(2) functionallity, allowing for simpler network protocol client modules.

IO::Async::Loop->connect's on_connect_error and IO::Async::Loop->listen's on_listen_error continuations now both receive errno information.

New direct name resolution methods on IO::Async::Resolver for getaddrinfo(3) and getnameinfo(3). The resolver is now directly accessible from the IO::Async::Loop.

IO::Async::Resolver supports deadline timeouts.

IO::Async::Stream->write supports taking a CODE reference to dynamically generate data for the stream on-demand.

IO::Async::Stream->write supports an on_flush callback.

The IO::Async::Loop->new magic constructor now caches the loop. This is useful for wrapping modules, other event system integration, etc..

Documentation has been rearranged to add new EVENTS sections, documenting the events that Notifier classes can fire either as callbacks in coderefs, or as methods on subclasses.

Various bugfixes, other documentation additions

Event loops and Jenga; or 24 Advent Calendar Events in One Go

2010-12-24T13:29:00.008Z

There are many event loops systems in Perl. Do they play together?

I was thinking about this recently, at my LPW2010 talk about IO::Async. In the hackathon the following day, I managed to write IO::Async::Loop::POE; a way to run IO::Async atop POE.

So I started thinking further; if you can run one event loop system on top of another, how high can we stack them? Can we build a tower, putting each atop the previous, growing taller. Each new layer we try to add would start to get harder, more difficult, increasing the chances the whole thing came crashing down. Sortof like a Jenga tower.

So what would a Perl event loop Jenga tower look like?

My attempt looks like this: (326 lines, paste.scsys.co.uk)

The output looks something like this:

$ perl jenga.pl
AnyEvent resolved 127.0.0.1:80
Glib reads Hello world!
POE reads Hello world!
POE resolved 127.0.0.1:80
IO::Async reads Hello world!
AnyEvent reads Hello world!
AnyEvent listener accepted
POE listener accepted
IO::Async resolved 127.0.0.1:80
AnyEvent connected received
POE connected received
IO::Async listener accepted
IO::Async connected received
Glib child exited 0
POE child exited 0
IO::Async child exited 0
AnyEvent child exited 0
Glib timer
POE timer
IO::Async timer
AnyEvent timer
^CIO::Async SIGINT
AnyEvent SIGINT
POE SIGINT
Stopping...

That's 24 events. Count them. It combines Glib, POE, IO::Async and AnyEvent. It performs a basic filehandle read, a child process watch, and a timed wait in each of these four systems. Because Glib lacks signal watching, only the other three perform this. The other three are also used to perform name resolution, socket listening, and socket connecting.

Everyone seems to be doing Advent Calendar blogs this year. 24 daily posts, each showing one small thing. Someone suggested I should write a Perl Event systems advent calendar. So perhaps here, consider this to be one. Except it has 24 windows all in one go.

As it turns out, it's possible to make this tower a little higher. There's a module to run Event beneath Glib; that is, it replaces the core polling function of Glib to use Event instead. And I suspect it may just about be possible to run Tk on Glib, and the POE on Tk.

At some point in the new year, I have some plans to turn this one-program script into a more useful resource of examples and translations. The Rosetta Stone for Unix provides a cross-reference for looking up Unix concepts between different systems. I feel that a similar attempt at Perl event loops could be quite useful too.

Perl - CPS - version 0.11

2010-12-19T11:43:00.006Z

CPS is a Perl module that provides several utilities for writing programs in Continuation Passing Style.

In a nutshell, CPS is a method of control flow within a program, which uses values passed around as continuations as a replacement of the usual call/return semantics. In Perl, this is done by passing CODE references; calling a function and passing in a CODE reference to be invoked with the result, rather than having the function return it. While at first this doesn't seem to be very useful, most of its power comes from the observation that the function doesn't need to invoke its continuation immediately; if it performs some IO operation or similar, it can perform this in an asynchronous fashion, invoking its continuation later. This style of coding is often associated with nonblocking or asynchronous event-driven programming. It is typical of such event systems as IO::Async.

A typical problem with implementing CPS control flow, is that all of the usual Perl control-flow mechanisms are built for immediate call/return semantics, where the use of CPS gets in the way. The CPS module provides utility functions for implementing control flow in a continuation passing style, by providing a set of functions that replace the usual Perl control-flow keywords. For example the Perl control structure of a foreach loop

foreach my $frob ( @frobs ) {
   my $wibble = mangle( $frob );
   say "Mangled $frob looks like $wibble";
}
say "All done";

becomes a call to CPS::kforeach

use CPS qw( kforeach );

kforeach( \@frobs, sub {
   my ( $frob, $knext ) = @_;
   kmangle( $frob, sub {
      my ( $wibble ) = @_;
      say "Mangled $frob looks like $wibble";
      $knext->();
   } );
}, sub {
   say "All done";
} );

We haven't really gained anything by doing this though. If the process of mangling a frob involves some IO tasks, perhaps talking to some remote server, then we'll spend most of our time waiting for it to respond when we could be sending multiple requests and waiting on their responses. We could likely save some time by running them concurrently.

I gave a talk at LPW2010 about Continuation Passing Style and CPS, the slides of which are available here.

After discussing CPS and IO::Async at LPW, I was talking with mst about his IPC::Command::Multiplex module. He came up with the idea for another control-flow function; a combination of kpar and kforeach, which I called kpareach. In use it looks exactly like kforeach, except that it starts the loop body for each item all at the same time, in parallel, rather than waiting for each one to finish before invoking the next.

This is a new addition to CPS version 0.11, which is now on CPAN. It is also one of the first new control-flow structures that doesn't have a direct plain-Perl counterpart; a demonstration of the usefulness of CPS in event-driven programming.

General Updates

2010-11-21T23:24:00.000Z

I have no big specific updates today. Instead, a list of lots of little things I've been working on:

IO::Socket::IP now has preliminary non-blocking connect support (version 0.05_003). This isn't quite a perfect solution because of blocking name resolvers, but see also Net::LibAsyncNS below.

Created a new CPAN dist, wrapping libasyncns, called Net::LibAsyncNS. This allows a simple way to asynchronise name resolver lookups.

Have fixed a few bugs in IO::KQueue, relating to dodgy handling of Perl scalars in the udata field. Some memory leak bugs still exist, but I believe these to be the kernel's fault. See below.

Spent some time on Freebsd-hackers@ arguing about kqueue and managing user data pointers. Long story short I believe kqueue API itself is missing a feature, making generic wrapping of it impossible from any high-level language, or properly by C libraries also.

Both talks I submitted for LPW2010 were accepted; on the subjects of CPS and IO::Async.

Net::Async::HTTP now has SSL support and can stream response body content as it arrives, rather than waiting for the whole response (version 0.08).

That's all for a quick update, but I may write about any or all of these topics in more detail later...

Perl - Test::Identity

2010-10-31T19:39:00.007Z

Today I uploaded a new module to CPAN; Test::Identity. It's possibly the quickest module I've ever written, from when I decided to write it, to when it was actually uploaded:

(18:02) sub Test::Identity::identical { is refaddr $_[0], refaddr $_[1], $_[2] } <== I'm about to write such in a module, unless anyone can suggest me a module that already has it
...
(19:21) * GumbyPAN CPAN Upload: Test-Identity-0.01 by PEVANS

I won't spend a long time explaining why, I'll just quote the docs:

This module provides a single testing function, identical. It asserts that a given reference is as expected; that is, it either refers to the same object or is undef. It is similar to Test::More::is except that it uses refaddr, ensuring that it behaves correctly even if the references under test are objects that overload stringification or numification.

It also provides better diagnostics if the test fails:

$ perl -MTest::More=tests,1 -MTest::Identity -e'identical [], {}'
1..1
not ok 1
#   Failed test at -e line 1.
# Expected an anonymous HASH ref, got an anonymous ARRAY ref
# Looks like you failed 1 test of 1.

$ perl -MTest::More=tests,1 -MTest::Identity -e'identical [], []'
1..1
not ok 1
#   Failed test at -e line 1.
# Expected an anonymous ARRAY ref to the correct object
# Looks like you failed 1 test of 1.

Perl - IO::Socket::IP

2010-10-18T05:32:00.004+01:00

IO::Socket::IP is a subclass of IO::Socket that provides a protocol-independent way to use IPv4 and IPv6 sockets. It provides an API compatible with the IPv4-only IO::Socket::INET, but does so in a way that ensures properly transparent IPv6 support.

The following example shows it has an identical API to the ::INET module:

use IO::Socket::IP;

my $sock = IO::Socket::IP->new(
   PeerHost => "www.google.com",
   PeerPort => "www",
) or die "Cannot construct socket - $@";

At this point, $sock is just another IO::Socket-derived filehandle, and supports all the usual methods and IO functionality. The only difference here is that, where IO::Socket::INET would have used a legacy gethostbyname call and made a PF_INET socket, IO::Socket::IP will use getaddrinfo (via Socket::GetAddrInfo), and will use either PF_INET or PF_INET6 as appropriate.

It's not yet a complete 100% API clone of ::INET though. While it supports all the methods, there are a few constructor arguments not yet supported, being Blocking and Timeout. The way that getaddrinfo returns a list of candidate addresses, to be tried in order, makes nonblocking support hard to do, and complicates the model for what timeout really means. For nonblocking connect support, better solutions already exist, such as IO::Async's Connector, which has always supported IPv6 via getaddrinfo. As for timeouts, eventually IO::Socket::IP should support it, but for now a local'ised $SIG{ALRM} and alarm call should suffice.

For almost all use cases, switching to using IO::Socket::IP should be a simple matter, because of the API similarities. Just adding an extra dependency (because ::IP isn't core), and substituting the package name in source should be enough.

Finally, in case you don't want to pull in an extra hard dependency, you might consider the following fragment I've used quite successfully:

   my $class = eval { require IO::Socket::IP
                         and "IO::Socket::IP" } ||
               do   { require IO::Socket::INET
                         and "IO::Socket::INET" };

   $socket = $class->new(
      PeerHost => $host,
      PeerPort => $port,
   ) or die "Cannot connect - $@\n";

There. Now you have no excuse for not being IPv6-ready.

Perl - IO::Async - version 0.30

2010-09-23T20:40:00.003+01:00

Yesterday, I put the next version of IO::Async on CPAN; version 0.30. This was primarily an update to add some new features, though also a few minor bugfixes and documentation updates were included too. Here I want to focus on a few of these new features.

The first of these new features is nothing groundbreaking in itself, but feeds into the others. It's simply the addition of IO::Async::Socket, a notifier subclass to contain a socket that isn't necessarily a stream (primarily SOCK_DGRAM or SOCK_RAW sockets such as UDP, PF_PACKET or PF_NETLINK). This neatens up a few rough edges with trying to put such sockets directly in IO::Async::Handle objects.

The second main new feature is the creation of the IO::Async::Protocol class, and IO::Async::Protocol::Stream subclass. These derive directly from IO::Async::Notifier rather than IO::Async::Handle, and are intended to be abstract containers of code, and not perform any IO operations directly. Instead, they contain a Handle or Stream object as a child notifier. By exposing an API identical to IO::Async::Stream, the IO::Async::Protocol::Stream should be a drop-in replacement for any modules trying to implement a network protocol.

With the addition of IO::Async::SSL, not every stream-like connection can be represented by IO::Async::Stream, so separating the transport layer from the protocol layer is required. This wasn't possible by subclassing, whereas object containment makes it much simpler.

Net::Async::FTP, Net::Async::HTTP, and Net::Async::IRC have all been updated to use it, and most other use cases should be simple to change.

The final main change is that $loop->connect and IO::Async::Listener now support direct on_stream or on_socket continuations, which will be provided an instance of Stream or Socket directly, rather than requiring the invoked code to wrap one. This can then be easily configured as a IO::Async::Protocol's transport.

Having made this change, it leads the way to transparent SSL support across all protocols, and possibly other concerns like SOCKS proxies, by extending the arguments to $loop->connect or Listener. But that's for another post...

Finally, I should announce that I've now started a channel on irc.perl.org called #ioasync, as the official IRC home for IO::Async. Feel free to drop by if you have any issues, comments, questions,...

Perl - overload::substr

2010-09-20T19:41:00.002+01:00

overload allows an object class to provide methods which Perl should use to implement certain operators, like numerical addition or string concatenation. One operator that overload doesn't allow to be provided, is substr.

overload::substr allows this to be overloaded. This allows objects that behave like a string, to specify to Perl how they will handle the substr operator.

$ cat example.pl 
#!/usr/bin/perl

use strict;
use feature qw( say );

package ExampleString;

use overload::substr;

sub new { return bless [ @_ ]; }

sub _substr
{
   my $self = shift;
   my ( $offs, $len, $replace ) = @_;

   return sprintf ">> %s between %d and %d <<", $self, $offs, $offs+$len;
}

package main;

my $str = ExampleString->new( "Hello, world" );

say substr( $str, 2, 5 );

$ perl example.pl
>> ExampleString=ARRAY(0x86dd9c8) between 2 and 7 <<

The module is still in its early days yet, but the basics appear to be working on all Perl versions back to 5.8. I also want to try extending it, so that split() and regexp matches with m// and substitutions with s/// also use the substr operation. The identity that

$1 == substr( $str, $-[1], $+[1] - $-[1] )

is sure to be useful here.

I need a good example to show it off with sometime. I have in mind a string-alike object with real positional cursors, which remember their contextual position even after edits in other parts of the string. But more on that later...

Module name suggestions: A proper IO::Socket for IPv4/IPv6 duallity

2010-09-06T14:59:00.007+01:00

I currently don't have a good name for a module I'd like to write, because I think it is very much required right now.

We have IO::Socket::INET. It wraps PF_INET, thus making it IPv4 only.

We have IO::Socket::INET6. It wraps either PF_INET or PF_INET6, despite its name. It also uses Socket6, thus restricting it to only working on machines capable of supporting IPv6.

Thus any author wanting to write code to communicate to the internet (apparently that's some new fad everyone's talking about this week) is presented a moral dilema: Support IPv6 at the cost of not working on older v4-only machines, or support older machines but be incapable of using IPv6.

I originally partially solved this problem some years ago by the creation of Socket::GetAddrInfo, a module that presents the interface of RFC2553's getaddrinfo(3) and getnameinfo(3) functions. This however is not enough for actually connecting and using sockets.

I'd therefore like to propose a new IO::Socket subclass that uses these and only these functions, for converting between addresses and name/service pairs.

use IO::Socket::YourNameHere;

my $sock = IO::Socket::YourNameHere->new(
   PeerHost    => "www.google.com",
   PeerService => "www",
);

printf "Now connected to %s:%s\n", $sock->peerhost, $sock->peerservice;

...

Since it would use Socket::GetAddrInfo, it can transparently support IPv4 or IPv6. Since it would only use Socket::GetAddrInfo, it will work in a v4-only mode on machines incapable of supporting IPv6, and will not be restricted to only IPv4 or IPv6 if and when some new addressing family comes along to replace IPv6 one day; as v6 is now trying to do with v4.

In order to provide an easy transition period, I'd also support additional IO::Socket::INET options where they still make sense; e.g. accepting {Local/Peer}Port as a synonym for {Local/Peer}Service. The upshot here ought to be that you can simply

sed -i s/IO::Socket::INET/IO::Socket::YourNameHere/

and suddenly your code will JustWork on IPv6 in a good >90% of cases.

Can anyone suggest me a better module name for this?

Edit 2010/09/07: We seem to be settling on IO::Socket::IP for this currently.

Edit 2010/09/23: We did indeed settle on IO::Socket::IP; this is now up on CPAN, and will be the subject of a future posting...

This cross-posted from module-authors@perl.

Test to assert object identity

2010-08-15T15:51:00.002+01:00

I've just copypasted the following test function into about the fifth different test script:

use Scalar::Util qw( refaddr );
sub identical
{
   my ( $got, $expected, $name ) = @_;

   my $got_addr = refaddr $got;
   my $exp_addr = refaddr $expected;

   ok( !defined $got_addr && !defined $exp_addr ||
          $got_addr == $exp_addr,
       $name ) or
      diag( "Expected $got and $expected to refer to the same object" );
}

Rather than continuing to copypaste it around some more, can anyone suggest a standard Test:: module that contains it? Failing that, if it's really honestly the case that nobody has yet felt it necessary to provide one, could someone suggest a suitable module to contain it?

This behaviour cannot be implemented using, say, is( $obj, $expected ) because that will attempt to compare numerical equality, which of course will fail for any object that overloads numberification or numeric comparison operators.

Perl - Config::XPath - new version 0.16

2010-08-10T00:23:00.004+01:00

Config::XPath is a Perl module for accessing configuration files using XPath queries. It provides some wrapping around XML::XPath for convenience in using a single config file, and easily fetching string, list or map values from it. It plays nicely alongside, for example, Module::PluginFinder (about which I shall write more another day) for easily building powerful configuration-driven plugin-based programs.

use Config::XPath;
use Module::PluginFinder;

my $conf = Config::XPath->new( filename => 'foomangler.conf' );
my $finder = Module::PluginFinder->new(
                search_path => 'FooMangler::Plugin',
                typefunc    => 'TYPE',
             );

my %plugins;

foreach my $plugin_conf ( $conf->get_sub_list( '/plugin' ) ) {
   my $name = $plugin_conf->get_string( '@name' );
   my $type = $plugin_conf->get_string( '@type' );

   $plugins{$name} = $finder->construct( $type, $plugin_conf );
}

Given a config file that perhaps looks like

<foomangler>
  <plugin type="hello" name="hello_world">
    <message>Hello, world</message>
  </plugin>
</foomangler>

We can implement a plugin for this system quite simply, and have it be automatically discovered by the plugin system, instances created, and passed in its configuration from the config file:

package FooMangler::Plugin::Hello;
use constant TYPE => "hello";

sub new
{
   my $class = shift;
   my ( $config ) = @_;

   my $message = $config->get_string( 'message' );
   ...
}

As well as providing one-shot reading support, it also has a subclass Config::XPath::Reloadable which allows for convenient reloading of config files. It itself keeps track of which XML nodes it has already seen, based on some defined key attribute, so it can determine additions and deletions. It will invoke callback functions when items are added or deleted, or their underlying config may have changed.

use Config::XPath::Reloadable;

my $conf = Config::XPath::Reloadable->new( filename => 'foomangler.conf' );
my $finder = Module::PluginFinder->new( ... );

$SIG{HUP} = sub { $conf->reload };

my %manglers;

$conf->associate_nodeset( '/mangler', '@name',
   add => sub {
      my ( $name, $mangler_conf ) = @_;
      my $type = $mangler_conf->get_string( '@type' );

      $manglers{$name} = $finder->construct( $type, $mangler_conf );
   },

   keep => sub {
      my ( $name, $mangler_conf ) = @_;

      $manglers{$name}->reconfigure( $mangler_conf );
   },

   remove => sub {
      my ( $name ) = @_;

      delete $manglers{$name};
   },
);

Now, whenever a SIGHUP signal is received, the config file is re-read. The configurations for all the current manglers are updated, new ones added, and old ones deleted.

I've just uploaded a new release, 0.16. This release finally gets rid of the awkward Error-based exceptions, instead using plain-old Carp-based string exceptions. This removes a dependency on the old, deprecated, and unsupported Error distribution.

I've also manually set the configure_requires element to set the required version of Module::Build down to 0.2808, which is what Perl 5.10.0 shipped with, rather than let it pick its own version, where it sets it to 0.36. Hopefully this should lead to no awkward "please upgrade Module::Build" on clean-slate installs. If this comes out OK I might start applying that by default across all my dists (where appropriate). It does seem a little awkward, but then I can't really think of a neater way for it to detect that - hard for it to know, for example, about random methods or functionality invoked during the Build.PL file itself, or bugs/features implicitly relied upon. Something to think about for next time, I feel...

Perl - List::UtilsBy

2010-07-30T22:32:00.005+01:00

List::UtilsBy is a module containing a number of list utility functions which all take a block of code to control their behaviour. Among its delights are a neat wrapping of sort by a custom accessor, optimisation, and rearrangement functions. The functions in this module are a loose collection of functions I've written or found useful over the past few months or so. I won't give a full overview here, you can read the docs yourselves; but I will give a brief description of a few functions.

One frequent question we often get in #perl on Freenode concerns how to sort a list of items by some property of the items, perhaps the value of an object accessor or the result of some regexp extraction. Sometimes the answer comes in variants on a theme of

@sorted_items = sort { $a->accessor cmp $b->accessor } @items;

@sorted_items = sort { ( $a =~ m/^(\d+)/ )[0] <=> ( $b =~ m/^(\d+)/ )[0] } @items;

Sometimes a mention of the Schwartzian Transform comes.

I decided to take this often-use pattern and find a nicer way to represent it. The result is the sort_by functional.

@sorted_items = sort_by { $_->accessor } @items;

@sorted_items = sort_by { ( $_ =~ m/^(\d+)/ )[0] } @items;

As well as neatness of code, this also has advantage of invoking the accessor only once per item, rather than once per item pair.

An operation I've often wanted to perform is to search a list of items for the best match by some metric. For this, there is max_by and variations.

$longest = max_by { length $_ } @strings;

$closest = min_by { distance_between( $_->location, $here ) } @places;

Finally, as a replacement for the often-used pattern

@array = grep { wanted $_ } @array;

We have

extract_by { not wanted $_ } @array;

As noted in the documentation, this is implemented by spliceing the unwanted elements out, not by assigning a new list, so this is safe to use on lists containing weak references, or tied variables.

My current Perl project - Circle

2010-07-19T21:05:00.006+01:00

Recently chromatic wrote that we should "tell the world, what are you working on with Perl?"

So, to answer this then, my current project is Circle, an IRC client. Actually, it's much more than an IRC client, but that will do as a first approximation.

Rather than being Just Another IRC client, this one is split into two programs; a backend server that runs on some machine somewhere, likely your co-located shell hosting box, or home server. This maintains all the connections to the IRC networks, persists the scrollback and so on; it is the guts of the logic. Then there's the frontend program, a lightweight GTK application that draws the UI for the backend logic. The frontend doesn't really understand IRC, the backend has no knowledge of GTK. Several readers may recognise something of the MVC pattern about this.

Without going into too much detail here (you can read the above link), this gives you the advantages of a real local native-UI client, plus the advantages of a persistent server. The UI interactions are local, no network latency or bandwidth to get in the way of line editing, backbuffer scrolling, window switching, and so on; yet all the data is persisted in the server so you can just disconnect the thin client and reconnect it from anywhere else.

A common way people usually solve this sort of problem is to run irssi in a screen session, and reconnect over SSH. The primary downside of this setup is it requires a low-latency, high-bandwidth connection to the server, as every keypress of the line editor to send your next line, will have to round-trip over that network. Every backbuffer operation, scrolling up and down, or switching between windows, has to redraw over the link. If that link has high latency, or low bandwidth, the user experience will suffer. If the network charges for bandwidth, you will end up paying many times to keep re-sending the same screenful of scrollback as you switch windows. By not having a real presence on the local desktop, irssi-in-screen also cannot take advantage of local desktop features such as notification sounds or highlight popups, nor can it access the local filesystem to perform DCC transfers or similar.

Another solution to remote persistant IRC is to run an IRC proxy server or bouncer, and point a regular IRC client at that. These either don't support backbuffer refills, or save and replay events, possibly by prefixing timestamps in the message text. They suffer many shortcomings by being a hacked-on proxy in front of an existing IRC client, which overall doesn't really support the disconnect/reconnect model. These solutions almost exclusively are also IRC-specific, and cannot integrate non-IRC (such as Instant Messaging) alongside.

Right now this is pretty-much all there is to it, though the design is such that it can accommodate much more. There's also a plain telnet-alike backend module, but it could quite easily accept Instant Messaging, Email, PIM, whatever. Right now, the only frontend is GTK, but nothing says one couldn't also be written for Qt, Windows, or any other GUI toolkit. I'm also slowly in the process of writing a terminal-mode one.

The code is available on CPAN:

Circle (backend)
Circle-FE-Gtk

Patches, they say, are Welcome.

Good code, bad tests

2010-07-01T22:39:00.000+01:00

I've been working on IO::Async::SSL recently. Both the previous upload, and the next one I shall shortly do, are fixes for failing smoke tests. Moreover, they're fixes for failing tests, of correct code.

The first concerns the semantics of the END block. The code looked roughly like

use Test::More
system( 'socat -help' ) == 0 or plan skip_all => "No socat";
plan tests => 3;

open my $kid, "-|", "socat", .... or die;
END { kill TERM => $kid }

The idea of this code was to ensure that socat is no longer running when the test exits. Of course, if the socat program isn't installed, the entire test is skipped. And the END block is still run. The result of this is that since $kid is undefined, the containing perl process is sent SIGTERM instead, and the test harness exits with an error. Ooops.

Now what I actually wanted here was

END { kill TERM => $kid if defined $kid }

I wonder if this situation would warrant a new block-alike, perhaps lets call it ENDX, which is only executed at END time if the line declaring it was actually reached in code. Perhaps it can be hacked up by

my @ends;
sub ENDX(&) {
   push @ends, $_[0];
}
END { $_->() for @ends }
...
ENDX { kill TERM => $kid };

The second failure is a curious one that starts off looking like an OS-specific bug in the code. All the Linux smoke boxes were giving fails on a particularly innocent-looking test

is( $client->peeraddr, $server->sockaddr, 'Client socket address' );

Under closer inspection, it seems that the way Linux returns socket addresses from the kernel doesn't initialise the "holes" in the address structure, whereas most other OSes do.

The result of this is that rendered as strings of bytes, the two addresses don't necessarily contain all the same bytes as each other, even though they represent the same address. The way to fix this one is to unpack the addresses, known to be AF_INET addresses, and use is_deeply instead:

use Socket qw( unpack_sockaddr_in );

is_deeply( [ unpack_sockaddr_in $client->peeraddr ],
           [ unpack_sockaddr_in $server->sockaddr ],
           'Client socket address' );

This doesn't look too neat this way, but perhaps there's some scope for considering a Test::Sockaddr module or somesuch, to neaten it up. Perhaps a little special-purpose though..

Why you should never use indirect object notation

2010-06-03T12:57:00.004+01:00

On the subject of indirect object notation, kappa writes "Do not use it when it causes problems".

It always causes problems.

$ perl -MO=Deparse -e 'package Container;
  sub new { shift->BUILD }
  sub BUILD { my $contained = new Contained() }'
package Container;
sub new {
    shift(@_)->BUILD;
}
sub BUILD {
    my $contained = new(Contained());
}
-e syntax OK

What you meant, perhaps, was

$ perl -MO=Deparse -e 'package Container;
  sub new { shift->BUILD }
  sub BUILD { my $contained = Contained->new() }'
package Container;
sub new {
    shift(@_)->BUILD;
}
sub BUILD {
    my $contained = 'Contained'->new;
}
-e syntax OK

Basically, indirect notation breaks any time you're actually writing an object class. If you start indirectly invoking constructors (likely), you'll accidentally hit your own. Any time you go adding a new method to your own class - bang; you've just broken any indirect calls to that named method on any other object you'd be using. That sounds excessively fragile to me. Code that used to mean one thing suddenly means a different thing when you add seemingly-unrelated code in another place. Fragile breakage by action-at-a-distance. It even depends on the order of the source code:

$ perl -MO=Deparse -e 'package Container;
  sub BUILD { my $contained = new Contained() }
  sub new { shift->BUILD }'
package Container;
sub BUILD {
    my $contained = 'Contained'->new;
}
sub new {
    shift(@_)->BUILD;
}
-e syntax OK

And given probably most of the time you're writing code, you're writing objects, right? Even if the function you're writing currently isn't in an object class, it probably uses objects. Maybe one day you'll decide that function is useful in an object, so you'll copypaste that code elsewhere. This probably means most of your code ought not use indirect notation. If already most of it isn't using it, just get out of the habit NOW, before it is too late, and pretend that indirect notation does not exist at all. You'll save yourself a lot of pain later on.

Weasels in the Code

2010-05-27T16:48:00.004+01:00

I've recently written the following utility method on an object:

sub _capture_weakself
{
   my $self = shift;
   my ( $code ) = @_;   # actually bare method names work too

   weaken $self;

   return sub { $self->$code( @_ ) };
}

It's quite a useful little method for creating a new CODE ref around either a given code ref, or a method on the object. This CODE ref captures a reference to the object, passing it as the first argument. This makes it useful for passing around elsewhere, perhaps as an event callback (as it happens, this method lives in IO::Async::Notifier). Because the object ref is stored weakly in this closure, it means the returned closure can safely be stored in the object itself without creating a cycle.

It's sufficiently useful that I feel sure this technique must have a name, but so far I'm failing to find one. I manage to keep reading the name here as "capture weasel". This gives me an idea - perhaps it could be said this closure has been weaseled; short for wea(k)sel(f)ed.

my $weasled_code = $notifier->_capture_weakself( sub {
     my $self = shift;
     my ( $x, $y ) = @_;
     ....
} );

$weaseled_code->( 123, 456 );

What does anyone think here? Does this technique have a name already? If not; does this seem suitable? I find it unlikely this name already exists somewhere else in CompSci (FOLDOC doesn't have a use in this sense), so it would be a good unique name...

PF_PACKET, Linux Socket Filters, and IPv6

2010-05-18T19:10:00.010+01:00

For diagnosing network-related problems, it's often useful to be able to capture packets transmitted or received by a machine. Linux implements a socket family, PF_PACKET, to this end. Sockets in this family receive raw datagrams containing packets received or transmitted on network interfaces. A network capture program creates such a socket, then sits in a loop receiving datagrams. Each datagram contains the bytes of the packet. The AF_PACKET address format gives information about, among other things, packet direction and interface number.

In situations where a machine is passing lots of traffic, such as a busy internet router, but the problem being diagnosed concerns only a narrow selection of this traffic, capturing every single packet would be very CPU-intensive. The application could inspect each packet and discard all of those not of interest to it. But since each packet requires a recvfrom(2) system call, this gets quite expensive in context switches, and wasted buffer space. A far more efficient scheme is to have the kernel filter the packets, or at least throw away most of the uninteresting ones.

In Linux, this is done using the SO_ATTACH_FILTER socket option. This option attaches a filter program, which executes on a simple virtual machine within the kernel. This machine is given access to inspect the bytes of the packet, and can return a value indicating whether the packet should be kept, or discarded. This machine is based on BSD's BPF mechanism, with some extensions.

This virtual machine is a register machine. It has a single accumulator (A), a single index register (X), and the various usual sorts of arithmetic and logical operations one would expect. To inspect the actual packet contents it has instructions to load data from the captured packet, in 8, 16, or 32bit unsigned quantities, into the accumulator. The program gives its answer by returning a number to the kernel, which should be the number of bytes of the packet to capture, or 0 to discard it entirely.

For example, lets consider the following example, which selects any TCP packets:

LD BYTE[9]
JEQ 6, 0, 2

LD len
RET A

RET 0

The LD BYTE[offs] instruction loads a byte from the packet (at offset 9, being where IPv4 headers keep their protocol number) into the A register. JEQ is a conditional jump instruction which compares the A register to the immediate constant (6, the protocol number of TCP). If they are equal it jumps to the first label; if not, the second. Both jump labels are unsigned integers, being the number of instructions forwards to skip. In the true case (i.e. the protocol is TCP), the length of the packet is loaded into the accumulator and returned from the filter. In the false case, the number 0 is returned, to discard the packet.

Of course, this filter isn't much use yet, because we don't know for sure it's even an IPv4 packet we've caught. The original BSD BPF doesn't define any metadata access scheme, so Linux has invented a way for programs to access this. It reserves the topmost 4KiB of packet buffer, to provide some virtual "registers" containing metadata. Since the load instruction offsets are 32bit integers, it is unlikely any real packet would ever be anywhere near this size, so in practice this works well.

In this extra data area, called the Ancillary Data, the first offset stores the ethertype of the packet (this field address has a symbolic name, SKF_AD_PROTOCOL). In IPv4's case, this will be 0x0800. We can extend our filter to look at this protocol number (not to be confused with IPv4's protocol field), and check it.

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 0, 4

LD BYTE[9]
JEQ 6, 0, 2

LD len
RET A

RET 0

We can further extend our filter, looking only for a certain TCP port number (such as 80, for http). Presuming an IPv4 header with no extra options, it is 20 bytes long, and therefore, the TCP header will start at offset 20:

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 0, 8

LD BYTE[9]
JEQ 6, 0, 6

LD HALF[20]
JEQ 22, 2, 0
LD HALF[22]
JEQ 22, 0, 2

LD len
RET A

RET 0

Of course, this filter isn't much use in the real world, because we made the rather large assumption that an IPv4 header would be 20 bytes long. This is where the index register becomes useful. We can load the index register, X, with the size of the IPv4 header, and access the TCP header relative to this. BPF provides a very special instruction, LDMSHX, for exactly this purpose. Given the address of a byte, it loads that byte, masking off the bottom 4 bits, and shifts up by 2 bits. This calculates the size in bytes of an IPv4 header, given the first byte.

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 0, 9

LD BYTE[9]
JEQ 6, 0, 7

LDMSHX BYTE[0]
LD HALF[X+0]
JEQ 22, 2, 0
LD HALF[X+2]
JEQ 22, 0, 2

LD len
RET A

RET 0

Finally, this filter is only of any use on a SOCKET_DGRAM socket; a socket where the kernel will throw away the link-level header (such as the Ethernet or PPP framing), and present only the network-level header. A packet capturing program would very likely be interested in those link-level bytes too, so would be using a SOCKET_RAW socket instead. In this case, we don't directly know the offset of the IPv4 header, but once again, Linux comes to our rescue. It provides a "virtual view" over the part of the packet from the start of the network header. It provides a virtual offset, NET, where load instructions read relative to the start of the network header, rather than from the start of the packet buffer as a whole.

On a SOCKET_RAW socket, this filter would probably be appropriate to select TCP port 80 over IPv4:

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 0, 9

LD BYTE[NET+9]
JEQ 6, 0, 7

LDMSHX BYTE[NET+0]
LD HALF[NET+X+0]
JEQ 22, 2, 0
LD HALF[NET+X+2]
JEQ 22, 0, 2

LD len
RET A

RET 0

This is internally implemented by carving up a further extra data area, 1MiB from the top of the packet buffer (defined by a constant SKF_NET_OFF). This offset gives a virtual view of the bytes in the packet, starting at the network protocol header.

Of course, I have been looking IPv4 quite specifically here. With the slowly-growing popularity of IPv6, it's inevitable that packet capture programs might want to capture IPv6 packets too.

IPv6 follows a different style from IPv4 in terms of its header options. In IPv4, all the IP-level options are stored in the header, one after another. The "header length" field in the header gives the total size of the header, with all these options. In IPv6, the header contains fewer fields (because things like fragmentation are now options). At the end of this header is the "next header" number, which is either a protocol number such as TCP or UDP, or gives an IPv6 extension header number (such as fragmentation control, or IPsec's authenticating header). Each option links on to the next with its own "next header" field.

This presents us something of a problem when it comes to packet capture filters. Recall how, for the JEQ instruction, the branch labels in both cases are unsigned integers? This is how BPF guarantees termination of a program in finite time - every jump has to be forwards. There can be no loops. Without loops, the program is guaranteed to terminate.

But now how do we parse these IPv6 headers? We can't write a while() loop in the program to walk down the headers, until we find a TCP one. Furtheremore, IPv6 doesn't define a standard header layout for all headers. Each header type puts its "next header" field in possibly a different place. Some headers are fixed-length, some carry a "header length" field of their own. It's a total mess, from the point of view of packet filtering.

What I would propose, is to create two new metadata constants:

A new Ancillary Data area field, SKF_AD_TRANSPROTO to store the transport level protocol.

A new data area offset, SKF_TRANS_OFF, to give a virtual view of the transport header.

This will allow us to very easily write a packet filter to capture TCP port 80, say, agnostic of IPv4 or IPV6. The filter program would look like:

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 9, 0
JEQ 0x86dd, 0, 8

LD WORD[AD_TRANSPROTO]
JEQ 6, 0, 6

LD HALF[TRANS+0]
JEQ 22, 2, 0
LD HALF[TRANS+2]
JEQ 22, 0, 2

LD len
RET A

RET 0

We've now created a filter which can detect the transport protocol of TCP, and inspect the transport header, without having to directly calculate its offset from hard-coded knowledge of the network protocol.

I would like to propose that Linux adopts these two constants, and finds a way to implement them. I have some thoughts on implementation but I will defer these to a later post; as this one has gone on quite long enough already. :)

Perl - ExtUtils::H2PM

2010-04-30T22:18:00.006+01:00

I've spent a lot of time lately writing modules that wrap Linux kernel features in some way or another. They all boil down to basically the same thing - export a bunch of constants, and structure packing/unpacking functions. Various bits of extra fancy interface code are sometimes nice, but most of the time these can be written in Pure Perl once the base bits are done.

It's always annoyed me that one has to write an XS module just to obtain these. It's a lot of extra work and bother, for something that ought to be so simple. So instead, I started thinking about it, how can I make this much simpler from Perl?

What I came up with is ExtUtils::H2PM; a module to make it trivially-easy to write this sort of boring module to wrap some constants and structures from OS's header files.

As a brief example, consider the following

use ExtUtils::H2PM;

module "Fcntl";

include "fcntl.h";

constant "O_RDONLY";

write_output $ARGV[0];

This is it. This is all the code you, as a module author, have to write. You store that in some file, lets call it Fcntl.pm.PL. You let the build system go about converting that into Fcntl.pm; which on my system now looks like this:

package Fcntl;
# This module was generated automatically by ExtUtils::H2PM from -e

push @EXPORT_OK, 'O_RDONLY';
use constant O_RDONLY => 0;

1;

This is a plain standard Perl module that can be installed and used, to provide that constant.

We can also create pack/unpack-style functions to wrap structure types too. Consider

use ExtUtils::H2PM;

module "Time";

include "time.h";

structure "struct timespec",
  members => [
    tv_sec  => member_numeric,
    tv_nsec => member_numeric,
  ];

write_output $ARGV[0];

For this we obtain:

package Time;
# This module was generated automatically by ExtUtils::H2PM from -e

use Carp;
push @EXPORT_OK, 'pack_timespec', 'unpack_timespec';

sub pack_timespec
{
   @_ == 2 or croak "usage: pack_timespec(tv_sec, tv_nsec)";
   my @v = @_;
   pack "q q ", @v;
}

sub unpack_timespec
{
   length $_[0] == 16 or croak "unpack_timespec: expected 16 bytes";
   my @v = unpack "q q ", $_[0];
   @v;
}

1;

This was done entirely automatically too.

In a real use-case, you'd be using this to wrap things like socket option constants/structures, and so on; cases where existing Perl functions wrap existing syscalls, you simply want to provide new option values or structures. I've now written several modules mostly or entirely relying on ExtUtils::H2PM to build them, freeing me of having to write all that XS code to implement them.

Linux::SocketFilter

Socket::Netlink

Socket::Netlink::Route

Socket::Netlink::Taskstats

Order matters even when it doesn't

2010-04-26T16:20:00.005+01:00

Revision control diffs are most readable when they aren't noisy. Operations that disturb the order of many lines in the file create noise which makes it hard to read the interesting change. YAML specifies mappings (hashes, to us perl-types), that are unordered associations of keys to values. Even though YAML doesn't put an ordering on those, sometimes we'd like to pretend that it does, so as to preserve the order when we load a file, edit the data, then dump it back to the file.

At work we store a YAML document in Subversion, which describes a lot of details about IPsec tunnels. In an ideal world this would be the initial source of the information. The world, as you may have observed, is not yet ideal, so this file is in fact back-scraped from information in the actual config files, to keep it up to date. Naturally that's done in Perl.

The YAML file stores a big mapping, each entry itself being a record-like mapping, containing details in named keys. This causes great trouble for our load/edit/dump script, because YAML doesn't specify an ordering in mapping keys. They'll be dumped in "no particular order".
This wouldn't normally be a problem, except that because it's stored in Subversion, a commit changing one line of actual detail might suddenly produce hundreds of lines of false diff, because of reordered keys.

To solve this, I had to apply Much Evil Hackery. The YAML Perl module, it turns out, has a data structure tied to a hash, which remembers the order of keys. By subclassing YAML::Loader and replacing its method to read a mapping into a hash ref, we can force it to use this structure instead. This alteration is transparent to the perl code inbetween, it just sees a normal hash. However, YAML::Dumper sees the ordering and preserves it when it writes out.

The upshot: Load/edit/dump of trees of mappings in YAML preserves ordering, allowing cleaner commits into revision control.

This has been suggested as a wishlist bug against YAML; see also https://rt.cpan.org/Ticket/Display.html?id=56741

Why I won't tell you what to do

2010-04-06T11:33:00.006+01:00

Often in #perl we get people asking a question, whether explicitly or implied, that requires us to pick a solution for them. I try hard not to do this. The closest I'll get is to listen to their description of the problem, and suggest some things which I think might help. Hardly ever do I suggest just one thing.

I do this because it's hard for us to know the entire surrounding context of a problem. If anyone else is like me, then there'll be days, weeks, maybe even months of history behind it; various attempts they've tried already, other bits and pieces of code, system, whatever, that they haven't been able to explain in the 5 minutes they tried to give us the problem. You can't explain a 1-month problem in 5-minutes, nor can I give a solution to it in similar timeframe.

What I can do is name a few things that I'd include in a shortlist of things to think about in more detail, were I to find myself with a similar problem. They can then go away and think about these things. Maybe he already was aware of them, and we've just given some more confidence that those might be correct. Or maybe he wasn't, so we've given him something new to read about. Either way, we've helped guide the decision process, without outright saying "thou shalt do this" - because, without having that month of context around it; for all we know it could be completely wrong. But it's a good start.

Perl - IO::Async

2010-03-31T08:30:00.000+01:00

IO::Async is:

a Perl module distribution
a generic eventing framework
suitable for all kinds of IO-bound tasks
a way to achieve IO concurrency
portable to Linux, Solaris, BSDs, other POSIXes
mostly-working on cygwin
able to use OS-specific optimisations where appropriate
able to use Glib for running GTK2-based GUI applications

IO::Async serves as a base layer for all kinds of IO-heavy programs.

Its design is centred around passing code references as continuations; handlers to invoke when a certain event happens, or after a certain operation has completed. By relying on lexical variable capture in these code references, these continuations can easily store state in normal variables.

It's a different approach than that taken by POE, the other eventing system for Perl, but I find this approach fits better in my head. If you dislike storing state in a hash to pass it around disjoint named functions loosely connected by event names, you might just want to give it a try...