2010/12/19

Perl - CPS - version 0.11

CPS is a Perl module that provides several utilities for writing programs in Continuation Passing Style.

In a nutshell, CPS is a method of control flow within a program, which uses values passed around as continuations as a replacement of the usual call/return semantics. In Perl, this is done by passing CODE references; calling a function and passing in a CODE reference to be invoked with the result, rather than having the function return it. While at first this doesn't seem to be very useful, most of its power comes from the observation that the function doesn't need to invoke its continuation immediately; if it performs some IO operation or similar, it can perform this in an asynchronous fashion, invoking its continuation later. This style of coding is often associated with nonblocking or asynchronous event-driven programming. It is typical of such event systems as IO::Async.

A typical problem with implementing CPS control flow, is that all of the usual Perl control-flow mechanisms are built for immediate call/return semantics, where the use of CPS gets in the way. The CPS module provides utility functions for implementing control flow in a continuation passing style, by providing a set of functions that replace the usual Perl control-flow keywords. For example the Perl control structure of a foreach loop
foreach my $frob ( @frobs ) {
my $wibble = mangle( $frob );
say "Mangled $frob looks like $wibble";
}
say "All done";
becomes a call to CPS::kforeach
use CPS qw( kforeach );

kforeach( \@frobs, sub {
my ( $frob, $knext ) = @_;
kmangle( $frob, sub {
my ( $wibble ) = @_;
say "Mangled $frob looks like $wibble";
$knext->();
} );
}, sub {
say "All done";
} );
We haven't really gained anything by doing this though. If the process of mangling a frob involves some IO tasks, perhaps talking to some remote server, then we'll spend most of our time waiting for it to respond when we could be sending multiple requests and waiting on their responses. We could likely save some time by running them concurrently.

I gave a talk at LPW2010 about Continuation Passing Style and CPS, the slides of which are available here.

After discussing CPS and IO::Async at LPW, I was talking with mst about his IPC::Command::Multiplex module. He came up with the idea for another control-flow function; a combination of kpar and kforeach, which I called kpareach. In use it looks exactly like kforeach, except that it starts the loop body for each item all at the same time, in parallel, rather than waiting for each one to finish before invoking the next.

This is a new addition to CPS version 0.11, which is now on CPAN. It is also one of the first new control-flow structures that doesn't have a direct plain-Perl counterpart; a demonstration of the usefulness of CPS in event-driven programming.

2010/11/21

General Updates

I have no big specific updates today. Instead, a list of lots of little things I've been working on:
  • IO::Socket::IP now has preliminary non-blocking connect support (version 0.05_003). This isn't quite a perfect solution because of blocking name resolvers, but see also Net::LibAsyncNS below.

  • Created a new CPAN dist, wrapping libasyncns, called Net::LibAsyncNS. This allows a simple way to asynchronise name resolver lookups.

  • Have fixed a few bugs in IO::KQueue, relating to dodgy handling of Perl scalars in the udata field. Some memory leak bugs still exist, but I believe these to be the kernel's fault. See below.

  • Spent some time on Freebsd-hackers@ arguing about kqueue and managing user data pointers. Long story short I believe kqueue API itself is missing a feature, making generic wrapping of it impossible from any high-level language, or properly by C libraries also.

  • Both talks I submitted for LPW2010 were accepted; on the subjects of CPS and IO::Async.

  • Net::Async::HTTP now has SSL support and can stream response body content as it arrives, rather than waiting for the whole response (version 0.08).
That's all for a quick update, but I may write about any or all of these topics in more detail later...

2010/10/31

Perl - Test::Identity

Today I uploaded a new module to CPAN; Test::Identity. It's possibly the quickest module I've ever written, from when I decided to write it, to when it was actually uploaded:
(18:02) sub Test::Identity::identical { is refaddr $_[0], refaddr $_[1], $_[2] } <== I'm about to write such in a module, unless anyone can suggest me a module that already has it
...
(19:21) * GumbyPAN CPAN Upload: Test-Identity-0.01 by PEVANS
I won't spend a long time explaining why, I'll just quote the docs:

This module provides a single testing function, identical. It asserts that a given reference is as expected; that is, it either refers to the same object or is undef. It is similar to Test::More::is except that it uses refaddr, ensuring that it behaves correctly even if the references under test are objects that overload stringification or numification.

It also provides better diagnostics if the test fails:
$ perl -MTest::More=tests,1 -MTest::Identity -e'identical [], {}'
1..1
not ok 1
# Failed test at -e line 1.
# Expected an anonymous HASH ref, got an anonymous ARRAY ref
# Looks like you failed 1 test of 1.

$ perl -MTest::More=tests,1 -MTest::Identity -e'identical [], []'
1..1
not ok 1
# Failed test at -e line 1.
# Expected an anonymous ARRAY ref to the correct object
# Looks like you failed 1 test of 1.

2010/10/18

Perl - IO::Socket::IP

IO::Socket::IP is a subclass of IO::Socket that provides a protocol-independent way to use IPv4 and IPv6 sockets. It provides an API compatible with the IPv4-only IO::Socket::INET, but does so in a way that ensures properly transparent IPv6 support.

The following example shows it has an identical API to the ::INET module:
use IO::Socket::IP;

my $sock = IO::Socket::IP->new(
PeerHost => "www.google.com",
PeerPort => "www",
) or die "Cannot construct socket - $@";
At this point, $sock is just another IO::Socket-derived filehandle, and supports all the usual methods and IO functionality. The only difference here is that, where IO::Socket::INET would have used a legacy gethostbyname call and made a PF_INET socket, IO::Socket::IP will use getaddrinfo (via Socket::GetAddrInfo), and will use either PF_INET or PF_INET6 as appropriate.

It's not yet a complete 100% API clone of ::INET though. While it supports all the methods, there are a few constructor arguments not yet supported, being Blocking and Timeout. The way that getaddrinfo returns a list of candidate addresses, to be tried in order, makes nonblocking support hard to do, and complicates the model for what timeout really means. For nonblocking connect support, better solutions already exist, such as IO::Async's Connector, which has always supported IPv6 via getaddrinfo. As for timeouts, eventually IO::Socket::IP should support it, but for now a local'ised $SIG{ALRM} and alarm call should suffice.

For almost all use cases, switching to using IO::Socket::IP should be a simple matter, because of the API similarities. Just adding an extra dependency (because ::IP isn't core), and substituting the package name in source should be enough.

Finally, in case you don't want to pull in an extra hard dependency, you might consider the following fragment I've used quite successfully:
   my $class = eval { require IO::Socket::IP
and "IO::Socket::IP" } ||
do { require IO::Socket::INET
and "IO::Socket::INET" };

$socket = $class->new(
PeerHost => $host,
PeerPort => $port,
) or die "Cannot connect - $@\n";
There. Now you have no excuse for not being IPv6-ready.

2010/09/23

Perl - IO::Async - version 0.30

Yesterday, I put the next version of IO::Async on CPAN; version 0.30. This was primarily an update to add some new features, though also a few minor bugfixes and documentation updates were included too. Here I want to focus on a few of these new features.

The first of these new features is nothing groundbreaking in itself, but feeds into the others. It's simply the addition of IO::Async::Socket, a notifier subclass to contain a socket that isn't necessarily a stream (primarily SOCK_DGRAM or SOCK_RAW sockets such as UDP, PF_PACKET or PF_NETLINK). This neatens up a few rough edges with trying to put such sockets directly in IO::Async::Handle objects.

The second main new feature is the creation of the IO::Async::Protocol class, and IO::Async::Protocol::Stream subclass. These derive directly from IO::Async::Notifier rather than IO::Async::Handle, and are intended to be abstract containers of code, and not perform any IO operations directly. Instead, they contain a Handle or Stream object as a child notifier. By exposing an API identical to IO::Async::Stream, the IO::Async::Protocol::Stream should be a drop-in replacement for any modules trying to implement a network protocol.

With the addition of IO::Async::SSL, not every stream-like connection can be represented by IO::Async::Stream, so separating the transport layer from the protocol layer is required. This wasn't possible by subclassing, whereas object containment makes it much simpler.

Net::Async::FTP, Net::Async::HTTP, and Net::Async::IRC have all been updated to use it, and most other use cases should be simple to change.

The final main change is that $loop->connect and IO::Async::Listener now support direct on_stream or on_socket continuations, which will be provided an instance of Stream or Socket directly, rather than requiring the invoked code to wrap one. This can then be easily configured as a IO::Async::Protocol's transport.

Having made this change, it leads the way to transparent SSL support across all protocols, and possibly other concerns like SOCKS proxies, by extending the arguments to $loop->connect or Listener. But that's for another post...

Finally, I should announce that I've now started a channel on irc.perl.org called #ioasync, as the official IRC home for IO::Async. Feel free to drop by if you have any issues, comments, questions,...

2010/09/20

Perl - overload::substr

overload allows an object class to provide methods which Perl should use to implement certain operators, like numerical addition or string concatenation. One operator that overload doesn't allow to be provided, is substr.

overload::substr allows this to be overloaded. This allows objects that behave like a string, to specify to Perl how they will handle the substr operator.
$ cat example.pl 
#!/usr/bin/perl

use strict;
use feature qw( say );

package ExampleString;

use overload::substr;

sub new { return bless [ @_ ]; }

sub _substr
{
my $self = shift;
my ( $offs, $len, $replace ) = @_;

return sprintf ">> %s between %d and %d <<", $self, $offs, $offs+$len;
}

package main;

my $str = ExampleString->new( "Hello, world" );

say substr( $str, 2, 5 );

$ perl example.pl
>> ExampleString=ARRAY(0x86dd9c8) between 2 and 7 <<
The module is still in its early days yet, but the basics appear to be working on all Perl versions back to 5.8. I also want to try extending it, so that split() and regexp matches with m// and substitutions with s/// also use the substr operation. The identity that
$1 == substr( $str, $-[1], $+[1] - $-[1] )
is sure to be useful here.

I need a good example to show it off with sometime. I have in mind a string-alike object with real positional cursors, which remember their contextual position even after edits in other parts of the string. But more on that later...

2010/09/06

Module name suggestions: A proper IO::Socket for IPv4/IPv6 duallity

I currently don't have a good name for a module I'd like to write, because I think it is very much required right now.

We have IO::Socket::INET. It wraps PF_INET, thus making it IPv4 only.

We have IO::Socket::INET6. It wraps either PF_INET or PF_INET6, despite its name. It also uses Socket6, thus restricting it to only working on machines capable of supporting IPv6.

Thus any author wanting to write code to communicate to the internet (apparently that's some new fad everyone's talking about this week) is presented a moral dilema: Support IPv6 at the cost of not working on older v4-only machines, or support older machines but be incapable of using IPv6.

I originally partially solved this problem some years ago by the creation of Socket::GetAddrInfo, a module that presents the interface of RFC2553's getaddrinfo(3) and getnameinfo(3) functions. This however is not enough for actually connecting and using sockets.

I'd therefore like to propose a new IO::Socket subclass that uses these and only these functions, for converting between addresses and name/service pairs.
use IO::Socket::YourNameHere;

my $sock = IO::Socket::YourNameHere->new(
PeerHost => "www.google.com",
PeerService => "www",
);

printf "Now connected to %s:%s\n", $sock->peerhost, $sock->peerservice;

...
Since it would use Socket::GetAddrInfo, it can transparently support IPv4 or IPv6. Since it would only use Socket::GetAddrInfo, it will work in a v4-only mode on machines incapable of supporting IPv6, and will not be restricted to only IPv4 or IPv6 if and when some new addressing family comes along to replace IPv6 one day; as v6 is now trying to do with v4.

In order to provide an easy transition period, I'd also support additional IO::Socket::INET options where they still make sense; e.g. accepting {Local/Peer}Port as a synonym for {Local/Peer}Service. The upshot here ought to be that you can simply
sed -i s/IO::Socket::INET/IO::Socket::YourNameHere/
and suddenly your code will JustWork on IPv6 in a good >90% of cases.

Can anyone suggest me a better module name for this?


Edit 2010/09/07: We seem to be settling on IO::Socket::IP for this currently.


Edit 2010/09/23: We did indeed settle on IO::Socket::IP; this is now up on CPAN, and will be the subject of a future posting...


This cross-posted from module-authors@perl.