Perl - Dual-life Socket

I've been working on how to dual-life the Socket module, so it can go on CPAN and give older Perl versions the benefits of recent additions; namely getaddrinfo(3)/getnameinfo(3) wrappings and other IPv6 support.

I have what's currently marked as an unauthorized release now on CPAN, though I'm hoping to get co-maint on it to release officially. In the meantime, it'd be useful to get some smoketest reports on what works and what doesn't. I've tested on Linux at perl 5.10.1, 5.12.4 and 5.14.1. Other OSes, especially MSWin32, would be most appreciated.

http://search.cpan.org/~pevans/Socket-1.94_03/

Perl - IO::Async - version 0.41

Wow; seems I haven't been writing these posts as much as I should be. I last wrote about version 0.34, and just now I've uploaded version 0.41 to CPAN. Quite a bit changed between then and now, here's a rundown of the most important bits:

• Added IO::Async::FileStream. This behaves like a read-only IO::Async::Stream, but reads its data from a regular file on the filesystem. Like tail -f, or File::Tail, it watches the file for changes in size, and follows appended data.


• Added IO::Async::Function. This is a true IO::Async::Notifier subclass to represent an asynchronous block of code; which was previously handled by DetachedCode. Being a true Notifier subclass gives it many advantages, and should provide a better base to build other things on. IO::Async::Resolver is now a subclass of Function, not DetachedCode.


• Added IO::Async::Process. This is another IO::Async::Notifier subclass, this time to represent an external process.


• Support encoding parameter in IO::Async::Stream. This allows the Stream to handle text in some encoding, rather than simply raw bytes.


• Support first_interval parameter to IO::Async::Timer::Periodic. If supplied, this will be the first wait time when the timer is started. In particular, if it is zero the timer's first invocation will happen immediately.


• Allow Loop->listen to be extended via extensions, similar to ->connect. Such extensions as IO::Async::SSL are already set up to make use of this.


• Distribution now uses Test::Fatal rather than Test::Exception. The former is smaller and simpler, whereas the latter relies on clever tricks to hack on caller(), which sometimes breaks some setups.


• Added convenient method in IO::Async::Loop for handling socket addresses; extract_addrinfo. This recognises strings for common networking constants; 'inet', 'inet6' and 'unix' as socket families, 'stream', 'dgram' and 'raw' as socket types. These convenient forms are also recognised by the ->connect and ->listen methods.


• Now prefers to use the IPv6 support functions found in Perl 5.14's Socket module; only falling back to using Socket::GetAddrInfo in earlier perls.

IPv6 in Perl

A lot of people are talking about IPv6 lately. And perhaps with good reason - it's been years in coming, but finally we're really starting to run out of IPv4 addresses. Tools like the IPv4 Address Report may put a certain amount of panic on things with their realtime Flash-based countdown widget, but the problem is real and does need sorting.

The latest development release of Perl; perl-5.13.9, now has full support for IPv6, and the new address handling functions specified in RFC 2553. The new functions all live in Socket.

As well as the low-level AF_INET6 constant, and the pack_sockaddr_in6 and unpack_sockaddr_in6 structure functions (already in place in 5.13.8 in fact), there is now the full set of getaddrinfo, getnameinfo and associated constants. These allow fully protocol-agnostic handling of connections and addresses. There is now enough in core to support IO::Socket::IP. This is a fully protocol-agnostic replacement of IO::Socket::INET.

use IO::Socket::IP;

my $sock = IO::Socket::IP->new(
   PeerHost    => "www.google.com",
   PeerService => "www",
) or die "Cannot construct socket - $@";

printf "Now connected to %s:%s\n", $sock->peerhost_service;

...

What could be simpler?

Perhaps now is the time to make a case for putting IO::Socket::IP itself in the core distribution, such that when perl-5.14.0 comes out, it will be properly ready for the next 30 years of the Internet?

IO::Socket::IP's API is designed to be a drop-in replacement for the IPv4-only IO::Socket::INET. Any program currently using INET ought to work exactly the same, by simply substituting IP instead, and now will also work on IPv6.

If you maintain a distribution that uses IO::Socket::INET, please try out IO::Socket::IP instead. I'd be very keen to hear from anyone who finds it doesn't JustWork in their situation.

See also my earlier post on the subject; Perl - IO::Socket::IP

Perl - IO::Socket::IP

IO::Socket::IP is a subclass of IO::Socket that provides a protocol-independent way to use IPv4 and IPv6 sockets. It provides an API compatible with the IPv4-only IO::Socket::INET, but does so in a way that ensures properly transparent IPv6 support.

The following example shows it has an identical API to the ::INET module:

use IO::Socket::IP;

my $sock = IO::Socket::IP->new(
   PeerHost => "www.google.com",
   PeerPort => "www",
) or die "Cannot construct socket - $@";

At this point, $sock is just another IO::Socket-derived filehandle, and supports all the usual methods and IO functionality. The only difference here is that, where IO::Socket::INET would have used a legacy gethostbyname call and made a PF_INET socket, IO::Socket::IP will use getaddrinfo (via Socket::GetAddrInfo), and will use either PF_INET or PF_INET6 as appropriate.

It's not yet a complete 100% API clone of ::INET though. While it supports all the methods, there are a few constructor arguments not yet supported, being Blocking and Timeout. The way that getaddrinfo returns a list of candidate addresses, to be tried in order, makes nonblocking support hard to do, and complicates the model for what timeout really means. For nonblocking connect support, better solutions already exist, such as IO::Async's Connector, which has always supported IPv6 via getaddrinfo. As for timeouts, eventually IO::Socket::IP should support it, but for now a local'ised $SIG{ALRM} and alarm call should suffice.

For almost all use cases, switching to using IO::Socket::IP should be a simple matter, because of the API similarities. Just adding an extra dependency (because ::IP isn't core), and substituting the package name in source should be enough.

Finally, in case you don't want to pull in an extra hard dependency, you might consider the following fragment I've used quite successfully:

   my $class = eval { require IO::Socket::IP
                         and "IO::Socket::IP" } ||
               do   { require IO::Socket::INET
                         and "IO::Socket::INET" };

   $socket = $class->new(
      PeerHost => $host,
      PeerPort => $port,
   ) or die "Cannot connect - $@\n";

There. Now you have no excuse for not being IPv6-ready.

Module name suggestions: A proper IO::Socket for IPv4/IPv6 duallity

I currently don't have a good name for a module I'd like to write, because I think it is very much required right now.

We have IO::Socket::INET. It wraps PF_INET, thus making it IPv4 only.

We have IO::Socket::INET6. It wraps either PF_INET or PF_INET6, despite its name. It also uses Socket6, thus restricting it to only working on machines capable of supporting IPv6.

Thus any author wanting to write code to communicate to the internet (apparently that's some new fad everyone's talking about this week) is presented a moral dilema: Support IPv6 at the cost of not working on older v4-only machines, or support older machines but be incapable of using IPv6.

I originally partially solved this problem some years ago by the creation of Socket::GetAddrInfo, a module that presents the interface of RFC2553's getaddrinfo(3) and getnameinfo(3) functions. This however is not enough for actually connecting and using sockets.

I'd therefore like to propose a new IO::Socket subclass that uses these and only these functions, for converting between addresses and name/service pairs.

use IO::Socket::YourNameHere;

my $sock = IO::Socket::YourNameHere->new(
   PeerHost    => "www.google.com",
   PeerService => "www",
);

printf "Now connected to %s:%s\n", $sock->peerhost, $sock->peerservice;

...

Since it would use Socket::GetAddrInfo, it can transparently support IPv4 or IPv6. Since it would only use Socket::GetAddrInfo, it will work in a v4-only mode on machines incapable of supporting IPv6, and will not be restricted to only IPv4 or IPv6 if and when some new addressing family comes along to replace IPv6 one day; as v6 is now trying to do with v4.

In order to provide an easy transition period, I'd also support additional IO::Socket::INET options where they still make sense; e.g. accepting {Local/Peer}Port as a synonym for {Local/Peer}Service. The upshot here ought to be that you can simply

sed -i s/IO::Socket::INET/IO::Socket::YourNameHere/

and suddenly your code will JustWork on IPv6 in a good >90% of cases.

Can anyone suggest me a better module name for this?

---

Edit 2010/09/07: We seem to be settling on IO::Socket::IP for this currently.

---

Edit 2010/09/23: We did indeed settle on IO::Socket::IP; this is now up on CPAN, and will be the subject of a future posting...

---

This cross-posted from module-authors@perl.

PF_PACKET, Linux Socket Filters, and IPv6

For diagnosing network-related problems, it's often useful to be able to capture packets transmitted or received by a machine. Linux implements a socket family, PF_PACKET, to this end. Sockets in this family receive raw datagrams containing packets received or transmitted on network interfaces. A network capture program creates such a socket, then sits in a loop receiving datagrams. Each datagram contains the bytes of the packet. The AF_PACKET address format gives information about, among other things, packet direction and interface number.

In situations where a machine is passing lots of traffic, such as a busy internet router, but the problem being diagnosed concerns only a narrow selection of this traffic, capturing every single packet would be very CPU-intensive. The application could inspect each packet and discard all of those not of interest to it. But since each packet requires a recvfrom(2) system call, this gets quite expensive in context switches, and wasted buffer space. A far more efficient scheme is to have the kernel filter the packets, or at least throw away most of the uninteresting ones.

In Linux, this is done using the SO_ATTACH_FILTER socket option. This option attaches a filter program, which executes on a simple virtual machine within the kernel. This machine is given access to inspect the bytes of the packet, and can return a value indicating whether the packet should be kept, or discarded. This machine is based on BSD's BPF mechanism, with some extensions.

This virtual machine is a register machine. It has a single accumulator (A), a single index register (X), and the various usual sorts of arithmetic and logical operations one would expect. To inspect the actual packet contents it has instructions to load data from the captured packet, in 8, 16, or 32bit unsigned quantities, into the accumulator. The program gives its answer by returning a number to the kernel, which should be the number of bytes of the packet to capture, or 0 to discard it entirely.

For example, lets consider the following example, which selects any TCP packets:

LD BYTE[9]
JEQ 6, 0, 2

LD len
RET A

RET 0

The LD BYTE[offs] instruction loads a byte from the packet (at offset 9, being where IPv4 headers keep their protocol number) into the A register. JEQ is a conditional jump instruction which compares the A register to the immediate constant (6, the protocol number of TCP). If they are equal it jumps to the first label; if not, the second. Both jump labels are unsigned integers, being the number of instructions forwards to skip. In the true case (i.e. the protocol is TCP), the length of the packet is loaded into the accumulator and returned from the filter. In the false case, the number 0 is returned, to discard the packet.

Of course, this filter isn't much use yet, because we don't know for sure it's even an IPv4 packet we've caught. The original BSD BPF doesn't define any metadata access scheme, so Linux has invented a way for programs to access this. It reserves the topmost 4KiB of packet buffer, to provide some virtual "registers" containing metadata. Since the load instruction offsets are 32bit integers, it is unlikely any real packet would ever be anywhere near this size, so in practice this works well.

In this extra data area, called the Ancillary Data, the first offset stores the ethertype of the packet (this field address has a symbolic name, SKF_AD_PROTOCOL). In IPv4's case, this will be 0x0800. We can extend our filter to look at this protocol number (not to be confused with IPv4's protocol field), and check it.

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 0, 4

LD BYTE[9]
JEQ 6, 0, 2

LD len
RET A

RET 0

We can further extend our filter, looking only for a certain TCP port number (such as 80, for http). Presuming an IPv4 header with no extra options, it is 20 bytes long, and therefore, the TCP header will start at offset 20:

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 0, 8

LD BYTE[9]
JEQ 6, 0, 6

LD HALF[20]
JEQ 22, 2, 0
LD HALF[22]
JEQ 22, 0, 2

LD len
RET A

RET 0

Of course, this filter isn't much use in the real world, because we made the rather large assumption that an IPv4 header would be 20 bytes long. This is where the index register becomes useful. We can load the index register, X, with the size of the IPv4 header, and access the TCP header relative to this. BPF provides a very special instruction, LDMSHX, for exactly this purpose. Given the address of a byte, it loads that byte, masking off the bottom 4 bits, and shifts up by 2 bits. This calculates the size in bytes of an IPv4 header, given the first byte.

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 0, 9

LD BYTE[9]
JEQ 6, 0, 7

LDMSHX BYTE[0]
LD HALF[X+0]
JEQ 22, 2, 0
LD HALF[X+2]
JEQ 22, 0, 2

LD len
RET A

RET 0

Finally, this filter is only of any use on a SOCKET_DGRAM socket; a socket where the kernel will throw away the link-level header (such as the Ethernet or PPP framing), and present only the network-level header. A packet capturing program would very likely be interested in those link-level bytes too, so would be using a SOCKET_RAW socket instead. In this case, we don't directly know the offset of the IPv4 header, but once again, Linux comes to our rescue. It provides a "virtual view" over the part of the packet from the start of the network header. It provides a virtual offset, NET, where load instructions read relative to the start of the network header, rather than from the start of the packet buffer as a whole.

On a SOCKET_RAW socket, this filter would probably be appropriate to select TCP port 80 over IPv4:

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 0, 9

LD BYTE[NET+9]
JEQ 6, 0, 7

LDMSHX BYTE[NET+0]
LD HALF[NET+X+0]
JEQ 22, 2, 0
LD HALF[NET+X+2]
JEQ 22, 0, 2

LD len
RET A

RET 0

This is internally implemented by carving up a further extra data area, 1MiB from the top of the packet buffer (defined by a constant SKF_NET_OFF). This offset gives a virtual view of the bytes in the packet, starting at the network protocol header.

Of course, I have been looking IPv4 quite specifically here. With the slowly-growing popularity of IPv6, it's inevitable that packet capture programs might want to capture IPv6 packets too.

IPv6 follows a different style from IPv4 in terms of its header options. In IPv4, all the IP-level options are stored in the header, one after another. The "header length" field in the header gives the total size of the header, with all these options. In IPv6, the header contains fewer fields (because things like fragmentation are now options). At the end of this header is the "next header" number, which is either a protocol number such as TCP or UDP, or gives an IPv6 extension header number (such as fragmentation control, or IPsec's authenticating header). Each option links on to the next with its own "next header" field.

This presents us something of a problem when it comes to packet capture filters. Recall how, for the JEQ instruction, the branch labels in both cases are unsigned integers? This is how BPF guarantees termination of a program in finite time - every jump has to be forwards. There can be no loops. Without loops, the program is guaranteed to terminate.

But now how do we parse these IPv6 headers? We can't write a while() loop in the program to walk down the headers, until we find a TCP one. Furtheremore, IPv6 doesn't define a standard header layout for all headers. Each header type puts its "next header" field in possibly a different place. Some headers are fixed-length, some carry a "header length" field of their own. It's a total mess, from the point of view of packet filtering.

What I would propose, is to create two new metadata constants:

• A new Ancillary Data area field, SKF_AD_TRANSPROTO to store the transport level protocol.


• A new data area offset, SKF_TRANS_OFF, to give a virtual view of the transport header.

This will allow us to very easily write a packet filter to capture TCP port 80, say, agnostic of IPv4 or IPV6. The filter program would look like:

LD WORD[AD_PROTOCOL]
JEQ 0x0800, 9, 0
JEQ 0x86dd, 0, 8

LD WORD[AD_TRANSPROTO]
JEQ 6, 0, 6

LD HALF[TRANS+0]
JEQ 22, 2, 0
LD HALF[TRANS+2]
JEQ 22, 0, 2

LD len
RET A

RET 0

We've now created a filter which can detect the transport protocol of TCP, and inspect the transport header, without having to directly calculate its offset from hard-coded knowledge of the network protocol.

I would like to propose that Linux adopts these two constants, and finds a way to implement them. I have some thoughts on implementation but I will defer these to a later post; as this one has gone on quite long enough already. :)