CPAN-based Experiments: A Reminder

I've been adding new features to core perl for a number of years. For the most part, all of the big things I've been adding have been near-copies of existing abilities in existing CPAN modules. As a reminder, this list contains:

perl 5.34:
try {} catch {}	from Syntax::Keyword::Try
perl 5.36:
builtins	from Scalar::Util and friends
defer {}	from Syntax::Keyword::Defer
booleans	this was actually new
perl 5.38:
class { }	from Object::Pad
||=, //= in signatures	from Sublike::Extended
perl 5.40:
field :reader, __CLASS__	from Object::Pad
perl 5.42:
my method, field :writer	from Object::Pad
lexical method invoke ->&	from Object::Pad::LexicalMethods
builtin any, all	from List::Util + List::Keywords
(upcoming in perl 5.44):
named :$params in signatures	from Sublike::Extended

I specifically bring attention to the distinguished booleans ability of perl 5.36, because it is the only major addition I believe I've added in recent years that didn't just come from a CPAN experiment. I would have done that on CPAN if I could have, but it involved such deep internal changes to the very fundamental building blocks in perl that I don't think it would have been possible. ^[1].

Everything else was.

At conferences and discussions I enjoy telling the story of the first item on this list. I had spent about 3 years experimenting with Syntax::Keyword::Try as a CPAN module. The nature of the experiment was not "can we get this working?". Getting it working was the easy bit. The hard part was the Perl-visible design of the syntax. We went through a number of iterations there before we ended up with something that looked and felt right, and seemed to gel nicely with other ideas. This was the point at which the idea got copied into core perl. That process involved just copy-pasting the unit tests, rewriting some documentation, and actually reïmplementing the code. The entire time for implementing the feature once it was solidly designed and tested took me about 5 days. That's all it took because we had that design.

When it came time to review the change to put it into core perl, it was easy to make the case that the design of it was right. It had existed for years on CPAN and been battle-tested by lots of users in lots of ways.

This overall principle has two key advantages:

• Because every idea has plenty of time (often years) of existing as a CPAN module, we have had time to feel around for any design flaws or usability issues. Remember the disaster that was perl 5.10's smartmatch and given/when? We're still to this day trying to back-pedal out of that one. I'm not going to confidently say that kind of thing wouldn't have happened if given/when was first a CPAN-based experiment, but it feels likely that if more people had tried using it in more ways, at least some of the issues would have been more apparently and the whole idea redesigned or abandoned entirely.
• Every one of these existing CPAN experiments works on existing versions of perl; often going back to quite old ones. Even the most complex of my syntax modules, Object::Pad, runs just fine on perl 5.22 which was released over 10 years ago now. This means that people don't have to wait for the next bleadperl or development point release just to try out these new ideas. They can install something from CPAN and use these new things right now in existing code, on existing systems.

These CPAN-based experiments have all worked because of the much faster iteration time on CPAN modules as compared to core perl. We get one major new release of perl every year. Sometimes I've been known to release three versions of a given CPAN module in a single week. That's at least a hundred times faster iteration speed to work on a new idea. I feel quite confident in saying that almost none of the features I listed above would have been possible by now had I been given only core perl to start experimenting in.

We also get a much gentler "adoption curve", if such is a way to phrase it. In core perl, we have "experimental" features, but once something becomes non-experimental, it's declared stable and basically must be supported by perl for all of time. CPAN modules feel like they have much more finer levels than that. Within Object::Pad for instance, there's definitely quite a range from "this is something I thought up the other day so I want to see how it plays out", to "here is a stable supported feature that should be copied into core perl". Ideas can slowly graduate along the scale at their own pace.

As a direct consequence of all of the above, we then have the Feature::Compat:: modules. Because core's try feature was directly and deliberately implemented as compatible with Syntax::Keyword::Try, there is a module called Feature::Compat::Try which simply enables the try feature on a sufficiently new perl, or pulls in the CPAN module for it on older perls. It gives authors of Perl code a smooth upgrade path towards using stable core-supported features. Once a Feature::Compat module exists for a feature you want to use, you can just use that module and not worry whether it is yet supported by the perl version you are running.

I started writing this post because I had a specific point to make about Object::Pad and roles, but I feel this has gone on quite long enough already so I will write that in a follow-up shortly.

Instead, for now I will end by reminding people not to be afraid of the word "experimental". It doesn't mean "we're not sure this thing works"; we know it works. The focus of the experiment is "do we like this?" - and by "we" here I specifically mean the entire Perl community - the core maintainers, the CPAN authors, the end-user developers. We don't want to end up in another smartmatch scenario of having a poorly-designed feature suddenly dropping into core untested but needing to be supported. This gradual experiment path through CPAN allows people to try out these new ideas and give feedback which helps guide their design to something solid and stable, that everyone can be happy about supporting long-term. But this entire mechanism only works if people use these modules and give that feedback.

It's probably no surprise to folks that I dogfood^[2] a lot of my own modules, but in some cases I am the only user of those modules that I know about. I am personally happy to continue designing core perl features just suited to my own personal use, but I suspect many other of you would have your own opinions to provide. Please all, try to remember to try things out. Talk to me about these things, use some of these CPAN experiments where and when you can. Or even if you can't directly use them for some reason, just look at them and talk to me anyway. The more feedback I get from more varied people across more varied use-cases, the more confidence I have in the correctness of these designs, and hence the more likely I am to spend the time to migrate them into the core language.

---

[1]: I would also have done 5.32's isa operator on CPAN too, but at the time we didn't have the PL_infix_plugin mechanism, which I later added in 5.38 to allow exactly this sort of thing in future. It has been very helpful in designing a number of new operators since then.

[2]: "Eating your own dog food" on Wikipedia

This post also appears on the perl5-porters mailing list.

Building for new ATtiny 2-series chips on Debian

I have previously written about how to build code for the ATtiny 1-series chips on Debian, outlining what files are missing from Debian in order to allow this. It seems, three years on, the same stuff is still missing - and moreso now that the new 2-series chips are available. Here now, is some more instructions on top of that to get code working for these newer chips as well.

As before, start off by downloading the "Atmel ATtiny Series Device Support" file from http://packs.download.atmel.com/. This is a free and open download, licensed under Apache v2. This file carries the extension atpack but it's actually just a ZIP file.

Note that by default it'll unpack into the working directory, so you'll want to create a temporary folder to work in:

$ mkdir pack

$ cd pack/

$ unzip ~/Atmel.ATtiny_DFP.2.0.368.atpack 
Archive:  /home/leo/Atmel.ATtiny_DFP.2.0.368.atpack
   creating: atdf/
   creating: avrasm/
   creating: avrasm/inc/
...

From here, you can now copy the relevant files out to where avr-gcc will find them:

$ sudo cp include/avr/iotn?*2[467].h \
    /usr/lib/avr/include/avr/
$ sudo cp gcc/dev/attiny?*2[467]/avrxmega3/*.{o,a} \
    /usr/lib/avr/lib/avrxmega3/
$ sudo cp gcc/dev/attiny?*2[467]/avrxmega3/short-calls/*.{o,a} \
    /usr/lib/avr/lib/avrxmega3/short-calls/

Unlike last time, you'll also need the device-specs files for avr-gcc itself to understand the new chips. You'll have to find the exact path on your system where the existing ones are, and then copy the new ones in there:

$ dpkg -S specs-atmega328
gcc-avr: /usr/lib/gcc/avr/5.4.0/device-specs/specs-atmega328

# So it appears to be /usr/lib/gcc/avr/5.4.0/device-specs

$ sudo cp gcc/dev/attiny?*2[467]/device-specs/* \
    /usr/lib/gcc/avr/5.4.0/device-specs/

Finally, there's one last task that needs doing. Locate the main avr/io.h file (it should live in /usr/lib/avr/include) and add the following lines somewhere within the main block of similar lines. These are needed to redirect from the toplevel #include <avr/io.h> towards the device-specific file.

#elif defined (__AVR_ATtiny424__)
#  include <avr/iotn424.h>
#elif defined (__AVR_ATtiny426__)
#  include <avr/iotn426.h>
#elif defined (__AVR_ATtiny427__)
#  include <avr/iotn427.h>
#elif defined (__AVR_ATtiny824__)
#  include <avr/iotn824.h>
#elif defined (__AVR_ATtiny826__)
#  include <avr/iotn826.h>
#elif defined (__AVR_ATtiny827__)
#  include <avr/iotn827.h>
#elif defined (__AVR_ATtiny1624__)
#  include <avr/iotn1624.h>
#elif defined (__AVR_ATtiny1626__)
#  include <avr/iotn1626.h>
#elif defined (__AVR_ATtiny1627__)
#  include <avr/iotn1627.h>
#elif defined (__AVR_ATtiny3224__)
#  include <avr/iotn3224.h>
#elif defined (__AVR_ATtiny3226__)
#  include <avr/iotn3226.h>
#elif defined (__AVR_ATtiny3227__)
#  include <avr/iotn3227.h>

Having done this we find we can now compile firmware for these new chips:

avr-gcc -std=gnu99 -Wall -Os -DF_CPU=20000000 -mmcu=attiny824 -flto -ffunction-sections -fshort-enums -o .build/firmware_t824.elf src/main.c
avr-size .build/firmware_t824.elf
   text    data     bss     dec     hex filename
   3054      24       9    3087     c0f .build/firmware_t824.elf
avr-objcopy -j .text -j .rodata -j .data -O ihex .build/firmware_t824.elf firmware_t824-flash.hex

Keep an eye on the Debian bug #930195, as hopefully one day these steps will no longer be necessary.

A troubling thought - smartmatch reïmagined

Preface: This is less a concrete idea, and more a rambling set of thoughts that lead me to a somewhat awkward place. I'm writing it out here in the hope that others can lend suggestions and ideas, and see if we can arrive at a better place.

I've been thinking about comparison operators lately - somewhat in the context of my new Syntax::Operator::Elem / Syntax::Operator::In module, somewhat in the context of smartmatch and the planned deprecations thereof, and partly in the context of my new match/case syntax.

Smartmatch Deprecations

For years now, smartmatch has been an annoying thorny design, and recently we've started making moves to get rid of it. In my mind at least, this is because it has a large and complex behaviour that is often unpredictable in advance. There are two distinct reasons for this:

1. It tries very hard to (recursively) distribute itself across container values on either side; saying that $x ~~ @y is true if any { $x ~~ $_ } @y for example; sometimes in ways that are surprising (e.g. how do you compare an array with a hash?)
2. It acts unpredictably with mixed strings or numbers; because those concepts are very fluid in perl and aren't well-defined

match/case and New Infix Operators

I've lately been writing some new ideas for new infix operators that Perl might want; partly because they're useful on their own but also because they're useful combined with the match/case syntax provided by Syntax::Keyword::Match. Between them all, these are intended as a replacement for the given/when syntax and its troublesome smartmatch. For example, to select an option based on a string comparison you can

match($x : eq) {
  case("abc") { say "It was the string abc" }
  case("def") { say "It was the string def" }
  case($y)    { say "It was whatever string the variable $y gives" }
}

This is much more predictable than given/when and smartmatch, because the programmer declared right upfront that the eq operator is being used here; there's no smartmatch involved.

Initially this feels like a great improvement on given/when and ~~, but it has lots of tricky cornercases to it. For example, the given/when approach can easily handle undef, whereas match/case using only the eq operator cannot distinguish undef from "". For this reason, I invented a new infix operator, called equ (provided by Syntax::Operator::Equ), which can:

say "Equal" if $x equ $y;  # true if they're both undef, or both
                           #   defined and the same string

match($x : equ) {
  # these two cases are now distinct
  case(undef) { say "It was undefined" }
  case("")    { say "It was the empty string" }

  default     { say "It was something else" }
}

Plus of course it also defines a new === operator which performs the numerical equivalent, able to distinguish undef from zero.

Syntax::Operator::Elem

Another operator I felt was required was one that can test if a given string (or number) is present in a list. For that, I wrote Syntax::Operator::Elem:

say "Present" if $x elem @ys;  # stringy

say "Present" if $x ∈ @ys;     # numerical

(Yes, that really is an operator spelled with a non-ASCII Unicode character. No I will not apologise :P)

These operators too have the "oops, undef" problem about them - which lead me briefly to consider adding two more that consider undef/"" or undef/zero to be distinct. Maybe I'd call them elemu and ... er.. well, Unicode doesn't have a variant of the ∈ operator that can suggest undefness. It was about at that point that I stopped, and wondered if really we're going about this whole thing the right way at all.

Smartmatch Reïmagined

I begin to think that if we go right back to the beginning, we might find that a huge chunk of this is unnecessary, if only we can find a better model.

During the 5.35 development series and now released in 5.36, Perl core has two improvements to what some might call its "type system":

• Real booleans - true and false are now first-class values distinct from 1 and zero/emptystring.
• Tracking of whether defined, nonboolean, nonreferential values began as strings or numbers; even if they have since evolved to effectively be both.

It is now possible to classify any given scalar value into exactly one of the following five categories:

undef

boolean

initially string

initially number

reference

I start to wonder whether, therefore, we have enough basis to create a better version of what the smartmatch operator tried (but ultimately failed) to be. For sake of argument, since I've already used one Unicode symbol I'm going to use another for this new one: The triple-bar identity symbol, ≡.

Lets consider a few properties this ought to have. First off, it should be well-behaved as an equality operator; it should be reflexive, symmetric and transitive. That is, given any values $x, $y and $z, all three of the following must always hold:

$x ≡ $x  is true                     # reflexive
$x ≡ $y  is the same as  $y ≡ $x     # symmetric
if $x ≡ $y and $y ≡ $z then $x ≡ $z  # transitive

Additionally, I don't think it ought to have any sort of distributive properties like $x ~~ @arr has. That sort of distribution should be handled at a higher level. (For example, the proposed caselist syntax of match/case.)

Because it only operates on pairs of scalars, this is already a much simpler kind of operator to think about. Because of the fact we can classify perl scalar values into these neat five categories, we can already write down five simple rules for when both sides are given the same category of scalar:

UNDEF	undef ≡ undef	is true
BOOL	$x ≡ $y	is true if $x and $y are both true, or both false
STR	$x ≡ $y	is true if $x eq $y
NUM	$x ≡ $y	is true if $x == $y
REF	$x ≡ $y	is true if refaddr($x) == refaddr($y)

I'd also like to suggest a rule that given any pair of scalars of different categories, the result is always false. This means in particular, that undef is never ≡ to any defined value (but never warns), that no boolean is ever ≡ to any non-boolean, and no reference is ever ≡ to any non-reference. I don't think anyone would argue with that.

Already this operator feels useful, because of the way it neatly handles undef as distinct from any number or string, we now don't need the equ or === operators.

The one problem I have with this whole model is what do we do with STR ≡ NUM; how do we handle code like the following:

my $x = "10";
say "Equivalent" if $x ≡ 10;

By my first suggestion, this would always be false. While it's predictable and simple, I don't think it's very useful. It would mean that whenever you want to e.g. perform a numerical case comparison on a value taken from @ARGV, you always have to "numify" it by doing some ugly code like:

match(0 + $ARGV[0] : ≡) {
  case(1) { ... }
}

This does not feel very perlish.

So maybe we can find a more useful handling of STR vs NUM. I can already think of several bad ideas:

• Pick the category on the righthand side
  Superficially this feels beneficial to the match/case syntax, but it soon falls down in a lot of other scenarios. Plus it is blatantly not symmetric, which we already decided any good equality test operator ought to be.
• The operator throws an exception
  This doesn't feel like the right way to go. Having things like UNDEF, BOOL and REF already neatly just yield false, means that you can safely mix strings/numbers and undef in match/case labels, for example, and all is handled nicely. To have NUM-vs-UNDEF yield false but NUM-vs-STR throw an exception feels like a bad model. Plus it would not be transitive.

About the only sensible model I can think of in this mixed case, is to say that

NUM ≡ STR  is true if both `eq` and `==` would say true

It's reflexive and symmetric. It feels useful. It does (what most people would argue is) the right thing for "10" ≡ 10.

Still, something at the back of my mind feels wrong about this design for an operator. Some situation in which is will be Obviously Terrible, and thus bring the whole tower crashing down. Perhaps it isn't truely transitive - there might be some set of values for which it fails. Offhand I can't think of one, but maybe someone can find an example?

It's a shame, because if we did happen to find an operator like this, then I think combined with match/case syntax it could go a long way towards providing a far better replacement for smartmatch + given/when and additionally solve a lot of other problems in Perl all in one go.

I'm sorry I don't have a more concrete and specific message to say there, other than that I've given (and will continue to give) this a lot of thought, and that I invite comment and ideas from others on how we might further it towards something that can really work in Perl.

Thanks all for listening.

Perl in 2022 - A Yearly Update

At the end of 2020, I wrote a series of articles on the subject of recent CPAN modules that provide useful syntax, or recent core features added to perl. The series ended with a bonus post looking forward to imagine what new additions might one day appear. I followed this up with a video-based talk at FOSDEM, titled "Perl in 2025", with yet more ideas considering how a Perl might look in a few more years' time.

Over the past twelve months, I have made progress on several of these ideas. Four of them have already become CPAN modules and thus are available for writing in Perl in 2022:

• match/case - Now available as Syntax::Keyword::Match.

  match($n : ==) {
     case(1) { say "It's one" }
     case(2) { say "It's two" }
     case(3) { say "It's three" }
  }

• any, all - Now available as syntax-level keywords from List::Keywords.

  if( any { $_->size > 100 } @boxes ) {
     say "There are some large boxes here";
  }

• multi sub - An early experiment in Syntax::Keyword::MultiSub.

  multi sub max()          { return undef; }
  multi sub max($x)        { return $x; }
  multi sub max($x, @more) { my $y = max(@more);
                             return $x > $y ? $x : $y; }

• equ, === - Available from Syntax::Operator::Equ, though at present is only usable via Syntax::Keyword::Match or a specially-patched version of perl.

  if($x equ $y) {
     say "Both are undef, or defined and equal strings";
  }
   
  if($i === $j) {
     say "Both are undef, or defined and equal numbers";
  }

Of the rest:

• in - I have the beginnings of some code but it's not yet on CPAN as it again requires a patched version of perl for pluggable infix operators.
• let and is - not started yet.

In addition, not mentioned in the original article, the latest development version of perl has gained:

• defer blocks.

  {
      say "This happens first";
      defer { say "This happens last"; }
   
      say "And this happens inbetween";
  }

• finally as part of try/catch.

  try {
      say "This happens first";
  }
  catch ($e) {
      say "Oops, it failed";
  }
  finally {
      say "This happens last in either case";
  }

• The builtin:: namespace, providing many new utility functions that ought to have been considered part of the core language - copying utilities from places like Scalar::Util and POSIX, as well as providing some new ones.

  say "The refaddr of my object is ", builtin::refaddr($obj);
  
  use builtin 'ceil';
  say "The next integer above the value is ", ceil($value);

• Real boolean values. These will be useful in many places, such as data serialisation and cross-language conversion modules.

  use builtin qw(true false isbool);
  
  sub serialise($v) {
    return $v ? 'true' : 'false' if isbool $v;
    return qq("$v");
  }
  
  say join ",", map { serialise($_) }
      0, 1, false, true, 'true';

Overall I'm happy with progress so far. A lot of things have been started, laying much of the groundwork for more work that can follow. Behind the scenes all of these syntax modules are now using the XS::Parse::Keyword module to do the bulk of their parsing. This is great for getting something powerful written quickly, and has good properties in terms of interoperability between modules - for example, the way the new infix operators already work with the match/case syntax.

Core perl is on-track for a summer release as usual; hopefully that will provide the new defer and finally syntax, builtin functions and boolean values. I hope to have as much success in 2022 as I did in 2021 at writing more of these things, and with any luck I'll be able to write another article like this next year explaining what new progress has been achieved towards the Perl in 2025 goal.

Perl UV binding hits version 2.000

Over the past few months I've been working on finishing off the libuv Perl binding module, UV. Yesterday I finally got it finished enough to feel like calling it version 2.000. Now's a good time to take a look at it.

libuv itself is a cross-platform event handling library, which focuses on providing nicely portable abstractions for things like TCP sockets, timers, and sub-process management between UNIX, Windows and other platforms. Traditionally things like event-based socket handling have always been difficult to write in a portable way between Windows and other places due to the very different ways things work on Windows as opposed to anywhere else. libuv provides a large number of helpful wrappers to write event-based code in a portable way, freeing the developer from having to care about these things.

A number of languages have nice bindings for libuv, but until recently there wasn't a good one for Perl. My latest project for The Perl Foundation aimed to fix this. The latest release of UV version 2.000 indicates that this is now done.

It's unlikely that most programs would choose to operate directly with UV itself, but rather via some higher-level event system. There are UV adapter modules for IO::Async (IO::Async::Loop::UV), Mojo (Mojo::Reactor::UV), and Future::IO (Future::IO::Impl::UV) at least.

The UV module certainly wraps much of what libuv has to offer, but there are still some parts missing. libuv can watch filesystems for changes of files, and provides asynchronous filesystem access access functions - both of these are currently missing from the Perl binding. Threadpools are an entire concept that doesn't map very well to the Perl language, so they are absent too. Finally, libuv lists an entire category of "miscellaneous functions", most of which are already available independently in Perl, so there seems little point to wrapping those provided by libuv.

Finally, we should take note of one thing that doesn't work - the UV::TCP->open and UV::UDP->open functions when running on Windows. The upshot here is that you cannot create TCP or UDP sockets in your application independently of libuv and then hand them over to be handled by the library; this is not permitted. This is because on Windows, there are fundamentally two different kinds of sockets that require two different sets of API to access them - ones using WSA_FLAG_OVERLAPPED, and ones not. libuv needs that flag in order to perform event-based IO on sockets, and so it won't work with sockets created without it - which is the usual kind that most other modules, and perl itself, will create. This means that on Windows, the only sockets you can use with the UV module are ones created by UV itself - such as by asking it to connect out to servers, or listen and accept incoming connections. Fortunately, this is sufficient for the vast majority of applications.

I would like to finish up by saying thanks to The Perl Foundation for funding me to complete this project.

Writing a Perl Core Feature - part 11: Core modules

Index | < Prev

Our new feature is now implemented, tested, and documented. There's just one last thing we need to do - update the bundled modules that come with core. Specifically, because we've added some new syntax, we need to update B::Deparse to be able to deparse it.

When the isa operator was added, the deparse module needed to be informed about the new OP_ISA opcode, in this small addition: (github.com/Perl/perl5).

--- a/lib/B/Deparse.pm
+++ b/lib/B/Deparse.pm
@@ -52,7 +52,7 @@ use B qw(class main_root main_start main_cv svref_2object opnumber perlstring
         MDEREF_SHIFT
     );
 
-$VERSION = '1.51';
+$VERSION = '1.52';
 use strict;
 our $AUTOLOAD;
 use warnings ();
@@ -3060,6 +3060,8 @@ sub pp_sge { binop(@_, "ge", 15) }
 sub pp_sle { binop(@_, "le", 15) }
 sub pp_scmp { maybe_targmy(@_, \&binop, "cmp", 14) }
 
+sub pp_isa { binop(@_, "isa", 15) }
+
 sub pp_sassign { binop(@_, "=", 7, SWAP_CHILDREN) }
 sub pp_aassign { binop(@_, "=", 7, SWAP_CHILDREN | LIST_CONTEXT) }

As you can see it's quite a small addition here; we just need to add a new method to the main B::Deparse package named after the new opcode. This new method calls down to the common binop function which is shared by the various binary operators, and recurses down parts of the optree, returning a combined result using the "isa" string in between the two parts.

A more complex addition was made with the try syntax, as can be seen at (github.com/Perl/perl5); abbreviated here:

+sub pp_leavetrycatch {
+    my $self = shift;
+    my ($op) = @_;
...
+    my $trycode = scopeop(0, $self, $tryblock);
+    my $catchvar = $self->padname($catch->targ);
+    my $catchcode = scopeop(0, $self, $catchblock);
+
+    return "try {\n\t$trycode\n\b}\n" .
+           "catch($catchvar) {\n\t$catchcode\n\b}\cK";
+}

As before, this adds a new method named after the new opcode (in the case of the try/catch syntax this is named OP_LEAVETRYCATCH). The body of this method too just recurses down to parts of the sub-tree it was passed; in this case being two scope ops for the bodies of the blocks, plus a lexical variable name for the catch variable. The method then again returns a new string combining the various parts together along with the required braces, linefeeds, and indentation hints.

We can tell we need to add this for our new banana feature, as currently this does not deparse properly:

leo@shy:~/src/bleadperl/perl [git]
$ ./perl -Ilib -Mexperimental=banana -MO=Deparse -ce 'print ban "Hello, world" ana;'
unexpected OP_BANANA at lib/B/Deparse.pm line 1664.
BEGIN {${^WARNING_BITS} = "\x10\x01\x00\x00\x00\x50\x04\x00\x00\x00\x00\x00\x00\x55\x51\x55\x50\x51\x45\x00"}
use feature 'banana';
print XXX;
-e syntax OK

We'll fix this by adding a new pp_banana in an appropriate place, perhaps just after the ones for lc/uc/fc. Don't forget to bump the $VERSION number too:

leo@shy:~/src/bleadperl/perl [git]
$ nvim lib/B/Deparse.pm 

leo@shy:~/src/bleadperl/perl [git]
$ git diff 
diff --git a/lib/B/Deparse.pm b/lib/B/Deparse.pm
index 67147f12dd..f6039a435d 100644
--- a/lib/B/Deparse.pm
+++ b/lib/B/Deparse.pm
@@ -52,7 +52,7 @@ use B qw(class main_root main_start main_cv svref_2object opnumber perlstring
         MDEREF_SHIFT
     );
 
-$VERSION = '1.56';
+$VERSION = '1.57';
 use strict;
 our $AUTOLOAD;
 use warnings ();
@@ -2824,6 +2824,13 @@ sub pp_lc { dq_unop(@_, "lc") }
 sub pp_quotemeta { maybe_targmy(@_, \&dq_unop, "quotemeta") }
 sub pp_fc { dq_unop(@_, "fc") }
 
+sub pp_banana {
+    my $self = shift;
+    my ($op, $cx) = @_;
+    my $kid = $op->first;
+    return "ban " . $self->deparse($kid, 1) . " ana";
+}
+
 sub loopex {
     my $self = shift;
     my ($op, $cx, $name) = @_;

This new function recurses down to deparse for the subtree, and returns a new string wrapped in the appropriate syntax for it. That should be all we need:

leo@shy:~/src/bleadperl/perl [git]
$ ./perl -Ilib -Mexperimental=banana -MO=Deparse -ce 'print ban "Hello, world" ana;'
BEGIN {${^WARNING_BITS} = "\x10\x01\x00\x00\x00\x50\x04\x00\x00\x00\x00\x00\x00\x55\x51\x55\x50\x51\x45\x00"}
use feature 'banana';
print ban 'Hello, world' ana;
-e syntax OK

Of course, this being a perl module we should remember to update its unit tests.

leo@shy:~/src/bleadperl/perl [git]
$ git diff lib/B/Deparse.t
diff --git a/lib/B/Deparse.t b/lib/B/Deparse.t
index 24eb445041..0fe6940cb3 100644
--- a/lib/B/Deparse.t
+++ b/lib/B/Deparse.t
@@ -3171,3 +3171,10 @@ try {
 catch($var) {
     SECOND();
 }
+####
+# banana
+# CONTEXT use feature 'banana'; no warnings 'experimental::banana';
+ban 'literal' ana;
+ban $a ana;
+ban $a . $b ana;
+ban "stringify $a" ana;

leo@shy:~/src/bleadperl/perl [git]
$ ./perl t/harness lib/B/Deparse.t 
../lib/B/Deparse.t .. ok     
All tests successful.
Files=1, Tests=321,  9 wallclock secs ( 0.14 usr  0.00 sys +  8.99 cusr  0.38 csys =  9.51 CPU)
Result: PASS

Because in part 10 we added documentation for a new function in pod/perlfunc.pod there's another test that needs updating:

leo@shy:~/src/bleadperl/perl [git]
$ ./perl t/harness ext/Pod-Functions/t/Functions.t 
../ext/Pod-Functions/t/Functions.t .. 1/? 
#   Failed test 'run as plain program'
#   at t/Functions.t line 55.
#          got: '
...
Result: FAIL

We can fix that by adding the new function to the expected list in the test file itself:

leo@shy:~/src/bleadperl/perl [git]
$ nvim ext/Pod-Functions/t/Functions.t

leo@shy:~/src/bleadperl/perl [git]
$ git diff ext/Pod-Functions/t/Functions.t
diff --git a/ext/Pod-Functions/t/Functions.t b/ext/Pod-Functions/t/Functions.t
index 2beccc1ac6..4d5b03e978 100644
--- a/ext/Pod-Functions/t/Functions.t
+++ b/ext/Pod-Functions/t/Functions.t
@@ -76,7 +76,7 @@ Functions.t - Test Pod::Functions
 __DATA__
 
 Functions for SCALARs or strings:
-     chomp, chop, chr, crypt, fc, hex, index, lc, lcfirst,
+     ban, chomp, chop, chr, crypt, fc, hex, index, lc, lcfirst,
      length, oct, ord, pack, q/STRING/, qq/STRING/, reverse,
      rindex, sprintf, substr, tr///, uc, ucfirst, y///
 
leo@shy:~/src/bleadperl/perl [git]
$ ./perl t/harness ext/Pod-Functions/t/Functions.t 
../ext/Pod-Functions/t/Functions.t .. ok     
All tests successful.
Files=1, Tests=234,  1 wallclock secs ( 0.04 usr  0.01 sys +  0.23 cusr  0.00 csys =  0.28 CPU)
Result: PASS

---

At this point, we're done. We've now completed all the steps to add a new feature to the perl interpreter. As well as all the steps required to actually implement it in the core binary itself, we've updated the tests, documentation, and support modules to match.

Along the way we've seen examples from real commits into the perl tree while we made our own. Any particular design of new feature will of course have its own variations and differences - there's still many parts of the interpreter we haven't touched on in this series. It would be difficult to try to cover all the possible ideas of things that could be added or changed, but hopefully having completed this series you'll at least have a good overview of the main pieces that are likely to be involved, and have some starting-off points to explore further to see whatever additional details might be required for whatever situation you encounter.

Index | < Prev

Writing a Perl Core Feature - part 10: Documentation

Index | < Prev | Next >

Now that have our new feature nicely implemented and tested, we're nearly finished. We just have a few more loose ends to tidy up. The first of these is to take a look at some documentation.

We've already done one small documentation addition to perldiag.pod when we added the new warning message, but the bulk of documentation to explain a new feature would likely be found in one of the main documents - perlsyn.pod, perlop.pod, perlfunc.pod or similar. Exactly which of these is best would depend on the nature of the specific feature.

The isa feature, being a new infix operator, was documented in perlop.pod: (github.com/Perl/perl5).

...
+=head2 Class Instance Operator
+X<isa operator>
+
+Binary C<isa> evaluates to true when left argument is an object instance of
+the class (or a subclass derived from that class) given by the right argument.
+If the left argument is not defined, not a blessed object instance, or does
+not derive from the class given by the right argument, the operator evaluates
+as false. The right argument may give the class either as a barename or a
+scalar expression that yields a string class name:
+
+    if( $obj isa Some::Class ) { ... }
+
+    if( $obj isa "Different::Class" ) { ... }
+    if( $obj isa $name_of_class ) { ... }
+
+This is an experimental feature and is available from Perl 5.31.6 when enabled
+by C<use feature 'isa'>. It emits a warning in the C<experimental::isa>
+category.

Lets now write a little bit of documentation for our new banana feature. Since it is a named function-like operator (though with odd syntax involving a second trailing named keyword), perhaps we'll write it in perlfunc.pod. We'll style it similarly to the case-changing functions lc and uc to get some suggested wording.

leo@shy:~/src/bleadperl/perl [git]
$ nvim pod/perlfunc.pod 

leo@shy (1 job):~/src/bleadperl/perl [git]
$ git diff | xml_escape 
diff --git a/pod/perlfunc.pod b/pod/perlfunc.pod
index b655a08ecc..319e9aab96 100644
--- a/pod/perlfunc.pod
+++ b/pod/perlfunc.pod
@@ -114,6 +114,7 @@ X<scalar> X<string> X<character>
 
 =for Pod::Functions =String
 
+L<C<ban>|/ban EXPR ana>,
 L<C<chomp>|/chomp VARIABLE>, L<C<chop>|/chop VARIABLE>,
 L<C<chr>|/chr NUMBER>, L<C<crypt>|/crypt PLAINTEXT,SALT>,
 L<C<fc>|/fc EXPR>, L<C<hex>|/hex EXPR>,
@@ -136,6 +137,10 @@ prefixed with C<CORE::>.  The
 L<C<"fc"> feature|feature/The 'fc' feature> is enabled automatically
 with a C<use v5.16> (or higher) declaration in the current scope.
 
+L<C<ban>|/ban EXPR ana> is available only if the
+L<C<"banana"> feature|feature/The 'banana' feature.> is enabled or if it is
+prefixed with C<CORE::>.
+
 =item Regular expressions and pattern matching
 X<regular expression> X<regex> X<regexp>
 
@@ -773,6 +778,15 @@ your L<atan2(3)> manpage for more information.
 
 Portability issues: L<perlport/atan2>.
 
+=item ban EXPR ana
+X<ban>
+
+=for Pod::Functions return ROT13 transformed version of a string
+
+Applies the "ROT13" transform to upper- and lower-case letters in the given
+expression string, returning the newly-formed string. Non-letter characters
+are left unchanged.
+
 =item bind SOCKET,NAME
 X<bind>

While this will do as a short example here, any real feature would likely have a lot more words to say than just this.

When editing POD files it's good to get into the habit of running the porting tests (or at least the POD checking ones) before committing, to check the formatting is valid:

leo@shy:~/src/bleadperl/perl [git]
$ ./perl t/harness t/porting/pod*.t
porting/podcheck.t ... ok         
porting/pod_rules.t .. ok   
All tests successful.
Files=2, Tests=1472, 34 wallclock secs ( 0.20 usr  0.00 sys + 33.79 cusr  0.15 csys = 34.14 CPU)
Result: PASS

While I was writing this documentation it occurred to me to write about how the function handles Unicode characters vs byte strings, so I was thinking more about how it actually does. It turns out the implementation doesn't work properly for that, as we can demonstrate with a new test:

--- a/t/op/banana.t
+++ b/t/op/banana.t
@@ -11,7 +11,7 @@ use strict;
 use feature 'banana';
 no warnings 'experimental::banana';
 
-plan 7;
+plan 8;
 
 is(ban "ABCD" ana, "NOPQ", 'Uppercase ROT13');
 is(ban "abcd" ana, "nopq", 'Lowercase ROT13');
@@ -23,3 +23,8 @@ my $str = "efgh";
 is(ban $str ana, "rstu", 'Lexical variable');
 is(ban $str . "IJK" ana, "rstuVWX", 'Concat expression');
 is("(" . ban "LMNO" ana . ")", "(YZAB)", 'Outer concat');
+
+{
+    use utf8;
+    is(ban "café" ana, "pnsé", 'Unicode string');
+}

leo@shy:~/src/bleadperl/perl [git]
$ ./perl t/harness t/op/banana.t 
op/banana.t .. 1/8 # Failed test 8 - Unicode string at op/banana.t line 29
#      got "pnsé"
# expected "pns�"
op/banana.t .. Failed 1/8 subtests 

This comes down to a bug in the pp_banana opcode function, which used the internal byte buffer of the incoming SV (SvPV) without inspecting the corresponding SvUTF8 flag. Such a pattern is always indicative of a Unicode support bug. We can easily fix this:

leo@shy:~/src/bleadperl/perl [git]
$ git diff pp.c
diff --git a/pp.c b/pp.c
index 9725806b84..3dbe21fadd 100644
--- a/pp.c
+++ b/pp.c
@@ -7211,6 +7211,8 @@ PP(pp_banana)
     s = SvPV(arg, len);
 
     mPUSHs(newSVpvn_rot13(s, len));
+    if(SvUTF8(arg))
+        SvUTF8_on(TOPs);
     RETURN;
 }
 

leo@shy:~/src/bleadperl/perl [git]
$ ./perl t/harness t/op/banana.t 
op/banana.t .. ok   
All tests successful.
Files=1, Tests=8,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.04 CPU)
Result: PASS

Writing good documentation is an integral part of the process of developing a new feature. Firstly it helps to explain the feature to users so they know how to use it. But often you find that the process of writing the words helps you think about different aspects of that feature that you may not have considered before. With that new frame of mind you sometimes discover missing parts to it, or uncover bugs or cornercases that need fixing. Make sure to spend time working on the documentation for any new feature - it is said that you never truely understand something until you try teach it to someone else.

Index | < Prev | Next >