A troubling thought - smartmatch reïmagined

Preface: This is less a concrete idea, and more a rambling set of thoughts that lead me to a somewhat awkward place. I'm writing it out here in the hope that others can lend suggestions and ideas, and see if we can arrive at a better place.

I've been thinking about comparison operators lately - somewhat in the context of my new Syntax::Operator::Elem / Syntax::Operator::In module, somewhat in the context of smartmatch and the planned deprecations thereof, and partly in the context of my new match/case syntax.

Smartmatch Deprecations

For years now, smartmatch has been an annoying thorny design, and recently we've started making moves to get rid of it. In my mind at least, this is because it has a large and complex behaviour that is often unpredictable in advance. There are two distinct reasons for this:

1. It tries very hard to (recursively) distribute itself across container values on either side; saying that $x ~~ @y is true if any { $x ~~ $_ } @y for example; sometimes in ways that are surprising (e.g. how do you compare an array with a hash?)
2. It acts unpredictably with mixed strings or numbers; because those concepts are very fluid in perl and aren't well-defined

match/case and New Infix Operators

I've lately been writing some new ideas for new infix operators that Perl might want; partly because they're useful on their own but also because they're useful combined with the match/case syntax provided by Syntax::Keyword::Match. Between them all, these are intended as a replacement for the given/when syntax and its troublesome smartmatch. For example, to select an option based on a string comparison you can

match($x : eq) {
  case("abc") { say "It was the string abc" }
  case("def") { say "It was the string def" }
  case($y)    { say "It was whatever string the variable $y gives" }
}

This is much more predictable than given/when and smartmatch, because the programmer declared right upfront that the eq operator is being used here; there's no smartmatch involved.

Initially this feels like a great improvement on given/when and ~~, but it has lots of tricky cornercases to it. For example, the given/when approach can easily handle undef, whereas match/case using only the eq operator cannot distinguish undef from "". For this reason, I invented a new infix operator, called equ (provided by Syntax::Operator::Equ), which can:

say "Equal" if $x equ $y;  # true if they're both undef, or both
                           #   defined and the same string

match($x : equ) {
  # these two cases are now distinct
  case(undef) { say "It was undefined" }
  case("")    { say "It was the empty string" }

  default     { say "It was something else" }
}

Plus of course it also defines a new === operator which performs the numerical equivalent, able to distinguish undef from zero.

Syntax::Operator::Elem

Another operator I felt was required was one that can test if a given string (or number) is present in a list. For that, I wrote Syntax::Operator::Elem:

say "Present" if $x elem @ys;  # stringy

say "Present" if $x ∈ @ys;     # numerical

(Yes, that really is an operator spelled with a non-ASCII Unicode character. No I will not apologise :P)

These operators too have the "oops, undef" problem about them - which lead me briefly to consider adding two more that consider undef/"" or undef/zero to be distinct. Maybe I'd call them elemu and ... er.. well, Unicode doesn't have a variant of the ∈ operator that can suggest undefness. It was about at that point that I stopped, and wondered if really we're going about this whole thing the right way at all.

Smartmatch Reïmagined

I begin to think that if we go right back to the beginning, we might find that a huge chunk of this is unnecessary, if only we can find a better model.

During the 5.35 development series and now released in 5.36, Perl core has two improvements to what some might call its "type system":

• Real booleans - true and false are now first-class values distinct from 1 and zero/emptystring.
• Tracking of whether defined, nonboolean, nonreferential values began as strings or numbers; even if they have since evolved to effectively be both.

It is now possible to classify any given scalar value into exactly one of the following five categories:

undef

boolean

initially string

initially number

reference

I start to wonder whether, therefore, we have enough basis to create a better version of what the smartmatch operator tried (but ultimately failed) to be. For sake of argument, since I've already used one Unicode symbol I'm going to use another for this new one: The triple-bar identity symbol, ≡.

Lets consider a few properties this ought to have. First off, it should be well-behaved as an equality operator; it should be reflexive, symmetric and transitive. That is, given any values $x, $y and $z, all three of the following must always hold:

$x ≡ $x  is true                     # reflexive
$x ≡ $y  is the same as  $y ≡ $x     # symmetric
if $x ≡ $y and $y ≡ $z then $x ≡ $z  # transitive

Additionally, I don't think it ought to have any sort of distributive properties like $x ~~ @arr has. That sort of distribution should be handled at a higher level. (For example, the proposed caselist syntax of match/case.)

Because it only operates on pairs of scalars, this is already a much simpler kind of operator to think about. Because of the fact we can classify perl scalar values into these neat five categories, we can already write down five simple rules for when both sides are given the same category of scalar:

UNDEF	undef ≡ undef	is true
BOOL	$x ≡ $y	is true if $x and $y are both true, or both false
STR	$x ≡ $y	is true if $x eq $y
NUM	$x ≡ $y	is true if $x == $y
REF	$x ≡ $y	is true if refaddr($x) == refaddr($y)

I'd also like to suggest a rule that given any pair of scalars of different categories, the result is always false. This means in particular, that undef is never ≡ to any defined value (but never warns), that no boolean is ever ≡ to any non-boolean, and no reference is ever ≡ to any non-reference. I don't think anyone would argue with that.

Already this operator feels useful, because of the way it neatly handles undef as distinct from any number or string, we now don't need the equ or === operators.

The one problem I have with this whole model is what do we do with STR ≡ NUM; how do we handle code like the following:

my $x = "10";
say "Equivalent" if $x ≡ 10;

By my first suggestion, this would always be false. While it's predictable and simple, I don't think it's very useful. It would mean that whenever you want to e.g. perform a numerical case comparison on a value taken from @ARGV, you always have to "numify" it by doing some ugly code like:

match(0 + $ARGV[0] : ≡) {
  case(1) { ... }
}

This does not feel very perlish.

So maybe we can find a more useful handling of STR vs NUM. I can already think of several bad ideas:

• Pick the category on the righthand side
  Superficially this feels beneficial to the match/case syntax, but it soon falls down in a lot of other scenarios. Plus it is blatantly not symmetric, which we already decided any good equality test operator ought to be.
• The operator throws an exception
  This doesn't feel like the right way to go. Having things like UNDEF, BOOL and REF already neatly just yield false, means that you can safely mix strings/numbers and undef in match/case labels, for example, and all is handled nicely. To have NUM-vs-UNDEF yield false but NUM-vs-STR throw an exception feels like a bad model. Plus it would not be transitive.

About the only sensible model I can think of in this mixed case, is to say that

NUM ≡ STR  is true if both `eq` and `==` would say true

It's reflexive and symmetric. It feels useful. It does (what most people would argue is) the right thing for "10" ≡ 10.

Still, something at the back of my mind feels wrong about this design for an operator. Some situation in which is will be Obviously Terrible, and thus bring the whole tower crashing down. Perhaps it isn't truely transitive - there might be some set of values for which it fails. Offhand I can't think of one, but maybe someone can find an example?

It's a shame, because if we did happen to find an operator like this, then I think combined with match/case syntax it could go a long way towards providing a far better replacement for smartmatch + given/when and additionally solve a lot of other problems in Perl all in one go.

I'm sorry I don't have a more concrete and specific message to say there, other than that I've given (and will continue to give) this a lot of thought, and that I invite comment and ideas from others on how we might further it towards something that can really work in Perl.

Thanks all for listening.

Perl in 2022 - A Yearly Update

At the end of 2020, I wrote a series of articles on the subject of recent CPAN modules that provide useful syntax, or recent core features added to perl. The series ended with a bonus post looking forward to imagine what new additions might one day appear. I followed this up with a video-based talk at FOSDEM, titled "Perl in 2025", with yet more ideas considering how a Perl might look in a few more years' time.

Over the past twelve months, I have made progress on several of these ideas. Four of them have already become CPAN modules and thus are available for writing in Perl in 2022:

• match/case - Now available as Syntax::Keyword::Match.

  match($n : ==) {
     case(1) { say "It's one" }
     case(2) { say "It's two" }
     case(3) { say "It's three" }
  }

• any, all - Now available as syntax-level keywords from List::Keywords.

  if( any { $_->size > 100 } @boxes ) {
     say "There are some large boxes here";
  }

• multi sub - An early experiment in Syntax::Keyword::MultiSub.

  multi sub max()          { return undef; }
  multi sub max($x)        { return $x; }
  multi sub max($x, @more) { my $y = max(@more);
                             return $x > $y ? $x : $y; }

• equ, === - Available from Syntax::Operator::Equ, though at present is only usable via Syntax::Keyword::Match or a specially-patched version of perl.

  if($x equ $y) {
     say "Both are undef, or defined and equal strings";
  }
   
  if($i === $j) {
     say "Both are undef, or defined and equal numbers";
  }

Of the rest:

• in - I have the beginnings of some code but it's not yet on CPAN as it again requires a patched version of perl for pluggable infix operators.
• let and is - not started yet.

In addition, not mentioned in the original article, the latest development version of perl has gained:

• defer blocks.

  {
      say "This happens first";
      defer { say "This happens last"; }
   
      say "And this happens inbetween";
  }

• finally as part of try/catch.

  try {
      say "This happens first";
  }
  catch ($e) {
      say "Oops, it failed";
  }
  finally {
      say "This happens last in either case";
  }

• The builtin:: namespace, providing many new utility functions that ought to have been considered part of the core language - copying utilities from places like Scalar::Util and POSIX, as well as providing some new ones.

  say "The refaddr of my object is ", builtin::refaddr($obj);
  
  use builtin 'ceil';
  say "The next integer above the value is ", ceil($value);

• Real boolean values. These will be useful in many places, such as data serialisation and cross-language conversion modules.

  use builtin qw(true false isbool);
  
  sub serialise($v) {
    return $v ? 'true' : 'false' if isbool $v;
    return qq("$v");
  }
  
  say join ",", map { serialise($_) }
      0, 1, false, true, 'true';

Overall I'm happy with progress so far. A lot of things have been started, laying much of the groundwork for more work that can follow. Behind the scenes all of these syntax modules are now using the XS::Parse::Keyword module to do the bulk of their parsing. This is great for getting something powerful written quickly, and has good properties in terms of interoperability between modules - for example, the way the new infix operators already work with the match/case syntax.

Core perl is on-track for a summer release as usual; hopefully that will provide the new defer and finally syntax, builtin functions and boolean values. I hope to have as much success in 2022 as I did in 2021 at writing more of these things, and with any luck I'll be able to write another article like this next year explaining what new progress has been achieved towards the Perl in 2025 goal.