I've been adding new features to core perl for a number of years. For the most part, all of the big things I've been adding have been near-copies of existing abilities in existing CPAN modules. As a reminder, this list contains:
| perl 5.34: | |
| try {} catch {} | from Syntax::Keyword::Try |
| perl 5.36: | |
| builtins | from Scalar::Util and friends |
| defer {} | from Syntax::Keyword::Defer |
| booleans | this was actually new |
| perl 5.38: | |
| class { } | from Object::Pad |
| ||=, //= in signatures | from Sublike::Extended |
| perl 5.40: | |
| field :reader, __CLASS__ | from Object::Pad |
| perl 5.42: | |
| my method, field :writer | from Object::Pad |
| lexical method invoke ->& | from Object::Pad::LexicalMethods |
| builtin any, all | from List::Util + List::Keywords |
| (upcoming in perl 5.44): | |
| named :$params in signatures | from Sublike::Extended |
I specifically bring attention to the distinguished booleans ability of perl 5.36, because it is the only major addition I believe I've added in recent years that didn't just come from a CPAN experiment. I would have done that on CPAN if I could have, but it involved such deep internal changes to the very fundamental building blocks in perl that I don't think it would have been possible. [1].
Everything else was.
At conferences and discussions I enjoy telling the story of the first item on this list. I had spent about 3 years experimenting with Syntax::Keyword::Try as a CPAN module. The nature of the experiment was not "can we get this working?". Getting it working was the easy bit. The hard part was the Perl-visible design of the syntax. We went through a number of iterations there before we ended up with something that looked and felt right, and seemed to gel nicely with other ideas. This was the point at which the idea got copied into core perl. That process involved just copy-pasting the unit tests, rewriting some documentation, and actually reïmplementing the code. The entire time for implementing the feature once it was solidly designed and tested took me about 5 days. That's all it took because we had that design.
When it came time to review the change to put it into core perl, it was easy to make the case that the design of it was right. It had existed for years on CPAN and been battle-tested by lots of users in lots of ways.
This overall principle has two key advantages:
- Because every idea has plenty of time (often years) of existing as a CPAN module, we have had time to feel around for any design flaws or usability issues. Remember the disaster that was perl 5.10's smartmatch and given/when? We're still to this day trying to back-pedal out of that one. I'm not going to confidently say that kind of thing wouldn't have happened if given/when was first a CPAN-based experiment, but it feels likely that if more people had tried using it in more ways, at least some of the issues would have been more apparently and the whole idea redesigned or abandoned entirely.
- Every one of these existing CPAN experiments works on existing versions of perl; often going back to quite old ones. Even the most complex of my syntax modules, Object::Pad, runs just fine on perl 5.22 which was released over 10 years ago now. This means that people don't have to wait for the next bleadperl or development point release just to try out these new ideas. They can install something from CPAN and use these new things right now in existing code, on existing systems.
These CPAN-based experiments have all worked because of the much faster iteration time on CPAN modules as compared to core perl. We get one major new release of perl every year. Sometimes I've been known to release three versions of a given CPAN module in a single week. That's at least a hundred times faster iteration speed to work on a new idea. I feel quite confident in saying that almost none of the features I listed above would have been possible by now had I been given only core perl to start experimenting in.
We also get a much gentler "adoption curve", if such is a way to phrase it. In core perl, we have "experimental" features, but once something becomes non-experimental, it's declared stable and basically must be supported by perl for all of time. CPAN modules feel like they have much more finer levels than that. Within Object::Pad for instance, there's definitely quite a range from "this is something I thought up the other day so I want to see how it plays out", to "here is a stable supported feature that should be copied into core perl". Ideas can slowly graduate along the scale at their own pace.
As a direct consequence of all of the above, we then have the Feature::Compat:: modules. Because core's try feature was directly and deliberately implemented as compatible with Syntax::Keyword::Try, there is a module called Feature::Compat::Try which simply enables the try feature on a sufficiently new perl, or pulls in the CPAN module for it on older perls. It gives authors of Perl code a smooth upgrade path towards using stable core-supported features. Once a Feature::Compat module exists for a feature you want to use, you can just use that module and not worry whether it is yet supported by the perl version you are running.
I started writing this post because I had a specific point to make about Object::Pad and roles, but I feel this has gone on quite long enough already so I will write that in a follow-up shortly.
Instead, for now I will end by reminding people not to be afraid of the word "experimental". It doesn't mean "we're not sure this thing works"; we know it works. The focus of the experiment is "do we like this?" - and by "we" here I specifically mean the entire Perl community - the core maintainers, the CPAN authors, the end-user developers. We don't want to end up in another smartmatch scenario of having a poorly-designed feature suddenly dropping into core untested but needing to be supported. This gradual experiment path through CPAN allows people to try out these new ideas and give feedback which helps guide their design to something solid and stable, that everyone can be happy about supporting long-term. But this entire mechanism only works if people use these modules and give that feedback.
It's probably no surprise to folks that I dogfood[2] a lot of my own modules, but in some cases I am the only user of those modules that I know about. I am personally happy to continue designing core perl features just suited to my own personal use, but I suspect many other of you would have your own opinions to provide. Please all, try to remember to try things out. Talk to me about these things, use some of these CPAN experiments where and when you can. Or even if you can't directly use them for some reason, just look at them and talk to me anyway. The more feedback I get from more varied people across more varied use-cases, the more confidence I have in the correctness of these designs, and hence the more likely I am to spend the time to migrate them into the core language.
[1]: I would also have done 5.32's isa operator on CPAN too, but at the time we didn't have the PL_infix_plugin mechanism, which I later added in 5.38 to allow exactly this sort of thing in future. It has been very helpful in designing a number of new operators since then.
[2]: "Eating your own dog food" on Wikipedia
This post also appears on the perl5-porters mailing list.
