Writing a Perl Core Feature - part 8: Interpreter internals

Index | < Prev | Next >

At this point we are most of the way to adding a new feature to the Perl interpreter. In part 4 we created an opcode function to represent the new behaviour, part 5 and part 6 added compiler support to recognise the syntax used to represent it, and in part 7 we made a helper function to provide the required behaviour. It's now time to tie them all together.

When we looked at opcodes and optrees back in part 4, I mentioned that each node of the optree performs a little part of the execution of a function, with child nodes usually obtaining some piece of data somewhere that gets passed up to parent nodes to operate on. I skipped over exactly how that all works, so for this part lets look at that in more detail.

The data model used by the perl interpreter for runtime execution of code is based around being a stack machine. Most opcodes that operate in some way on regular perl data values do so by interacting with the data stack (often simply called "the stack"; though this is sometimes ambiguous as there are in fact several stacks within the perl interpreter). As the interpreter walks along an optree invoking the function associated with each opcode, these various functions either push values onto the stack, or pop values already there back off it again, in order to use them.

For example, in part 4 we saw how the line of code my $x = 5; might get represented by an optree of three nodes - an OP_SASSIGN with two child nodes OP_CONST and OP_PADSV.

When this statement is executed the optree nodes are visited in postfix order, with the two child BASEOPs running first in order to push some values to the stack, followed by the assignment BINOP afterwards, which takes those values back off the stack and performs the appropriate assignment.

Lets now take a closer look at the code inside one of the actual functions which implements this. For example, pp_const, the function for OP_CONST consists of three short lines:

PP(pp_const)
{
    dSP;
    XPUSHs(cSVOP_sv);
    RETURN;
}

Of these three lines, all four symbols are in fact macros:

1. dSP declares some local variables for tracking state, used by later macros
2. cSVOP_sv fetches the actual SV pointer out of the SVOP itself. This will be the one holding the constant's value
3. XPUSHs extends the (data) stack if necessary, then pushes it there
4. RETURN resynchronises the interpreter state from the local variables, and arranges for the opcode function to return the next opcode, for the toplevel instruction loop

The pp_padsv function is somewhat more complex, but the essential parts of it are quite similar; the following example is heavily paraphrased:

PP(pp_padsv)
{
    SV ** const padentry = &(PAD_SVl(op->op_targ));
    XPUSHs(*padentry);
    RETURN;
}

This time, rather than the cSVOP_sv which takes the SV out of the op itself, we use PAD_SVl which looks up the SV in the currently-active pad, by using the target index which is stored in the op.

When the isa feature was added, its main pp_isa opcode function was actually quite small: (github.com/Perl/perl5).

--- a/pp.c
+++ b/pp.c
@@ -7143,6 +7143,18 @@ PP(pp_argcheck)
     return NORMAL;
 }
 
+PP(pp_isa)
+{
+    dSP;
+    SV *left, *right;
+
+    right = POPs;
+    left  = TOPs;
+
+    SETs(boolSV(sv_isa_sv(left, right)));
+    RETURN;
+}
+

Since OP_ISA is a BINOP it is expecting to find two arguments on the stack; traditionally these are called left and right. This opcode function simply takes those two values and calls the sv_isa_sv() function, which returns a boolean truth value. The boolSV helper function returns an SV pointer to represent this boolean value, which is then used as the result of the opcode itself.

As a small performance optimsation, this function decides to only POP one argument, before changing the top-of-stack value to its result using SETs. This is equivalent to POPing two of them and PUSHing its result, except that it doesn't have to alter the stack pointer as many times.

For more of a look at how the stack works, you could also take a look at another post from my series on Parser Plugins: Part 3a - The Stack.

Lets now take a look at implementing our banana feature for real. Recall in part 4 we added the pp_banana function with some placeholder content that just died with a panic message if invoked. We'll now replace that with a real implementation:

leo@shy:~/src/bleadperl/perl [git]
$ nvim pp.c 

leo@shy:~/src/bleadperl/perl [git]
$ git diff pp.c
diff --git a/pp.c b/pp.c
index 93141454e1..bced3d23ea 100644
--- a/pp.c
+++ b/pp.c
@@ -7203,7 +7203,15 @@ PP(pp_cmpchain_dup)
 
 PP(pp_banana)
 {
-    DIE(aTHX_ "panic: we have no bananas");
+    dSP;
+    const char *s;
+    STRLEN len;
+    SV *arg = POPs;
+
+    s = SvPV(arg, len);
+
+    PUSHs(newSVpvn_rot13(s, len));
+    RETURN;
 }
 
 /*

Now lets rebuild perl and try it out:

leo@shy:~/src/bleadperl/perl [git]
$ make -j4 perl
...

leo@shy:~/src/bleadperl/perl [git]
$ ./perl -Ilib -E 'use experimental "banana"; say ban "Hello, world!" ana;'
Uryyb, jbeyq!

Well it certainly looks plausible - we've got back a different string of the same length, with different letters but in the same capitalisation and identical non-letter characters. Lets compare with something like tr to see if it's correct:

leo@shy:~/src/bleadperl/perl [git]
$ echo "Uryyb, jbeyq!" | tr "A-Za-z" "N-ZA-Mn-za-m"
Hello, world!

Seems good. But it turns out we've still missed something. This function has a memory leak. We can demonstrate it by writing a small example that calls ban ... ana a large number of times (say, a thousand), and printing the total count of SVs on the heap before and after. There's a handy function in perl's unit test suited called XS::APItest::sv_count we can use here:

leo@shy (1 job):~/src/bleadperl/perl [git]
$ ./perl -Ilib -I. -MXS::APItest=sv_count -E \
  'use experimental "banana";
   say sv_count();
   ban "Hello, world!" ana for 1..1000;
   say sv_count();'
5321
6321

Oh dear. The SV count is a thousand higher afterwards than before, suggesting we leaked an SV on every call.

It turns out this is because of an optimisation that the interpreter uses, where SV pointers on Perl data stack don't actually contribute to reference counting. When values get POP'ed from the stack we don't have to decrement their refcount; when values get pushed we don't increment it. This saves an amount of runtime performance to not have to be adjusting those counts all the time. The consequence here is that we have to be a bit more careful when returning newly-constructed values. We must mark the value as mortal, which means we are saying that its reference count is somehow artificially high (because of that pointer on the stack), and perl should decrement the reference count at some point soon, when it next discards temporary values.

Because this sort of thing is done a lot, there is a handy macro called mPUSHs, which mortalizes an SV when it pushes it to the data stack. We can call that instead:

$ git diff pp.c
...
+    mPUSHs(newSVpvn_rot13(s, len));
+    RETURN;
 }
 
 /*

Now when we try our leak test we find the same SV count before and after, meaning no leak has occurred:

leo@shy:~/src/bleadperl/perl [git]
$ ./perl -Ilib -I. -MXS::APItest=sv_count -E ...
5321
5321

We may be onto a winner here.

Index | < Prev | Next >