Devel::MAT investigation into C - part 1

I have previously written about using Devel::MAT to find the cause of memory leaks in Perl programs. While it is often able to find the reasons for memory growth in a great number of cases, there are times when it cannot alone find the whole story, and we must combine other tools to finish our investigation. Here we'll look at one particular case that - like so many others - begins with the observation that a long-running Perl process slowly consumes ever-more memory from the host operating system, the longer it runs. We'd like to find out why, and fix it.

The process under investigation this time is a long-running script that regularly polls a number of sites over HTTP (well, HTTPS in fact) to collect up and aggregate some data from them. The rate of memory leak is observed to correlate with how often it performs this polling, so it suggests that the leak is likely caused by something that is done on every iteration of the program losing the same amount each time. The regular iteration of the program provides us a convenient point to attach some memory debugging to begin our investigation.

A slow increase of memory could be caused by a Perl-visible accumulation of SVs, or could be something only visible at the C layer. Before we start investigating with the heavy guns of Devel::MAT, we might do well to try to classify what's going on with something more lightweight. In this case, Devel::Gladiator provides us a simple way to count the total number of SVs on the Perl heap, to see if it's increasing. We keep a count of the previous total, so we can print the difference per iteration as well:

use Devel::Gladiator qw( arena_ref_counts );
use List::Util qw( sum );

my $lasttotal = 0;

...

    # per iteration:
    my $total = sum values %{ arena_ref_counts };
    printf "SVs %d (+%d)\n", $total, $lasttotal - $total;
    $lasttotal = $total;

We can now run this and get an initial idea of the growth of the Perl heap:

SVs 51940 (+51940)
SVs 60171 (+8231)
SVs 59655 (-516)
SVs 59652 (-3)
SVs 59655 (+3)
SVs 59657 (+2)
SVs 59660 (+3)
SVs 59663 (+3)
SVs 59665 (+2)
SVs 59668 (+3)

After a few iterations of initial startup and settling down, this program is clearly leaking actual Perl-visible SVs at around 2 or 3 per iteration, so it makes sense to get Devel::MAT involved. We can easily insert that into the iteration loop:

use Devel::MAT::Dumper;

my $iter = 0;
...

    # per iteration:
    $iter++;
    Devel::MAT::Dumper::dump( "perl-$iter.pmat" );

Now when we run this we get a sequence of numbered .pmat files, one per iteration, that we can throw into pmat-leakreport to look for leaked scalars. Because the first few files will contain a lot of noise from program startup, we'll skip those and take from, in this case, the 4th onwards:

$ pmat-leakreport -1 perl-{4,5,6,7,8}.pmat
LEAK[1] UNDEF() at 0x55efee189120
LEAK[1] UNDEF() at 0x55efee1ae1f8
LEAK[1] UNDEF() at 0x55efeda8c5e8
LEAK[1] UNDEF() at 0x55efee1acd30

This gives us four exciting candidates we can take more of a look at using the pmat shell:

$ pmat perl-8.pmat 
Perl memory dumpfile from perl 5.26.2
Heap contains 55890 objects
pmat> identify 0x55efee189120
UNDEF() at 0x55efee189120 is:
└─not found

pmat> identify 0x55efee1ae1f8
UNDEF() at 0x55efee1ae1f8 is:
└─not found

pmat> identify 0x55efeda8c5e8
UNDEF() at 0x55efeda8c5e8 is:
└─not found

pmat> identify 0x55efee1acd30
UNDEF() at 0x55efee1acd30 is:
└─not found

Well that's not very helpful. We have four scalars that contain no defined values, which aren't referenced by anywhere that Devel::MAT can see. In order to continue investigating this, we need to dig deeper with more tools. Perl itself provides an optional ability to write a more detailed log of memory allocation operations, controlled by a variable called PERL_MEM_LOG. According to perlhacktips(1):

PERL_MEM_LOG

If compiled with -DPERL_MEM_LOG (-Accflags=-DPERL_MEM_LOG), both memory and SV allocations go through logging functions, which is handy for breakpoint setting.

Now, we can't just set that environment variable on any normal perl binary, because the logging functions are sufficiently expensive at runtime that they are usually disabled on normal builds. We have to request them specifically with the use of some compile-time flags. To achieve this, it's easiest to use perlbrew to compile a custom binary for our purposes:

$ perlbrew install perl-5.26.2 --debug -Accflags=-DPERL_MEM_LOG

That'll take a while to build, but when it's ready we'll continue our investigation in part 2...

---

For another memory-leak investigation involving pmat-leakreport, see also the Binary.com tech blog post Differential Memory Leak Analysis with Devel::MAT.

Unicode 9 emoji support in pangoterm/libvterm

A small update to libvterm now means that it understands the width of all the characters in Unicode 9. Combined with a suitable font (such as, in this case, the fonts-noto-color-emoji debian package), this now means all the Unicode emoji characters can be nicely displayed by pangoterm.

Devel::MAT::Dumper independent distribution

I've just released a new version of Devel-MAT, now with the heap dumping part split out into its own distribution, Devel-MAT-Dumper.

There's two reasons for this. Firstly, by being split into its own distribution, the heap-dumper part is smaller and lighter to install on places like production servers as it needs no non-core dependencies. Since the heapdump files still need analysis somewhere, you'll still have to install the full Devel-MAT distribution on a machine - such as a workstation or laptop - but you no longer need the analysis tools on every target machine.

Secondly, since the two parts are now in independent distributions, they can have different Perl version requirements. I want to keep the ability to analyse Perl programs running on versions from 5.10 onwards (and maybe sometime see if I can push that back to 5.8), but now the distributions are split I have begun to make use of 5.14 features in some of the analysis tools. This means that you can still analyse programs running on 5.10, but you will need an installation of 5.14 or later to run the tooling.

Looking a little further afield, I plan to continue writing more tooling, perhaps published as independent distributions in places too. It's still the case that almost every time I run into some situation that needs memory analysis like this, I end up writing more tools to help, so the flow of additional tooling will continue for a while yet.

---

I'm also keen to hear success stories from others (or even failure ones - what couldn't you work out and what might I do to help?) to help guide the creation of more tooling. I'm available to consult with people to help you with your own problems also - send me a note to

leonerd [at] leonerd [dot] org [dot] uk

and we can discuss your requirements.

Async/await in Perl - control flow for asynchrony

I've decided that my Future::AsyncAwait CPAN module is sufficiently non-alpha that I've started migrating a few of my less critical code into using it. I thought I'd pick a few of the Device::Chip drivers for various kinds of chip, because they're unlikely to be particularly involved in anyone's real deployment code, as really I only wrote those to test out some ideas on the chips before writing microcontroller code in C for them. These seemed like good candidates to begin with.

Here's an example of a function in the previous version, using Futures directly. The code had lots of syntactical noise, some ->then chaining and the Future::Utils::repeat loop not looking like a regular foreach loop. You can just-about read what's going on but it's clear there's a lot of machinery noise getting in the way of really understanding the code.

By rewriting all the logic using await expressions inside an async sub we arrive at a version that much closer resembles the sort of thing you'd write in straight-line synchronous code. In reading it you can just skim over the awaits while looking at it and read it like synchronous code.

A question you might begin to ask at this point is why I'd choose to implement this particular set of syntax or semantics, of the various possibilities for how to manage asynchronous control flow. Aside from its general neatness and applicability to Futures (which I've already worked with at length), there's one key reason: The async/await syntax here is the exact same thing as implemented in Python 3, ES6, C#5, Dart, even Rust is currently considering adopting it Yes, it's nice to have a good concurrency model built into the language, but it's considerably stronger if it's the same as the consensus among a variety of other languages too.

Some language references for them:

Python	Tasks and coroutines
JavaScript	async function
C#	Asynchronous Programming
Dart	Dart Language Asynchrony Support

If the four quite semantically-different languages of Python, JavaScript, C# and Dart can all come to the same idea, then maybe it has merit. I honestly think that given a few years, async/await could become as ubiquitous as if or while loops, to the level of "well obviously our language has that". This is why I wanted to steal it into Perl. In ten years time it might look as foolish for a language not to have an async/await construct, as it does today for it not to have a try/catch or a switch.

Ah... more on that subject another day perhaps ;)

Developing against $HOME/lib libraries and LD_LIBRARY_PATH

I've come up against an awkward situation using libtool, and I wonder if I could ask for some advice on what I'm probably doing wrong here.

I'm developing two different C libraries; lets just call them A and B. A is fairly standalone, and B depends on A.

Using libtool, I can develop on A fine. In particular, it has some internal unit-tests which I run using libtool --mode=exec. These run OK.

Using libtool, I can develop on B just fine, provided the A library is installed as a real system package. The linker can find it by the normal mechanisms and it lives in /usr/lib along with other system things. All is good. Library B also has some unit-tests that are executed with libtool --mode=exec. So far so good.

But now suppose I have a possible change I want to make that needs edits in both libraries. I don't want to build a new system package yet and upset all the other users on this box. That's fine, I can just install library A into my $HOME directory using

$ make install PREFIX=$HOME

In order to be able to test programs that use library A, I'm going to need to set LD_LIBRARY_PATH now so they can find the library here in preference to the system-installed one, so in my .profile I use

export LD_LIBRARY_PATH=$HOME/lib

But now, that upsets every use of libtool --exec to run any of the unit tests for my libraries in their source code (for both libraries A and B). With LD_LIBRARY_PATH set in the environment, the libtool wrapper script no longer sets it up at all, meaning that libtool --exec on a unit test just sees the system-packaged versions of each library. This means I can't test the new code I just write in library A, because the unit-test executables don't see it.

I could make unit tests for library A itself run fine by entirely defeating the LD_LIBRARY_PATH mechanism again, if I

$ LD_LIBRARY_PATH= make test

But what of library B? It has to be able to see its own locally-build library in $CWD/.libs and library A in $HOME/lib. So I could

$ LD_LIBRARY_PATH=`pwd`/.libs:$LD_LIBRARY_PATH make test

But at that point I really feel like I am fighting against libtool, rather than it helping me do this right.

So, what am I doing wrong here?

Alternatively, am I using this entire setup in its intended way, but libtool just isn't playing ball with me?

More Perl memory leaks with Devel::MAT - another use-case story

I've written a second post about Devel::MAT on the company blog, following up my first one.

Again, rather than copy the entire thing I'll just link to it here instead:

Tracing Perl memory leaks with Devel::MAT, part 2

As a little preview, I'll add that this one has screenshots:

Tracing Perl memory leaks with Devel::MAT - a use-case story

I've lately been working a lot on the analysis tools around Devel::MAT, the toolset used for analysing memory leaks, unbounded growth, and other memory-related issues affecting Perl programs.

I wrote a blog post about it on the company blog. Rather than copy the entire thing I'll just link to it here instead:

Tracing Perl memory leaks with Devel::MAT