Ideas and Plans for Memory Profiling with NYTProf

XXX This mostly a dumping ground for thoughts, and links at the moment. Somewhere to gather the info so it'll be easier to find when someone wants to work on it.

It's also focussed on runtime profiling of total memory usage over time (rather than a detailed analysis of who 'owns' what memory as some particular point in time, like the end of the program).

NYTProf currently only measures time and supports a limited range of "clocks" (realtime vs cputime). For profiling memory we need to add a new kind of "clock" that measures memory usage. Since we're generalizing the concept of what gets measured (and how we get the info from the system) a better name than "clock" would be "probe".

Conjectural terminology:

"Probe" means some measuring mechanism like get_clock(), times(), getrusage(),
that may yield multiple pieces of information with a single call.

"Measure" is one specific item generated by a probe.

Probe "time"   uses times(), measures: "time.user", "time.user+sys" etc
Probe "clock"  uses clock_gettime(), measures: "clock.realtime", "clock.monotonic" etc
Probe "rusage" uses getrusage(), measures: "rusage.majflt", "rusage.nvcsw" etc
Probe "memory" measures: "memory.bytes", "mem.allocs" etc
Probe "arena"  measures: "arena.svs", "arena.bytes" etc

Generalize the concepts of probes. Have a structure defining a 'probe' with pointers to functions to get the values, subtract values to get relative ticks, return the tick units etc. Give them names and attributes (cpu, realtime etc). User could then pick a probe by name. By default we'd pick the best available realtime probe. Use the subtraction logic where we currently handle times in the statement and subroutine profilers.

Email threads

"Memory profiling in Devel::NYTProf?: - June 2009 http://groups.google.com/group/develnytprof-dev/browse_frm/thread/1df4cba3001cd4e4/136812b44e9f7631 Talking about the problems of measuring memory usage of the whole process re: http://blog.robin.smidsrod.no/index.php/2009/05/26/memory-footprint-of-popular-cpan-modules

"Memory profiling possibilities in NYTProf" - September 2009 http://groups.google.com/group/develnytprof-dev/browse_frm/thread/c711c132216a3cea/035012e3dc2971ec This includes a detailed overview of the issues.

"profiling memory" - Dec 2009 http://groups.google.com/group/develnytprof-dev/browse_frm/thread/5ffd24200866b0c1/201b58c18d826aaa Nicholas Clark offers an experimental patch that intercepts malloc and free and makes NYTProf measure memory usage.

Possibly Relevant Perl Modules

Per-process memory information:

http://search.cpan.org/perldoc?Devel::Mallinfo

Arena, stash, and pad based memory reporters:

http://search.cpan.org/perldoc?Devel::Gladiator
http://search.cpan.org/dist/Internals-DumpArenas/
http://search.cpan.org/dist/Internals-GraphArenas/
http://search.cpan.org/perldoc?Devel::DumpSizes
http://search.cpan.org/perldoc?Devel::Arena

Per-object memory size reporters:

http://search.cpan.org/perldoc?Devel::Size
http://search.cpan.org/perldoc?Devel::Size::Report

Others:

http://search.cpan.org/dist/Devel-Memalyzer/
http://search.cpan.org/perldoc?Devel::Memalyzer::Plugin::ProcSmaps

Other Items of Interest

"Memory Efficient Perl" slides by jjore

http://docs.google.com/present/view?id=dg7kgpct_24cjs3c9fv
http://diotalevi.isa-geek.net/~josh/090402/frontend.png

Other profile/memory visualization tools

http://netjam.org/spoon/viz/
http://java.dzone.com/announcements/visualvm-12-great-java

http://blogs.perl.org/users/alex_balhatchet/2012/01/debugging-memory-use-in-perl---help.html

http://stackoverflow.com/questions/8715611/can-i-use-dtrace-on-os-x-10-5-to-determine-which-of-my-perl-subs-is-causing-the