Rational Girl

Attempting to be rational while dreaming of 3.141592653589793...

profiling code

Ipython is my favorite interactive environment. So lets set up profiling in Ipython. You need to install three key packages (Im assuming you have ipython installed, this example is based on ipython 0.13). For each package I show three possible ways to install

  • using easy_install on a machine where you have sudo
  • using easy_install on a machine you do NOT have sudo, into your own local directory
  • using pip

ipython version 0.13

install line_profile

http://pythonhosted.org/line_profiler/

sudo easy_install line_profiler

easy_install --prefix /path/to/users/local line_profiler

pip install line_profiler

install psutil

https://code.google.com/p/psutil/

sudo easy_install psutil

easy_install --prefix /path/to/users/local psutil

pip install psutil

memory_profiler

https://pypi.python.org/pypi/memory_profiler

sudo easy_install memory_profiler

easy_install --prefix /path/to/users/local memory_profiler

pip install memory_profiler

Create ipython profile

http://ipython.org/ipython-doc/dev/config/overview.html

To generate the default configuration files, do:

$> ipython profile create profiler

$> mkdir ~/.config/ipython/profile_profiler/extensions

ipython profile create profiler

creates ipython_config.py in ~/.config/ipython/profile_profiler

The location may be different for you, but when you call the command it tells you where it put the new profiler directory (cause thats the right thing to do)...

If you dont see this you can locate your ipython config directory which will contain your profiles

ipython locate

in the ~/.config/ipython/profile_profiler/extensions directory create two new files

line_profiler_ext.py

import line_profiler

def load_ipython_extension(ip):
    ip.define_magic('lprun', line_profiler.magic_lprun)

memory_profiler_ext.py

import memory_profiler

def load_ipython_extension(ip):
    ip.define_magic('memit', memory_profiler.magic_memit)
    ip.define_magic('mprun', memory_profiler.magic_mprun)

Next update ipython_config.py to use these extensions

c.InteractiveShellApp.extensions = [
    'line_profiler_ext',
    'memory_profiler_ext'
    ]

c.TerminalIPythonApp.extensions = [
    'line_profiler_ext',
    'memory_profiler_ext'
    ]

Now you can call your profiler profile

ipython --profile=profiler

Python 2.7.3 |EPD 7.3-2 (64-bit)| (default, Apr 11 2012, 17:52:16)
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

IPython profile: profiler

In [1]: %memit?
In [2]: %lprun?
In [3]: %timeit?

Lets look at a simple example profiling MaskedArray methods

Time Profile

In [11]: from numpy.ma import MaskedArray
In [12]: jnk = np.random.random(100)
In [13]: jnk = MaskedArray(jnk, jnk>.6)

In [14]: %timeit jnk.mean()
10000 loops, best of 3: 29.1 us per loop

In [15]: %timeit np.mean(jnk[jnk <= .6])
10000 loops, best of 3: 150 us per loop

Line Profile

Usually you run this on a file to profile the code line by line

%lprun -f <filename>.py <method>

Memory Profile

%memit -r <R> statement

this gets the max memory usage over 100 loops

In [22]: %memit -r 100 jnk.mean()
maximum of 100: 50.273438 MB per loop

In [23]: %memit -r 100 np.mean(jnk[jnk <= .6])
maximum of 10: 50.273438 MB per loop