Further Adventures in Cython Profiling

Profiling Tools

After getting some criticism on my last post, I decided to dig a bit deeper into best profiling tools for Python/Cython. While I am familiar with Python's cProfile and time modules, my good friend, Shapr pointed me towards py-spy as a very low overhead option implemented in Rust. This profiler would also give me visual feedback in the form of a flame graph, so I could see which parts of my code were taking the longest.

Most of the information here has been pulled together from various sites linked through the py-spy repo, but I haven't seen a full config in a single place with the given constraints and demands that I had. One of the biggest limitations of CProfile was that any of the cdef methods I have defined in my code (which I need in order to release the GIL) can't be picked up by typical Python profilers. py-spy is able to do this through using --subprocesses and --native flags as seen below.

In my Cython program I had to add the following to my setup.py file

"ext_modules": cythonize(extensions, emit_linenums=True, compiler_directives={'language_level': 3})

Compiling with this default option means that the generated cpp code has line numbers that correspond to the original Cython code in the .pyx files. You can check that this has worked by looking in the Cython generated cpp files for lines that look like this:

#line 212 "pymaxion/particle_system.pyx"

I also included the following directive in the headers for all my .pyx files:

# cython: linetrace = True

My personal machine is a Mac, which works fine for some of the functionality of py-spy, except for profiling native functions (meaning being able to get at the cpp processes in my code that I really wanted to see!). As a result, I made a docker container and ran py-spy with Linux.

My Dockerfile:

FROM python:3.6.12-buster

COPY . /code

RUN apt-get update
RUN apt-get -y install sudo

RUN pip install -r /code/requirements.txt

ENV PATH=$PATH:/code/src
ENV PYTHONPATH "${PYTHONPATH}:/code/src"

RUN cd /code/src && python setup.py build_ext --inplace

WORKDIR /code/tests/open_mp_tests

I used python 3.6 in my image based on some open issues on py-spy which said they couldn't get subprocesses to work with newer versions of Python. My requirements for the program are pretty minimal, only cython and numpy at this point for the actual program execution, as well as py-spy. The other parts of the Dockerfile are to install sudo (you'll need to run py-spy as root in the container) and to automatically compile my Cython library.

Once I had an image, I ran the following to get a container with the right permissions.

docker run -td --cap-add SYS_PTRACE $IMAGE_NAME

After that, I could run the actual py-spy profiler on my sample file.

sudo docker exec -t $CONTAINER_NAME bash -c "PYTHONPATH=/code/src &&
py-spy record --subprocesses --native -o profile.svg -- python cython_test.py""

docker cp $CONTAINER_NAME:/code/tests/open_mp_tests/profile.svg ~/open_mp_tests/

The lines above allow me to produce an interactive flamegraph, which is then copied to the original directory.