Python Profiling

There are two types of profiling.

Python comes with two modules for deterministic profiling: cProfile and profile. Both are different implementations of the same interface. The former is a C extension with relatively small overhead, and the latter is a pure Python module. As the official documentation says, the module profile would be suitable when we want to extend the profiler in some way. Otherwise, cProfile is preferred for long-running programs. Unfortunately, there is no built-in module for statistical profiling, but we will see some external packages for it.

For the sake of example, we are going to write a small Python script like this:

The function computation is a naive implementation for the sum of two lists. Here, we use two lists of 1000000 elements. The rest of the code is just a preparatory for the next section.

Let’s find out where our program spends the most time (pretty obvious, right?).

As we can see, the profiler has recorded information for each function of our code. How can we interpret the results?

We can also sort the functions by some criteria. For example:

We can use cProfile as a module rather than as a script to run more complex analyses. The most basic use of this module is the following:

“A picture is worth a thousand words”.

2. Secondly, we run gprof2dot to generate a png image:

This is the result:

As we can see, the graph shows both metrics, the total and cumulative time, and the number of calls.

To install it, just type:

Using this tool is straightforward; you just have to type this:

As I mentioned before, we can attach the profiler in runtime. In that case, we would use the PID of the process.

After that, we will see something like this:

As we see, the profiler is sampling periodically and updating the information.

To install it, just type:

Now, we are going to generate an HTML report for our code:

We can open the generated HTML file with our default browser:

As you can see, the output of this tool is easy to interpret and distraction-free. This is because the main purpose of it is to show the slowest functions of your code, that’s all. When we have tons of functions, this is a huge advantage.

As I said before, there are many other tools for profiling. Feel free to use the most suitable tool for you. Here I list you some of them:

Let’s find out the slowest function of our code. This code is not for production and the maximum execution time in my computer is 5–10s. Also, the length is not excessive, so I will try cProfile.

The list is incomplete. One of the biggest drawbacks of cProfile is that it prints the internal functions of the libraries you use, making the output really messy. In most of the cases, we cannot optimize code from external libraries.

Now we have the output we were looking for, without messy lines, just the slowest functions of our code.

Thank you for having come this far. Now I would like to remind you of the key points of this article:

Related posts:

Threads have been a must, unavoidable pain in large scale application. We spend lots of time creating, managing and resolving conflicts. We take into consideration every single bit. Things like…

What is an Encorepreneur? The Urban Dictionary defines an Encorepreneur as “ a person nearing retirement and undertaking a new venture. Just a few months back, the term Encorepreneur might have been…

I have mixed feelings about Mexico City. While I enjoy its lively culture, rich history, and bustling atmosphere, there is one aspect of it that I absolutely cannot stand: Friday nights. My…