Profiling — cProfile

PythonMidpython

The Question

How do you profile Python code with cProfile?

What a Strong Answer Covers

• function call time
• "not CPU
• "python -m cProfile

Senior-Level Answer

Profiling is the process of measuring where a program spends its time, so you can direct optimization efforts at the actual bottleneck rather than guessing. Python's built-in `cProfile` module is a deterministic profiler that instruments every function call and records call counts, total time, and cumulative time per function.

The simplest usage is from the command line: `python -m cProfile -s cumulative my_script.py`. The `-s cumulative` flag sorts output by cumulative time — time spent in a function *including* all callees. This is usually the most useful sort for finding the root of slowness. Other sort keys: `tottime` (time in the function excluding callees — useful for finding tight CPU loops), `ncalls` (call frequency), `filename`.

From within code, you wrap the target section: ```python import cProfile with cProfile.Profile() as pr: result = expensive_function() pr.print_stats(sort='cumulative') ```

For programmatic analysis, dump to a file and load with `pstats`: ```python pr.dump_stats('output.prof') import pstats stats = pstats.Stats('output.prof') stats.strip_dirs().sort_stats('cumulative').print_stats(20) ``` `strip_dirs()` removes path prefixes for cleaner output; `print_stats(20)` shows only the top 20 functions.

For visualization, **snakeviz** (`pip install snakeviz; snakeviz output.prof`) renders an interactive sunburst chart of the call tree in a browser — far easier to navigate than raw text for large profiles. **gprof2dot** generates call-graph DOT files.

cProfile has overhead because it instruments *every* function call — typically 10-50x slowdown. For production profiling of a live service, **py-spy** is a sampling profiler that attaches to a running process without modification or restart, reads the Python stack via ptrace, and produces flamegraphs with minimal overhead.

Profile the *same workload* you are optimizing — synthetic microbenchmarks often mislead. Use `timeit` for isolated micro-benchmarks of specific expressions. For memory profiling (not CPU), use `memory_profiler` or `tracemalloc` (built-in since Python 3.4).

What Separates a 2/3 from a 3/3

Common Mistakes

• Sorting by call count instead of cumulative time — call count alone doesn't reveal expensive functions.
• Not knowing the difference between tottime (excluding callees) and cumtime (including callees) — misreading these leads to optimizing the wrong function.
• Running cProfile in production — the deterministic overhead (every call instrumented) makes it unsuitable for live traffic.
• Skipping visualization — raw cProfile text output is hard to navigate for non-trivial programs.

Follow-Up Questions

• What is the difference between a deterministic profiler (cProfile) and a sampling profiler (py-spy)? — Deterministic: hooks every function call — complete data, significant overhead. Sampling: periodically reads the call stack — low overhead, statistical approximation. Sampling is for production; deterministic for dev.
• How would you profile a specific slow endpoint in a running Flask/Django application without restarting it? — Use py-spy to attach by PID. Or add cProfile instrumentation via a middleware decorator scoped to the target endpoint, writing to a temp file per request.
• What does a flamegraph show and how do you read it? — X-axis: proportion of samples (not time order). Y-axis: call stack depth. Width of a bar = time spent in that function+callees. Look for wide bars near the top — those are the actual hotspots.
• How would you use tracemalloc to find which code is responsible for a memory leak? — tracemalloc.start() → run workload → take snapshot → compare snapshots with display_top(). Shows file/line with highest allocation delta, enabling surgical investigation.

Related Questions

• GIL — What It Is and What It Protects
• ThreadPoolExecutor & ProcessPoolExecutor
• Decorators — Under the Hood
• Generators & yield
• Context Managers