py-spy: Python Sampling Profiler

Warnings

• breaking py-spy frequently requires elevated permissions to profile running processes. On Linux, attaching to an existing process usually necessitates `sudo` or modification of `ptrace_scope`. For Docker or Kubernetes containers, the container must be launched with `--cap-add SYS_PTRACE` to allow `py-spy` to read process memory. On macOS, running `py-spy` as root (`sudo`) is generally required.
  Fix: Run `py-spy` with `sudo` when attaching to existing processes. For Docker, add `--cap-add SYS_PTRACE` to your `docker run` or `docker-compose.yml` configuration. Ensure `ptrace_scope` is configured appropriately on Linux if `sudo` is not desired.
• gotcha Older versions of `py-spy` might not fully support profiling newer Python interpreters due to changes in CPython's internal ABI. For example, Python 3.12 and 3.13 support was added in recent `py-spy` releases (v0.4.0+), and using older `py-spy` versions with these Python versions could lead to errors or inaccurate profiling.
  Fix: Always use the latest available `py-spy` version to ensure compatibility with the Python interpreter you are profiling. `pip install --upgrade py-spy`.
• gotcha On macOS, System Integrity Protection (SIP) prevents `py-spy` from profiling Python interpreters installed at `/usr/bin`. Attempting to profile such interpreters will fail.
  Fix: Profile Python interpreters installed via alternative methods (e.g., Homebrew, Anaconda, pyenv) or within an active virtual environment, as these are typically located outside `/usr/bin`.
• gotcha Profiling very idle Python programs or those with highly regular, periodic activity can result in misleading or inaccurate flame graphs and `top` output due to sampling aliasing. `py-spy`'s default sampling rate might coincidentally align with program's internal ticks, leading to over- or under-reporting of activity.
  Fix: Use the `--idle` flag to include idle frames in the profile, or the `--nonblocking` option to avoid pausing the target (though this may introduce occasional sampling errors). Consider adjusting the sampling rate (`-r` option) to a non-standard value to mitigate aliasing, although it might not fully eliminate the issue.
• gotcha py-spy is a CPU sampling profiler only and does not provide functionality for memory profiling. It cannot diagnose memory leaks, track memory allocation/deallocation, or optimize memory usage.
  Fix: For memory analysis, use dedicated Python memory profiling tools (e.g., `memory_profiler`, `objgraph`, built-in `tracemalloc`) in conjunction with `py-spy` for a comprehensive performance overview.

Install

• pip install py-spy Install with pip

Quickstart

This quickstart demonstrates how to use `py-spy` to record a flame graph (`profile.svg`) of a running Python process. It first starts a simple Python script in the background, then attaches `py-spy` to its process ID (PID) to capture profiling data. For live viewing, `py-spy top --pid <PID>` is also a common command. Remember that `py-spy` often requires elevated privileges (e.g., `sudo`) or specific system capabilities (like `SYS_PTRACE` in Docker) to function correctly when attaching to existing processes.

import time
import subprocess
import os

# Create a dummy Python script to profile
python_script_content = """
import time
import sys

def busy_loop(iterations):
    result = 0
    for i in range(iterations):
        result += i * i
    return result

def main():
    print(f"[{os.getpid()}] Starting my_app.py...")
    for _ in range(5):
        busy_loop(1_000_000)
        time.sleep(0.5)
    print(f"[{os.getpid()}] my_app.py finished.")

if __name__ == "__main__":
    main()
"""

with open("my_app.py", "w") as f:
    f.write(python_script_content)

print("Running my_app.py in the background...")
# Start the target Python script in the background
# In a real scenario, you'd profile an already running process by PID
# For quickstart, we launch it and then 'profile' it (though py-spy can launch directly too)
process = subprocess.Popen([sys.executable, "my_app.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
pid = process.pid
print(f"Target Python process PID: {pid}")

# Wait a bit for the script to start
time.sleep(2)

print("Recording profile with py-spy...")
# Example: Record a flame graph of the running process
# On Linux, you might need 'sudo' if attaching to an existing process (not a child)
# On Docker, ensure --cap-add SYS_PTRACE is used
# os.environ.get is not directly applicable here as py-spy is a CLI tool
# but the principle for quickstarts is to show a runnable example.
try:
    # Using subprocess.run for simplicity, would typically be a direct shell command
    # For this example, we assume necessary permissions are available (e.g., running as root or correct ptrace_scope)
    # In a real shell, you'd do: py-spy record -o profile.svg --pid <PID>
    result = subprocess.run(["py-spy", "record", "-o", "profile.svg", "--pid", str(pid)], capture_output=True, text=True, check=True)
    print("py-spy stdout:", result.stdout)
    print("py-spy stderr:", result.stderr)
    print("Flame graph saved to profile.svg")
except subprocess.CalledProcessError as e:
    print(f"Error running py-spy: {e}")
    print("py-spy stdout:", e.stdout)
    print("py-spy stderr:", e.stderr)
    print("HINT: You might need to run this with 'sudo' or ensure appropriate ptrace permissions.")
except FileNotFoundError:
    print("Error: py-spy command not found. Ensure py-spy is installed and in your PATH.")
finally:
    # Clean up the background process
    if process.poll() is None:
        process.terminate()
        process.wait(timeout=5)
    # Clean up the dummy script
    os.remove("my_app.py")
    if os.path.exists("profile.svg"):
        print("To view the profile, open profile.svg in a web browser.")