Edit

Analyze GPU profiling flame graphs on Azure Kubernetes Service (AKS) (Preview)

GPU profiling flame graphs show which functions consume the most GPU memory. This article explains how to interpret the flame graphs generated by the GPU profiling tools for Azure Kubernetes Service (AKS).

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Before you begin, set up GPU profiling and confirm that profiling data is available in Pyroscope and Grafana.

Note

The first profiling run might not show every symbol, including symbols from startup operations such as model loading, while the profiler analyzes the workload and resolves symbols. Run the workload again after the initial analysis to capture these symbols in subsequent profiles.

Understand flame graphs

A flame graph visualizes profiled call stacks. Each bar represents a function, and the stacked bars show the call chain, or which function called another function. The width of each bar represents the amount of the measured resource, such as CPU time or allocated GPU memory, that flows through that function.

Key rule: The wider a bar, the more of the measured resource flows through that function.

The following sample flame graphs are captured from a vLLM inference workload.

Screenshot of the initial collapsed flame graph view in Grafana for a vLLM workload, showing stacked bars that represent GPU memory allocation call stacks.

Tip

Use Expand all groups in Grafana's flame graph panel to see the full call stack without collapsing. Use the Search box to find specific functions or keywords. To prevent the panel from collapsing again, pause the dashboard auto-refresh by setting the refresh interval to Off in the upper-right corner of the Grafana dashboard while you inspect the expanded flame graph.

Screenshot of the flame graph in Grafana for a vLLM workload after selecting Expand all groups, showing the full call stack with individual function frames for GPU memory allocations.

Tip

Use Focus block to focus on a specific allocation path.

Read the symbols

Flame graph labels follow these conventions:

Symbol format Meaning
Foo > bar class Foo: method def bar()
Foo > __init__ Constructor of class Foo
bar (alone) Standalone def bar() function
<interpreter trampoline> CPython overhead. You can ignore this symbol.
<raw-address>, for example, 0x7f151 Native C or CUDA code with no Python symbol available.

Examples include:

  • GPUModelRunner / _allocate_kv_cache_tensors: A method on a class. Read this symbol as class GPUModelRunner: def _allocate_kv_cache_tensors(self).
  • LlamaMLP / __init__: A constructor called when creating a LlamaMLP(...) object.
  • _compile_fx_inner: A standalone module-level function that isn't inside a class.

Understand self versus total allocation

Use the Self and Total values to determine where the resource is consumed:

  • Total is the resource consumed by a function and everything it calls. A function can have a large total but allocate nothing itself because it's part of the call chain.
  • Self is the resource consumed directly by the function, excluding its child functions. A high self value means this function is where the resource is consumed.

For example, if GPUModelRunner._allocate_kv_cache_tensors has 55.1 GB of self allocation, it's the function that calls torch.empty() to create the key-value (KV) cache tensors.

Use the following navigation techniques:

  • Start with leaf nodes, which are bars with nothing above them. Their entire width represents self allocation.
  • Skip wide bars with zero self allocation when you search for the source of an allocation. These functions orchestrate calls to other functions.
  • Investigate wide bars with high self allocation as potential optimization targets.

Find the largest resource consumer

Use the following steps to identify hotspots:

  1. Look at the widest bars at the top of the graph. These leaf functions allocate memory. The wider the bar, the more memory it consumes.

  2. Check self versus total allocation. A wide bar at the bottom with self: 0 is part of the call chain. Follow it upward until you find a bar with high self allocation.

  3. Read the call stack from bottom to top to understand why the function was called. The following example ends with the function that performs the allocation:

    <raw-address>, for example, 0x7f151       -> native code entry
<interpreter trampoline>                 -> CPython dispatch
<module>                                 -> script top level
EngineCoreProc.run_engine_core           -> vLLM engine startup
EngineCore.__init__                      -> engine initialization
EngineCore._initialize_kv_caches         -> KV cache setup
Worker.initialize_from_config            -> worker setup
GPUModelRunner.initialize_kv_cache       -> model runner
GPUModelRunner._allocate_kv_cache_tensors -> actual allocation

Use the following table as a quick reference:

Goal What to look for
Find what allocates the most memory Widest leaf bar at the top of the stack
Find what's responsible for the most memory Widest bar at the bottom of the stack
Identify optimization targets Bars with wide self allocation
Identify functions you can skip Wide bars with zero self allocation

Next steps

  • Last updated on