USENIX ATC '17: Visualizing Performance with Flame Graphs

Brendan Gregg, Senior Performance Architect, Netflix
Flame graphs are a simple stack trace visualization that helps answer an everyday problem: how is software consuming resources, especially CPUs, and how did this change since the last software version? Flame graphs have been adopted by many languages, products, and companies, including Netflix, and have become a standard tool for performance analysis. They were published in "The Flame Graph" article in the June 2016 issue of Communications of the ACM, by their creator, Brendan Gregg.
This talk describes the background for this work, and the challenges encountered when profiling stack traces and resolving symbols for different languages, including for just-in-time compiler runtimes. Instructions will be included generating mixed-mode flame graphs on Linux, and examples from our use at Netflix with Java. Advanced flame graph types will be described, including differential, off-CPU, chain graphs, memory, and TCP events. Finally, future work and unsolved problems in this area will be discussed.
View the entire USENIX ATC '17 program at www.usenix.org/conference/atc...

[@shuyijia3755]

Quick explanation to flame graph starts at 12:47

[@kalyanasundaramgnanasekara9986]

I understood about profiler works, interpret the flame graphs, types of flame graphs. after watched this video I ordered the 'System Performance' and 'BPF programming guide' books which are written by Brendan Gregg. Thanks.

[@goverdhank]

Thanks for sharing ! You rock

[@milequinze]

Awesome, thanks!

[@SoeaOu]

Great talk, thanks.

[@ronneilcamara5071]

We've been troubleshooting Jenkins issues for almost 4 months now. The "top" tool shows kworker is at 100% cpu utilization. We can't figure out what the culprit really is. Using strace to attach to the process doesn't do anything. Jenkins java process is hung or frozen. Any tips on how we should debug this issue?

[@WALLACE9009]

What happens when function C is called by function A and also by function B? I kind of see C parallel to A and B but not on top of A and B.

[@maoandmi]

Good stuff!

[@sid6576]

Legendary

[@MikhailMatrosov]

How do you display situation when a function appears on different call stacks? E.g. if F() does 100% of the job in your app, but 50% times it is called from A(), and 50% times it is called from B()? What would flame graph for this setup be?

[@Darleer]

good question. I am no flamegraph expert so just wonder if you've found an answer yet?

[@MikhailMatrosov]

Nope, still mystery to me :)

[@nicoster2]

Mikhail Matrosov then you get 2 towers A-F, B-F, each with 50% width

[@williammeng2217]

good stuff.

[@xl000]

Do you have any plans to have Netflix in North Korea and Syria ?