Advancing Spark - Getting Started with Ganglia in Databricks

As a first video back for 2022, we thought we'd take a look back at one of the most useful, but overlooked tools within a Databricks Administrator's toolbelt. We hear from many people in the community that they're having trouble monitoring their clusters, figuring out how utilised they are and diagnosing performance problems. Ganglia is an incredibly useful (but initially intimidating) tool that's baked in to the Databricks workspace!
In this video Simon walks through how he uses Ganglia when looking at a specific load problem, what you can do, what you can't do, and gives you what you need to get started monitoring your cluster performance
We've dug into the Spark UI previously, so if you're just getting started, check it out here: • Advancing Spark - Unde...
As always, get in touch if Advancing Analytics can help you on your analytics journey

[@bramlangelaar]

Thanks for the video! I just started working with Databricks, and used Ganglia a bit. Now I am walking into some out of memory issues, I should definitely use it more.

[@mamamiakool]

Belated new year wishes Simon !! Great to see you back and sharing incredible detailed insights into Advanced Performance tuning with Spark. More power to you for doing the good work :)

[@hubert_dudek]

Ganglia is nice but in databricks it really should be replaced by some databricks native solution as it is quite "funny" to include smth where most tabs are not working and include png screenshots of previous states

[@jaimehernandez4333]

I appreciated this video. Thanks!

[@anonymouslyyours5605]

Very nicely explained. Can you add how we can get data from ganglia to generate alerts

[@thiagojuliao2214]

Is there a way to hit some ganglia endpoint to collect the metrics without consulting the frontend page? Great video btw!!

[@ravirajuvysyaraju123]

Very informative

[@RodrigoCastroHdz]

I thought Ganglia was no longer maintained in favor of other tools like grafana or Prometheus

[@AdvancingAnalytics]

We don't see any updates to Ganglia, but it remains baked into the Spark UI, so it's worth understanding how it works if you don't want to sync your logs to another store and build your own dashboards!

[@omarcubano2698]

Anything about gpu monitoring?

[@user-nx9kz8yi8t]

this guys is so very irritating
'

[@AdvancingAnalytics]

Hah, I aim to please 😅

[@kriandir]

@@AdvancingAnalytics Don't take a random internet comment to heart, you did a very well job of explaining it in an easy to follow and entertaining way