Advancing Spark - Getting Started with Databricks AutoML

It can be daunting figuring out where to start when tackling a new data science project, that's why we've seen a whole range of automated machine learning tools cropping up over the last few years. The offering from Databricks is fairly new, coming out earlier this year, but has some nice features that set it apart from the rest.
In this video, Simon invites Gavi, a Principal Data Scientist with Advancing Analytics to give us an overview of Databricks AutoML and run through a quick first model.
For more details around the AutoML offering - see databricks.com...
And, as always, if you need help on your advanced analytics journey, come say hello and see if Advancing Analytics can get you there.

[@knildawild]

Thank you for this interesting introdution to AutoML!

[@venkateshkothapalli]

Great stuff Simon!

[@guybourne1284]

Cool video - love the notebook outputs. On new topics, it'd be great to see some content from you guys on the intersection of data engineering and data science. Deployment, monitoring, management of data drift - all the MLOps goodness. On the merch...that sticker... I'm going to have to find a way to get some over in Australia!

[@AdvancingAnalytics]

Absolutely, MLOps is a big topic that we spend a ton of time with clients on. We'll knock out the basics of the machine learning persona first, but it's on our roadmap!
And yeah... "the sticker" was a joke at first, but people kept asking us for them! Sorry about the lack of Oz delivery, we're limited to what the youtube "merch provider" supplies out the box!
Simon

[@seoexperimentations6933]

Would love some more videos on auto-ml. Azure AutoML is great too, it can do feature engineering for you (creating new features) and deploy the models to a kubernetes cluster. Would love to know if that's possible in databricks as well.

[@AdvancingAnalytics]

More planned! Especially looking at how Databrick's Featurestore integrates to make AutoML better. I'll see if we can round up some Azure ML coverage too!
Simon

[@saids.4307]

💓

[@miguelangelfernandezguerre1681]

Hi Simon, geat stuff! I noticed that in the demo AutoML fits only scikit-learn kind models. I wonder if that is the default or AutoML decided to use scikit-learn instead of sparkml because of the dataset size that you provided to it. In my data science projects I'm never 100% sure about when should I use scikit-learn or sparkml for medium-large datasets to train the machine learning pipeline. Do you see many teams fitting ML models with sparkml or it is better to use scikit-learn in order to deploy better the project as a python library? Nevertheless, how do these teams deploy machine learning models built using sparkml functions?
I kind of prefer use scikit-learn because at the end of the project I know how to package all my code into a python library and deploy it easier (and pay the price for very long training times). I do not know if I build the whole machine learning pipeline (feature enginnering + ml model) using pyspark (sparkml) the process could be the same and I can wrap all the project as a python library (I like packaging as a python library in order to include unit tests, config files and all other software engineering best practices). Do you have any reference about how to "package" the whole machine learning project for production?
Thank you so much in advanced for your answer and all your fantastic content,
Miguel

[@scivanpoon]

I am learning now. how can I get AutoML in Databrick Community Edition? Thanks!

[@zhangbrandon3104]

Yes, baddly want to know