Coding Bayesian Optimization (Bayes Opt) with BOTORCH - Python example for hyperparameter tuning

Bayesian Optimization is one of the most common optimization algorithms. While there are some black box packages for using it they don't allow a lot of custom changes and are not well suited for all problems. Facebook AI released a library called Botorch which enables the customization of all different layers of Bayes Opt (from GP-model up to the acquisition function). In this video, you get a top-level overview of how to code a Bayesian optimization from scratch and what to have in mind. Based on this knowledge you can then dive deeper into the single subparts to improve your own algorithm. It is a python based library!
Theory for BayesOpt: • Bayesian Optimization ...
BOTORCH: botorch.org
Links for Chapters:
0:00 Intro
0:35 Show test function
2:26 Generate initial samples
7:05 One Bayes Opt iteration
17:56 Optimization Loop
28:55 Outro
-------------------------------------------------------------------------------
Data Science to go: paretos.com

[@eloisedecarvalho3003]

Thank you so much! Waiting for the Multi Objective tutorial :)

[@anassbellachehab9821]

Extremely valuable! There are few to no other resources out there on botorch or gpytorch. Thanks

[@antoniosarusic5833]

Amazing! I am a beginner in this field and this helped me a lot with getting started. It would be very cool if you could make a video where you apply this to a more complex problem with multiple imputs.

[@user-pv9id2cr2g]

Good Video!Looking forward for your new videos.

[@sambitmishra1500]

I just started working on my research to implement BO and this video pops up. Perfect timing!

[@sambitmishra1500]

@@paretos-com It revolves around trying to optimise parameters in a CFD solver called PyFR.
Since the simulations take a long time, optimising the parameters using Bayesian Optimisation seems like the way to go.

[@sambitmishra1500]

@@paretos-com If there is any possibility, please make a video on Integer optimisation using Botorch. I can't seem to find how to do it. I just know that I must use botorch.optim.optimize.optimize_acqf_discrete.

[@xXOmbra8Xx]

Thank you, thank you, thank you! Very helpful!

[@taofeektejuosho3789]

Great video!
I would like you to go deeper with the acquisition functions.

[@gregoryhabermeyer8850]

Great Video !

[@jean-lucjannink9221]

0. Thanks!
1. Would be nice if you posted the code you end up with somewhere
2. If I'm not mistaken, your objective function here is exact: there is no stochastic noise. The next step that would interest me most would be what to do when the objective function is noisy. Can you then still use SingleTaskGP and ExactMarginalLogLikelihood?
3. Thanks!

Good video! For deep learning hyperparameter optimization, do you think BoTorch suffices, or is the Ax library more convenient? Also, do you prefer raw BoTorch instead of Ax, and why? Will you do some video on Ax?

[@leochan3265]

it is very helpful

[@navidhaghmoradi6606]

Thanks a lot for the Video. it was really good and well explained for each step. Do you have the second video which includes the multiobjective optimization coding by Botorch?

[@gabrieldariouribeguerra9117]

Good video, I have try to run this algorithm with constraints in the optimize_acqf, but I cant get any result, can you explain how I can use the inequality consraint into the optimize?

[@Data_In_real_world]

just a little question i want to optimize multiple parameters in black box function in bayesian optimization can you please make a video on that

[@shift_escape]

Hi, great tutorial! I just implemented a version of this for my research thanks to your video.
If you're planning on continuing this series, could you elaborate a bit more on applying constraints for this optimization? For an instance, I have a requirement in which if candidate = [x1, x2, x3], then sum x1 + x2 + x3

[@shift_escape]

@@paretos-com Thanks. In your opinion does this sort of optimization require normalization of inputs before training the surrogate model?

[@EpicMarcTV]

Thank you very much for this tutorial, it was very helpful. Do you know if there are other tutorials or resources that use BoTORCH for Bayesian Optimisation but using multiple inputs/parameters and using the MultiTaskGP instead of SingleTaskGP ?

[@nandni22jain]

Hi, did you find any tutorial or resources related to it. It will be very helpful if you can share it.

[@babyzupa1837]

Can you give example about Convolutional neural network featuring bayesian optimization?

[@hansergonzalez6646]

Do you guys think it is better to use Pytorch or Tensorflow for BO?

[@Data_In_real_world]

i hAVE LITTLE QUERY here you have prepared a random data but i know the data boundaries and i want to use them in my objective function how can i do this

[@atulwithsmile]

can botorch do multi-fidelity optimisation/modelling?