Відео

Screenbrush.

This notebook uses Marimo, you can find a livestream on that tool as well as a podcast with one of the creators on our YT channel.

Thanks!

Not sure what you mean with "modal is 👽no more", could you elaborate?

@@probabl_ai Sure Modal is alien no more.

We like to think so, yeah :)

Thank you! Cheers!

Thanks!

My pleasure

Enjoy!

Thank you! Cheers!

Will do!

your channel is amazing , I love your content especially cause provide so much clarity in comparison to what we learn in uni.

@@mberoakoko24 Happy to hear it!

love your videos Vincent, keep them coming <3 <3

Also, wanted to do the internship, won't say I was the perfect candidate, would have applied. Now it is gone :') :')

Totally agree - Sci-kit learn is NOT easy to learn. A GOOD resource to learn the most important parts would be fantastic. And tell us what REALLY are the vital bits to understand...it's big...

I agree. Its not about the price but the annoying thing of certification expiry. This is the reason this certification would never be popular and eventually a newer/better package would replace scikit. This is missed opportunity.

We appreciate the feedback but would also like to clarify a few things here. 1. The reason why there is an expiry is because the best practices change over time. The library is evolving (2 years represent 4 releases), and so is knowledge. We want to make sure you're up to date for good practices. Just to give one example: a few years ago we might've recommended imbalanced-learn for imbalanced classification use-cases but have since dropped this recommendation because calibration tends a better avenue for those kinds of problems. Our recent podcast dives into this topic to those who are interested. It is because of this "techniques have to be re-evaluated"-phenomenon that we also set an expiry date on the certification. If the recommendations changes over time, the certification should also get an update. Whenever somebody wants to renew their certification they can do so at a reduced price and with a shorter exam. We're still working out the details of this, but we don't plan on charging the same full amount during a certification renewal/update. 2. We are working on a good resource for the certification exam as we understand that it can be frustrating to prepare for an exam without having a clear guide on what is expected. For now we have the official scikit-learn MOOC (which is a for-free resources that we invested in), as well as the details on the certification site that describe the expectations. We are working on adding material that is more focussed on the certification. Note that the video description contains some extra links now to these aforementioned resources (adding them in the comments usually triggers the YT anti-spam mechanism). 3. We hope that people can appreciate that we do our best to make this certification accessible and affordable but that we also hope to develop a stream of income that helps fund the scikit-learn project. There are many other courses and certification providers out there that easily charge ten times our listed price while they contribute *nothing* to the maintenance of the project.

Technically, if you have access to a Jupyter enviornment you will always have access to the docstring via help(thing). Dunno about the rest of the docs though.

Alas, in principle folks only have access to the scikit-learn doc, not pandas. That said, we try to avoid making really elaborate pandas/numpy stuff. That said, and as mentioned below, you still have access to docstrings from Jupyter.

Oh it is a super cool project!

A cheap drawing tablet and an app called screenbrush.

You might enjoy this TIL on the topic: koaning.io/til/optimal-seeds/ That said, lets think for a moment what the `random_seed` is meant for. The reason that it exists is to allow for repeatability of the experiment. When we set the seed, we hope to ensure that future runs give us the same result. That's a useful mechanism, but notice how it is not a mechanism that is meant as a control lever of an algorithm. That's why I find there is something dirty about finding an "optimal" algorithm by changing the seed value. It's meant to control consistency, not model performance. There is a middle path though: use randomized search. Adding a random seed here is "free" in the sense that it does not blow up the search space but it might allow you to measure the effect of randomness in hindsight. Does this help?

@@probabl_ai Makes sense to me! Thanks.

You can follow the official Scikit-learn MOOC. The Associate certification level is based on it.

Thank You!

Happy to hear it!

Out of curiosity, did you look at the hyperparameters and did these also show regions of better performance?

It's more that you are battling the fact that luck may work against you.

How so? Is the optimisers curse a DnD thing?

​@@probabl_ai I am not aware of one, but the name certainly sounds like it could be referring to a D&D optimizer. As someone with a lot of science and D&D content showing up in my feed, I honestly half thought it was related too. Haha.

The original hash has *a lot* of entropy. The odds of getting a collision there is very small. But we reduce that entropy by a lot when we introduce a set size for the vocab. In the example here I take 10_000, which is a tiny set of values compared to the original possible values that the hash could give. The concern isn't so much that the original integer from the hash can collide, rather that the (hash_int % vocab) size might. When you look at it this way, do you still have the same concern? Now that I think more of it, you are right to be critical of what I proposed here. On reflection, I think that we can (and should) do much better than a sliding window because this still introduces a relationship from (window a) and (window a + 1). Instead it might be better to just calculate a big hash and to chop it up into non-overlapping segments.

​@@probabl_ai I think I have more holes in my knowledge than I realised, I'll learn more about hashes

@@cunningham.s_law No worries, feel free to keep asking questions because it is totally possible that I have made a mistake.

MMH3 hash seems to only generate 16 bytes (128 bits). Isn't sliding window kinda limited in that case?

@@Mayur7Garg as mentioned before, the sliding window is indeed suboptimal. But the number of bits could still work for a bloom vectoriser. You need a few thousands of buckets, not a million.

I've even seen this happen on convex models actually. Granted, when this happened it was related to the `max_iter` variable being too low so it isn't converging properly. Bit of an edge case, but devil is in the details.

It's not a wrong perspective, but the phenomenon that is being described is the counterintuitive situation where adding more hyper-parameters may make the "best performance" statistic less reliable. Hypothesis testing also tends to be a bit tricky in the domain of hyper-parameters too, mainly around the question of what underlying distribution/test you can assume.

If the datasource is open, say on Github, then you should be able to fetch it as you would normally in Python. However, if you need private keys then it is a different story. I would not do anything with private data here because everything you share with Marimo now is public on the internet.

@@probabl_aiNoted. Thank you.

Typically the hyperparameters do not have "gradients". To use Pytorch as an analogy, the weights of a neural network might be differentiable, but the amount of dropout isn't. Not to mention that a lot of algorithms in scikit-learn aren't based on gradient algorithms.

Statistical tests can certainly be a good idea for sure, but confirming the results with a visual can be a nice insurance policy either way.

Factor analysis finds linear relationships, which is good, but there are important nonlinear relationships between hyperparameters, especially for complex models and/or datasets (learning rate vs batch size for neural networks is one common example of this).

@@joshuaspeckman7075 Good point!

A uniform distribution might also be suspicious. But the main lesson is that you can indeed apply a healthy amount of doubt when looking at these stats. They can give you an optimistic version of reality and it can be very tempting to get fooled by good numbers.

Goodhearts law is indeed something to look out for. When a metric becomes a target, it is not a great metric anymore.

I might be a bit careful there. By using a random seed you will sample from a distribution that is defined by the hyperparams. Change the hyperparams and you will have another distribution and I don't know if you can claim anything general about the difference of these two distributions upfront.

@probabl_ai As I said fixed hyper params. I am only updating the random seed. The idea is to establish some sort of likelihood of the model's score distribution just due to chance. So instead of saying that the model score is 0.7 for some specific random seed value, I can say something like model score is 0.7±0.1 where the latter is the std of the scores or that the model scores over 0.6 for 95% of random seed values.

Is a random state similar to initial starting point? So with a bad random state you end up in a local minima

In order to make a point sometimes you first have to show how not to do it, that way it is easier to motivate the alternative route. Happy to hear you like the content!

d0h! My bad! Just added a link to the notebook in the shownotes of this video.

There is certainly a balance there yeah. Not everyone has unlimited compute. The simplest answer is to just try and remain pragmatic. Make sure you invest a little bit in visuals and don't trust your numbers blindly. Really think about what metric matters and really try to think about what problem needs to be solved in reality.

For deep models, initializing weights sampled from a relatively small variance Gaussian distribution has been shown to give faster convergence. Andrew Karpathy doesn’t touch on it in his making GPT video, but if you go to the GitHub code you can see the change. Also, adding a weight-size penalty to the loss can encourage the model to come up with more general parameters, but this can be very delayed (grokking). I have seen several gradient and double descent methods that basically “pick up the signal” early, though. Remember for nontrivial tasks and good architecture this is more of icing on cake tho.