One of the best videos I’ve seen where I get a good insight into the thought process of a data scientist in terms of the whole process of applying ML to a business function from model selection (considering the pros and cons) to data collection and deployment considerations. Please do more of such videos!
Helpful mock interview! I felt like a reasonable question at the beginning would have been what does the historical data look like? That would have eliminated him having to make a bunch of assumptions that impacted his decisions, such as are the classes imbalanced, are there lots of features/samples.
AUC is not really a good metric for unbalanced datasets. The x-axis metric depends on the number of negatives. Better use a F-ROC curve or precision-recall curve and the area under the precision-recall curve (also called average precision).
Great interview and answer! Just felt the first 8 min is not really related to the question asked if interview time is limited maybe straight to the point?
first 8 min is gold, in terms of interview technique.. establishing context, reaffirming the understanding of the request gives yourself time to think about the prob... as well as effectively communicate with the stakeholder (here the interviewer). Particularly important if you are interviewing with a company that wants ds to have client facing skills
@16:47, during the shadow model, not sure if why it needs to be done, ensure the distribution of our real-time prediction is equal to training distribution? can someone help elaborate that? much appreciated!
I believe the concern here is data drift, or just generally making sure that your model is predicting for a dataset with a similar distribution to the one that the model was trained on. For example, if the model was trained 5 months ago, the distributions in the underlying dataset that it was trained on have likely changed and so the model's learnings are outdated. This becomes less so of an issue with real time systems where the model is being re-trained regularly, but comparing the distribution of features and the output variable that the model was trained on to the distribution of features and output variable that the model is predicting on is still useful because it gives you confidence that the model "learned" on a similar dataset.
Interpretability nowadays has nothing to do with model selection thanks to some open source packages like Shap and Lime. Even with so called black box model, individual prediction and directional contribution are now both explainable.
No cross questioning, no product level metric discussion, no revenue saved simulation for the past if model have been deployed before, no AUC drawbacks discussed, no feature engineering discussion, no different set of input data discussion, no sampling to address imbalance - NOTHING !! Really trivial and basic from both the sides.
To an extent I agree with your concerns, although the candidate did prefer PR AUC over standard AUC. I think Jay should not be so laid back, to simulate a real interview.
impressive! He is so organized and thorough. Probably the best candidate I saw on your channel
One of the best videos I’ve seen where I get a good insight into the thought process of a data scientist in terms of the whole process of applying ML to a business function from model selection (considering the pros and cons) to data collection and deployment considerations. Please do more of such videos!
Helpful mock interview! I felt like a reasonable question at the beginning would have been what does the historical data look like? That would have eliminated him having to make a bunch of assumptions that impacted his decisions, such as are the classes imbalanced, are there lots of features/samples.
1 of the best candidates I've seen so far
AUC is not really a good metric for unbalanced datasets. The x-axis metric depends on the number of negatives. Better use a F-ROC curve or precision-recall curve and the area under the precision-recall curve (also called average precision).
This seemed like a really real interview!
Great interview and answer! Just felt the first 8 min is not really related to the question asked if interview time is limited maybe straight to the point?
first 8 min is gold, in terms of interview technique.. establishing context, reaffirming the understanding of the request gives yourself time to think about the prob... as well as effectively communicate with the stakeholder (here the interviewer). Particularly important if you are interviewing with a company that wants ds to have client facing skills
@16:47, during the shadow model, not sure if why it needs to be done, ensure the distribution of our real-time prediction is equal to training distribution? can someone help elaborate that? much appreciated!
I believe the concern here is data drift, or just generally making sure that your model is predicting for a dataset with a similar distribution to the one that the model was trained on. For example, if the model was trained 5 months ago, the distributions in the underlying dataset that it was trained on have likely changed and so the model's learnings are outdated. This becomes less so of an issue with real time systems where the model is being re-trained regularly, but comparing the distribution of features and the output variable that the model was trained on to the distribution of features and output variable that the model is predicting on is still useful because it gives you confidence that the model "learned" on a similar dataset.
Interpretability nowadays has nothing to do with model selection thanks to some open source packages like Shap and Lime. Even with so called black box model, individual prediction and directional contribution are now both explainable.
You make it sound like explainability is solved. I wish it were true!
No cross questioning, no product level metric discussion, no revenue saved simulation for the past if model have been deployed before, no AUC drawbacks discussed, no feature engineering discussion, no different set of input data discussion, no sampling to address imbalance - NOTHING !! Really trivial and basic from both the sides.
To an extent I agree with your concerns, although the candidate did prefer PR AUC over standard AUC. I think Jay should not be so laid back, to simulate a real interview.
from someone who is the interviewer at Square, this is definitely what they want to test for. Totally misleading directions😂
Can you clarify ? Sounds like it is definitely what they do NOT want to test, given the totally misleading directions.