I really like how you take up two roles by changing your voice, background and clothes. Smart strategy, keeping using it, it definitely made me feel very engaging.
Hi Emma, I wanted to ask something regarding the success metrics. I was thinking that for new users who signup, if they have uploaded their profile photo, there would be a higher chance that the connection requests they sent would be accepted as the profile would look more genuine. Since, one major goal of linkedin is to enable users to build connections,। think Percentage of connection requests accepted should be one success metric. The other metric that I was thinking could be number of jobs applied per user, since users also join LinkedIn to apply to job opportunities. Wanted to know your thoughts on this. Am I thinking in the right direction?
Hey Emma, thanks for your great video! For this question, if there is a follow-up question asking us how we can resolve this conflict metrics result? Then will it be the type of Diagnose the program question, and do you have some good idea in terms of answering this question?
Thank you so much for this sharing! I have learned a lot! Moreover, there are lots of metrics for performance and A/B testing. Do you have any recommendations about learning metrics and their real-world practice? Is the book you showed in the video a great resource? Thanks!
For real-word practice, I highly recommend Trustworthy Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu. For your 2nd question, I wonder why do you want to find the "sum-up"? Different metrics can be used to measure different things. I guess you mean if we can combine different metrics into a signal metric? It's possible but difficult, there are a few examples in the book in Chapter 7.
Awesome video! I would love to hear how you think about tradeoffs between different metrics before you recommend 2 success and 1 guardrail. Don't ever stop!
This is exactly what I'm looking for! I have been preparing for DS interviews. I haven't found other videos talking about the metric questions in details. I will check out the book you recommended!
For success metric, would average number of job candidate matches a good metric? I am guessing companies would like genuine candidates and the matches might go up?
It would be a good metric as long as it aligns with the goal and the data can be easily collected. For more about defining metrics you can check out this video ua-cam.com/video/SuXc5ckvlJ8/v-deo.html.
We finally have someone who aces all those DS but not PM style product sense questions!! And I am tried of 100 metrics for one case study. Love, love, love it!
Great video, and you have organized it very well throughout! one quick question, in terms of a true engagement metric would #connections per new user (e.g., by first week or first month) be more telling than #invites sent per user per day. The idea is that the user photo will also lead to more acceptances of user's invites and they are also likely to get more invites from other people.
Yes, absolutely you could use #connections. I don't think it's a more "telling" metric compared with # of invites sent. Connections is one step down the funnel after sending/receiving invites and imo we should track all of them.
Emma, your video is one of the things I look forward each week. Thanks so much for spending so much time and effort to make such awesome videos! I have recommended your channel to everyone in my network who are looking for DS jobs.
I am currently applying for a CSR position in a BPO and an interviewer asked me "What are your metrics?" Can you give an example answer please? I need to know if I responded correctly.
Hey Emma, I was one of the commenters who thought you should do these types of videos. And I'm so happy you did! I definitely think these videos are very helpful to aspiring data scientists who are interviewing at these big tech companies. Keep it up!
One very important guardrail metric here needs to be around quality of the engagement. I.e., whether the lack of a photo (or its addition) causes more invites that are *not accepted*. This could be an indicator of spam activity.
Hi Emma, me again! Saw an interesting question the other day: Suppose DoorDash is ramping up a feature that could allow merchants to set promotions (such as free delivery, 10% off) at certain period of time, how do you measure the success of the feature?
As mentioned in the video, before defining metrics, it's helpful to start with the business goal. In this problem, the goal could be to increase revenue and/or number of users, which can help you define metrics. For running A/B tests, you can refer to this video ua-cam.com/video/X8u6kr4fxXc/v-deo.html and my blog post towardsdatascience.com/7-a-b-testing-questions-and-answers-in-data-science-interviews-eee6428a8b63
I took help of gpt to come up with the success metrics for this feature with the goal of increasing revenue: During promotion: 1. Increase in Order Volume 2. Average Order Value 3. Merchant revenue growth What do you think on them?
Hey Emma, just had a metrics interview yesterday and then saw your video. Looking for another video that need to come up with many metrics to keep tracking and show how to probe using the metrics. Good job, Keep it up!
This is awesome. In the past I would just answer the question without hesitation and maybe only cover 1 straightforward point or 2, your answer is so complete and flawless! Thanks Im learning a lot here!
Hi Emma, I am a beginning learner of A/B testing. When I saw you mentioned 'guardrail metric', I though you are talking about invariant metrics for sanity check. I supposed you would say that the experiment setup is incorrect since we didn't pass sanity check. But actually you didn't treat it as invariant metric. Could you explain more about it?
Guardrail metrics are not invariant. They can change unexpectedly, which we want to check during or after the experiments. Eg. when rolling out a new feature, we want to improve say the revenue (goal metric) but don't want the latency (guardrail metric) to be worse.
Thanks Emma for these videos on Data Science interview! Do you mind making a video on why and how you made the switch from Data Scientist to Software Engineer?
Thanks for the idea! There are people ask me the same question. It may be an interesting topic but I feel it may be a bit tangent of the content of the channel so far. I want to make this channel more about how to help people get their dream jobs (by sharing interview guidance) and less about me :) But feel free to message me if you have any specific questions.
Hey Emma, while describing 2nd success metric i.e. "Usefulness of the feature", you said that "this metric should be significantly larger than that of control group" I didn't get it. It would be very helpful if you could rephrase it.
The metric should be able to measure whether the new feature is useful. And we want to see the value is much larger in the treatment group to prove it is indeed useful.
@@emma_ding Hello Emma. I have a clarification question for it. The percentage of users uploading photos includes uploading during signup+afterwards, or just during sign-up. If it's the latter, then we cannot have control group. Thanks.
I really like how you take up two roles by changing your voice, background and clothes. Smart strategy, keeping using it, it definitely made me feel very engaging.
This is among the most exceptional content I've ever encountered. Thank you Emma! Keep posting and enlightening us.
This is the best resource I have ever found! Thanks so much!
Hi Emma, I wanted to ask something regarding the success metrics. I was thinking that for new users who signup, if they have uploaded their profile photo, there would be a higher chance that the connection requests they sent would be accepted as the profile would look more genuine. Since, one major goal of linkedin is to enable users to build connections,। think Percentage of connection requests accepted should be one success metric. The other metric that I was thinking could be number of jobs applied per user, since users also join LinkedIn to apply to job opportunities. Wanted to know your thoughts on this. Am I thinking in the right direction?
So clear. That's so helpful. All of us would be happy to see more! Thanks a lot!
3:08 gold 6:56 Invites sent per user per day
I have no excuse to fail a metrics interview after watching this
Awesome content, thanks so much Emma!
Hey Emma, thanks for your great video! For this question, if there is a follow-up question asking us how we can resolve this conflict metrics result? Then will it be the type of Diagnose the program question, and do you have some good idea in terms of answering this question?
for measure user engagement, is it better to add a time constraint? eg, average invites sent out per user per day (within the first 30day)
Thank you so much for this sharing! I have learned a lot! Moreover, there are lots of metrics for performance and A/B testing. Do you have any recommendations about learning metrics and their real-world practice? Is the book you showed in the video a great resource? Thanks!
For example, you have mentioned the SUCCESS Metrics & GUARDRAIL Metrics. where could we find the sum-up of different kinds of metrics? Thank you :)
For real-word practice, I highly recommend Trustworthy Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu. For your 2nd question, I wonder why do you want to find the "sum-up"? Different metrics can be used to measure different things. I guess you mean if we can combine different metrics into a signal metric? It's possible but difficult, there are a few examples in the book in Chapter 7.
The FIRST ever channel that I won't skip any minute when viewing for all the video you upload
l
Awesome video! I would love to hear how you think about tradeoffs between different metrics before you recommend 2 success and 1 guardrail. Don't ever stop!
This is exactly what I'm looking for! I have been preparing for DS interviews. I haven't found other videos talking about the metric questions in details. I will check out the book you recommended!
You are awesome and lovely. High quality production level videos and lectures.
For success metric, would average number of job candidate matches a good metric?
I am guessing companies would like genuine candidates and the matches might go up?
It would be a good metric as long as it aligns with the goal and the data can be easily collected. For more about defining metrics you can check out this video ua-cam.com/video/SuXc5ckvlJ8/v-deo.html.
We finally have someone who aces all those DS but not PM style product sense questions!! And I am tried of 100 metrics for one case study. Love, love, love it!
Wow! Very helpful - your video gives me hope, thank you!
so useful
Great video, and you have organized it very well throughout! one quick question, in terms of a true engagement metric would #connections per new user (e.g., by first week or first month) be more telling than #invites sent per user per day. The idea is that the user photo will also lead to more acceptances of user's invites and they are also likely to get more invites from other people.
Yes, absolutely you could use #connections. I don't think it's a more "telling" metric compared with # of invites sent. Connections is one step down the funnel after sending/receiving invites and imo we should track all of them.
@@emma_ding Yep makes sense
@@hohhhhhhhhhaaaa I have the same idea. Thanks for bringing this up! Agree that making connections is one step down the funnel after sending invites.
Emma, your video is one of the things I look forward each week. Thanks so much for spending so much time and effort to make such awesome videos! I have recommended your channel to everyone in my network who are looking for DS jobs.
I am currently applying for a CSR position in a BPO and an interviewer asked me "What are your metrics?" Can you give an example answer please? I need to know if I responded correctly.
Just failed my LinkedIn interview… I wish I had seen your video one day earlier…
Hey Emma, I was one of the commenters who thought you should do these types of videos. And I'm so happy you did! I definitely think these videos are very helpful to aspiring data scientists who are interviewing at these big tech companies. Keep it up!
This is so helpful! I just failed one product sense interview. But with that failed experience and this video, I know how to improve!
One very important guardrail metric here needs to be around quality of the engagement. I.e., whether the lack of a photo (or its addition) causes more invites that are *not accepted*. This could be an indicator of spam activity.
Hi Emma, do I always have to run an experiments to find out causal effect.
Interesting video. Your postings are always helpful and what I'm looking forward to every week.
Hi Emma, me again! Saw an interesting question the other day: Suppose DoorDash is ramping up a feature that could allow merchants to set promotions (such as free delivery, 10% off) at certain period of time, how do you measure the success of the feature?
It would be great to see how you would tackle this question! Thank you~
As mentioned in the video, before defining metrics, it's helpful to start with the business goal. In this problem, the goal could be to increase revenue and/or number of users, which can help you define metrics. For running A/B tests, you can refer to this video ua-cam.com/video/X8u6kr4fxXc/v-deo.html and my blog post towardsdatascience.com/7-a-b-testing-questions-and-answers-in-data-science-interviews-eee6428a8b63
I took help of gpt to come up with the success metrics for this feature with the goal of increasing revenue:
During promotion:
1. Increase in Order Volume
2. Average Order Value
3. Merchant revenue growth
What do you think on them?
Thanks for your hard work Emma. I am looking at all your videos today and yes loving the content ❤️
🚀 Above and Beyond ! Emma Thanks for making impact !
Hey Emma, just had a metrics interview yesterday and then saw your video. Looking for another video that need to come up with many metrics to keep tracking and show how to probe using the metrics. Good job, Keep it up!
Thanks for sharing, Emma! Your video helped me a lot.
I'm sure you will help many more people like me. Keep up the good work!
Amazing Chanel!!! I wish I found it earlier. QQ how would you think uploading profile photo will help detect fraud can you elaborate?
Linkedin and facebook may have ML models checking profile photos. It will increase the difficulty registering accounts for non-legitimate purpose.
This is awesome. In the past I would just answer the question without hesitation and maybe only cover 1 straightforward point or 2, your answer is so complete and flawless! Thanks Im learning a lot here!
Hi Emma, I am a beginning learner of A/B testing. When I saw you mentioned 'guardrail metric', I though you are talking about invariant metrics for sanity check. I supposed you would say that the experiment setup is incorrect since we didn't pass sanity check. But actually you didn't treat it as invariant metric. Could you explain more about it?
Guardrail metrics are not invariant. They can change unexpectedly, which we want to check during or after the experiments. Eg. when rolling out a new feature, we want to improve say the revenue (goal metric) but don't want the latency (guardrail metric) to be worse.
Thank you for this video. Can you please upload some multiple testing videos. Thank you Emma!
Absolutely! What specific questions on multiple testing do you have in mind?
Really would like to see you to come up with both success metric and counter metric at different business phase(AARRR).
AARRR is another interesting framework. Stay tuned for future videos!
Found your videos are all extremely well-thought and applicable in interviews and day-to-day work. Thank you very much for doing this Emma!
Thanks Emma for these videos on Data Science interview!
Do you mind making a video on why and how you made the switch from Data Scientist to Software Engineer?
Thanks for the idea! There are people ask me the same question. It may be an interesting topic but I feel it may be a bit tangent of the content of the channel so far. I want to make this channel more about how to help people get their dream jobs (by sharing interview guidance) and less about me :)
But feel free to message me if you have any specific questions.
Hey Emma,
while describing 2nd success metric i.e. "Usefulness of the feature", you said that "this metric should be significantly larger than that of control group"
I didn't get it.
It would be very helpful if you could rephrase it.
The metric should be able to measure whether the new feature is useful. And we want to see the value is much larger in the treatment group to prove it is indeed useful.
@@emma_ding Hello Emma. I have a clarification question for it. The percentage of users uploading photos includes uploading during signup+afterwards, or just during sign-up. If it's the latter, then we cannot have control group. Thanks.
This is great! thanks
Really usefull. Thanks
Solid content!