A few questions/suggestions: 1. Shouldn't the laundry list of proposed metrics be prioritized by some logidc and a North Star metric selected? I did not hear a summary/recommendation. 2. Shouldn't the North Star metric be related or be the same as the NS for FB Feed, which is time spent? The candidate does a good job explaining the value of the reactions to the user, e.g. a less fatiguing browsing experience, but there is no connection between the value and the NS metric. 3. Finally, shouldn't the candidate pick some counter-metrics to ensure that the proposed metrics are not "gamed" or are not obscuring the real picture. I think, the interviewer captured this by suggesting to look at the segments of the users who adopt the reactions feature, in order not to obscure a spike in reactions per post per user coming from a small subset of users. Overall, I think, the interviewer feedback was on the money.
Another interesting question which comes to mind after watching this is why does not UA-cam add reactions for all their videos? What would they like to measure? Define success for it?
That's an interesting question. So your question is "Should UA-cam support reactions?" I can see Google asking that question in a PM interview. I'd love to see how people would attempt to answer it.
Good but it couldve been better. Definitely not a PM level. Since he is still in school and got an APM, makes sense. PM at Facebook is 5-6 years experience dude,
To really determine the success, don't you have to define a quantitative level which you think indicates success. For example, the interviewee says one of the metrics is number of engagement per user to see if that increased? Increased by how much?
Thanks for posting your feedback.The amount really depends on the resources, the velocity you're going after, and the level of product maturity. In a real job, yes you have to have a number in mind but in the interview setting, it will be hard for the interviewee to come up with a number. And the interviewers are ok if you don't come up with a number. If they ask for it, you can explain the information you need to determine that number.
Firstly, I guess the metric is not "number of engagement per user" but the metric is "user engagement" - number of users engaged with our product. Second, this being an interview the interviewer is certainly not looking at a precise answer which is accurate to the decimal point, the interviewee is evaluated based on his thought process. To say, the metric here is to track how one whittles down and arrives at his answer and certainly not to gauge the accuracy of it. Do you agree Bijan?
Thanks for sharing. Appreciated the way clarification was sought. Shouldn't the success metrics have been categorized by user segments from the outset? Success for advertisers might not necessarily mean a great user experience for a user not wanting ads. Diving into the "Definition of Metrics", shouldn't the metrics proposed be led by a hypothesis and coupled with a possible test framework? Assuming Reactions' success is defined by ad conversion gains (CVR), then shouldn't "quality of engagement" have been framed as (in brief) "My hypothesis is there's a strong correlation between users giving positive Reactions and downstream CVR. To test this, we could structure an A/B test where the treatment group is exposed to only more "positive" Reaction selections with the aim of "driving positivity" and therefore CVR". But do we really want to limit the freedom of Reactions? This could be paired with measuring the test's effect(s) on user engagement (e.g. screen time on FB / # of subsequent Reactions or Posts in x days).
thanks babayaga. Here are my answers to your questions. Shouldn't the success metrics have been categorized by user segments from the outset? Your goal / objective will take care of ensuring that you're focusing on the right user group here. shouldn't the metrics proposed be led by a hypothesis and coupled with a possible test framework? Not necessary in metrics questions. You can do that in AB testing interview questions. but in metrics questions, the interviewer wants to see you describe at a high level how you'd measure the success of the product. hope it helps. thanks for your comments.
A few questions/suggestions:
1. Shouldn't the laundry list of proposed metrics be prioritized by some logidc and a North Star metric selected? I did not hear a summary/recommendation.
2. Shouldn't the North Star metric be related or be the same as the NS for FB Feed, which is time spent? The candidate does a good job explaining the value of the reactions to the user, e.g. a less fatiguing browsing experience, but there is no connection between the value and the NS metric.
3. Finally, shouldn't the candidate pick some counter-metrics to ensure that the proposed metrics are not "gamed" or are not obscuring the real picture. I think, the interviewer captured this by suggesting to look at the segments of the users who adopt the reactions feature, in order not to obscure a spike in reactions per post per user coming from a small subset of users.
Overall, I think, the interviewer feedback was on the money.
If you have any questions or feedback about this mock interview, feel free to post them here and we'll answer them :)
Another interesting question which comes to mind after watching this is why does not UA-cam add reactions for all their videos? What would they like to measure? Define success for it?
That's an interesting question. So your question is "Should UA-cam support reactions?" I can see Google asking that question in a PM interview. I'd love to see how people would attempt to answer it.
Good but it couldve been better. Definitely not a PM level. Since he is still in school and got an APM, makes sense. PM at Facebook is 5-6 years experience dude,
Thank you for the feedback and sharing your view. Really appreciate it:)
To really determine the success, don't you have to define a quantitative level which you think indicates success. For example, the interviewee says one of the metrics is number of engagement per user to see if that increased? Increased by how much?
Thanks for posting your feedback.The amount really depends on the resources, the velocity you're going after, and the level of product maturity. In a real job, yes you have to have a number in mind but in the interview setting, it will be hard for the interviewee to come up with a number. And the interviewers are ok if you don't come up with a number. If they ask for it, you can explain the information you need to determine that number.
Firstly, I guess the metric is not "number of engagement per user" but the metric is "user engagement" - number of users engaged with our product. Second, this being an interview the interviewer is certainly not looking at a precise answer which is accurate to the decimal point, the interviewee is evaluated based on his thought process. To say, the metric here is to track how one whittles down and arrives at his answer and certainly not to gauge the accuracy of it. Do you agree Bijan?
Thanks for sharing. Appreciated the way clarification was sought.
Shouldn't the success metrics have been categorized by user segments from the outset?
Success for advertisers might not necessarily mean a great user experience for a user not wanting ads.
Diving into the "Definition of Metrics", shouldn't the metrics proposed be led by a hypothesis and coupled with a possible test framework?
Assuming Reactions' success is defined by ad conversion gains (CVR), then shouldn't "quality of engagement" have been framed as (in brief) "My hypothesis is there's a strong correlation between users giving positive Reactions and downstream CVR. To test this, we could structure an A/B test where the treatment group is exposed to only more "positive" Reaction selections with the aim of "driving positivity" and therefore CVR".
But do we really want to limit the freedom of Reactions? This could be paired with measuring the test's effect(s) on user engagement (e.g. screen time on FB / # of subsequent Reactions or Posts in x days).
thanks babayaga. Here are my answers to your questions.
Shouldn't the success metrics have been categorized by user segments from the outset?
Your goal / objective will take care of ensuring that you're focusing on the right user group here.
shouldn't the metrics proposed be led by a hypothesis and coupled with a possible test framework?
Not necessary in metrics questions. You can do that in AB testing interview questions. but in metrics questions, the interviewer wants to see you describe at a high level how you'd measure the success of the product.
hope it helps. thanks for your comments.
Super feedback, I learnt more from the feedback.
Glad to hear you found it valuable Abhi :)