Hi James, I have a question on NDCG or any other ranking aware metrics. How does these metrics work where you have millions of products/items. What I mean is if we have millions of items, then it means we have to first label (manually) all the million items for relevance /rank. And then when our model predicts we use NDCG. Isn't this a big drawback of NDCG. Can you please suggest what is better approach to rank if we don't have relevance labeled data. Thanks in
Biggest problem is labeling the product whether it is relevant or not. It is not possible to label each search. Meanless if you can't handle with that.
Yeah data prep as usual with ML is the hard part, if you're interested in evaluation methods for IR *without* labeled data look into online metrics for eval (and training)
Hi , I have a query If I am working on a song recommendation project by using Spotify API data set, I have used models like cosine similarity, matrix factorization, knn , Latent Semantic Analysis (LSA) model, Correlation Distance method. Now I am confused about how should I approach for evaluation metric in this system.
1. I got confused at 18:29 when predicted is a nicely increasing sequence making me think are those ranks or item ids. I was also thinking whether the len of intersection act_set & pred_set could simply be len(act_set), then i realized this example here is a very special case where act_set is subset of pred_set. If act_set contains value 9, then we can't use len(act_set) alone and the formula in video is required. 2. Similar to question nikhil goel asked in comments section 2 weeks before this, where does 13:46 actual_relevant data come from? It looks manually labelled, and this labelling occurs per query making it super unscalable?. 3. Assuming we accept manual labelling how is the 0-4 range determined? I feel like drift is a problem, when todays 4 becomes tomorrows' 3 as value judgements change, does this mean relabelling all results again? 4. I noticed some metrics aggregate across queries and k, and some are only within 1 query across k, in what scenarios do we use each? 5. I didn't expect a *relk in AP@K formula, why do we ignore certain precision at certain k? Feels like artificially increasing metrics for the sake of it, which becomes ineffective if every query does it
IN MRR, when our search result doesnt inclued the result that we want, for your example if we want to search for cats and we find only dogs, how can we calculate MRR ? can we give it a big number for exemple rank 20 for all Not included results? 1/20
Hi James, I have a question on NDCG or any other ranking aware metrics. How does these metrics work where you have millions of products/items. What I mean is if we have millions of items, then it means we have to first label (manually) all the million items for relevance /rank. And then when our model predicts we use NDCG. Isn't this a big drawback of NDCG. Can you please suggest what is better approach to rank if we don't have relevance labeled data. Thanks in
21:23 Statistically there is probably a cat in the box on image 3
Very helpful, thank you!
Your videos are impressive and very informative mate. 👌
thanks!
Biggest problem is labeling the product whether it is relevant or not. It is not possible to label each search. Meanless if you can't handle with that.
Yeah data prep as usual with ML is the hard part, if you're interested in evaluation methods for IR *without* labeled data look into online metrics for eval (and training)
Hi , I have a query If I am working on a song recommendation project by using Spotify API data set, I have used models like cosine similarity, matrix factorization, knn , Latent Semantic Analysis (LSA) model, Correlation Distance method. Now I am confused about how should I approach for evaluation metric in this system.
Amazing Explanation. So clear. Very helpful
1. I got confused at 18:29 when predicted is a nicely increasing sequence making me think are those ranks or item ids. I was also thinking whether the len of intersection act_set & pred_set could simply be len(act_set), then i realized this example here is a very special case where act_set is subset of pred_set. If act_set contains value 9, then we can't use len(act_set) alone and the formula in video is required.
2. Similar to question nikhil goel asked in comments section 2 weeks before this, where does 13:46 actual_relevant data come from? It looks manually labelled, and this labelling occurs per query making it super unscalable?.
3. Assuming we accept manual labelling how is the 0-4 range determined? I feel like drift is a problem, when todays 4 becomes tomorrows' 3 as value judgements change, does this mean relabelling all results again?
4. I noticed some metrics aggregate across queries and k, and some are only within 1 query across k, in what scenarios do we use each?
5. I didn't expect a *relk in AP@K formula, why do we ignore certain precision at certain k? Feels like artificially increasing metrics for the sake of it, which becomes ineffective if every query does it
Probably best explanation out there.
Super informative and great..thanks
Clearly explained. Thank you
What a video, hats off!
Good video !
love your videos but why do you always seem so sad
thanks! idk I'm happy I promise lol
Hi James! can u make some vedios of updating Models if we Keep on getting data(e.g Biweekly)
cool idea! I'll add to the list :)
IN MRR, when our search result doesnt inclued the result that we want, for your example if we want to search for cats and we find only dogs, how can we calculate MRR ? can we give it a big number for exemple rank 20 for all Not included results? 1/20
yes as you said - or use another metric that better fits to your scenario
@@jamesbriggs Thank you for your answer
Super nice .. Thanks
This video is great.
Very helpful ❣️