The Notebook LM allows only 20 papers in each notebook. So on the whole dataset its performance would be similar to Gemini Pro. But I guess it would be interesting to test on particular paper examples. How do they score the quality of responses? Manually? For people not familiar with QUASPER (I also had to look it up): paperswithcode.com/dataset/qasper "QASPER is a dataset for question answering on scientific research papers. It consists of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers."
Great job of explaining and comparing. I look foreward to your next videos on NBLM and LLM and ChatGpt's
I wonder how it would perform on the Qasper benchmark
The Notebook LM allows only 20 papers in each notebook. So on the whole dataset its performance would be similar to Gemini Pro.
But I guess it would be interesting to test on particular paper examples.
How do they score the quality of responses? Manually?
For people not familiar with QUASPER (I also had to look it up):
paperswithcode.com/dataset/qasper
"QASPER is a dataset for question answering on scientific research papers. It consists of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers."