Mastering LLM Evaluation: Metrics and Methodologies

In this final lab, you will focus on evaluating large language models (LLMs) programmatically.
You will learn to compare LLMs using methods like blue score and Rouge score, but these methods have limitations. The lab introduces a more effective approach: using a third language model as a judge to compare LLMs.
The scores are being assigned based on comparing responses from different models. GPT-3.5 is used as the judge in this case, but any model could serve. The lab concludes by encouraging you to further explore model evaluation, watch additional lectures on H2O LLM evaluation, and consider taking a quiz for certification.
Feel free to take a look at a more detailed presentation of our LLM EvalGPT app made by Andreea Turcu at the following link: Introducing H2O LLM EvalGPT
Instructions to access H2O.ai EvalGPT: You can gain access publicly to H2O.ai EvalGPT via the following link: evalgpt.ai
Please be aware that the h2oGPT exercise featured in the current video (found in the One Step Further section of LAB 4 accompanying this notebook) is solely for demonstration purposes. The endpoint used in the demonstration will not function for you.
You can access the influencers_data.csv file at the following link: LinkedIn Influencers' Data
The Link for the Python LAB 5 can be found here: LAB 5 - Evaluation.ipynb
To access h2oGPT for learning purposes, visit our h2oGPT platform using the link provided: gpt.h2o.ai.
You'll have open access using the credentials:
username: guest
password: guest