SQL Generation Evals: LLMs-as-a-Judge

LLM-as-a-Judge is a popular and scalable technique to evaluate LLMs for tasks including toxicity classification, sentiment classifier, and text-to-SQL tasks. However, LLM-as-a-Judge based evaluation has certain limitations and points of contention - circular methodology (using 1 LLM to evaluate another LLM) and disregard for database schema or distribution. In this session, we will discuss an experiment we designed to evaluate the performance of the LLM-as-a-Judge Eval for text-to-SQL tasks. We’ll take you through a framework to compare LLM-as-a-Judge approach with a data distribution-based Eval approach for text-to-SQL tasks. We will also discuss some interesting cases that came up in our research highlighting the pitfalls of LLM-as-a-Judge approach and some suggestions on how this approach can be enhanced to account for those limitations.