Evaluating LLM outputs

[CalderJohnson]

We seek to demonstrate that retrieval implementing backtranslation of queries and subsequent fusion of docs leads to a higher quality answer to the user's question than retrieval using the raw query.

This issue page can be used to discuss this problem as well as share updates.

I've started a brainstorming doc with some preliminary ideas for the approach: https://docs.google.com/document/d/1K0o5Se4UJ-5RubWynGUd0IrdZcYCxtcVGxFOXo-nO9k/edit?tab=t.0

[CalderJohnson]

I think I found a good benchmarking dataset for us to run our experiment on: https://huggingface.co/datasets/rungalileo/ragbench

The source paper: https://arxiv.org/pdf/2407.11005

It has queries, lists of ideal docs to retrieve, and ideal answers.

[CalderJohnson]

Reiterating what we discussed in the meeting:

1. We will run RePair on the MS-MARCO subset of the RAGBench query set (MS-MARCO is convenient since RePair is already set up to work with MS-MARCO).
2. Using our preexisting MS-MARCO indexes, we'll get top 100 docs retrieved using original queries as well as the fused top 100 original, German BT, and Chinese BT retrieved docs.
3. Calculate precision, recall, NDCG metrics at k=1, k=5, and k=10 retrieved docs (comparing original retrieved docs to ground truth, and retrieved docs after RAG fusion to ground truth).
4. Generate a response (GPT-3.5 Turbo for now) from original retrieved docs and retrieved docs after RAG fusion. Compare both to ground truth using ROUGE, BLEU, and cosine similarity of embeddings.
5. Once we're done this "proof of concept" run, we'll re-run using more LLM types and backtranslation languages for a greater degree of coverage in the study.

[CalderJohnson]

Update:

I've begun to work on this and upon reading the RAGBench paper more closely it seems I had a misconception. The RAGBench dataset is not for evaluating RAG pipelines directly, It's for finetuning an LLM (DeBERTa is mentioned in the paper) to evaluate RAG systems.

They describe a system of criteria for evaluating a RAG response, TRACe:

• UTilization: How much of that relevant content is used in the answer.
• Relevance: How much of the retrieved content is actually relevant.
• Adherence: Fewer hallucinations.
• Completeness: Whether the answer covers all relevant content.

The GPT-3.5 responses are of varying quality and do not represent an ideal response/"golden answer".

The authors provide a prompt to get an LLM to predict these TRACe headers. They show GPT-4 has ~95% agreement rate with human evaluators when predicting TRACe metrics.

They finetuned DeBERTa on their dataset and showed good results but for some reason this checkpoint is not published.

For how to adjust our approach, I think we should take an LLM with comparable power to GPT-4 but open weights so we can run it on our server free of charge. We should use the prompt the authors supply for predicting TRACe headers and show that this LLM with this prompt is accurate when compared to the ground truth in the dataset. We should then adjust step 4 of the plan above and instead of comparing to "ground truth responses" which don't exist, use our prompt to predict TRACe headers.

Apologies for the oversight. Soon we should discuss the validity of this approach.

[CalderJohnson]

I've been working on this branch on my fork: https://github.com/CalderJohnson/RePair/tree/benchmark

Currently, I have the code in place to use the methodology described in the paper to get an LLM to evaluate a RAG response (ie. Predict the TRACe scores).

I can only run very small models on my personal machine (without massive wait times) so currently the models are struggling to evaluate the outputs correctly. However I did my best to follow the exact methodology of the paper so I'm optimistic that with larger models on the lab's server we'll be able to get more accurate results.

Everything is modular so once good results are obtained (ie. A high degree of accuracy when compared to the ground truth values in RAGBench) it should be smooth to apply the evaluation approach to benchmarking RePair's RAG pipeline and go through with our experiment.