Analyzing the Impact of Various Indexing Techniques on Retrieval-Augmented Generation (RAG) Performance in Closed-Domain Question Answering

Abstract

Rapid and recent advancements in large language models (LLMs) have become the driving force of many Natural Language Processing (NLP) applications, revolutionizing tasks such as Question Answering (QA) with the aim of improving human-computer interaction. LLMs have been explored to be capable of predicting reasonably good answers for provided questions using its ability to memorize information seen during training of the model. But to what extent can these models remember training data? Retrieval-Augmented Generation (RAG) augments LLMs with knowledge to avoid relying on their memorization capabilities by combining generative capabilities of LLMs with retrieval-based methods to enhance answer accuracy and relevance of answers. With or without RAG, the aim is to mitigate the generation of plausible but non factual responses by LLMs which is hallucination. RAG has demonstrated significant performance in reducing hallucination and improving accuracy, speed and relevance of LLM generation in QA tasks. This research evaluates how different indexing methods such as BM25, DPR, and hybrid techniques affect RAG system performance in closed-domain QA, examining accuracy of retrieval and generation using performance metrics such as hit rate, precision, faithfulness and f1-score. The study also considers passage length variability and domainspecific adaptation across multiple datasets and domains with the hypothesis that focusing on a single domain can enhance system performance. Experiments are carried out on multiple datasets within both closed and open domains, using pre-trained models. This study explores optimal indexing strategies for RAG systems, balancing accuracy and efficiency. Results show that increasing passage length and retrieved samples improves hit rate and recall but reduces precision. The best performance achieved was 91% Hit-rate@9 and 93% relevancy using the BioASQ dataset.