Reduzindo alucinações e aprimorando a qualidade de respostas em sistemas alimentados por large language models através de re-ranking de documentos para aplicações em domínios específicos

Abstract

This work addresses the problem of hallucinations in Large-Scale Language Models (LLMs) by improving Retrieval Augmented Generation (RAG) systems. Specifically, it proposes a document re-ranking model, based on BERT architecture, to refine the results retrieved in a RAG system, prioritizing more relevant documents and, consequently, mitigating the generation of incorrect or unfounded information (hallucinations) by LLMs. To train and evaluate the model, an innovative dataset was created from ten Undergraduate Thesis Projects (TCCs) from the State University of Maranhão (UEMA), using Natural Language Processing (NLP) techniques for text extraction and automatic generation of questions with different levels of relevance (scores 1, 3 and 5). The BERTimbau re-ranking model was trained to classify question and document pairs according to their relevance. Experimental results demonstrate that the model achieves 92% accuracy in relevance classification and an MRR of 0.7367, indicating a significant improvement in the ordering of retrieved documents compared to an approach without re-ranking (MRR of 0.4140). The qualitative analysis illustrates the model's ability to discern between different levels of semantic relationship between questions and documents. This work contributes to the advancement of the state of the art in RAG, providing an effective method to reduce hallucinations in LLM and improve the reliability of information generated in applications in the Portuguese language domain, specifically in the context of UEMA's final course projects