Class Introduction

Function

The RAGEvaluator class provides a unified interface for RAG system assessment, offering support for custom evaluation metrics and multilingual prompt adaptation (such as English and Chinese).

Prototype

from mx_rag.evaluate import RAGEvaluator
RAGEvaluator(llm: LLM, embeddings: Embeddings)

Parameters

Parameter	Data Type	Required/Optional	Description
llm	LLM	Required	Communicates with an LLM. It must be inherited from langchain.llms.base.LLM. For details, see Text2TextLLM.
embeddings	Embeddings	Required	Vectorizes user questions during embedding evaluation. The value must be one of TextEmbedding or TEIEmbedding instances.

Example

from datasets import Dataset

from mx_rag.embedding.service import TEIEmbedding
from mx_rag.llm import Text2TextLLM, LLMParameterConfig
from mx_rag.utils import ClientParam
from mx_rag.evaluate import RAGEvaluator

llm = Text2TextLLM(
    base_url="https://ip:port/v1/chat/completions",
    model_name="model name",
    llm_config=LLMParameterConfig(temperature=0.1, top_p=0.8),
    client_param=ClientParam(ca_file="/path/to/ca.crt"),
)
embeddings = TEIEmbedding(url="https://ip:port/embed", client_param=ClientParam(ca_file="/path/to/ca.crt"),)

sample_queries = [
    "Who proposed the theory of relativity?"
    "Who won the Nobel Prize twice for her research on radioactivity?"
    "What are Isaac Newton's contributions to science?"
]
expected_responses = [
    "Albert Einstein proposed the theory of relativity, which changed our understanding of time, space, and gravity."
    Marie Curie was a physicist and chemist who conducted groundbreaking research on radioactivity and won the Nobel Prize twice.
    Isaac Newton formulated the laws of motion and universal gravitation, laying the foundation for classical mechanics.
]
retrieved_contexts = [["Albert Einstein's theory of relativity completely changed our understanding of time, space, and gravity.",
                       "Albert Einstein's theory of relativity revolutionized our understanding of time, space, and gravity.",
                       "Albert Einstein redefined our views of time, space, and gravity by proposing the theory of relativity."],
                      ["Marie Curie was an outstanding physicist and chemist who made groundbreaking contributions to radioactivity research and won the Nobel Prize twice."
                       "As a physicist and chemist, Marie Curie made groundbreaking contributions to radioactivity research and won the Nobel Prize twice."
                       "Marie Curie was a famous physicist and chemist who won the Nobel Prize twice for her groundbreaking work in radioactivity research."],
                       ["Isaac Newton proposed the laws of motion and universal gravitation, laying the foundation for the development of classical mechanics."
                       "Isaac Newton made foundational contributions to the establishment of the classical mechanics system by explaining the laws of motion and universal gravitation."
                       "Isaac Newton's laws of motion and universal gravitation laid the cornerstone for the formation of classical mechanics."],
                      ]
# Model answers
responses = ["Albert Einstein proposed the theory of relativity",
             "Marie Curie",
             "He explained the laws of motion and universal gravitation, laying the foundation for classical mechanics."]
dataset = []
for query, contexts, response, reference in zip(sample_queries, retrieved_contexts, responses, expected_responses):
    dataset.append(
        {
            'user_input': query,
            'response': response,
            'retrieved_contexts': contexts,
            'reference': reference
        }
    )
evaluation_dataset = Dataset.from_list(dataset)
evaluator = RAGEvaluator(llm=llm, embeddings=embeddings)
metrics = ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]
result = evaluator.evaluate(
    metrics=metrics,
    dataset=evaluation_dataset.to_dict(),
    language="chinese"
)
print(result)

Parent topic: RAGEvaluator