time module in Python. Here's how you can do it:LlamaIndex library to load your dataset and create an index. The time it takes to create this index includes the time it takes to embed the whole dataset. Here's an example: import time
from llama_index.core import download_loader
from llama_index.readers.wikipedia import WikipediaReader
from llama_index.core import VectorStoreIndex
loader = WikipediaReader()
documents = loader.load_data(pages=["Berlin"]) # replace with your dataset
start_time = time.time()
index = VectorStoreIndex.from_documents(documents)
end_time = time.time()
print("Total time elapsed for embedding the whole dataset: {}".format(end_time - start_time))
get_query_embedding method from the embedding model you are using. Here's an example: query = "Example query."
start_time = time.time()
query_result = embed_model.get_query_embedding(query) # replace embed_model with your model
end_time = time.time()
print("Total time elapsed for embedding a single query: {}".format(end_time - start_time))
aget_agg_embedding_from_queries method. This method is asynchronous and gets aggregated embedding from multiple queries. Here's an example:qa_dataset, you can iterate over the queries and use the time module in Python to measure the time it takes to embed each query. Here's an example:import time
from llama_index.core.evaluation import EmbeddingQAFinetuneDataset
# Assuming you have your qa_dataset
# qa_dataset = generate_question_context_pairs(nodes, llm=llm, num_questions_per_chunk=2)
queries = qa_dataset.queries.values()
for query in queries:
start_time = time.time()
query_embedding = embed_model.get_query_embedding(query) # replace embed_model with your model
end_time = time.time()
print("Time elapsed for embedding the query '{}': {}".format(query, end_time - start_time))embed_model with the actual model you are using for embedding. This model should have a method get_query_embedding that takes a query as input and returns the embedding.generate_question_context_pairs function does not create embeddings. This function is used to generate a set of (question, context) pairs over a given unstructured text corpus. It uses a language model (LLM) to auto-generate questions from each context chunk. The output is an EmbeddingQAFinetuneDataset object which contains a set of ids mapping to queries and relevant document chunks, as well as the corpus itself.from llama_index.core.evaluation import generate_question_context_pairs, EmbeddingQAFinetuneDataset # Assuming you have your nodes and llm # nodes = ... # llm = ... qa_dataset = generate_question_context_pairs(nodes, llm=llm, num_questions_per_chunk=2)
qa_dataset, you can use the RetrieverEvaluator class from the LlamaIndex library. Here's an example of how you can do it:from llama_index.core.evaluation import RetrieverEvaluator
# Assuming you have your retriever
# retriever = ...
# Define the metrics
metrics = ["mrr", "hit_rate"]
# Create the evaluator
retriever_evaluator = RetrieverEvaluator.from_metric_names(metrics, retriever=retriever)
# Evaluate the dataset
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
# Calculate the metrics
metric_dicts = []
for eval_result in eval_results:
metric_dict = eval_result.metric_vals_dict
metric_dicts.append(metric_dict)
# Calculate the mean of the metrics
full_df = pd.DataFrame(metric_dicts)
hit_rate = full_df["hit_rate"].mean()
mrr = full_df["mrr"].mean()
print(f"Hit Rate: {hit_rate}, MRR: {mrr}")retriever placeholder with your actual retriever. This retriever should be an instance of a class that implements the Retriever interface from the LlamaIndex library.qa_dataset, you can iterate over the queries and use the evaluate method of the RetrieverEvaluator class from the LlamaIndex library. Here's an example of how you can do it:from llama_index.core.evaluation import RetrieverEvaluator
# Assuming you have your retriever
# retriever = ...
# Define the metrics
metrics = ["mrr", "hit_rate"]
# Create the evaluator
retriever_evaluator = RetrieverEvaluator.from_metric_names(metrics, retriever=retriever)
# Iterate over the queries
for query_id, query in qa_dataset.queries.items():
# Get the expected documents for this query
expected_docs = qa_dataset.relevant_docs[query_id]
# Evaluate the query
eval_result = retriever_evaluator.evaluate(query, expected_docs)
# Print the metrics for this query
print(f"Query: {query}, Metrics: {eval_result.metric_vals_dict}")retriever placeholder with your actual retriever. This retriever should be an instance of a class that implements the Retriever interface from the LlamaIndex library.qa_dataset and evaluate each one using the RetrieverEvaluator. Here's how you can do it:from llama_index.core.evaluation import RetrieverEvaluator
# Assuming you have your retriever
# retriever = ...
# Define the metrics
metrics = ["mrr", "hit_rate"]
# Create the evaluator
retriever_evaluator = RetrieverEvaluator.from_metric_names(metrics, retriever=retriever)
# Iterate over the queries
for query_id, query in qa_dataset.queries.items():
# Get the expected documents for this query
expected_docs = qa_dataset.relevant_docs[query_id]
# Evaluate the query
eval_result = retriever_evaluator.evaluate(query, expected_docs)
# Print the metrics for this query
print(f"Query ID: {query_id}, Metrics: {eval_result.metric_vals_dict}")retriever placeholder with your actual retriever. This retriever should be an instance of a class that implements the Retriever interface from the LlamaIndex library.RetrieverEvaluator class from the LlamaIndex library, it returns an EvaluationResult object for each query. This object contains the values of the metrics (like MRR and hit rate) for that specific query.from llama_index.core.evaluation import RetrieverEvaluator
# Assuming you have your retriever
# retriever = ...
# Define the metrics
metrics = ["mrr", "hit_rate"]
# Create the evaluator
retriever_evaluator = RetrieverEvaluator.from_metric_names(metrics, retriever=retriever)
# Iterate over the queries
for query_id, query in qa_dataset.queries.items():
# Get the expected documents for this query
expected_docs = qa_dataset.relevant_docs[query_id]
# Evaluate the query
eval_result = retriever_evaluator.evaluate(query, expected_docs)
# Print the metrics for this query
print(f"Query ID: {query_id}, Metrics: {eval_result.metric_vals_dict}")eval_result.metric_vals_dict is a dictionary that contains the values of the metrics for the current query. You can access the MRR and hit rate for the current query using eval_result.metric_vals_dict['mrr'] and eval_result.metric_vals_dict['hit_rate'], respectively.get_nodes_from_documents method from the node_parser object does not convert the documents into embeddings. This method is used to parse the documents into nodes, where each node represents a chunk of text from the document.from llama_index.core.node_parser import SentenceSplitter node_parser = SentenceSplitter(chunk_size=512) nodes = node_parser.get_nodes_from_documents(documents)
documents is a list of documents, and node_parser is an instance of SentenceSplitter, which is a type of node parser. The get_nodes_from_documents method splits each document into chunks of a specified size (512 in this case), and returns a list of nodes, where each node represents a chunk.nodes list. for idx, node in enumerate(nodes):
node.id_ = f"node_{idx}"enumerate(nodes) is a Python built-in function that allows you to loop over a list (in this case, nodes) and have an automatic counter (idx in this case). node.id_ = f"node_{idx}" is setting the ID of each node to be a string that starts with "node_" followed by the index of the node in the list. For example, the first node in the list will have an ID of "node_0", the second node will have an ID of "node_1", and so on.for idx, node in enumerate(nodes):
node.id_ = f"node_{idx}"nodes list is assigned a unique ID in the format of "node_{index}", where {index} is the position of the node in the list.