Processed 6 documents in 130.94 seconds
Total number of pages processed: 6
Average time per document: 21.82 seconds
Average time per page: 21.50 seconds
Doc 4 took 16.62 seconds
  Page 1 took 14.89 seconds
Doc 2 took 39.05 seconds
  Page 1 took 38.35 seconds
Doc 6 took 38.55 seconds
  Page 1 took 37.80 seconds
Doc 5 took 93.89 seconds
  Page 1 took 85.75 seconds
Doc 1 took 129.99 seconds
  Page 1 took 128.76 seconds
Doc 3 took 130.94 seconds
  Page 1 took 129.01 seconds

File "/Users/joshuasabol/Library/Caches/pypoetry/virtualenvs/llama-app-backend-CfJQzey9-py3.11/lib/python3.11/site-packages/llama_index/llama_dataset/generator.py", line 105, in from_documents
    nodes = run_transformations(
...
File "/Users/joshuasabol/Library/Caches/pypoetry/virtualenvs/llama-app-backend-CfJQzey9-py3.11/lib/python3.11/site-packages/llama_index/node_parser/text/sentence.py", line 147, in split_text_metadata_aware
    raise ValueError(
ValueError: Metadata length (1493) is longer than chunk size (1024). Consider increasing the chunk size or decreasing the size of your metadata to avoid this.

NODE_PARSER_CHUNK_SIZE = 3000
NODE_PARSER_CHUNK_OVERLAP = 200

# set context for llm provider
gpt_35_context = ServiceContext.from_defaults(
    llm=OpenAI(model="gpt-3.5-turbo-1106",
               temperature=0.1,
               chunk_size=NODE_PARSER_CHUNK_SIZE,
               chunk_overlap=NODE_PARSER_CHUNK_OVERLAP,
    )
)

# instantiate a DatasetGenerator
dataset_generator = RagDatasetGenerator.from_documents(
    parent_nodes,
    service_context=gpt_35_context,
    num_questions_per_chunk=2,  # set the number of questions per nodes
    show_progress=True,
)

User: What is the patient's history with Dr. Woods?
**************************************************
=== Calling Function ===
Calling function: fabc5870-ce73-4ee0-9d09-c2c1158b1bdd with args: {
  "query": "chief complaint",
  "filter_key_list": ["content_info"],
  "filter_value_list": ["Dr. Woods"],
  "filter_operator_list": ["=="],
  "filter_condition": "AND"
}

# Create a new instance of SimpleDocumentStore to clear it --> DOES'T WORK
logger.info(f"Current docstore: {json.dumps(storage_context.docstore, indent=4, sort_keys=True, default=custom_serializer)}")
docstore = SimpleDocumentStore()
logger.info(f"New docstore: {json.dumps(storage_context.docstore, indent=4, sort_keys=True, default=custom_serializer)}")

# Create a new instance of SimpleIndexStore to clear it --> DOES'T WORK
logger.info(f"Current index_store: {json.dumps(storage_context.index_store, indent=4, sort_keys=True, default=custom_serializer)}")
index_store = SimpleIndexStore()
logger.info(f"New index_store: {json.dumps(storage_context.index_store, indent=4, sort_keys=True, default=custom_serializer)}")

# RESULT: The values of each are the same before and after resetting the instance

Query:  What's the X-ray details?
Creating tool: xray
Index xray_f30dc4df-d3f1-4926-9820-31d5867d0aa1 has 97 nodes
Tool xray has the description: A set of XRAY medical record for patient **************, Date of Birth: **************, that were exported from the user's electronic medical record system.
Total Nodes count: 97
Total Nodes: {...}
Filtered Nodes count: 60
BM25 Nodes count: 60
BM25 Node 1 of 60...

base_retriever Index ID: xray_f30dc4df-d3f1-4926-9820-31d5867d0aa1
Base Nodes count: 100 # ---> HERE'S THE ISSUE
Base Node 1 of 100...

UniqueViolation: duplicate key value violates unique constraint "data_pg_vector_store_pkey"
DETAIL:  Key (id)=(93735) already exists.

relevancy_result = judges["relevancy"].evaluate(
    query=example.query,
    response=prediction.response,
    contexts=prediction.contexts,
)

Query tokens: 14
Response tokens: 141
Context tokens: 38395
Remaining tokens: -30358