Find answers from the community

Home
Members
alvarojauna
a
alvarojauna
Offline, last seen 3 months ago
Joined September 25, 2024
a
alvarojauna
·

Json

hi! i think i found a bug or a mistake and i need help. i want to upload nodes to azure and save it in a txt. then i want to downlad and create a vector store index.

to achieve this i run the following commands:

HERE IS THE PART FROM DOCUMENTS TO NODES:
documents = SimpleDirectoryReader("/content/data").load_data()

embed_model = OpenAIEmbedding()
splitter = SemanticSplitterNodeParser(
buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model
)

nodes = splitter.get_nodes_from_documents(documents)

nodes_str = [x.dict() for x in nodes]
data=str(nodes_str)


ruta_en_dbfs = "/dbfs/nodes_architectures.txt"

Escribir la variable en el archivo en DBFS

with open(ruta_en_dbfs, 'w') as archivo:
archivo.write(data)

print("Variable guardada en DBFS con éxito.")

here are some commands of azure that i skipped and then:
with open(local_file_path, "rb") as data:
blob_client.upload_blob(data, overwrite=True)


HERE IS THE PART FROM NODES IN TXT FORMAT TO ADD IT TO VECTOR STORE INDEX (THIS IS IN ANOTHER SCRIPT):
with open(file_path, 'r', encoding='utf-8') as file:
#with open(file_path, 'r') as file:
# Leer cada línea en el archivo
my_list = file.readlines()

nodes_str = json.loads(my_list[0])
nodes = [TextNode.parse_obj(x) for x in nodes_str]

index = VectorStoreIndex(nodes_str)




This worked fine until the last big update of llamaindex. Now i found my_list[0] with ' instead of " so i run the following command: mi_lista=my_list[0].replace("'", """). But if i have in text for example microsoft"s, then the JSON format is not correct

Also, Now in nodes_str it appears like 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': instead of "relationships": {"1": {"node_id": so there is no good format for JSON
8 comments
a
L
hi there! i am getting embeddings from a 20k pages pdf but suddenly i got this error:
BadRequestError: Error code: 400 - {'error': {'message': "'$.input' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference.", 'type': 'invalid_request_error', 'param': None, 'code': None}}

Why is this happening?
8 comments
a
L
a
alvarojauna
·

PDF

Hi everyone, I have a large PDF containing 20,000 pages. After receiving the prompt response, how can I identify and show the page where the information was sourced from?
2 comments
a
W
hi, anyone knows why this happened?
46 comments
L
a
hi there! how can i convert llama_index.schema.TextNode
to str or list?
31 comments
L
a
hi there! do u know how can i add chat history so previous prompts are also considerated?
1 comment
L
hi there! how can i add the source of the obtained embeddings in the final prompt? I mean, the pages where information is obtained from PDF
5 comments
T
a
Hi everyone, I have a large PDF containing 20,000 pages. After receiving the prompt response, how can I identify and show the page where the information was sourced from?
1 comment
T