Find answers from the community

Updated 3 months ago

Json

hi! i think i found a bug or a mistake and i need help. i want to upload nodes to azure and save it in a txt. then i want to downlad and create a vector store index.

to achieve this i run the following commands:

HERE IS THE PART FROM DOCUMENTS TO NODES:
documents = SimpleDirectoryReader("/content/data").load_data()

embed_model = OpenAIEmbedding()
splitter = SemanticSplitterNodeParser(
buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model
)

nodes = splitter.get_nodes_from_documents(documents)

nodes_str = [x.dict() for x in nodes]
data=str(nodes_str)


ruta_en_dbfs = "/dbfs/nodes_architectures.txt"

Escribir la variable en el archivo en DBFS

with open(ruta_en_dbfs, 'w') as archivo:
archivo.write(data)

print("Variable guardada en DBFS con éxito.")

here are some commands of azure that i skipped and then:
with open(local_file_path, "rb") as data:
blob_client.upload_blob(data, overwrite=True)


HERE IS THE PART FROM NODES IN TXT FORMAT TO ADD IT TO VECTOR STORE INDEX (THIS IS IN ANOTHER SCRIPT):
with open(file_path, 'r', encoding='utf-8') as file:
#with open(file_path, 'r') as file:
# Leer cada línea en el archivo
my_list = file.readlines()

nodes_str = json.loads(my_list[0])
nodes = [TextNode.parse_obj(x) for x in nodes_str]

index = VectorStoreIndex(nodes_str)




This worked fine until the last big update of llamaindex. Now i found my_list[0] with ' instead of " so i run the following command: mi_lista=my_list[0].replace("'", """). But if i have in text for example microsoft"s, then the JSON format is not correct

Also, Now in nodes_str it appears like 'relationships': {<NodeRelationship.SOURCE: '1'>: {'node_id': instead of "relationships": {"1": {"node_id": so there is no good format for JSON
L
a
8 comments
Probably you should be using node.json() instead of node.dict() ?
after downloading the nodes from azure in txt format with json format that you said
well before upload nodes to azure i have to convert the json format to string like str(nodes)
here the empty space are filled with \ but i dont want that
how can i change it?
Add a reply
Sign up and join the conversation on Discord