Find answers from the community

Updated last year

hey all probably a super easy q, but how

cchirag

hey all probably a super easy q, but how can I assign metadata to a each respective document?

21 comments

LLogan M

Just as a dictionary
document.metadata = {"key": "val"}

LLogan M

or Document(text=text, metadata=metadata)

cchirag

thanks @Logan M so I have a directory full of documents

cchirag

I need to assign seperate metadata to each

LLogan M

If you use SimpleDirectoryReader(), some metadata will be assigned automatically (including filename)

So you could post-process the output list of documents from there and use the filenames to add more metadata

cchirag

ok so loop through the documents object and do document.metadata = {'foo' : 'bar'}

LLogan M

yup, ez pz

cchirag

I do see this in the constructor file_metadata of SimpleDirectoryReader, anything of value thre?

LLogan M

That is a function that takes a filename and returns a dictionary

LLogan M

If you override it, then you also override the default metadata

LLogan M

Could be useful as well

LLogan M

Plain Text

def get_metadata(filename):
  return {"filename": filename}

document = SimpleDirectoryReader(..., file_metadata=get_metadata).load_data()

cchirag

ah perfect thanks for the help!

cchirag

also while I have you for a second. How can I get the LLM to return the associated metadata value, in my case the "url" key which I plan to add to the output of the LLM response?

LLogan M

the response object contains the nodes that were used to write the response

You can access these nodes to fetch the metadata

Plain Text

response = query_engine.query("..")
print(response.source_nodes[0].node.metadata)

LLogan M

If you want the LLM to actually write the URL in the response, you should be able to just include that in your query and it will figure it out I think

cchirag

oh interested, I assume then the context the LLM gets the metadata in the prompt?

LLogan M

Yessir -- You can actually control this though

LLogan M

https://docs.llamaindex.ai/en/stable/module_guides/loading/documents_and_nodes/usage_documents.html#advanced-metadata-customization

cchirag

awesome thanks again for your help! you are invaluable to this community!

ddigital_dream64

Hi! Just commenting here so that the thread gets saved for me. Thank you!

Add a reply