Find answers from the community

Updated 11 months ago

Metadata

At a glance

The community member is using OpenAI and service_context with nodes, and when the data is retrieved from indexing, it is sent to the language model (LLM) with metadata like page_label and file_path. The community member wants to send just the context without this metadata. Another community member suggests setting the excluded metadata keys to "page_label" and others to exclude them when loading the documents. The community members discuss where this step should be done, either on the documents from the reader or the nodes from the parser.

kkemoalkoul

Hi Guys
I use OpenAI and service_context with nodes
and when the data is retrieved from indexing they send to llm with metadata like
page_label and file_path, so i want to send just the context without this metadata , how i can do it?

4 comments

LLogan M

When you load your documents, set the excluded metadata keys

document.excluded_llm_metadata_keys = ["page_label", ...]

kkemoalkoul

Can u clarify where must do this step?
This is my code

Plain Text

documents = SimpleDirectoryReader("/content/data").load_data()

node_parser = SimpleNodeParser.from_defaults(chunk_size=512, chunk_overlap=20)
nodes = node_parser.get_nodes_from_documents(documents)

LLogan M

Could be either on the documents from the reader, or the nodes from the parser, either would work

kkemoalkoul

thank u

Add a reply