hey from 0 6 2 llama index we are seeing

SSiddhant Saurabh

hey from 0.6.2 llama_index, we are seeing that Document extra parameter is also getting added to text while adding to pinecone vector store?
how can we remove them? (this is polluting/disturbing the search with embedding while using pinecone)
cc:

15 comments

SSiddhant Saurabh

Document don't have extra parameters in the text field but when they go under

Plain Text

vector_store = PineconeVectorStore(
            pinecone_index=self.pinecone_index,
            metadata_filters=metadata_filters,
            namespace="org-" + payload["organisation"],
        )
storage_context = StorageContext.from_defaults(vector_store=vector_store)
pc_index = GPTVectorStoreIndex.from_documents(
                documents,
                pinecone_index=self.pinecone_index,
                storage_context=storage_context,
                service_context=service_context,
            )

the text in pinecone have extra parameters.
gone through the code but still not able to figure out
please help

SSiddhant Saurabh

cc: @Logan M @ravitheja

LLogan M

I don't think there's a way to ignore it at the moment. But @disiok has been working on metadata changes

SSiddhant Saurabh

facing same above issue with 0.6.5 version

ddisiok

@Logan M do you know what we are injecting automatically?

LLogan M

@Siddhant Saurabh do you have an example of the extra info you are seeing?

ddisiok

oh I see, I think what happens is:

our reader automatically injects extra_info to document
the node parser propagates that to the Nodes
when we do Node.get_text() it gets injected automatically

LLogan M

Yea that sounds like what I was thinking of 👀

ddisiok

There's a include_extra_info flag in the SimpleNodeParser which we can toggle as a dumb fix

ddisiok

for this particular case

ddisiok

but ya we need to be more explicit about what is being injected where, and give user control over it

kkartik9257

ah okay so we can just switch this off for now

SSiddhant Saurabh

hey @disiok and @Logan M
when we use SimpleNodeParser(include_extra_info=False)
then we don't have "google\ntimestamp: 1684345677.945844\nchunk_size: 600\nchunk_overlap: 150\nnum_indexes: 74" in the text of the node
but then extra_info become {} empty

how can we not include "google\ntimestamp: 1684345677.945844\nchunk_size: 600\nchunk_overlap: 150\nnum_indexes: 74" not in the text of the node, but only keep then in extra_info of the node?

SSiddhant Saurabh

its solved in 0.6.8 thanks

ddisiok

Yay

Add a reply

Find answers from the community

hey from 0 6 2 llama index we are seeing