Embedding token count when upserting nodes in pinecone
Embedding token count when upserting nodes in pinecone
At a glance
The community member is experiencing issues with the token count when upserting nodes in Pinecone. They note that the metadata seems to be counted towards the token count, which they have confirmed through testing. Another community member responds that by default, metadata is included in LLM and embed model calls, but that fields can be excluded programmatically. They provide a link to documentation on advanced metadata customization.
Hi there, I'm getting errors with regards to embedding token count when upserting nodes in Pinecone.
Can anyone give me info as to how the token count for the Node object is counted? It looks like the metadata is counted towards the token count, my testing seems to confirm this
