The community member is trying to load API response data into a document and a vector store, and is specifically interested in handling "abstract" and "key points" as separate fields. The community members discuss whether it's possible to use two different retrievers for these two attributes, or if it's better to merge them into a single "text" field and use metadata to differentiate them. One community member suggests loading the "abstract" and "key points" as metadata, which would allow the community member to filter or parse them out later as needed.
Hello everyone, I am trying to load api response into document and into vector store later. If for example my api response with "absctract" and "key points" , I know that i can set up document with two different fields but what about index and vector store retrievers, is there any way that i can use two different retrievers on the two attributes?
Could you clarify on how you're looking to use the two attributes for retrieval? If you're looking to filter or parse out a specific one you could load them as metadata
hello Teemu, so my reasoning is since i have gotten structured data i would want to keep that way since both fields are equally important, didnt really make sense to me to merge everything together as something like "text". But I probably could just merge everything into one and just chunk load into documents, just wanting to see if it is possible to separate them and utilize it later to make sure when user asked things that is related to keypoints it will be retrieved by the embedding. Sorry if none of these makes sense, just started reading on the documentation last week and I am still trying to figure out what is the best way to do things.
basically what I am aiming to do is to have an agent to perform research for me, where it will call arxiv api to retrieve papers then consolidate information based on my initial prompt. So currently still trying to figure out the first part on data loading.
