LlamaIndex
Log in

Find answers from the community

Home
Members
ddashed
d
ddashed
Offline, last seen 2 months ago
Joined September 25, 2024
dddashed
·

I am using a customer retriever with a

I am using a customer retriever with a keyword Table retriever and a vector index retriever. Previously, I was making my indexes for these retrievers as I used them, but I have switched to indices which have been created elsewhere and persisted to storage. I have them stored and pulled from storage according to the persisting docs (i assigned ids and pull indexes off the id's), but when I query my retriever query engine I am now receiving this error when comparing embeddings. Its
Plain Text
ValueError: shapes (3072,) and (1536,) not aligned: 3072 (dim 0) != 1536 (dim 0).
As 3072 is double 1536, I'm pretty sure its duplicating something somewhere but i'm not sure. If anyone knows as to why this happens please let me know. Code used to persist and load :
Plain Text
    vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
    keyword_index = SimpleKeywordTableIndex(nodes, storage_context = storage_context)
    vector_index.set_index_id("slide_vector_index")
    keyword_index.set_index_id('slide_keyword_index')
    os.makedirs(f"/data/notes-bin/{file_name}/DocStore")
    vector_index.storage_context.persist(persist_dir=f"/data{file_name}/DocStore")
    keyword_index.storage_context.persist(persist_dir=f"/data/{file_name}/DocStore")
Plain Text
    storage_context = StorageContext.from_defaults(
    docstore=SimpleDocumentStore.from_persist_dir(persist_dir=f"/data{file_name}/DocStore"),
    vector_store=SimpleVectorStore.from_persist_dir(
        persist_dir=f"/data/{file_name}/DocStore", namespace="default"),
    index_store=SimpleIndexStore.from_persist_dir(persist_dir=f"/data/{file_name}/DocStore"),)


    vector_index = load_index_from_storage(storage_context, index_id="slide_vector_index")
    keyword_index = load_index_from_storage(storage_context, index_id="slide_keyword_index")
5 comments
d
T
dddashed
·

``` nodes = storage_context.docstore.

Plain Text
 nodes = storage_context.docstore.docs
is this the proper way to pull nodes out of a stoage context?
1 comment
L
dddashed
·

from llama_index.core import Settings,

from llama_index.core import Settings, Getting this Error message with updated llama_index. ImportError: cannot import name 'Settings' from 'llama_index.core' (unknown location). Having the same problem trying to import Documents from core
23 comments
a
d
L
dddashed
·

Question about streaming responses

Question about streaming responses through an endpoint. I get it to stream into my servers terminal but I can't get it to return and stream to the client side. realize this might not be the place to ask this but idk so throwing it out there
4 comments
d
L
dddashed
·

I am trying to make some queries against

I am trying to make some queries against my custom query engine in parallel. I have tried to use both concurrent futures and multiprocessing and both are throwing me random errors. My pipeline works just fine when I simply use a for loop but I want to speed up the processing time as waiting for my response is ineffective. If anyone has used a query engine in parallel with no problems please let me know!
6 comments
L
d
dddashed
·

Sql

Hi, I am looking at the recursive retriever and query engine (https://docs.llamaindex.ai/en/stable/examples/query_engine/pdf_tables/recursive_retriever.html). It seems that camelot and llama-index-core, require different versions of the SQLAlchemy package. To get around this, seems like a way would be different venv's for camelot and llama_index. any other solutions or ideas?
1 comment
L
dddashed
·

Splitting

Question about customs transformations with ingestion pipelines. I am looking to create a custom sentence splitter type thing where i am splitting primary on a regex separator vs primarily on chunk size. Can i use sentence splitter or do I need an alternate? any examples of something similar?
1 comment
L