Hi I see a very strange problem when

SSeaCat

Hi, I see a very strange problem when parsing a text of ~60,000 symbols - this code (the from_documents function) never ends, I can't figure out why, it worked very well till today (even with much bigger documents):

Plain Text

document = Document(text)
document.doc_id = data_source_id
service_context = ServiceContext.from_defaults(chunk_size=chunk_size)#1024)
vector_store = QdrantVectorStore(client=get_qrant_client(), collection_name=project_id)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(
    [document], storage_context=storage_context, service_context=service_context
)

If I go into the function from_documents, this code is hanging:

Plain Text

            return cls(
                nodes=nodes,
                storage_context=storage_context,
                service_context=service_context,
                **kwargs,
            )

18 comments

LLogan M

How long did you try waiting? Just curious?

SSeaCat

10-20 minutes or more. Usually for texts of such length, it takes no more than a minutes, usually much less

LLogan M

I see you looked at where it's hanging. Do you know what file that line of code was from?

LLogan M

That's definitely a long time to not be working

SSeaCat

It's from BaseIndex/base.py, line 97
Actually, I suspect the text itself but I can't figure out what's wrong with it. Maybe there are some prohibited symbols or something like that?

LLogan M

Hmmm maybe? Is it in a different language or from a problematic data source?

SSeaCat

It's in English, scraped from one website, nothing usual at the first glance

LLogan M

Maybe as a sanity test, does this code run well for you?

Plain Text

from llama_index.embeddings import OpenAIEmbedding

embed_model = OpenAIEmbedding()

embeddings = embed_model.get_text_embedding("It is raining cats and dogs here!")
print(len(embeddings))

SSeaCat

Interesting, let me try, do you want me to test with that text or with this one?

SSeaCat

Hmmm I have the exception "cannot import name 'OpenAIEmbedding' from 'llama_index.embeddings'"

SSeaCat

I guess should be llama_index.embeddings.openai 😉

SSeaCat

Okay, it worked, the output: 1536

LLogan M

Nice! That means at least the embeddings can be generated 😅

LLogan M

And it's not some other problem related to servers or keys

SSeaCat

Which server? It's working with any other text, just doesn't with that "problematic" one

SSeaCat

keys can't be involved either

LLogan M

Openai servers I meant, like there's no specific network issue with them

SSeaCat

Ahhh okay... right sorry didn't see you "not"😆 Let me check the text, maybe it really has some suspicious symbols

Add a reply

Find answers from the community

Hi I see a very strange problem when