Hi All I need help as I am really

AAtaliba Miguel

Hi All, I need help as I am really struggling to get it to work streamlit file_uploader with llamaIndex. def configure_qa_chain(uploaded_files):
docs = []
with tempfile.TemporaryDirectory() as temp_dir:
for file in uploaded_files:
with tempfile.NamedTemporaryFile(delete=False, dir=temp_dir) as tmp_file:
tmp_file.write(file.getvalue())
tmp_file_path = tmp_file.name
docs.append(tmp_file_path)
loader = PyPDFLoader(file_path=tmp_file_path)
docs = loader.load()
nodes = SimpleNodeParser(text_splitter=" ") .get_nodes_from_documents([docs])
MONGO_URI = os.environ["MONGO_URI"]
MONGODB_DATABASE = "LlamaIndex_Chat"
docstore = MongoDocumentStore.from_uri(uri=MONGO_URI)
docstore.add_documents(nodes)
storage_context = StorageContext.from_defaults(docstore=MongoDocumentStore.from_uri(uri=MONGO_URI, db_name=MONGODB_DATABASE),
index_store=MongoIndexStore.from_uri(uri=MONGO_URI, db_name=MONGODB_DATABASE),)
docstore = MongoDocumentStore.from_uri(uri=MONGO_URI, db_name=MONGODB_DATABASE)
nodes = list(docstore.docs.values())
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
llm = ChatOpenAI( model_name="gpt-3.5-turbo", temperature=0, openai_api_key=openai.api_key,
streaming=True)
llm_predictor = LLMPredictor(llm=llm)
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
list_index = GPTListIndex(nodes, storage_context=storage_context, service_context=service_context)
vector_index = GPTVectorStoreIndex(nodes, storage_context=storage_context, service_context=service_context)
keyword_table_index = GPTSimpleKeywordTableIndex(nodes, storage_context=storage_context, service_context=service_context)
vector_response = vector_index.as_query_engine().query(user_query)
return vector_response

35 comments

LLogan M

whats the error?

AAtaliba Miguel

ValidationError: 1 validation error for SimpleNodeParser text_splitter value is not a valid dict (type=type_error.dict)

AAtaliba Miguel

Attachment

LLogan M

nodes = SimpleNodeParser(text_splitter=" ") .get_nodes_from_documents([docs])

text splitter can't be a space, it has to be an actual object 👀

You can likely just use the defaults

nodes = SimpleNodeParser.from_defaults().get_nodes_from_documents([docs])

AAtaliba Miguel

Thanks @Logan M, tried it but now yielding AttributeError: 'list' object has no attribute 'get_content'

AAtaliba Miguel

Attachment

LLogan M

whoops one more error

Plain Text

nodes = SimpleNodeParser.from_defaults().get_nodes_from_documents(docs)

LLogan M

No need for square brackets, it's already a list

AAtaliba Miguel

I guess something might be going wrong now yielding another error - AttributeError: 'Document' object has no attribute 'get_content'

AAtaliba Miguel

I have been struggling to use streamlit file_uploader ever since. As I plan to query own documentation not via path but rather via uploading files from everywhere where located. I understand the Path being too static and cumbersome

AAtaliba Miguel

I really appreciate support here or any exisitng code snippet that uploads documents via streamlit and then fetch it to nodes and storage_contex, service_context and alike

LLogan M

Hmm that error is kind of wild, the Document object definitely has a get_content() method lol

AAtaliba Miguel

How shall I go about it? Please I am really struggling with this for ages and I can't get it to work. Any suggestions on the path forward are very welcomed please

EEmanuel Ferreira

which version are you using?

AAtaliba Miguel

llama-index 0.8.16

EEmanuel Ferreira

can you provide us a repo with a reproducible version? will facilitate a lot to help you

AAtaliba Miguel

Honestly @Emanuel Ferreira the stage I am now it is just the .py file that I can share with you. I haven't gone to repo's on Github yet. If you fine I can send the file across just now or DM you.

EEmanuel Ferreira

https://gist.github.com/

paste here and send me pls

AAtaliba Miguel

@Emanuel Ferreira this is my very first time on this, hope it works fine. There it goes <script src="https://gist.github.com/achilela/5fcd85d690cfe4680a27e6f99f4bc226.js"></script>

AAtaliba Miguel

Pls let know if it worked fine you accessing it

EEmanuel Ferreira

https://gist.github.com/achilela/5fcd85d690cfe4680a27e6f99f4bc226

worked, I will do some tests and let you know

AAtaliba Miguel

Really appreciate @Emanuel Ferreira

EEmanuel Ferreira

can you send me the requirements file?
pip freeze > requirements.txt

EEmanuel Ferreira

then I can have the same versions/packages as you

AAtaliba Miguel

Not sure if done it correctly pip freeze > requirements.txt ~/LLMWorkshop/Experimental
Lama_QA_Retrieval/llamaIndex_streamlit_chat.py

EEmanuel Ferreira

I see you aren't using venv, so it get all your packages (even the ones which isn't on the project)

EEmanuel Ferreira

but that's ok, I can find a way here

AAtaliba Miguel

Really appreciate @Emanuel Ferreira

AAtaliba Miguel

Hi @Emanuel Ferreira, do not meant to push, was just wondering are there any updates on the uploaded_file.

EEmanuel Ferreira

Hi! I'll be trying to take a look on it today yet

EEmanuel Ferreira

@Ataliba Miguel Your issue is that your current document coming from the PDF doesn't follow a format that the node parse request

Plain Text

loader = PyPDFLoader(file_path=tmp_file_path)
docs = loader.load()

then sending your docs to something like that, can solve

Plain Text

from llama_index import Document

def parse_document(docs):
    documents = []
    for document in docs:
        # Manually create a new metadata dictionary and exclude specific keys
        documents.append(
            Document(
                text=document.page_content if hasattr(document, 'pagecontent') else "",
                id=document.metadata.get("id", None),
                metadata=document.metadata,
            )
        )

    return documents

EEmanuel Ferreira

I didn't run all the code until make it work, because there's a lot of things and envs, but you definetly can go to the next steps with that

EEmanuel Ferreira

will mention @Logan M if want to correct me or complement with something

EEmanuel Ferreira

@Ataliba Miguel

Your issue is that it's a langchain document

@Logan M suggested easier than that is to use a langchain Loader

Plain Text

from llama_index import Document

document = Document.from_langchain_format(langchain_document)

so you can go through your docs array and format to a llamaindex document

AAtaliba Miguel

Thanks @Emanuel Ferreira. I will try it out in approx 1/2 hour. Just finishing work now. Will keep you posted. Appreciate.

Add a reply

Find answers from the community

Hi All I need help as I am really