Find answers from the community

Updated 3 months ago

Hello Community, today I am trying to

Hello Community, today I am trying to refresh my vector index and I'm following this tutorial https://docs.llamaindex.ai/en/stable/module_guides/indexing/document_management.html# . However, it happened TypeError as the screenshot 1. I understood that keyword argument 'delete_from_docstore' is duplicated in delete_ref_doc() function. I was trying to go to base llama_index/indices/base.py and commented out delete_from_docstore=True as the screenshot 2. And then, it happened KeyError error as screenshot 3, but I dont understand why it refers to def delete(self, doc_id: str) , and I can't find def delete(self, doc_id: str) in base.py, I only see delete_ref_doc function there. What should I do now, guys? Please help me. Thank you.

Plain Text
documents = SimpleDirectoryReader(input_dir="sleep", filename_as_id=True).load_data()
print(f"No of documents: {len(documents)}")
# rebuild storage context
storage_context = StorageContext.from_defaults(persist_dir="storage/sleep")
# load index from storage
index = load_index_from_storage(storage_context=storage_context)
print("Successfully load index from storage!")
# refresh the index
refreshed_docs = index.refresh_ref_docs(documents=documents, update_kwargs={"delete_kwargs": {"delete_from_docstore": True}})
print(refreshed_docs)  # true values indicate that the value were refreshed
print(f"No of inserted/refreshed documents: {sum(refreshed_docs)}")
Attachments
3.png
1.png
2.png
W
J
C
65 comments
Are you trying this with latest version?
Hi @WhiteFang_Jr I'm using llama-index==0.9.15
Let me check at my side once
@WhiteFang_Jr Thank you, the function worked fine when I inserted new documents. However, when I removed some text in my files, and ran func refresh_ref_docs. This error will happen.
#❓py-issues-and-help
ImportError: cannot import name 'add_builder_config' from 'st_utils' (C:\Users\chait\Downloads\rags-main\st_utils.py)
Traceback:
File "C:\Users\chait\miniconda3\envs\llma\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 534, in _runscript
exec(code, module.dict)
File "C:\Users\chait\Downloads\rags-main\1🏠_Home.py", line 4, in <module>
from st_utils import (
getting this error anyone with solution??
@CHAITANYA DESHPANDE it said it cant import add_builder_config from st_utils. So I guess you have to go to the source code st_utils.py to find whether add_buider_config is there?
Got the same error , multiple values for delete_from_docstore, Let me see if I can give you a small fix
the code has it
idk why its not recognising
Check if it requires to declare a init file for recognition
Great. Thank you. I saw someone had this error 2 month ago. However, they still havent had solution for this yet.
I have a init file but it has nothing
what to do
any idea?
can you try to write from .st_utils import * in that init.py file?
ModuleNotFoundError: No module named 'core.st_utils'
now getting this error
in which file? the code you sent here doesnt have core.st_utils in it.
ModuleNotFoundError: No module named 'core.st_utils'
Traceback:

Explain
File "C:\Users\chait\miniconda3\envs\llma\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 534, in _runscript exec(code, module.dict)File "C:\Users\chait\Downloads\rags-main\1🏠_Home.py", line 4, in <module>
from st_utils import (
File "C:\Users\chait\Downloads\rags-main\st_utils.py", line 2, in <module>
from core.agent_builder.loader import (
File "C:\Users\chait\Downloads\rags-main\core__init__.py", line 2, in <module>
from .st_utils import *
this seem like the core.st_utils file
Hey I think I got it
So if you check this method: https://github.com/run-llama/llama_index/blob/e7090975a1807b6c30c132c65464cb51dba3804a/llama_index/indices/base.py#L284


Here you already have a boolean flag for removing doc from the docstore
delete_from_docstore
and while popping the kwargs dict also gives you the same key.

I commented the kwargs part and it is working now
Wait I see that you already tried this πŸ˜…

BUt I did not get any error after commenting the kwargs part
Both of them worked for me @Jimmy Phan
Commenting down the delete_from_docstore itslef and the kwargs part
Actually I did comment down delete_from_docstore at first. You can see my second screenshot above. And it had keyerror (see my third screenshot). And I still found this key (doc_id) in my index. Tmr I will try to comment out kwargs part and let u know soon. Thank you.
Attachment
3.png
I'm sorry I'm sure about this error. Which python package are u using in your project? Is it streamlit? If it's streamlit, I suggest that you should go to streamlit thread to find a proper solution for your issue.
ok
Thanks for the help
Did you clone from RAG repo recently?
Working for me
Attachment
image.png
you are using Conda???
Just did the following
  • Fresh clone
  • Fresh env
  • Saved openai key in .streamlit/secrets.toml file
  • ran it using streamlit run filename
Yes with conda
Saved openai key in streamlit folder
on the repo they said to store it in .secret.toml
ig thats why i am getting the error
i will reclone it and retry
secrets.toml goes inside .streamlit folder
updated my answer
wait i just check its exactly there
I would suggest you try putting fresh cloning/installtion a try
thanks for the help though
where to create the .streamlit folder ???
it needs to be created independent right
i mean not inside any big folder like core ,pages or something
In the root folder only
Hey its still showing some error
ImportError: cannot import name 'add_builder_config' from 'st_utils' (C:\Users\chait\Downloads\rags-main\rags-main\st_utils.py)
From which folder are you running the code?
\rags-main>streamlit run 1_🏠_Home.py
so in the anaconda prompt
got this error
KEY ERROR:: 'openai_key'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\chait\miniconda3\envs\rags\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 534, in _runscript exec(code, module.dict) File "C:\Users\chait\Downloads\rags-main\rags-main\1🏠_Home.py", line 4, in <module>
from st_utils import (
File "C:\Users\chait\Downloads\rags-main\rags-main\st_utils.py", line 2, in <module>
from core.agent_builder.loader import (
File "C:\Users\chait\Downloads\rags-main\rags-main\core\agent_builder\loader.py", line 6, in <module>
from core.builder_config import BUILDER_LLM
File "C:\Users\chait\Downloads\rags-main\rags-main\core\builder_config.py", line 12, in <module>
os.environ["OPENAI_API_KEY"] = st.secrets.openai_key
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chait\miniconda3\envs\rags\Lib\site-packages\streamlit\runtime\secrets.py", line 296, in getattr
raise AttributeError(_missing_attr_error_message(key))
AttributeError: st.secrets has no attribute "openai_key". Did you forget to add it to secrets.toml or the app settings on Streamlit Cloud? More info: https://docs.streamlit.io/streamlit-cloud/get-started/deploy-an-app/connect-to-data-sources/secrets-management
can you show me your project structure once
ok wait i will share the image
Open AI API key stored in secrets.toml file
Not sure why you getting this then as the project structure looks right.
Can you make sure the key is actually openai_key only
[api_keys]
openapi_key = "KEY"
stored it like this
the key name is different in your case, it should be openai_key
see this: os.environ["OPENAI_API_KEY"] = st.secrets.openai_key
AttributeError: st.secrets has no attribute "openai_key". Did you forget to add it to secrets.toml or the app settings on Streamlit Cloud? More info
Attachment
Screenshot_2023-12-14_183904.png
i am still getting the same error though i have changed it
reran
got this error now importError: cannot import name 'add_builder_config' from 'st_utils' (C:\Users\chait\Downloads\rags-main\rags-main\st_utils.py)
Hi, I tested your solution and it worked perfrectly. I'm not sure why I did the same (comment out delete_from_docstore param) yesterday, but it didn't work. Maybe I didn't restart the python kernel haha. Thank you @WhiteFang_Jr . And one more question, is it a bug from their source code? Can I push a fixed code to their repository? Because I dont think it's a good way to modify package's source code locally.
Hey, and instead modifying base.py. I just used refreshed_docs = index.refresh_ref_docs(documents=documents) and it still worked πŸ™‚
Awesome πŸ’ͺ
Yep I tried this as well but it was pretty late so didn't want to disturb. πŸ˜…

I guess document may need update
Yeah, I think so too. Btw, may you help me with another issue? I posted that issue (related to CustomExtractor()). It has async and await mechanism which I'm not familiar to deal with πŸ₯²
Sure, Just answered there
@WhiteFang_Jr i am still getting the same error
can you pls help
Add a reply
Sign up and join the conversation on Discord