Find answers from the community

Updated 9 months ago

Pytest collects zero test cases. Am I

Pytest collects zero test cases. Am I missing something? 🧵
V
L
S
57 comments
Console log:
Plain Text
~/P/llama_index  conda[base]   call_to..ndling ±  poetry shell                     
Spawning shell within /Users/Bob/Library/Caches/pypoetry/virtualenvs/llama-index-VqnWDOI7-py3.10
source /Users/Bob/Library/Caches/pypoetry/virtualenvs/llama-index-VqnWDOI7-py3.10/bin/activate.fish

Welcome to fish, the friendly interactive shell
Type help for instructions on how to use fish
 ~/P/llama_index  conda[base]   call_to..ndling ±  source /Users/Bob/Library/Caches/pypoetry/virtualenvs/llama-index-VqnWDOI7-py3.10/bin/activate.fish      
(llama-index-py3.10)  ~/P/llama_index  conda[base] py[llama-index-VqnWDOI7-py3.10]   call_to..ndling ±      
(llama-index-py3.10)  ~/P/llama_index  conda[base] py[llama-index-VqnWDOI7-py3.10]   call_to..ndling ±  poetry install --with dev,docs                      
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: llama-index (0.10.20)
(llama-index-py3.10)  ~/P/llama_index  conda[base] py[llama-index-VqnWDOI7-py3.10]   call_to..ndling ±  pytest tests                
===================================================================================== test session starts =====================================================================================
platform darwin -- Python 3.11.8, pytest-7.2.1, pluggy-1.3.0
rootdir: /Users/Bob/Projects/llama_index
plugins: httpserver-1.0.8, mock-3.11.1, anyio-3.7.1, dotenv-0.5.2
collected 0 items

==================================================================================== no tests ran in 0.01s ====================================================================================
(llama-index-py3.10)  ✘  ~/P/llama_index  conda[base] py[llama-index-VqnWDOI7-py3.10]   call_to..ndling ±  


(Sorry; fish looks kinda messy without colors.)
I'm on this revision:

Plain Text
$ git log | head
commit dbf3d9a3a658b7449e2d1889c745eaf299bebf30
Author: Emanuel Ferreira <contatoferreirads@gmail.com>
Date:   Tue Mar 19 17:18:10 2024 -0300
make test is giving me this:
Plain Text
$ make test
pants --no-local-cache --changed-since=origin/main --changed-dependents=transitive test
make: pants: No such file or directory
make: *** [test] Error 1

which I suppose I can fix with brew install pantsbuild/tap/pants, but I hope someone can mention this in https://github.com/run-llama/llama_index/blob/main/CONTRIBUTING.md.
In PyCharm CE, although I pointed it to the Poetry environment (which I discovered by running which python) that poetry shell leads me to, oddly it cannot find the pytest module.
Attachment
Screenshot_2024-03-19_at_21.09.37.png
You should be able to provide the path to the tests folder you want to rub

For example, to test core
pytest llama-index-core/tests
Plain Text
$ pytest llama-index-core/tests   
ImportError while loading conftest '/Users/Bob/Projects/llama_index/llama-index-core/tests/conftest.py'.
llama-index-core/tests/conftest.py:8: in <module>
    from llama_index.core.base.llms.types import LLMMetadata
E   ModuleNotFoundError: No module named 'llama_index.core.base'
Did you do pip install -e ./llama_index_core ?

To me, that error says you may have an older version of llama-index installed in your venv (which would break things)
You miiiight need a fresh venv
Plain Text
$ pip install -e ./llama_index_core
ERROR: ./llama_index_core is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).
That's... a new one
Is that the proper path?
Oh I put underscores
It should be dashes
LOL I just copied without checking
but I just nuked my Poetry venv. Let me re-install real quick.
Note to future readers: I'm following this: https://stackoverflow.com/a/70064450/1147061
Plain Text
Successfully built llama-index-core
Installing collected packages: llama-index-core
  Attempting uninstall: llama-index-core
    Found existing installation: llama-index-core 0.10.20
    Uninstalling llama-index-core-0.10.20:
      Successfully uninstalled llama-index-core-0.10.20
Successfully installed llama-index-core-0.10.20.post3
$ pytest llama-index-core/tests                                               (base)

ImportError while loading conftest '/Users/Bob/Projects/llama_index/llama-index-core/tests/conftest.py'.
llama-index-core/tests/conftest.py:8: in <module>
    from llama_index.core.base.llms.types import LLMMetadata
E   ModuleNotFoundError: No module named 'llama_index.core.base'
Here's my binaries:
Plain Text
$ which poetry 
/opt/homebrew/Caskroom/miniforge/base/bin/poetry
$ which python
/Users/Bob/Library/Caches/pypoetry/virtualenvs/llama-index-VqnWDOI7-py3.10/bin/python
Maybe cd llama-index-core and then run pytest tests ?

Either that, or you have some global install of llama-index maybe? Outside of a venv?
Ah you're right:
Plain Text
$ pip uninstall llama-index 
Found existing installation: llama-index 0.9.21
Uninstalling llama-index-0.9.21:
  Would remove:
    /opt/homebrew/Caskroom/miniforge/base/bin/llamaindex-cli
    /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/llama_index-0.9.21.dist-info/*
    /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/llama_index.pth
Proceed (Y/n)? y
  Successfully uninstalled llama-index-0.9.21
But that oddly didn't solve the issue...
(Fresh venv again now? 😅 )
Let me check one thing first.
I know that git sometimes keep empty dirs around when upstream moves file around.
Nope. A fresh clone from main + a fresh venv didn't help.
I'm reinstalling pytest now. My current copy is installed via brew. Maybe it doesn't respect Poetry venv Python binary???
Ohhhhh
Attachment
Screenshot_2024-03-19_at_22.35.24.png
This is a fake binary specifically calling the Python binary at /opt/homebrew/opt/python@3.11/bin/python3.11.
Die, you pytest!
Attachment
Screenshot_2024-03-19_at_22.36.31.png
Ok. Now it worked.
@Logan M , do you think we should add pytest as a dev dependency to pyproject.toml?
Oh wait it's already there. It's not. I was looking at an old revision.
I think this is why my Poetry venv kept defaulting to the brew-installed copy of pytest earlier today.
Was it intentional to remove pytest from the dev packages group?
It's the v0.10.0 commit that removed the pytest-related packages from the top-level pyproject.toml: https://github.com/run-llama/llama_index/commit/369973f4e8c1d6928149f0904b25473faeadb116
Attachment
Screenshot_2024-03-19_at_22.54.12.png
I wonder if we should add it back or remove pytest tests as an option to test the repo altogether.
Pytest is a lot easier than using pants imo
I'll leave the add-pytest-back-or-not discussion for you official maintainers to discuss.
I'm still catching myself up on what the heck is this Pants thing.
i generally wouldnt expect contributors to learn pants lol
its mostly there so that our cicd can work in a monorepo
It's a cool tech, not gonna lie
I had fun setting it up haha, solves a very cool problem
@Logan M , can you verify if these 2 unit tests are failing on main branch on your end?
Plain Text
FAILED tests/agent/react/test_react_agent.py::test_astream_chat_basic - AttributeError: 'NoneType' object has no attribute 'wait'
FAILED tests/agent/react/test_react_agent.py::test_async_add_step - AttributeError: 'NoneType' object has no attribute 'wait'

Here's how I set up my env:
Plain Text
cd llama-index-core # ! not the root dir.
poetry shell
poetry install --with dev,docs
pytest


Revision c367767a899a1471f6e468f804eb43091396db5a.
I can confirm these pass for me

If you scroll up, you should see the reason it's failing. Curious what the actual error is/line of code
I opened a small PR. Let's see if the CI passes there.
https://github.com/run-llama/llama_index/pull/12137
You're right. With that merge commit reverted, tests passed.
But then I wonder how it escaped unit testing.
My experience getting the tests to run are as follows:

  1. I tried following the contributing.md guide: running pytest <package>tests/ directly, running make tests, even running the pants test command manually. None of the commands worked as I expected (running the tests). In the end, I had to navigate to this discord, find out other people were experiencing similar issues and resolve them manually in order to get the tests to run.
  2. I expected all packages required to run the project's tests would be installed as part of --with dev option (or perhaps --with test), especially for the core package. Because this was not the case, I had to run poetry add: pytest, pytest_mock, pytest_asyncio, tree_sitter_languages.
  3. After installing the required dependencies, I got failing tests
FAILED llama-index-core/tests/agent/react/test_react_agent.py::test_astream_chat_basic - AttributeError: 'NoneType' object has no attribute 'wait'
FAILED llama-index-core/tests/agent/react/test_react_agent.py::test_async_add_step - AttributeError: 'NoneType' object has no attribute 'wait'

I've spent quite some time on getting the project to run now, but still haven't been able to run all tests successfully. This is not so much of a critique, but more of a list of "gotchas" for other aspiring contributors to get things up and running. Perhaps it helps someone and feel free to add to this list if I'm missing something.

Would love to contribute more to the project, but getting the project up and running makes it harder to do that.
Let me rewrite that.
Add a reply
Sign up and join the conversation on Discord