Pytest collects zero test cases. Am I

Pytest collects zero test cases. Am I missing something? 🧵

57 comments

Console log:

Plain Text

~/P/llama_index  conda[base]   call_to..ndling ±  poetry shell                     
Spawning shell within /Users/Bob/Library/Caches/pypoetry/virtualenvs/llama-index-VqnWDOI7-py3.10
source /Users/Bob/Library/Caches/pypoetry/virtualenvs/llama-index-VqnWDOI7-py3.10/bin/activate.fish

Welcome to fish, the friendly interactive shell
Type help for instructions on how to use fish
 ~/P/llama_index  conda[base]   call_to..ndling ±  source /Users/Bob/Library/Caches/pypoetry/virtualenvs/llama-index-VqnWDOI7-py3.10/bin/activate.fish      
(llama-index-py3.10)  ~/P/llama_index  conda[base] py[llama-index-VqnWDOI7-py3.10]   call_to..ndling ±      
(llama-index-py3.10)  ~/P/llama_index  conda[base] py[llama-index-VqnWDOI7-py3.10]   call_to..ndling ±  poetry install --with dev,docs                      
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: llama-index (0.10.20)
(llama-index-py3.10)  ~/P/llama_index  conda[base] py[llama-index-VqnWDOI7-py3.10]   call_to..ndling ±  pytest tests                
===================================================================================== test session starts =====================================================================================
platform darwin -- Python 3.11.8, pytest-7.2.1, pluggy-1.3.0
rootdir: /Users/Bob/Projects/llama_index
plugins: httpserver-1.0.8, mock-3.11.1, anyio-3.7.1, dotenv-0.5.2
collected 0 items

==================================================================================== no tests ran in 0.01s ====================================================================================
(llama-index-py3.10)  ✘  ~/P/llama_index  conda[base] py[llama-index-VqnWDOI7-py3.10]   call_to..ndling ± 

(Sorry; fish looks kinda messy without colors.)

VVicent W.

I'm on this revision:

Plain Text

$ git log | head
commit dbf3d9a3a658b7449e2d1889c745eaf299bebf30
Author: Emanuel Ferreira <contatoferreirads@gmail.com>
Date:   Tue Mar 19 17:18:10 2024 -0300

VVicent W.

make test is giving me this:

Plain Text

$ make test
pants --no-local-cache --changed-since=origin/main --changed-dependents=transitive test
make: pants: No such file or directory
make: *** [test] Error 1

which I suppose I can fix with brew install pantsbuild/tap/pants, but I hope someone can mention this in https://github.com/run-llama/llama_index/blob/main/CONTRIBUTING.md.

VVicent W.

In PyCharm CE, although I pointed it to the Poetry environment (which I discovered by running which python) that poetry shell leads me to, oddly it cannot find the pytest module.

Attachment

LLogan M

You should be able to provide the path to the tests folder you want to rub

For example, to test core
pytest llama-index-core/tests

VVicent W.

Plain Text

$ pytest llama-index-core/tests   
ImportError while loading conftest '/Users/Bob/Projects/llama_index/llama-index-core/tests/conftest.py'.
llama-index-core/tests/conftest.py:8: in <module>
    from llama_index.core.base.llms.types import LLMMetadata
E   ModuleNotFoundError: No module named 'llama_index.core.base'

LLogan M

Did you do pip install -e ./llama_index_core ?

To me, that error says you may have an older version of llama-index installed in your venv (which would break things)

LLogan M

You miiiight need a fresh venv

VVicent W.

Plain Text

$ pip install -e ./llama_index_core
ERROR: ./llama_index_core is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).

LLogan M

That's... a new one

LLogan M

Is that the proper path?

Oh I put underscores

It should be dashes

Ohhhh

LOL I just copied without checking

VVicent W.

but I just nuked my Poetry venv. Let me re-install real quick.

VVicent W.

Note to future readers: I'm following this: https://stackoverflow.com/a/70064450/1147061

VVicent W.

Plain Text

Successfully built llama-index-core
Installing collected packages: llama-index-core
  Attempting uninstall: llama-index-core
    Found existing installation: llama-index-core 0.10.20
    Uninstalling llama-index-core-0.10.20:
      Successfully uninstalled llama-index-core-0.10.20
Successfully installed llama-index-core-0.10.20.post3
$ pytest llama-index-core/tests                                               (base)

ImportError while loading conftest '/Users/Bob/Projects/llama_index/llama-index-core/tests/conftest.py'.
llama-index-core/tests/conftest.py:8: in <module>
    from llama_index.core.base.llms.types import LLMMetadata
E   ModuleNotFoundError: No module named 'llama_index.core.base'

VVicent W.

Weird...

VVicent W.

Here's my binaries:

Plain Text

$ which poetry 
/opt/homebrew/Caskroom/miniforge/base/bin/poetry
$ which python
/Users/Bob/Library/Caches/pypoetry/virtualenvs/llama-index-VqnWDOI7-py3.10/bin/python

LLogan M

Maybe cd llama-index-core and then run pytest tests ?

Either that, or you have some global install of llama-index maybe? Outside of a venv?

VVicent W.

Ah you're right:

Plain Text

$ pip uninstall llama-index 
Found existing installation: llama-index 0.9.21
Uninstalling llama-index-0.9.21:
  Would remove:
    /opt/homebrew/Caskroom/miniforge/base/bin/llamaindex-cli
    /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/llama_index-0.9.21.dist-info/*
    /opt/homebrew/Caskroom/miniforge/base/lib/python3.10/site-packages/llama_index.pth
Proceed (Y/n)? y
  Successfully uninstalled llama-index-0.9.21

LLogan M

Ooo

VVicent W.

But that oddly didn't solve the issue...

LLogan M

(Fresh venv again now? 😅 )

VVicent W.

Let me check one thing first.

VVicent W.

I know that git sometimes keep empty dirs around when upstream moves file around.

VVicent W.

Nope. A fresh clone from main + a fresh venv didn't help.

VVicent W.

I'm reinstalling pytest now. My current copy is installed via brew. Maybe it doesn't respect Poetry venv Python binary???

VVicent W.

Ohhhhh

Attachment

VVicent W.

This is a fake binary specifically calling the Python binary at /opt/homebrew/opt/python@3.11/bin/python3.11.

VVicent W.

Die, you pytest!

Attachment

VVicent W.

Ok. Now it worked.

VVicent W.

@Logan M , do you think we should add pytest as a dev dependency to pyproject.toml?

VVicent W.

~~Oh wait it's already there~~. It's not. I was looking at an old revision.

VVicent W.

@Logan M , check out this: https://github.com/run-llama/llama_index/blob/1736097a5902ef39fd289a645ad9765058f1c8b3/pyproject.toml#L118

pytest = "7.2.1" was in pyproject.toml 3 months ago (0.9.23), but today it's not: https://github.com/run-llama/llama_index/blob/main/poetry.lock.

VVicent W.

I think this is why my Poetry venv kept defaulting to the brew-installed copy of pytest earlier today.

VVicent W.

Was it intentional to remove pytest from the dev packages group?

VVicent W.

It's the v0.10.0 commit that removed the pytest-related packages from the top-level pyproject.toml: https://github.com/run-llama/llama_index/commit/369973f4e8c1d6928149f0904b25473faeadb116

Attachment

VVicent W.

I wonder if we should add it back or remove pytest tests as an option to test the repo altogether.

LLogan M

Pytest is a lot easier than using pants imo

VVicent W.

I'll leave the add-pytest-back-or-not discussion for you official maintainers to discuss.

VVicent W.

I'm still catching myself up on what the heck is this Pants thing.

LLogan M

i generally wouldnt expect contributors to learn pants lol

LLogan M

its mostly there so that our cicd can work in a monorepo

VVicent W.

It's a cool tech, not gonna lie

LLogan M

I had fun setting it up haha, solves a very cool problem

VVicent W.

@Logan M , can you verify if these 2 unit tests are failing on main branch on your end?

Plain Text

FAILED tests/agent/react/test_react_agent.py::test_astream_chat_basic - AttributeError: 'NoneType' object has no attribute 'wait'
FAILED tests/agent/react/test_react_agent.py::test_async_add_step - AttributeError: 'NoneType' object has no attribute 'wait'

Here's how I set up my env:

Plain Text

cd llama-index-core # ! not the root dir.
poetry shell
poetry install --with dev,docs
pytest

Revision c367767a899a1471f6e468f804eb43091396db5a.

LLogan M

I can confirm these pass for me

If you scroll up, you should see the reason it's failing. Curious what the actual error is/line of code

VVicent W.

Here's the full log: https://pastebin.com/EApe15BJ

VVicent W.

I opened a small PR. Let's see if the CI passes there.
https://github.com/run-llama/llama_index/pull/12137

LLogan M

Hmm I wonder if that's related to this PR
https://github.com/run-llama/llama_index/pull/12116

VVicent W.

You're right. With that merge commit reverted, tests passed.

VVicent W.

But then I wonder how it escaped unit testing.

SShady El Gewily

My experience getting the tests to run are as follows:

I tried following the contributing.md guide: running pytest <package>tests/ directly, running make tests, even running the pants test command manually. None of the commands worked as I expected (running the tests). In the end, I had to navigate to this discord, find out other people were experiencing similar issues and resolve them manually in order to get the tests to run.
I expected all packages required to run the project's tests would be installed as part of --with dev option (or perhaps --with test), especially for the core package. Because this was not the case, I had to run poetry add: pytest, pytest_mock, pytest_asyncio, tree_sitter_languages.
After installing the required dependencies, I got failing tests

FAILED llama-index-core/tests/agent/react/test_react_agent.py::test_astream_chat_basic - AttributeError: 'NoneType' object has no attribute 'wait'
FAILED llama-index-core/tests/agent/react/test_react_agent.py::test_async_add_step - AttributeError: 'NoneType' object has no attribute 'wait'

I've spent quite some time on getting the project to run now, but still haven't been able to run all tests successfully. This is not so much of a critique, but more of a list of "gotchas" for other aspiring contributors to get things up and running. Perhaps it helps someone and feel free to add to this list if I'm missing something.

Would love to contribute more to the project, but getting the project up and running makes it harder to do that.

VVicent W.

Let me rewrite that.

VVicent W.

https://github.com/run-llama/llama_index/pull/12203

Add a reply

Find answers from the community

Pytest collects zero test cases. Am I