I am trying to use `IngestionPipeline`

At a glance

The community member is trying to use the IngestionPipeline locally, but it has a dependency on llama_index_client which comes from a poetry module mapping that is not installed by default. Even when installing llamaindex-py-client with pip, there is no way to get this mapping. The community member is wondering if they can use IngestionPipeline standalone.

In the comments, another community member mentions that the mapping is only used in the CICD, and that llamaindex-py-client is included as a direct dependency of llama-index-core, which the ingestion pipeline comes from. Another community member suggests checking if pip show llamaindex-py-client shows anything, and that pip install llamaindex-py-client should install it if not.

The community member then confirms that the dependency was not properly installed, but was missed in the Dockerfile which had a --no-deps for the pip install. They were able to fix the dependency issue with the help of another community member.

The community members also discuss issues with the RedisVectorStore and a large PR to refactor it to use

Useful resources

GGeoloeG

I am trying to use IngestionPipeline to be executed locally but it has a dependency on llama_index_client which comes from a poetry module mapping:

Plain Text

poetry_requirements(
    name="poetry",
    module_mapping={"llamaindex-py-client": ["llama_index_client"]},
)

which is not installed by default, and even when installing llamaindex-py-client with pip, there is no way to get this mapping.

Therefore, I am wondering if we are able to use IngestionPipeline standalone ?

8 comments

LLogan M

hmm, this mapping is only used in our CICD, specifically by a tool called pants -- curious how this is causing issues?

llamaindex-py-client is included as a direct dependency of llama-index-core, which the ingestion pipeline comes from

GGeoloeG

https://github.com/run-llama/llama_index/blob/37c95965426bddae82cec1ad49d3aa82e8bfe819/llama-index-core/llama_index/core/ingestion/pipeline.py#L15

and the corresponding trace in message.txt

LLogan M

@GeoloeG I've never seen anyone have this issue before 😅 does pip show llamaindex-py-client show anything? pip install llamaindex-py-client should install it if not

GGeoloeG

indeed, it was not properly installed... but missed it in the Dockerfile which had a --no-deps for the pip install, let's see what are the conflicts...

GGeoloeG

fixed the dep issue! Thanks @Logan M

GGeoloeG

playing around with the RedisVectorStore, it seems that redisearch is a dep, but again is not installed with redis (5.0.1), will investigate and add this as dep if really needed for the integration

LLogan M

Actually, there's a large PR to refactor the redis vector store to use redisVL lol

LLogan M

Should be merged in a few days

Add a reply

Find answers from the community

I am trying to use `IngestionPipeline`