Find answers from the community

Updated 4 months ago

Llamaindex workflow compatibility with Redpanda and Confluent

At a glance

The community members are discussing whether the LlamaIndex Workflow can work with Redpanda, a Kafka-compatible message queue, or if it requires Confluent Kafka. Some community members note that Redpanda implements the Kafka protocol and could potentially be used as a substitute, while others mention that the LlamaIndex Workflow may not actually use a message queue, but rather single-process asyncio queues. The discussion also touches on the minimum setup required for the LlamaIndex Workflow to use Kafka, and whether Redpanda could be used in a Docker deployment. Overall, the community members seem to believe that Redpanda could potentially work with the LlamaIndex Workflow, but there is no definitive answer provided.

Useful resources
Does LlamaIndex Workflow work with Redpanda? Or it needs to be confluent?
L
d
12 comments
I've never used redpanda -- not sure what that would look like
It implements the Kafka protocol but is might lighter weight.
C++ and Raft impl of Kafka standard.
hmm, workflows don't use a message queue. Its all just single-process asyncio queues πŸ‘€

redpanda could be a llama-deploy integration in the future though, if the existing kafka message queue doesn't work with it
So do you need to install Kafka broker for it to work?
What's the minimum setup for Llama workflow that uses Kafka ?
I don't see any reason for a docker deployment why RedPanda could not substitute here https://github.com/run-llama/llama_deploy/blob/main/docker/docker-compose-kafka.yml
Because it implements the Kafka protocol. Can you please check this. Thanks
assuming the protocol is the same, i'd assume it just works 🀷
Yes, that's what I am hearing from RedPanda as well. I think it should work
Add a reply
Sign up and join the conversation on Discord