Find answers from the community

Updated 8 months ago

How can we execute batch inference

How can we execute batch inference optimally using query_pipeline?
While the documentation provides insights on asynchronous, parallel, and multi-root operations, it lacks details on batch inference methods.
Would employing asynchronous execution be the most effective approach?"
Z
d
L
3 comments
you can use multiprocessing pool
Yea, batch is not technically implemented yet
Add a reply
Sign up and join the conversation on Discord