LlamaIndex

Log inLog into community

Find answers from the community

Updated 10 months ago

It's unfortunate that we have to `time.

It's unfortunate that we have to `time.

ttimothybeamish

·

It's unfortunate that we have to time.sleep(0.01) in the StreamingAgentChatResponse.
In JavaScript, there's a trick to set a timeout of 0ms for upcoming work and it achieves the same goal of allowing other workers to process things ahead of our work, but allows our work to proceed as quickly as possible. Can we do that in Python too? What happens if we time.sleep(0) instead?

L

t

27 comments

I think without this, users were reporting cpu usage of 100% lol

let me find the original issue

https://github.com/run-llama/llama_index/issues/10290
https://github.com/run-llama/llama_index/pull/10339

I'm not sure if time.sleep(0) will have the same effect in python 🤔

the user seemed to indicate huggingface uses the same time.sleep lol

ttimothybeamish

Copilot suggests it will work but...it's copilot so unsure

ttimothybeamish

In theory time.sleep(...) is a blocking function so the python process should release the GIL.

ttimothybeamish

That's what we're doing here, right? Releasing the GIL so another process can add things to a Queue?

I thiiiink thats the intention yes

https://discuss.python.org/t/time-sleep-0-yield-behaviour/27185

apparently it works? For windows at least

ttimothybeamish

Looks promising.
(also just because someone at HuggingFace does something, doesn't make it right)

lol of course

ttimothybeamish

According to StackOverflow, the decision of "who goes next" is up to the scheduler of the OS.

So I think this time.sleep(0) trick will work. Your overall responses should be faster but:
1) it will still consume 100% CPU however
2) it will be polite and let other threads go (as long as the scheduler allows it)

isnt consuming 100% CPU an issue though? 😅 Perhaps I should try replicating this again

but I see the point, it should allow other tasks to be scheduled and run properly

ttimothybeamish

It's not an issue as long as you're polite and letting other threads have a turn

ttimothybeamish

And it'll only be 100% for the duration of the streamed response

I'll try to get to this change today at some point

ttimothybeamish

This response especially leads me to believe this is a good trick in this case

(github has been popping off with issues/PRs this week lol)

ttimothybeamish

Yeah fair - v0.10 was a big change. Our team (which uses Bazel) had issues with v0.10 over some Bazel build files that were in the release

oh hey there was an issue about that! I need to fix that too :PSadge:

ttimothybeamish

I think we raised the issue 🙂

haha nice 😅 Yea I need to edit every pyproject.toml and then also version bump + publish every package. I should let this run overnight tonight tbh

I didn't expect the BUILD thing to be an issue when adding pants, whoops

ttimothybeamish

Pull Request to sleep for 0 seconds in 3 cases, to release the GIL but gives others a turn.

Add a reply

Sign up and join the conversation on Discord