Find answers from the community

Updated 10 months ago

It's unfortunate that we have to `time.

It's unfortunate that we have to time.sleep(0.01) in the StreamingAgentChatResponse.
In JavaScript, there's a trick to set a timeout of 0ms for upcoming work and it achieves the same goal of allowing other workers to process things ahead of our work, but allows our work to proceed as quickly as possible. Can we do that in Python too? What happens if we time.sleep(0) instead?
L
t
27 comments
I think without this, users were reporting cpu usage of 100% lol
let me find the original issue
I'm not sure if time.sleep(0) will have the same effect in python πŸ€”
the user seemed to indicate huggingface uses the same time.sleep lol
Copilot suggests it will work but...it's copilot so unsure
In theory time.sleep(...) is a blocking function so the python process should release the GIL.
That's what we're doing here, right? Releasing the GIL so another process can add things to a Queue?
I thiiiink thats the intention yes
apparently it works? For windows at least
Looks promising.
(also just because someone at HuggingFace does something, doesn't make it right)
According to StackOverflow, the decision of "who goes next" is up to the scheduler of the OS.

So I think this time.sleep(0) trick will work. Your overall responses should be faster but:
1) it will still consume 100% CPU however
2) it will be polite and let other threads go (as long as the scheduler allows it)
isnt consuming 100% CPU an issue though? πŸ˜… Perhaps I should try replicating this again
but I see the point, it should allow other tasks to be scheduled and run properly
It's not an issue as long as you're polite and letting other threads have a turn
And it'll only be 100% for the duration of the streamed response
I'll try to get to this change today at some point
This response especially leads me to believe this is a good trick in this case
(github has been popping off with issues/PRs this week lol)
Yeah fair - v0.10 was a big change. Our team (which uses Bazel) had issues with v0.10 over some Bazel build files that were in the release
oh hey there was an issue about that! I need to fix that too :PSadge:
I think we raised the issue πŸ™‚
haha nice πŸ˜… Yea I need to edit every pyproject.toml and then also version bump + publish every package. I should let this run overnight tonight tbh
I didn't expect the BUILD thing to be an issue when adding pants, whoops
Pull Request to sleep for 0 seconds in 3 cases, to release the GIL but gives others a turn.
Add a reply
Sign up and join the conversation on Discord