Rate limit reached for text-embedding-ada-002
no matter how I adjust batch size. Changing to use_async = false fixes this but makes it very very slow and batch size dont apply when use_async = false for some reason. How can I embed a lot of documents quickly while without getting rate limit error? thank you!Generating embeddings: 45%| | 18/40 [01:11<01:27, 3.99s/it]
openai.error.RateLimitError: Rate limit reached for text-embedding-ada-002 in organization **** on tokens per min. Limit: 1000000 / min. Current: 992848 / min.
Generating embeddings: 88%|β | 17500/19999 [02:54<00:28, 86.19it/s]
, it stays /19999 no matter what batch size I set