Apologies in advance, as my knowledge

PPunkbit

Apologies in advance, as my knowledge regarding OpenAI, LLM, etc, are pretty poor. But as a software developer, given tools like Llama Index, it helps reduce friction.

I've done a few model tests to figure out which provides reasonable computation considering cost. I found that GPT-4 didn't work for me, because of the rate limit–I'd expect to have the ability to make a request at least–, and gpt-4-1106-preview worked but its "preview" with further limitations. In any case, GPT-4 is pricier to GPT-3.5. Although, GPT-3.5 doesn't seem to work that well.

So, this brings me into questioning how to determine the number of TPM my request requires. The use case is scrapping several pages with the help of SimpleWebPageReader, I believe that the number of URL sources might have a big role on this, correct?

For this reason, I'm thinking that it is best to create a single page with FAQ instead and put this instead of the multiple URL sources.

Any hints or suggestions to improve performance?

4 comments

TTeemu

Are you on the free plan? The rate limits on the paid plan should be enough to handle it. The scraping + indexing part shouldn't really incur rate limits since you're not actively usually calling an LLM during it. You'll incur embedding token usage but the paid plan has like 5mil / min rate limits I think

TTeemu

But for looking at TPM you can check your OpenAI usage page or use openai.log = "debug" to track your API requests

PPunkbit

@Teemu when I set a sitemap (multiple URLs) it doesn't work at all.

I haven't found how to set the openai.log = "debug" based in the documentation with the current setup I have. By simple importing and setting to "debug" doesn't seem to work at all. This seems because I use llama_index.llms import OpenAI instead

Plain Text

...
from llama_index.llms import OpenAI
import openai

openai.log = "debug"
app = FastAPI()
loader = SitemapReader()

llm = OpenAI(temperature=0.1, model="gpt-3.5-turbo")
service_context = ServiceContext.from_defaults(llm=llm)

Attachments

PPunkbit

I'm on tier 1 btw

Add a reply

Find answers from the community

Apologies in advance, as my knowledge