I have a suggestion.

ssansmoraxz

I have a suggestion.

There is an issue trying to incorporate the prompts for various languages. Sometimes it goes swimmingly like gpt4 - claude or vice versa.
Other times it struggles. Like trying to use those prompts with multilingual models, small models, custom specialized models, yada yada.

So how about some centralized hub for storing these prompt sets that can be dynamically set through Settings? The community can create and share entire promptsets.

18 comments

LLogan M

Thats definitely in the roadmap 🙂

LLogan M

First step is finding all the prompts in the codebase and centralizing them in some sort of registry

LLogan M

Then from there, customizing is straightforward

LLogan M

Probably a P2 priority right now though internally

ssansmoraxz

Are you all tracking issues outside github?
Other than the language one don't see corresponding issue

LLogan M

yea we have an internal list of priorities/projects, the prompt template thing is one of them 👍

For full transparency, the current planned stuff for the next month or two

improved callbacks (better interface, more coverage)
better docs (focus an actual API documentation, rather than flooding with random notebooks)
converting underlying query engines/agents to use new query pipeline syntax (and improve the query pipeline while doing this)
refactor the current KG Index, introduce better abstractions for building/querying KGs

This is on top of normal bug fixes and other small things of course

ssansmoraxz

Speaking of callbacks, not sure if it's there in your tracker, but some LLMs do have their own built-in token-counters. Probably some abstraction can be made to reflect actual values from it, rather than guesswork from tiktoken

LLogan M

openai (and APIs that have a similar api) already pull token usage from it 👍

LLogan M

https://github.com/run-llama/llama_index/blob/6fb1fa814fc274fe7b4747c047e64c9164d2042e/llama-index-core/llama_index/core/callbacks/token_counting.py#L53

ssansmoraxz

OK. bedrock's responses are different for different providers.

Will take a dig into this later.

LLogan M

classic bedrock 😆 Singular API definition? nahhh

ssansmoraxz

I wonder how vertex does it

ssansmoraxz

Or any of the public APIs for other LLMs

ssansmoraxz

I think the logic may be that the vendors get to define their APIs. Using REST through Anthropic's public API or through bedrock (or Cohere or other vendors) will be exactly the same.

LLogan M

ah I see, that makes sense. But also extremely hard to support nicely in a framework haha

ssansmoraxz

yes

I wonder if we can define some abstraction layer and pass it to the provider specific API class.
Like for Anthropic provider use the antropic class rather than redefining everything in bedrock class.

LLogan M

Yea I was thinking the same. Like, its basically just a different auth layer on top of antropic for example

LLogan M

probably the class could be refactored to handle this 🤔

Add a reply

Find answers from the community

I have a suggestion.