Classify

At a glance

The community members are discussing the use of large language models (LLMs) like OpenAI and BERT-like models for categorizing and routing incoming requests in a Kubernetes-based system. The post suggests using an LLM to determine if a request is "likely to be about YAML" or "free-form and not really YAML". The comments explore the trade-offs between using an LLM and a BERT-based model, noting that LLMs may be easier to use but less reliable for more complex categorization tasks, while BERT-based models require more effort to set up but can be more reliable. The community members also discuss the need for more sophisticated request preprocessing and routing in the future, and how an LLM could potentially be used for this purpose.

tthoraxe

Not specifically a llamaindex question, and possibly not even LLM.
In the context of Kubernetes, I can use an LLM like OpenAI to "categorize" questions as being "likely to be about yaml" or "free-form and not really yaml"
for example:

query: give me a resource definition to run mongodb
answer: yaml

query: help me understand why this pod is crashing
answer: free form

Someone suggested this could potentially be achieved with BERT-like "sentiment-analysis". It just seems like using an LLM to do this is overkill/resource intensive.

9 comments

LLogan M

Yea exactly, a BERT model could be used to classify sentences

The one downside compared to LLMs is there's not really a "reliable out-of-the-box" model for doing this.

Usually you take a base model like BERT and add a classification layer on top, and then train it for your specific task.

The hard part about this is getting quality training data. Whereas with an LLM, you can usually get ok-ish results with little effort

tthoraxe

yes there's definitely a cost/trade-off

tthoraxe

calls to OpenAI are not free
hosting an LLM requires a GPU
training a BERT/sentiment system can be hard to get good data for

LLogan M

Yea that pretty much sums it up!

Personally, the cost involved with calling openai seems worth it compared to the alternatives 😅

But you'll find LLMs are only good at this task if you have a small set of labels. Once you go beyond a handful, I feel the smaller BERT approach would be more reliable.

At a previous job we classified products out of over 400 categories. No way would I use an LLM for that 😅

tthoraxe

conceptually we are designing a system that can "route" incoming requests to the right endpoint service. in this case, right now, the only service is a yaml generator. so we want to know "therequest is for yaml?" and then either reject it "sorry, i can only YAML" or pass it along

tthoraxe

in the future we will need more sophisticated analysis of the incoming query to be able to route appropriately, or filter, or whatever

tthoraxe

basically a bunch of "pre-processing" of the prompt before deciding the next thing (rag? just pass through? something else?)

LLogan M

Yea that makes sense. And in that case, an LLM makes sense I think?

Because you can combine the routing and preprocessing into a single LLM call as well

With openai, I would be using their function calling feature that let's it output a structured pydantic object to capture these details.

With other LLMs, structured outputs (like asking for JSON) works too

LLogan M

The concept is very similar to how the sub question query engine works

Add a reply

Find answers from the community

Classify