Maybe double check that the pipeline is returning a string.
Usually it returns a list of dict, and you need to do something like response = pipeline(...)[0]["generated_text"]
Hmmmm... I'll check that out
Yea, the _call function of the custom llm needs to return a string, otherwise you get that error
That worked! Now I'm getting "Larger chunk overlap (20) than chunk size (-354)". Hmm
Thats a classic 🫠 once the prompt helper settings and chunk size limit are tweaked, weird stuff can happen
I think there is a fix just released yesterday that might fix this.
Otherwise, you'll have tonplay around with chunk size settings I think 😞
Played around with them, got me to the next error:
The size of Tensor a must match the size of tensor b at non-singleton dimension 3.
Searching seems to suggest that it's a problem with pictures, but my indices should only contain text, because it using a directory reader to parse pdf files.
Hmmm does the error tell you if it's embeddings or the llm?
Looks like it's in the transformers/models/gpt_neox/modeling_gpt_neox.py
torch.where(causal_mask, attn_scores, mask_value
)
I was breaching the input token limit... 😒
Need to make sure the max input size is set in the prompt helper to be small enough 👀
Now just trying the right combination of values to avoid the "Larger chunk overlap than chunk size" error.
What llama index version do you have btw? I think 0.6.0a6 should have a fix for that
If you have that version, I guess the fix is not perfect hah
Updated, but seems like many things have changed and my code is broken all over the place! 😖
I'll need to work it later, but some of the changes look like they are going help streamline what I am trying to do...
Oof, yea many changes. Sorry I thought maybe you were already on 0.6.x lol
But the changes are good!
No worries, yeah I might go back to the drawing board with how I'm setting up this app, there might be some opportunity to make it better with some of the changes.
