Find answers from the community

s
F
Y
a
P
Updated 2 years ago

Is there a way to change encoding on

Is there a way to change encoding on SimpleDirectoryReader().load_data()?

I am loading in the articles with some of them having special characters.
Attachments
image.png
image.png
j
E
6 comments
atm no, it's a good point, we should have this support soon
Alright so at least I can rest on that for the moment instead of going crazy to figure out how to do it. Thanks
I found a temporary solution by making a custom prompt and telling it to spell check everything @jerryjliu0
haha that means an extra LLM call right?
@Erik will try to take a look at this today, sorry about that
Works better with extra call yes but I as for my use, I was only concerned about the output of final answer being in weird characters because of the decoding. So just telling it to spell check everything in the custom QA_PROMPT_TMPL did the thing @jerryjliu0
Add a reply
Sign up and join the conversation on Discord