CSV

LLi

Hello, I am new to Llama_index. I am working on a side project which is a RAG app. I have a few csv files and want to read/parse/clean the data, then embedding one field. May I know what is the best way to read a csv file and convert each row to an object? The first row is fieldname list. I tried SimpleDirectoryReader, but all the content of a row will be in the text field of a node. I searched here, only find this thread: https://discord.com/channels/1059199217496772688/1202932306982080572/1202932309754384384 . The bot suggested parsing the csv file and building Node manually. Is there any other better way? Thanks!

6 comments

WWhiteFang_Jr

You can try this CSV loader from llamahub: https://llamahub.ai/l/file-pandas_csv?from=all

WWhiteFang_Jr

Check the code here: https://github.com/run-llama/llama-hub/blob/main/llama_hub/file/pandas_csv/base.py

WWhiteFang_Jr

You can read per row or concat all rows together using concat_rows flag.

concat_rows (bool): whether to concatenate all rows into one document.

LLi

Thanks for your suggestion. But I found it can not meet my needs. I write a simple reader based on SimpleCSVReader which can choose the text field and other fields as metadata. Not sure if this is the best way, but it works for me. https://gist.github.com/shenli/36f7514a809535a76d9042fa2a15e58a

WWhiteFang_Jr

Yeah looks good if this fits your case. Thats what I find cool about the code written by LlamaIndex team. It's super easy to understand and tweak as per our requirements.

LLi

That's super helpful. Thanks!

Add a reply

Find answers from the community

CSV