Storage

Question

For those who are creating datasets for evaluating their LLMs, where are you storing typically those datasets?

Logan M · Answer

S3, git lfs, huggingface datasets

Logan M · Answer

Could also be any sql-ish db, or nosql, depending on what's in the dataset

Ryan · Answer

If s3, or lfs, are you storing as json typically?

Logan M · Answer

yea itd just be a json blob. You could compress it if its really huge, but JSON is nice for less complexity

Logan M · Answer

(plus then you can just dump/load a pydantic model for example)

Ryan · Answer

Okay thanks. For my use-case, the datasets are smaller so I'll probably stick to JSON + git for now until I need something more robust.

Find answers from the community