llama_index/PandasIndexDemo.ipynb at mai...

ttrungbb

Hi all when i use PandasIndex the same with demo https://github.com/jerryjliu/llama_index/blob/main/examples/struct_indices/PandasIndexDemo.ipynb then i use index.save_to_disk('index_csv.json') it work ok , but when i load it again by index = GPTPandasIndex.load_from_disk('ndex_csv.json') i got error "ValueError: Only one of documents or index_struct can be provided." Anyone get this error ? Thanks

12 comments

LLogan M

With a pandas index, all the data is actually in the dataframe. So save to index isn't actually saving anything (Although yes, this is still a bug)

You can re-build the index and pass in the dataframe for the same results

ttrungbb

@Logan M So each time we query we have to re-build the index 😦 . Thanks for your answer

LLogan M

Building the index should be fast if the dataframe is loaded already 🤔 it would be the same speed if it was loading from disk

If you can, keep the index loaded in some global variable maybe?

ttrungbb

Thanks

ttrungbb

@Logan M Do you have any ideas if csv file is too large

LLogan M

Hmm I know pandas 2.0 is supposed to handle bigger files better 🤔

ttrungbb

thanks

ttrungbb

@Logan M Sorry i have other question about Composability . i have 2 GPTPandasIndex , index1 and index2. Then make graph like this graph =

ComposableGraph.from_indices(
      GPTListIndex,
      [index1, index2], 
      index_summaries=[index_detail_sum, index_sum_sum],
    max_keywords_per_chunk=100
)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/llama_index/indices/composability/graph.py", line 114, in from_indices
    assert isinstance(index.index_struct, V2IndexStruct)
AssertionError

Do you have any ideas ? Thanks

LLogan M

Hmmm this looks like a bug. I don't think pandas index got migrated to V2 (from the looks of the error) 🙃

ttrungbb

@Logan M So have to wait ?:(

LLogan M

I think so. Maybe file an issue on github so it can be tracked (or you can try tackling the PR yourself!)

ttrungbb

Yes thank i will post this issue

Add a reply

Find answers from the community

llama_index/PandasIndexDemo.ipynb at mai...