Find answers from the community

Updated 4 months ago

Nodes

At a glance

i have a very basic question, when we generate nodes as below. Will the len(nodes) differ with respect to hardware? the reason for this question is when i see the len(nodes) on my Mac M1 Pro i get it as approx 900 but when i use kaggle or colab with p100 GPU i get only 60

Note: Assume my chunk size is constant at Settings.chunk_size=512

documents = SimpleDirectoryReader("./data/").load_data()
node_parser = SentenceSplitter()
nodes = node_parser.get_nodes_from_documents(documents)

3 comments

LLogan M

The nodes only depend on how much data you load. There is no gpu or hardware dependent operations involved here

LLogan M

I'm guessing you aren't loading the same data on each machine?

ppavanmantha

That’s what my understanding was, I am loading the same dataset any have will check it and thanks to clarify my doubt.

Add a reply