LlamaIndex
Log in

Find answers from the community

Home
Members
Fried cheese
F
Fried cheese
Offline, last seen 3 months ago
Joined September 25, 2024
FFried cheese
·

can i limit the size of summary

can i limit the size of summary generated by the summary extractor?
2 comments
F
L
FFried cheese
·

I am not able to use one of the loaders

I am not able to use one of the loaders (https://llamahub.ai/l/web-unstructured_web?from=all). Can any one check on this?
2 comments
F
W
FFried cheese
·

```

Plain Text
node_parser = SentenceSplitter(chunk_size=512,chunk_overlap=20)
base_nodes = node_parser.get_nodes_from_documents(documents,show_progress=True)
base_index = VectorStoreIndex(base_nodes, service_context=service_context)
base_retriever = base_index.as_retriever(similarity_top_k=2)

is this code block same as
Plain Text
service_context = ServiceContext.from_defaults(embed_model=embed_model,chunk_size=512,llm=llm)
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context, service_context=service_context
)
2 comments
F
L
FFried cheese
·

Metadata

Im trying to get summaries+ generated questions for a html read using unstructured, and parsed using node parser. It has been extract summaries since 30 minutes just for 1 html, is this normal? I am using a T4 with llama 2 in colab.
9 comments
F
L
FFried cheese
·

Asynci

Will be help if someone lets me know if this is an issue on my side (because its defined in the function, and I'm referring this from https://docs.llamaindex.ai/en/stable/examples/retrievers/recursive_retriever_nodes.html#metadata-references-summaries-generated-questions-referring-to-a-bigger-chunk)
3 comments
F
L
FFried cheese
·

I am using unstructured loader to load

I am using unstructured loader to load documents in my chroma vector store. Now before i run vectorstorefromdocuemnts to load them in to my db, can i get the nodes in a variable? I want them in a variable so that i can setup a recursive retriever for questions+ summaries
Plain Text
dir_reader = SimpleDirectoryReader('./data/download', file_extractor={
  ".html": UnstructuredReader(),
})
documents = dir_reader.load_data()
2 comments
F
r
FFried cheese
·

Hey. I'm getting this weird key error

Hey. I'm getting this weird key error after creating an index from documents extracted via unstructured module (html here). if rebuilt the index, and the key changed from -1 to 0, and now its stuck on 1. Any idea?
18 comments
F
L
FFried cheese
·

does anyone know how to fix this?

does anyone know how to fix this?
16 comments
F
L
FFried cheese
·

I want to prevent myself from

I want to prevent myself from accidentally using too much of my open ai key, so I want to check the header returned. is it possible, when i am using the extractor to generate summaries?
1 comment
r