Chunks

At a glance

I'm trying to break the text into chunks of 500, with a 20 overlap each. But the following doesn't seem to be working. Any suggestions on what I'm doing wrong here?

Plain Text

llm_predictor = LLMPredictor(llm=OpenAI(temperature=0,
                                        model_name=llama_model,
                                        max_tokens=512))

#prompt helper
context_window = 4096
num_output = 512 # set number of output tokens
chunk_overlap_ratio = 0.04 #set chunk overlap ratio, where .04 of 500 = 20 overlap
chunk_size_limit = 500 #set limit of chunk size

prompt_helper = PromptHelper(context_window, num_output, chunk_overlap_ratio, chunk_size_limit)


service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper = prompt_helper)

index = VectorStoreIndex.from_documents(documents, service_context = service_context, show_progress = True)

Suggestions are welcome!

5 comments

WWhiteFang_Jr

You'll need to pass the chunk_size kwarg or define node_parser in the service_context

For ex:

Plain Text

from llama_index import ServiceContext, LLMPredictor, OpenAIEmbedding, PromptHelper
from llama_index.llms import OpenAI
from llama_index.text_splitter import TokenTextSplitter
from llama_index.node_parser import SimpleNodeParser

llm = OpenAI(model='text-davinci-003', temperature=0, max_tokens=256)
embed_model = OpenAIEmbedding()
node_parser = SimpleNodeParser.from_defaults(
  text_splitter=TokenTextSplitter(chunk_size=1024, chunk_overlap=20)
)
prompt_helper = PromptHelper(
  context_window=4096,
  num_output=256,
  chunk_overlap_ratio=0.1,
  chunk_size_limit=None
)

service_context = ServiceContext.from_defaults(
  llm=llm,
  embed_model=embed_model,
  node_parser=node_parser,
  prompt_helper=prompt_helper
)

WWhiteFang_Jr

For more ref: https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/root.html

yyaya90

Thanks. Can you explain the difference between the chunk_size for the text splitter and the chunk_size_limit in the prompt helper? Similarly, what is the point of the chunk_overlap_ratio? Not sure I understood this distinction yet.

WWhiteFang_Jr

chunk_size: Defines the chunk size, Basically the fixed size of node object. Nodes will not exceed this size.

chunk_size_limit : In this case, it defines the size of chunk of information that is to be passed to LLM for response generation.

WWhiteFang_Jr

chunk_overlap_ratio: Found this in the code for this:
chunk_overlap_ratio (float): Chunk overlap as a ratio of chunk size

Add a reply

Find answers from the community

Chunks