LlamaIndex
Log in

Find answers from the community

Updated 2 months ago

Docstore

ggamecode8
·
Hello, can I have some clarification on what the docstore is intended to store? Chunks or full document?

The documentation here states chunks: https://docs.llamaindex.ai/en/stable/module_guides/storing/docstores/

However, I have found that the ingestion pipeline stores the full document text before chunking when using document management.

https://docs.llamaindex.ai/en/stable/module_guides/loading/ingestion_pipeline/#document-management

Thank you!
L
f
g
7 comments
LLogan M
Both 🙂
LLogan M
A document and a chunk are essentially the exact same class under the hood
LLogan M
The docstore can store both
ffarzzy528
cheesyfishes
ffarzzy528
whats your thoughts on contextual retreival from anthropic?
LLogan M
its just metadata/chunk enrichment -- nothing new really. Us and other libraries have had similar modules for ages. The cooler part about it is leveraging prompt caching to make it scalable
ggamecode8
Awesome thanks for the clarification Logan
Add a reply
Sign up and join the conversation on Discord