what is the difference between TokenSplitter and SentenceSplitter, which items are the bes

1. TokenTextSplitter:

from llama_index.core.node_parser import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=1024,
    chunk_overlap=20,
    separator=" ",
)
nodes = splitter.get_nodes_from_documents(documents)

1. SentenceSplitter:

from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(
    chunk_size=1024,
    chunk_overlap=20,
)
nodes = splitter.get_nodes_from_documents(documents)

1. First, set up your retriever:

from llama_index.core import VectorStoreIndex
from llama_index.core.retrievers import VectorIndexRetriever

# Assuming you have already created your index
index = VectorStoreIndex.from_documents(documents)

# Create a retriever
retriever = VectorIndexRetriever(
    index=index,
    similarity_top_k=10  # Adjust as needed
)

1. Define your postprocessors. LlamaIndex offers various postprocessors, such as KeywordNodePostprocessor, SimilarityPostprocessor, and others. Here's an example of setting up multiple postprocessors:

from llama_index.core.postprocessor import KeywordNodePostprocessor, SimilarityPostprocessor

# Create postprocessors
keyword_postprocessor = KeywordNodePostprocessor(
    required_keywords=["your_keyword"],
    exclude_keywords=["exclude_word"]
)

similarity_postprocessor = SimilarityPostprocessor(similarity_cutoff=0.7)

# Combine postprocessors into a list
node_postprocessors = [keyword_postprocessor, similarity_postprocessor]

1. Create a query engine that combines the retriever and postprocessors:

from llama_index.core.query_engine import RetrieverQueryEngine

query_engine = RetrieverQueryEngine.from_args(
    retriever,
    node_postprocessors=node_postprocessors
)

1. Now you can query the retriever with the postprocessors applied:

response = query_engine.query("Your query here")
print(response)

1. First, import the necessary classes:

from llama_index.core.objects import ObjectRetriever
from llama_index.core.base.base_retriever import BaseRetriever
from llama_index.core.postprocessor import BaseNodePostprocessor
from llama_index.core.schema import NodeWithScore, QueryBundle

1. Create your base retriever and node postprocessors:

# Assume you have already created your base_retriever
base_retriever: BaseRetriever = ...

# Create your node postprocessors
node_postprocessors: List[BaseNodePostprocessor] = [
    # Add your postprocessors here
]

1. Create an ObjectRetriever that wraps your base retriever and applies the postprocessors:

retriever = ObjectRetriever(
    retriever=base_retriever,
    object_node_mapping=lambda x: x,  # Identity mapping for NodeWithScore objects
    node_postprocessors=node_postprocessors
)

1. Now you can use this retriever to fetch and postprocess nodes without going through an LLM:

query = "Your query here"
retrieved_nodes = retriever.retrieve(query)

1. Purpose:
   The primary purpose of an ObjectRetriever is to retrieve objects based on a query. It's particularly useful when you want to index and retrieve custom Python objects, such as tools, database schemas, or any other structured data.

1. Functionality:
   
   • It wraps a base retriever (like a vector store retriever) and adds the ability to work with objects.
   • It can automatically apply node postprocessors to the retrieved results.
   • It converts between nodes (used by the underlying index) and objects.

1. Common Use Cases:
   
   • Tool Retrieval for Agents: One of the most common uses is to retrieve relevant tools for AI agents. This is demonstrated in the OpenAI Agent with Tool Retrieval example.
   • Indexing and Retrieving Custom Objects: It can be used to create an index of any type of Python object and then retrieve those objects based on queries.

1. How it works:
   
   • It uses an underlying index (often a VectorStoreIndex) to store and retrieve nodes.
   • It employs an object-node mapping to convert between objects and nodes.
   • When retrieving, it first uses the base retriever to get relevant nodes, then applies any specified node postprocessors, and finally converts the nodes back to objects.

1. Example Usage:
   Here's a simplified example of how you might use an ObjectRetriever:

Create an ObjectIndex

1. The system checks if a document is already in the document store based on its ID.
2. If the document is not in the store, or if the hash of the document has been updated, it will update the document in the document store and run the transformations.

1. If the document with this ID doesn't exist, it will be inserted.
2. If the document exists but its content has changed (different hash), it will be updated.
3. If the document exists and hasn't changed, the operation will effectively do nothing.

1. Automatic ID Generation: When you use a document loader like SimpleDirectoryReader without specifying document IDs, LlamaIndex will generate unique IDs for each document automatically. These IDs are typically UUID-based.

1. Implications for Upserts: If you're using the DocstoreStrategy.UPSERTS as in your previous pipeline configuration, the behavior will be slightly different:

• For new documents (first-time ingestion), they will be inserted with their automatically generated IDs.
• For subsequent ingestions, since the IDs are generated anew each time, the system will treat these as new documents rather than updates to existing ones.

1. Potential for Duplicates: Without consistent doc_ids across ingestions, you might end up with duplicate content in your index if you ingest the same documents multiple times.

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader("./data", filename_as_id=True).load_data()

1. Creating Documents Manually:

from llama_index.core import Document

documents = []
for i, text in enumerate(parsed_texts):
    doc = Document(
        text=text,
        doc_id=f"custom_id_{i}"  # or any other unique identifier
    )
    documents.append(doc)

1. Using Metadata for Identification:

for item in json_data:
    doc = Document(
        text=item['text'],
        doc_id=item['unique_identifier'],
        metadata=item.get('metadata', {})
    )
    documents.append(doc)

1. Generating Consistent IDs:

import hashlib

for text in parsed_texts:
    doc_id = hashlib.md5(text.encode()).hexdigest()
    doc = Document(text=text, doc_id=doc_id)
    documents.append(doc)