llama-index would help me here. I would like to integrate with Athena and read the schema of the specified table, craft the sql query, execute it on Athena and then return the results on something like a streamlit application. From the users perspective i would like them to enter something like From the event model can you show me the number of minutes viewed by device between the dates 20230101 and 20230131 inclusive. Is this something that i can do via llama-index or should i be using another framework. My approach would be to read the event schema from Athena, pass it in as part of the context, along with the user request and then send it to OpenAPI```json
[
{
"question": "What problem does Bitcoin aim to solve in the context of online payments?"
},
{
"question": "How does Bitcoin propose to prevent double-spending without a trusted third party?"
},
{
"question": "What role does the proof-of-work chain play in the Bitcoin network?"
},
{
"question": "Why is the longest chain in the Bitcoin network considered authoritative?"
},
{
"question": "How does the Bitcoin network ensure its integrity against attackers?"
},
{
"question": "What are the inherent weaknesses of the trust-based model in traditional electronic payments?"
},
{
"question": "How does the requirement for mediation by financial institutions affect transaction costs and sizes?"
},
{
"question": "What impact does the possibility of transaction reversal have on merchants and customers?"
},
{
"question": "In what way does Bitcoin's peer-to-peer network maintain minimal structure?"
},
{
"question": "How does the ability to make non-reversible payments benefit transactions for non-reversible services?"
}
]def process_document(dbdir):
chroma_client = chromadb.PersistentClient(path=dbdir)
chroma_collection = chroma_client.get_or_create_collection("bitcoin")
vector_store = ChromaVectorStore(chroma_collection)
llm = OpenAI(model="gpt-4-0125-preview")
loader = PyMuPDFReader()
docs = loader.load_data(file_path=os.path.join(os.path.dirname(__file__), "..", "docs", "bitcoin.pdf"))
for doc in docs:
doc.id_ = hashlib.sha256(doc.text.encode('utf-8')).hexdigest()
click.echo(f"Loaded {len(docs)} documents")
embed_model = OpenAIEmbedding()
extractors = [
SemanticSplitterNodeParser(buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model),
TitleExtractor(nodes=5),
SummaryExtractor(summaries=["prev", "self", "next"]),
QuestionsAnsweredExtractor(questions=10, metadata=MetadataMode.EMBED),
KeywordExtractor(keywords=5),
embed_model
]
pipeline = IngestionPipeline(transformations=extractors, vector_store=vector_store, cache=IngestionCache())
processed_nodes = pipeline.run(documents=docs, show_progress=True, store_doc_text=True, store_doc_metadata=True)
click.echo(f"Processed {len(processed_nodes)} nodes")An error occurred: 'TextNode' object has no attribute 'get_doc_id'
from llama_index import ServiceContext service_context = ServiceContext.from_defaults(chunk_size=1000)