So I have built this llamaindex based app and used FastAPI to create the relevant APIs.
My app has a feature to chat with the video transcript. Now for this I have exposed 2 APIs - one to return the ContextChatEngine object and the other to return a response whenever the user types the query, with this object by calling query_engine.query().
But I am not able to return this ContextChatEngine object because the ContextChatEngine class is not serializable/deseriazable.Caling the API is throwing "TypeError: cannot pickle 'builtins.CoreBPE' object" and creating a custom response class is throwing an error too. Any idea how to fix this?
So the experience is like this - The moment you submit the video link, I create the ContextChatEngine object by calling the "/get_chat_engine" endpoint and then on subsequent submissions of query inputs I call the "/chat" endpoint. This is the code :-
@app.get("/get_chat_engine", response_model=None) def get_chat_engine(yt_video_link: str): index = initialize_index(yt_video_link)
system_prompt = f""" You are a friendly and helpful mentor whose task is to \ use ONLY the context information and no other sources to answer the question being asked.\ If you don't find an answer within the context, SAY 'Sorry, I could not find the answer within the context.' \ and DO NOT provide a generic response."""
I think you will have this problem with any API framework
By providing the chat interface over the API, I mean the backend should probably be managing all the converstations/engines. Then you could have API endpoints like @app.post(/chat/{user_id}) -- where messages get posted to a specific chat engine, managed by some ID? 🤔
yea possibly! or it could even be managed in-memory depending on the scale
Basically, serializing this stuff is pretty hard. It's probably better to cache active convserations in memory, and long term load/re-load the index and chat engine after a certain period of inactivity
Otherwise, you'll need to send over information needed to re-construct the chat engine on the client side