The community member has built a LlamaIndex-based app with FastAPI APIs, including one to return a ContextChatEngine object and another to return a response when the user queries. However, the community member is unable to return the ContextChatEngine object because it is not serializable/deserializable, resulting in a "TypeError: cannot pickle 'builtins.CoreBPE' object" error.
The community members discuss potential solutions, such as serializing the main settings needed to reconstruct the chat engine on the client side or providing the chat interface itself over the API. The latter option is suggested as a better design, where the backend manages all the conversations and engines, and the client interacts with API endpoints like "/chat/{user_id}" to post messages to a specific chat engine.
The community members also discuss that this issue is not specific to FastAPI, but would likely occur with any API framework, as serializing complex objects like chat engines or indexes can be challenging. The suggested solution is to either cache active conversations in memory and load/reload the index and chat engine as needed, or send the information required to reconstruct the chat engine on the client side.
So I have built this llamaindex based app and used FastAPI to create the relevant APIs.
My app has a feature to chat with the video transcript. Now for this I have exposed 2 APIs - one to return the ContextChatEngine object and the other to return a response whenever the user types the query, with this object by calling query_engine.query().
But I am not able to return this ContextChatEngine object because the ContextChatEngine class is not serializable/deseriazable.Caling the API is throwing "TypeError: cannot pickle 'builtins.CoreBPE' object" and creating a custom response class is throwing an error too. Any idea how to fix this?
So the experience is like this - The moment you submit the video link, I create the ContextChatEngine object by calling the "/get_chat_engine" endpoint and then on subsequent submissions of query inputs I call the "/chat" endpoint. This is the code :-
@app.get("/get_chat_engine", response_model=None) def get_chat_engine(yt_video_link: str): index = initialize_index(yt_video_link)
system_prompt = f""" You are a friendly and helpful mentor whose task is to \ use ONLY the context information and no other sources to answer the question being asked.\ If you don't find an answer within the context, SAY 'Sorry, I could not find the answer within the context.' \ and DO NOT provide a generic response."""
I think you will have this problem with any API framework
By providing the chat interface over the API, I mean the backend should probably be managing all the converstations/engines. Then you could have API endpoints like @app.post(/chat/{user_id}) -- where messages get posted to a specific chat engine, managed by some ID? 🤔
yea possibly! or it could even be managed in-memory depending on the scale
Basically, serializing this stuff is pretty hard. It's probably better to cache active convserations in memory, and long term load/re-load the index and chat engine after a certain period of inactivity
Otherwise, you'll need to send over information needed to re-construct the chat engine on the client side
