Hi everyone! Has anyone managed to get

HHamed

Hi everyone! Has anyone managed to get source_nodes before the response is generated in the streaming cases?

sec-insight is doing something similar by showing the SubQuery questions before the response is generated but the code is way too complex for my understanding (using anyio and different channels for each process). I was wondering if there's a simpler way?

I tried with something like this but doesn't work :/

Plain Text

streaming_response = chat_engine.stream_chat(last_message.content)

message = {
    "type": "sources",
    "data": [str(i) for i in streaming_response.source_nodes],
}
yield "Context received, before LLM call: " + str(time.time())
yield "\n\n"

for token in streaming_response.response_gen:
    yield token

Using it for streaming in FastAPI backend.

7 comments

LLogan M

sec-insights is using a custom callback handler to grab nodes. It allows them to be shown to the user much faster.

I'm curious why the above doesn't work though? Seems like it should at first glance

HHamed

Thanks Logan! It seems there are these topics here:
Streaming source:

The frontend implementation is also important. It needs to parse the object as it comes. At the moment using Vercel/AI & NextJS which doesn't support such a functionality.

Fetching the sources:

I think a better way to run the chat_engine would be to use the astream_chat to (hopefully) get the sources before full generation. However, it might be needed to do a manual retrieval.
In case of manual retrieval, not sure how to add it as the context in chat() method (without passing it as a big message so the response object will have correct metadata).

Any code example for how to do 1 & 2 would be great (frontend api call and async chat stream)

P.s. I intentionally switched to synced version because of issues in PGVectorStore . Even though I've set Async Conn Str, it errors out with an error about pgbouncer

Attachment

kkevingoed

@Hamed did you manage to get this resolved, just ran into the same error with pgbouncer.

HHamed

I used non async version. Seems the error is only in async method (achat, astream_chat,...)

kkevingoed

Not ideal... :/

LLogan M

Do the suggested hints make sense above? (I am a pgvector noob, but if theres a fix we can make in the code, that would be nice)

AAlexanderGo

Need help please, has anybody has deployed next.js rag to vercel and also get only short chat messages, even when set the TOKEN to 4000 in route.ts? in npm run dev mode its working fine, deployed to vercel the answer stops after afew lines

Add a reply

Find answers from the community

Hi everyone! Has anyone managed to get