I think internally query engine does a retrieve + llm step , i think u can try to do them individually and maybe even enable some debug logging @Utine ,
rr = index.as_retriever()
rr.retrieve("Some question which has revelant data in vec store")
and see its output. Then similarly can try for llm:
Settings.llm.predict("some relevant data from above + the original question u want to ask")