It probably depends on your use case. Have you measured where your bottleneck is? Might be helpful to quantify where you're at now and what your goal is (and more importantly,
why that's your goal)
If your index is the issue, it might be that a different type of index is better suited to the structure of your data (and the types of queries users are likely to make). You may also want to try preprocessing your data, e.g. annotating it with helpful metadata, to increase retrieval speed.
Not a direct answer to your question but some general thoughts here:
https://context.ai/post/stop-faking-it-till-you-make-it