The post discusses the possibility of OpenAI offering a context length of up to 32,000 tokens. Community members express excitement about the potential impact this could have on tools like GPT-index, allowing for more context to be stored. However, there are also concerns about the computational resources and costs required to utilize such a large token window. Some community members suggest that users would want to see the utility of the current 4,000 token limit validated before moving to 32,000 tokens.
yeah i'm excited about how this will augment llamaindex. there's obviously still compute/cost considerations in putting 32k tokens into a single LLM call, and I'd love to see how the expanded cost window expands current use cases + still introduces necessary cost/latency sondierations
Yeah, it’s not clear to me the scaling limitations/factors of gpt-3+. Like, are 32k token windows for most users a plausible near future goal? Or are the computational resources needed for that untenable at a massive scale 🤷🏽♂️
