Retrieving Text with BM25 in Korean and Japanese

The community members are discussing how to use the BM25 retriever with Korean or Japanese languages. The original poster encountered an error, "ValueError: ko not recognized", as the BM25 library only supports English stopwords by default, along with a few other languages. The community members suggest using the OpenAI tokenizer without stemming, or providing a custom regex pattern for tokenization. They also discuss the challenges of using BM25 for multilingual search, and mention a related library called BMX. The community members share some potential solutions, such as setting skip_stemming=True and using English stopwords, although they acknowledge that this is not an ideal solution. Overall, the discussion focuses on finding ways to adapt the BM25 retriever to work with non-English languages.