Find answers from the community

Updated 11 months ago

Keyword

At a glance

Is it a known issue that when using the keyword extractor for document with some english terms but mostly other content it stores all of the keywords in english?

Or maybe if we wanted to fix that we would need to make a custom extractor?

10 comments

LLogan M

That makes sense to me since the prompt is in English.

You'd probably have to rewrite the extraction prompt in another language?

NNiels

Yeah will probably have to add "make sure to keep the keywords in the original language" or something

NNiels

Probably doesn't make sense to add to llama index itself?

LLogan M

Oh lol the template is hard-coded
https://github.com/run-llama/llama_index/blob/78af3400ad485e15862c06f0c4972dc3067f880c/llama-index-core/llama_index/core/extractors/metadata_extractors.py#L195

NNiels

Yeah i noticed

LLogan M

Yea maybe. In general multilingual isn't really accounted for anywhere

NNiels

Kind of a limitation to be fair

NNiels

But i guess a custom keyword extractor would resolve that for us

LLogan M

Yea I agree -- in general, it's on the roadmap to centralize all prompts into some central registry to make things easier to customize.

NNiels

Cool!

Add a reply