Keywords

Nothing is exposed to the user in a nice way, so no documentation for this lol

I was just reading the source code thinking about how this could work.

Right now, if you use a keyword index, the LLM finds the keywords for you (both from the documents and queries)

@Logan M can we modify the .json keyword index to include the keywords that were going to be added to the node?

If the nodes have ids we can track them

You could try hacking the the json yes, at least as a temp workaround

this could then be exposed as a function that gets the node id and the list of keywords to be added. Would that be more permanenet solution?

Yup that's what I was thinking haha. Although using node ids can be a pain, since they are random

Setting keywords for entire doc ids could work too

i think the most elegant approach would be like if they are random ket the llm figure it out cause we probably dont have specific keywords in mind anyway. But for those we have internal ids, we create the nodes ourselves

same with doc id, just nother function added to the document class

I would honestly write all of this myself but i havent contributed to open source project before. kind of nervous haha

Do it! You have been fairly active on here giving good advice. Everyone is extremely happy to merge useful features like this 💪

Happy to help you out if you get stuck

thanks for the encouragement @Logan M ! I'll do it!

@Logan M For context, we split transcripts into logical segments (our own version of doc nodes), and each of these segments has an associated summary, keywords, context items, etc. This is done with "traditional" ML and would serve as useful metadata, so inserting at the node level would be ideal!

@Logan M sent a DM regarding a PR

@aleks_wordcab why dont you add this to extra info object of the node? You can easily modify the node extra info variable and add or remove object properties.

@BioHacker is there a way to use this extra info somehow during queries

We’lll have to think about that when we meet.

Extra info is already embedded in the nodes. So you are good to go

that's great!