Find answers from the community

Updated 2 months ago

is there way to indexed local repository

is there way to indexed local repository similar to github reader which provides all the meta data?
W
B
2 comments
You can use codesplitter and do recursive reading using SimpleDirectoryReader. That should work for local code reading.

https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/?h=codesplitter#codesplitter
@WhiteFang_Jr hey i tried with simple directory and code split but its not able to code split
def dir_reader(path):
logger.debug(f"Reading from {path}")
documents= SimpleDirectoryReader(input_dir=path,
recursive=True).load_data()
return documents

def get_nodes(documents,language):
logger.debug(f"Parsing {len(documents)} documents, language {language}")
splitter = CodeSplitter(
language=language,
chunk_lines=config.CHUNK_LINES,
chunk_lines_overlap=config.CHUNK_LINES_OVERLAP,
max_chars=config.MAX_CHARS,
)
nodes = splitter.get_nodes_from_documents(documents)
return nodes

storage context


def index(nodes,storage_context=None):
logger.debug(f"Indexing {len(nodes)} nodes")
index = VectorStoreIndex(
nodes=nodes,
# storage_context=storage_context,
show_progress=True,
)
Attachment
image.png
Add a reply
Sign up and join the conversation on Discord