Find answers from the community

Updated 6 months ago

is there way to indexed local repository

At a glance

The post asks if there is a way to index a local repository similar to GitHub's reader, which provides all the metadata. In the comments, a community member suggests using codesplitter and SimpleDirectoryReader to recursively read the local code. Another community member tried this approach but encountered issues with the code splitting. They provided some sample code demonstrating the process of reading the documents, parsing them, and indexing the nodes. However, there is no explicitly marked answer in the comments.

Useful resources
is there way to indexed local repository similar to github reader which provides all the meta data?
W
B
2 comments
You can use codesplitter and do recursive reading using SimpleDirectoryReader. That should work for local code reading.

https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/?h=codesplitter#codesplitter
@WhiteFang_Jr hey i tried with simple directory and code split but its not able to code split
def dir_reader(path):
logger.debug(f"Reading from {path}")
documents= SimpleDirectoryReader(input_dir=path,
recursive=True).load_data()
return documents

def get_nodes(documents,language):
logger.debug(f"Parsing {len(documents)} documents, language {language}")
splitter = CodeSplitter(
language=language,
chunk_lines=config.CHUNK_LINES,
chunk_lines_overlap=config.CHUNK_LINES_OVERLAP,
max_chars=config.MAX_CHARS,
)
nodes = splitter.get_nodes_from_documents(documents)
return nodes

storage context


def index(nodes,storage_context=None):
logger.debug(f"Indexing {len(nodes)} nodes")
index = VectorStoreIndex(
nodes=nodes,
# storage_context=storage_context,
show_progress=True,
)
Attachment
image.png
Add a reply
Sign up and join the conversation on Discord