Log in
Log into community
Find answers from the community
View all posts
Related posts
Did this answer your question?
๐
๐
๐
Powered by
Hall
Inactive
Updated 2 months ago
0
Follow
is there way to indexed local repository
is there way to indexed local repository
Inactive
0
Follow
B
Bhavya Giri
6 months ago
ยท
is there way to indexed local repository similar to github reader which provides all the meta data?
W
B
2 comments
Share
Open in Discord
W
WhiteFang_Jr
6 months ago
You can use codesplitter and do recursive reading using SimpleDirectoryReader. That should work for local code reading.
https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/?h=codesplitter#codesplitter
B
Bhavya Giri
6 months ago
@WhiteFang_Jr hey i tried with simple directory and code split but its not able to code split
def dir_reader(path):
logger.debug(f"Reading from {path}")
documents= SimpleDirectoryReader(input_dir=path,
recursive=True).load_data()
return documents
def get_nodes(documents,language):
logger.debug(f"Parsing {len(documents)} documents, language {language}")
splitter = CodeSplitter(
language=language,
chunk_lines=config.CHUNK_LINES,
chunk_lines_overlap=config.CHUNK_LINES_OVERLAP,
max_chars=config.MAX_CHARS,
)
nodes = splitter.get_nodes_from_documents(documents)
return nodes
storage context
def index(nodes,storage_context=None):
logger.debug(f"Indexing {len(nodes)} nodes")
index = VectorStoreIndex(
nodes=nodes,
# storage_context=storage_context,
show_progress=True,
)
Attachment
Add a reply
Sign up and join the conversation on Discord
Join on Discord