MRU dev econ course

bruh what. what is this magic lmao

eh didnt pass. RIP lol. will look again in the AM

@invictus hahaha wait what index did you try?

i'm happy to help advise on this, seems like a fun experiment

the simple one

just ran on the starter example lol

Got it. When you said ~grouped 5 sentences together, what does that mean?

You could try turning up the similarity_top_k (currently it's just set to 1)
So something like index.query("<query_str>", similarity_top_k=k)

i'll try that

ah i just meant when i was naively doing this before finding gpt-index i just fed 5 sentences at a time to embeddings to try and fake a sense of context lol

but now i just feed the entire transcript to gpt-index lol

the transcript is also noisy right now; not sure if that's degrading performance

the transcript csv is the following format:
video_title | sentence | link

really just interested in the "sentence"

the rest is metadata

ooh interesting. yeah i do wonder if preprocessing it a little would hep

Would one mega document be best? Or a txt file for every video 🧐

Maybe ill add a YouTube connector to the library

Very ez

number of documents shouldn't matter i think

regardless we'll chunk it up under the hood

Thanks! I’ll try again in the AM. East coast here

so i fed the entire course transcript all into one doc. still not great results. might be some prompt engineering i should explore. try asking the test questions in different ways or something.

sometimes it works really well and feels magical--then sometimes it's wrong. but that's the same with chatgpt too lol

i wonder if its worth storing each lecture as its own document, and then using the tree index

you could consider using index composability https://gpt-index.readthedocs.io/en/latest/how_to/composability.html

or maybe playing with other index forms generally (summarization over documents docs)

that looks promising! thanks

*to define graph-like structures over your data. out of curiosity, when you say it fails, what types of questions are you asking and what responses is it getting?

i just had a funny one lol

"Why is brain drain not a problem for the country people are leaving from?"

Answer: "Brain drain is not a problem for the country people are leaving from because the individuals who are leaving are typically young and have more to gain by uprooting themselves and moving to a new area. This means that the country is not losing its most productive members of society, and the individuals who are leaving are likely to send money back to their families in their home country, which can help to improve the economy. Additionally, the individuals who are leaving are more likely to be males than females, which means that the country is not losing its most valuable resource - its female population. Furthermore, the country may benefit from the knowledge and skills that the individuals who are leaving have acquired, which can help to improve the country's economic growth and reduce pollution. Additionally, the money sent back to the home country can be used to invest in environmental protection measures, which can help to reduce pollution and improve the country's economic growth." lol

For most countries in the world the most important source of growth is what? Choose one of the following: a) Increases in capital b) Increases in education c) Local research and development d) Ideas developed in other countries."

it did choose that right 😄

q: "Which are some reasons why firms of a given type are more productive in wealthier countries? Choose your answers from the following list; more than one may apply: a) Less nepotism b) Higher levels of trust c) More intense subsidies d) Restrictions on firm size in wealthier countries"

a: "a) Less nepotism
b) Higher levels of trust
d) Restrictions on firm size in wealthier countries
e) Access to more innovative ideas from other countries" 🙃

invented an option e lol