The community members are interested in creating an automated system to generate test questions for students based on uploaded documents. They wonder if this would be possible using LlamaIndex out-of-the-box or if it would require significant custom code. The comments discuss the challenges of using language models to generate fair and accurate test questions, as they may produce problems that are impossible to solve or require knowledge beyond what students are expected to have. One community member suggests exploring existing software like DeltaMath or Khan Academy that can generate math problems by changing numbers in a template. Another community member asks about the possibility of "finding a question to a fact", such as asking the model to generate a question for which the answer is "Paul Revere".
We like to create a system that could create test questions for students based on documents we upload. Think of it like an automated school test generator. To help it a little it should be possible to add topics which are important to be asked about as well as give the number of questions to be created. Would something like this be possible with Llama index like out of the box or would you expect loads of custom code to be generated. Ignoring ui and stuff on the side that would be needed anyways.
It wouldn't be immediate, but it wouldn't take too much code. The problem you'd be likely to encounter is that LLMs aren't perfect and could easily generate problems that aren't possible to solve, or require solving techniques unknown to students, or require background knowledge students aren't expected to know. By changing the prompt, you could limit this, but it would still be fairly likely that an LLM would generate "impossible" or unfair questions. Or, it would generate a question and the answer it would think is correct would be incorrect.
To do it, you would essentially just need digital files of all of the example tests, and not too much code.
But it could be worth checking if ChatGPT Plus could do something like this on a small scale. You might find success there.
I see, someties my thinking is to much short I guess 🙂 I was thinking it would be possible to force the LLM to find a question to a fact. But I get your argument about fairness and the way how to ask, that could become a real issue... I have to check as one of the student guides at my company told about a system like this he is using. but maybe I have to see myself about how much AI is behind it 😄
Well there are some softwares which will come up with example problems for, say, math, which is easier. Check out DeltaMath or Khan Academy, for example. But I think that works by just changing the numbers in a certain type of question.
For example, are you envisioning asking the LLM: "Give me a question related to the American war for independence for which the answer is Paul Revere", and it would tell you, "Who said the British are coming" ?
