Automated correction of open-ended questions based upon database of prior corrected answers

[roelvisje]

Dear all,

Currently me and my colleague are doing some research in automated answering of open-ended items (short answer).

Example 1:
what is the name of the capital of France?
answer:
student 1: Paris
student 2: the capital is Paris
student 3: Paris is the capital

You can see, keyword here is Paris of course.

Probably such type of question can already be implemented in Chamilo (just set a keyword that is being checked), i didn't look into that yet if it is already here.

example 2 is more difficult to grade:

Explain why the price of gold is so high
student 1: gold is a precious material
student 2: the value of gold is determined by volatile stockmarket
student 3: gold is a precious commodity marketed on the stockmarket.

So you can see: these answers are more difficult to measure.
Now what if you have many students retaking an exam with the same question. You could make use of already graded answers that resemble the answer of the student. Currently we have some routines that are able to do exactly the above: give an estimate of the question if it is correct or incorrect.
We correct the answer for spelling mistakes
We use a lemmatizer for tagging the words
We use e.g. random forrest statistics to create a trained model for the question based upon the answers that were already manually graded. Depending on the question type we get a correlation of computer scored items with teacher scored items of around 0.78. so that means the computer is pretty good capable of distinguishing good from not good. Then sometimes you can see that the computer is not sure about an answer. that answer you can manually check ofcourse.

So for example 2 it would be nice if it could be implemented in chamilo. I know it is maybe somewhat out of scope of chamilo, but it would make chamilo unique in its capabilities compared to other products on the market.
If interested i will try to share our code in the future for example 2 (we are building a prototype website for april where we want to have students and teachers play with it). i hope this code will be compatible with chamilo of course.

[lonesomewalker]

This is a BAD idea. The computer IS NOT GOOD.
Has appeared in several news, that students were able to feed the answer slots with the needed buzzwords to get the grade.

Seriously: you get paid for your job to correct the answers/void the poor ones, so i suggest you should do it.

Edit: if you want, send me your test site, and i'll write a bot that will scramble your system.
Either with SEVERAL wrong answers -> so the algorithm is void.
Or with spamming the answer slot with "maybe words" which are related.

I suggest to realize that as a plugin, so the core of Chamilo is still able to deliver a good UX.

[roelvisje]

@lonesomewalker

True, i never said a computer is perfect, it sure makes mistakes, we know that too. But I can say based upon our results taken from national exams for certain questions the computer is able to make 10% less mistakes compared to teachers. So for those kind of questions you can for sure use computer marking. A teacher wouldn't do it better.

I know the articles and what their problems were; we tackled that in some ways:
1: the sentences need to resemble sentences; so only spamming keywords is not possible (hence the use of a lemmatizer etc). using only keywords will get you a strange sentence: "water tree bridge car" that is not a sentence with articles etc, so such an answer gets flagged as trying to cheat :).
2: due to the random forrest algorithm when you target this with keywords, it means you will get a very low score, since the majority of the answer is incorrect (low score) words. (hence, hacking the system like that is not doing you any good),
3: only first answer given is checked, so we check the position of the possible correct words / n-grams.

But if you are able to understand dutch and write such a bot to try it out, you are more then welcome once we have the site active. the more people try to hack the algorithm etc the better we get feedback on its pitfalls. Probably it works also for english ofcourse, but we dont have enough material for that i think. but if the site is up we can always look at it too.

[lonesomewalker]

Hi @roelvisje,

But I can say based upon our results taken from national exams for certain questions the computer is able to make 10% less mistakes compared to teachers.

Okay, valid point :D

1: the sentences need to resemble sentences; so only spamming keywords is not possible (hence the use of a lemmatizer etc). using only keywords will get you a strange sentence: "water tree bridge car" that is not a sentence with articles etc, so such an answer gets flagged as trying to cheat :).

Believe it or not, you can't prevent this.
Just copy the keywords in a so called "article spinner" and you get sentences.
This is programmed in minutes, and also sentences like: "i am not sure, maybe Paris or London, or even Madrid" would be a no-go but totally valid for your system. (and i know a lot about this; technical seo, machine learning... you name it)

2: due to the random forrest algorithm when you target this with keywords, it means you will get a very low score, since the majority of the answer is incorrect (low score) words. (hence, hacking the system like that is not doing you any good),

See above.

3: only first answer given is checked, so we check the position of the possible correct words / n-grams.

Again, see above. The race between algorithm and human in this case can only be won by computers. ;-)

But if you are able to understand dutch and write such a bot to try it out, you are more then welcome once we have the site active.

This is not a matter of language, you'll see sooner or later.
(but yes, Luuk from our team will possibly assist me there ;-) )

the more people try to hack the algorithm etc the better we get feedback on its pitfalls.

Universities already did that. Google it.

Probably it works also for english ofcourse, but we dont have enough material for that i think. but if the site is up we can always look at it too.

The universities which did that (in english) realized that it is a poor idea to try solve this with a computer.
Too many students spammed their system, and so the algorithm learned in the first time wrong solutions (which can be solved by flagging) and later was beaten by randomizers with the right keywords (not solved till now).

But maybe you are smarter than them (i mean that seriously!) because your solution goes another way.
I don't want to hold you back with this, but i recommend to let this run separately as a (deeply integrated, but optional) plugin (because this also needs a special setup i think, like plagiarism checker)...

[roelvisje]

We have items that have been in different tests. E.g. 2017 and 2019 exam. Per item we have around 1000-2000 manually graded answers. These items have been double checked by another expert. So we train our models on 1 year and then let them guess the score on the other year.

The maybe Paris or London one is a good example: in our system we also see this. Then the rule applies: first answer given is checked. So Paris or London is correct. London or Paris is incorrect. But I know you can say; maybe small city X or Paris.. small city X won't be in our system. Still the "maybe" "or Paris" part will give you lower score due to the fact that answers with or Paris usually are incorrect answers, since something before was seen as first answer and graded as incorrect. So n-gram or Paris means typical failed answer. But I like the idea ;-)

You can also see it as a system to have formative (practice tests) where students can also use open ended questions and get feedback without having a teacher to grade it. For remote computer based testing or testing without a locked computer of course you can use randomizers etc, but I think our approach would filter those out too.

But I agree it would probably be more a plugin kind of way for chamilo. :-)

[ywarnier]

I'm interested in your research @roelvisje
Anything we can do to make teacher's life easier by "pre-correcting" something and still offering the possibility to the teacher to review (or give the teacher feedback that will accelerate correction) is good. We can always set that as "beta" state for a few versions.
We should be starting to work on "more intelligent algorithms" after the release of Chamilo 2.0 (which we hope will be in early 2021). Right now, we are focused on that first goal, then we will have the bases to develop more features more quickly and will be able to start looking into this.
I'm temporarily setting this issue as a goal for 2.0 but it's rather to check it for 2.1 or something (which isn't a planned version yet but will certainly happen). I will move the target as we go.
One of the items that worries us is the ease of installation. We want most of our users to be able to install the software necessary to do the work. I'm not sure what you're using (if only PHP or not) but with PHP 8 (with dynamic C-libraries loading) a whole set of new possibilities open in that regard.

[ywarnier]

So as we are preparing Chamilo 2.0 for within a few months and we have integrated 2 LLMs (and plan to be able to integrate more in the future), I think this feature request can change form and become an AI-supported feature, where we ask an AI to analyze the answers and provide a suggested score and feedback.

This would simply become a button on the correction/feedback page where the teacher can click the button next to each question and that would ask the AI if the answer is correct.

This would work with a prompt like this:
"In language {language of the course}, for the question: '' {and the following answers (only if answers are offered in the initial question} in the context of {course title}, provide a score between 0 and {max score for the question} on one line, then some feedback on the next line for the following answer: '{the answer provided by the student}.'"

The answer could then immediately fill the feedback form and the teacher would only have to review/modify before saving.

This action would be saved in track_e_default as ai_use_question_feedback with the ID of the answer (from track_e_attempt) and the user ID of the teacher, so we can confirm later on if the feature was used (for auditing reasons).