I have been checking various #AI #LLM systems with #Moodle #AIText to evaluate the correctness of programming code. Students would submit trivial amounts of code and it would point out errors such as missing braces or parenthesis.
The results are wildly inconsistent across multiple systems. It will give an accurate evaluation on one submission and then overlook an error that a casual human observer would spot on the next.
Tested against #OpenAI, #ChatGPT Google Gemini, Anthropic & others.
It is always a pleasure to get a well asked question on one of my #Moodle plugins. This question was about my #AIText question type which uses an external #AI #LLM to generate feedback on students free text responses. On a side note I have found #Google #Gemini works at least as well or better than #OpenAI #ChatGPT models.
It was an issue I had not documented until now and as many people use it for language learning, it matters.
https://github.com/marcusgreen/moodle-qtype_aitext/issues/32
I have been thinking about how to allow more flexible prompting for my #Moodle #AIText question type. At the moment the prompt entered into the editing form is “wrapped”, which limits what a question author can do. I have created a github wiki page that proposes a “raw” mode to allow full flexibility for more experienced developers.
You can see the description here
https://github.com/marcusgreen/moodle-qtype_aitext/wiki/Raw-prompt-mode
Feedback appreciated
#EdTech #Education #AI #LLM
I have been experimenting with how my #Moodle #aitext question type which uses an external AI/LLM to process students responses can process questions that the Regular Expression Short answer question type was designed for, see
https://docs.moodle.org/500/en/Regular_Expression_Short-Answer_question_type
I think the answer is "quite well". It deals with mis-spellings and similar issues without having to learn regex or anticipate every permutation on an answer.
An academic paper has been published that uses my #Moodle #AIText #question. AIText uses an external #AI #LLM system to evaluate free text responses to questions.
The paper is called
“Exploring AI to automate EFL corrective written feedback in the first language”
And covers use for English Language learners in Japan.
You can read it at
https://www.castledown.com/journals/tltl/article/view/tltl.v7n1.2208
Matt Bury @matbury has written a summary of a paper that highlights weaknesses in recent studies & outlines principles for conducting research that focuses on real learning outcomes using #AI This bit jumped out at me.
“No-intervention” control groups (comparing AI to doing nothing) show only that AI is better than no support, offering limited insight into its instructional quality compared to alternative methods.”
My #AIText #Moodle question type uses an external AI/LLm to evaluate student responses. I have developed a branch which allows the creation of multiple test responses for use when editing a question. You can see the branch and test it out at
https://github.com/marcusgreen/moodle-qtype_aitext/tree/response_test
It is beta code but I am fairly confident it works as expects and I intend merging it into the main branch in the near future. Feedback would be apprciated
I have briefly tested my #AIText #question type with #Moodle 5 and it appears to work as expected. It gives feedback and (optionally) marks to a student response based on prompt to the #LLM. In this case it was OpenAI with gpt-4.1 With thanks to @PhilippMemmel who gave me an essential hint about the code.
My next stop will be to test against
https://github.com/mebis-lp/moodle-local_ai_manager
Which is the Mebis-lp (Bavaria) AI back end which is a good alternative to the core Moodle AI subsystem.
A user of my #Moodle #AIText question type (free text with #LLM feedback) was telling me how they supplied as much as possible of the answers in the #prompt to improve the quality of responses. This reminded me that no matter how interested I am in how my code is used, there is no substitute for it being used “in anger” by a real #teacher.
(*anger is used as a metaphor in this context, I prefer it if teachers are not actually angry using my plugins)
I have been working on updating the question editing interface of my #Moodle #AIText question type, which uses an external Large Language model such as #ChatGPT or #Ollama to give feedback on student responses. It is mainly a visual tidy up rather than additional functionality. The video shows before and after.
I now need a translation into German of the string
“Sample answer evaluation”
You can see the branch with this code here
https://github.com/marcusgreen/moodle-qtype_aitext/commits/responserunjs/
#edtech #education #ui #llm
One of the delights in creating free software is when I receive code contributions. A great example is the prompt test functionality of my #Moodle #AIText question type. This was provided by Justin Hunt of poodll.com.
Poodll is a suite of software aimed at language learners, you can find them here
https://moodle.org/plugins/?q=Poodll
Many are subscription based services (i.e. cost money) and are well worth the price.
I have been downloading Large Language Models for testing with #ollama and #openwebui I am evaluating them specifically for my #Moodle #AIText question type, which is available under the #GPL from
https://github.com/marcusgreen/moodle-qtype_aitext
You can experiment with it here
https://examulator.com/cg/
I want to see how they deal with #translation and language evaluation. The question type is popular with #TEFL, or teaching english as a foreign language
If you are interested (and concerned) about the use of #AI in education such as with the #Moodle #AI Subsystem and the things similar to my #AIText question type, Jon Fila has an online course available called
Inclusive AI Strategies™
I have followed his output for a while and I like his approach.
Find out more at
https://inclusiveaistrategies.com
The JALTCALL Podcast, recorded live on January 25th at the Sojo University SUTLF conference in Kumamoto, Japan features Professor Rob Hirschel explaining how he uses the AI Text #Moodle plugin which I created.
Programmer and JALTCALLwebmaster Luc Gougeon adds specific details to the conversation about Open Source #AI and #Ollama (large language models).
I met Rob and Luc in Japan in February last year and I find them inspiring.
Rather than purely generative #AI I consider my #Moodle #GPL #AIText question type to be "Evaluative AI". By this I mean that rather than refining and summarising text, or generating new text it breaks down text it is given and then evaluates it. This might be seen as a Marketing difference rather than a technical difference, but I am fine with that if it helps understanding…..
https://github.com/marcusgreen/moodle-qtype_aitext/wiki/Evaluative-AI
I have been compiling a list of countries where I know people are running my Moodle AIText question type with students. So far I have
Australia
Germany
Japan
Vietnam
Turkey
Israel
Ukraine
Taiwan
Spain
Can anyone add to that list?
#AIText is a #Moodle question type that uses an external LLM e.g. #ChatGPT to give instant feedback to student responses.
You can find the (GPL) code here
https://github.com/marcusgreen/moodle-qtype_aitext
And you can try a live example here
https://examulator.com/cg
The main branch of my #Moodle #aitext question type now contains code to allow it to work with two additional “back ends”, which are the Moodle 4.5 AI Subsystem and the Mebis Local AI Manager
See release notes here
https://github.com/marcusgreen/moodle-qtype_aitext
The former main branch has been renamed to “classic”
Mebis-lp have a fascinating alternative to the AI subsystem that shipped with #Moodle 4.5. It has a nice interface and features that make it well worth trying out. I like it enough that I have been adapting my #aitext question type to work with both “back ends”. I have had a huge amount of help from the Mebis people who look after the Moodle systems in schools for the German state of Bavaria.
https://moodle.org/plugins/view.php?id=3424
I also recommend their other plugins
https://moodle.org/plugins/view.php?id=3424
I have a branch of my #Moodle #AIText question type that merges in the significant contributions from #mebis
They have used it with large numbers of students. The branch is here, please test and give feedback.
https://github.com/marcusgreen/moodle-qtype_aitext/tree/405_core_ai
It offers a choice of two additional “back ends”, which are the Moodle 4.5 AI Subsystem and the Mebis Local AI Manager
See release notes here https://github.com/marcusgreen/moodle-qtype_aitext/blob/405_core_ai/changelog.md
Test against #ChatGPT 4o here
https://examulator.com/cg
