Deng et al.’s (2024) meta-analysis found ChatGPT ‘enhances academic performance’, but performance isn’t learning. The studies were small, short and mostly university-level; self-efficacy gains weren’t significant; reduced mental effort is a warning sign. ‘The only meaningful question concerns how the technology is used in specific settings.‘
Evidence discussed
- Deng et al. (2024) Computers & Education 105224 (meta-analysis).
- Stadler, Bannert & Sailer (2024) Computers in Human Behavior 160: LLMs reduce mental effort but compromise depth of inquiry.
- Zhai, Wibowo & Li (2024) Smart Learning Environments 11: over-reliance harms cognitive abilities (systematic review).
- UNESCO (2024) AI competency framework for students.
- Historical panics: writing (Socrates), print, “Is Google making us stupid?” (Carr 2008).
Implications for builders and teachers
- Measure retention and transfer with delayed tests, not immediate scores.
- Track self-efficacy and mental effort alongside performance.
- Attempt first, then AI help. Protect productive effort (cf. “desirable difficulties”).
- Plan AI use as a variable for each task and group, not a yes/no decision.
Apply in my context (Claude)
- This is the same distinction as The illusion of learning - correct answers are not acquisition. For IELTS writing with AI feedback, require a learner draft before any AI interaction and a revision rationale after it.
Connections
Critical AI literacy is hard to teach · AI guidelines for teachers - six principles · Spacing and interleaving beat massed practice