Analysing the Elements of AI MOOC’s peer review shows common design faults: ratings with no comment box, easy scales, incentives that reward speed, weak control of junk answers, and staff grading that makes the peer stage feel pointless.
Design rules
- Give reviewers a sample answer and a short, specific rubric (e.g. on topic · complete · sound arguments · easy to understand).
- Add an open justification box, at least for low ratings.
- Monitor for copied or junk answers, and provide a report route.
- If you reward reviewing more peers, check that it doesn’t reward speed over quality.
- Be clear whether staff checks affect the result.
Related: reusable audio feedback (2022)
Short, group-neutral audio files (“done well” / “watch out”) reusable across terms, always with a transcript. Captions aren’t enough for screen readers.
Apply in my context (Claude)
- Directly relevant to peer review in the RS Learning System: require a written reason for every score, and show reviewers an anchor example first. That’s what makes peer assessment reliable (calibration).
Connections
Written correction is a workload and uptake problem · Gamification as pointsification (incentive design)