NC State's AICG Framework Cuts MAE by 50.3% on Math Answer Grading

A team from the Department of Computer Science at North Carolina State University has proposed AICG, a framework for automatically grading short math answers. Conventional transformer-based automated short answer grading (ASAG) models learn to predict a score and nothing else. AICG adds an estimate of the student's skills and a correction for likely help from generative AI. On 49,402 answers that instructors graded over 19 semesters, mean absolute error (MAE) fell by 50.3% relative to RoBERTa regression. Quadratic weighted kappa (QWK, which measures agreement between the model and the instructor while accounting for the severity of disagreements) rose from 0.798 to 0.927.

Why a single score is not enough

A math answer is a mix of text, formulas and unfinished derivations. Partial credit goes for the reasoning shown, not just the final result. The authors give an example: a student correctly applies conditional independence twice, then stops before Bayes' rule and writes that they don't know how to continue. The score records the points earned but doesn't say which skill failed. Two answers with the same grade can hide different levels of understanding, and a model trained only on a scalar label generalizes worse to unfamiliar solution patterns.

To address this, AICG has two branches that share a RoBERTa encoder. The first grades an individual answer with three heads: a regression head produces a continuous score, an ordinal head produces an ordered grade, and a classification head picks one of five categories. The final grade comes from the ordinal head.

The second branch uses a Q-matrix built by experts, which links each task to the skills it tests (there are ten). From it, the model predicts skill mastery before and after the course, as well as knowledge gain. The target values come from G-DINA, a cognitive diagnosis model that infers skill mastery from answer patterns.

The predicted grades are not fed into the cognitive branch, so it serves as a separate source of signal for the shared encoder.

Ablations show that the main gain comes specifically from the Q-matrix-based cognitive signal.

How the framework accounts for AI assistance

A fluent, correct answer today may be someone else's solution. It will earn a high grade but overstate the student's independent mastery. The authors therefore trained a separate RoBERTa-based AI-assistance detector: contrastive pretraining first, then a linear classifier and temperature calibration.

The training data was built as follows. Part of the text of a real student answer (30–50%) was masked, and an offline DeepSeek model filled in the gaps, so 50–70% of the original text was preserved. Fully generated correct answers were also placed in the "AI assistance" class. From 26,039 real answers, the team took 14,044 and built one AI version of each, yielding 28,088 balanced examples. Validation on other generators used a leave-one-generator-out scheme, and on the F14–S17 semesters it used 11,458 labeled examples.

The probability of assistance is converted into a soft reliability weight with a lower bound, with no hard threshold. A suspicious answer has less influence on the mastery estimate and the G-DINA targets, but it is not discarded. The weight does not affect the grade of the answer itself. At a threshold of 0.5, the detector flagged 2,896 of 13,874 answers from semesters F22–F24 (20.87%). Weighting cut overall MAE by a further 4.2% relative to the cognitive model without weights, and consistently improved MAE and QWK in the last four semesters of the generative AI era.

What remains in question

AICG was compared against five strong baselines in an expanding-window setup: each semester was evaluated using only the semesters before it. This is closer to real deployment than a random split. The result still has limitations.

All the data comes from one university and one subject area. Experts compiled the list of ten skills and the Q-matrix by hand, and that work would have to be repeated for another course. The detector is trained on synthetic assistance (DeepSeek completing masked answers), not on real cases of students using ChatGPT.

The 20.87% figure therefore remains a model estimate, not a confirmed rate of cheating. In addition, the excerpt provided in the source cuts off before describing the remaining baselines and the full tables, so the comparison with each of them should be checked against the full version of the paper.

The practical takeaway for developers of educational products is this. To improve automated grading, you can add a structured skill signal as an auxiliary task rather than only scaling up the model. At the same time, it is better to reduce the reliability of suspect data softly, through weights, than to delete suspicious answers outright.