Evidence
What has been measured so far, and what has not.
Three studies, run by Rubrica's author between March and July 2026, compare its grades with human scores and with a second automated grader, and check how stable they are. All three ran on development builds, before the first pilot build, and have not been re-run on the current version. Each is small, and each is described here with its limits.
Reading these numbers. ICC and weighted kappa measure how closely two sets of scores agree. 1 means identical, and 0 means no agreement beyond chance. A 95% confidence interval (CI) is the range the true value likely falls in. A wide range means a small sample. Standard deviation is the typical spread of a set of scores. In the research table, d is an effect size: about 0.2 is small, 0.5 medium, and 0.8 large. I² shows how much results vary between studies.
Compared with human scores
Rubrica's scores set beside human-assigned scores from another instructor's course.
Macroeconomics midterm
July 2026 · 25 exams1.26 ptsAverage difference per exam on the structured section (46.5 points, 38 items)
0.868Whole-exam agreement with human scores, ICC(A,1), n = 25, 95% CI 0.42 to 0.96
85%Of structured-section scores matched exactly (810 of 949)
3.88 ptsAverage difference on the one 25-point essay
What it showsOn the structured section, Rubrica differed from human scores by 1.26 points per exam on average, 2.7% of the section, with a mean difference of -0.28. Whole-exam consistency with human scores, ICC(C,1), was 0.922 (95% CI 0.83 to 0.96).
What it does not showOn the essay, Rubrica scored 3.64 points lower than the human on average, with the largest gaps on the strongest essays. Instructors should review essay scores closely. Only 8 of 25 essay scores matched. One course, and the essay and structured questions differ in format, length, and content, so the study does not show why the essay differed more. A larger study is planned.
Agreement between two graders
The same exams were graded a second time by a separate automated grader from another provider, given matched rubric content and answer pages.
Undergraduate economics course
March 2026 · 30 exams · 840 items96%Of items within one point (808 of 840)
0.15 ptsAverage difference per question
70%Exams with the same letter grade (21 of 30), before safeguards. All 9 differences were to the neighboring letter.
What it showsThe two graders scored individual items closely. Seven of the nine differences fell inside the window where Rubrica re-grades an exam. Two fell outside it. Agreement after that re-grade has not been measured.
What it does not showAgreement between two automated graders is not accuracy against a human. Eight of the nine letter differences were on one exam version. Methods: ICC(3,1) 0.966 (95% CI 0.93 to 0.98), and quadratic weighted kappa 0.883 on letter grades.
Repeat runs
Ten final exams were each graded three separate times to see how much scores move from one run to the next.
Final exams, graded three times
June 2026 · 10 exams10 / 10Kept the same letter grade on every run
0.91 ptsTypical variation in total score between runs (standard deviation, averaged across exams)
What it showsGrades were stable from run to run on this sample.
What it does not showStability is not accuracy. One course, ten exams, three repeats.
Safeguards in the grading pipeline
Rubrica checks its own work and flags answers for a person to check. Some safeguards depend on the rubric having been enhanced. An enhanced rubric decomposes a rubric into partial credit tiers, common errors, and expected methodology.
Second look near cutoffsAny exam within 1.5 percentage points of a letter-grade line is graded again. If the two passes give different letters, the total is averaged and flagged.
Multiple choice re-readA multiple-choice zero is read a second time when Rubrica can identify the chosen letter and its page.
Handwriting flagsWhen the model reports doubt about what a student wrote, that answer is flagged for the instructor.
Consistency checksFeedback that looks vague is rewritten. Feedback that seems to contradict the score is checked, which can change the score, and the change is flagged for the instructor.
Review before class sendA class-wide email goes only to exams marked reviewed. Marking an exam with open flags takes an explicit confirmation that clears them. Exports and single reports are available at any time, so the instructor decides when a grade is final.
Earlier scores keptScores are saved before most safeguards run. Both passes of a near-cutoff exam are shown in Review.
What has not been measured
- Learning outcomes.
- Quality of Rubrica's feedback against a human's.
- Whether flags catch the errors that matter.
- Agreement after safeguards.
- Any third-party replication.
- Time saved. The figure on this site is an estimate.
- Whether scores differ for students who write in a second language, or by handwriting.
We have not measured whether Rubrica's feedback improves learning. The app records whether a report was sent, not whether a student read it.
What the research says
These studies are about feedback in general, not about Rubrica.
| Finding | Source |
|---|---|
| Elaborated feedback 0.49, correct answer 0.32, correctness only 0.05. Computer-based, item-level, 40 studies, 70 effects. Delayed feedback lowered effects. | Van der Kleij, Feskens & Eggen (2015), Review of Educational Research 85(4). doi:10.3102/0034654314564881 |
| Average effect d = 0.48 after removing extreme values. I² = 83.40%. 17% of effects were negative. Corrective feedback d = 0.46. | Wisniewski, Zierer & Hattie (2020), Frontiers in Psychology 10. doi:10.3389/fpsyg.2019.03087 |
| Average d = .41, but over a third of interventions lowered performance. Effectiveness falls as attention moves toward the self and away from the task. | Kluger & DeNisi (1996), Psychological Bulletin 119(2). doi:10.1037/0033-2909.119.2.254 |
| "Research studies have shown that, if pupils are given only marks or grades, they do not benefit from the feedback." Formative assessment effects 0.4 to 0.7, ages 5 to undergraduates. | Black & Wiliam (1998), Phi Delta Kappan, "Inside the Black Box" |
| In 48 university English learners over six weeks, ChatGPT feedback and human tutor feedback showed no difference in learning outcomes. Quasi-experimental, no no-feedback group. | Escalante, Pack & Barrett (2023), Int. J. Educ. Technol. High. Educ. doi:10.1186/s41239-023-00425-2 |
Report and citation
A versioned report on Zenodo describes the software and the three studies. It is the reference to cite.
Fraser, T. (2026). Rubrica v0.1.10: software description and validation report [Report]. Zenodo. https://doi.org/10.5281/zenodo.23179233
A detailed technical overview is at travisfraser.com/rubrica.
Questions about the studies?
hello@tryrubrica.comContact us