Podcasts/Grading and Providing Feedback/Training Teaching Assistants

Grading and Providing Feedback: Training Teaching Assistants – Calibration

A major concern when grading assignments in large classes/sections is the need to divide the grading load among instructors and teaching assistants, while maintaining consistency. This guide is designed to help you run a calibration workshop to achieve that consistency.

Before

1: Design a specific rubric

Design a rubric that provides an objective guide as to how every mark should be distributed. Start by stating what a student needs to include in their answer to obtain 1 mark in a question. Then, show specifically what additional material they need to include to obtain all variations in marks, up to the maximum available. Such an approach should leave a grader in no doubt as to what they should award any given answer.

If possible, do this for open-ended essay questions, as well as short-answer questions. Providing ambiguous criteria, such as award between 5 marks (for exceptional transitions) and 0 marks (for no transitions or very weak transitions) will result in very confused graders. Conversely, stating something quantifiable (such as count the number of individual errors in transitions and deduct 1 mark for each from a starting 5 marks) makes it easier for graders to be objective.

2: Select a handful of papers to be graded by everyone

If the assignment has been completed in previous years or terms, select a variety of papers that showcase a range of grades (some high, some low, and some in the middle). Try to choose 3-5 papers to give enough of a sample and provide variety without providing so many that graders are overworked before the real grading begins.

If the assignment is a new one, compile some fictional, expected answers that should obtain a range of grades when graded with your rubric.

Ask your graders to work alone and use your rubric to grade these assignments.

3: Hold a workshop where all graders discuss the grades they assigned

Go through each graded question with everyone, and discuss the grades that were assigned. Where these differ, discuss the reasons for such differences with the graders and troubleshoot the rubric (and graders’ interpretations) together until everyone is confident they would award the same grades for each question.

4: Revise rubric

It may be necessary to make changes to the rubric where graders provided different grades for certain questions. If the same answer to a given question sees a wide range of grades from graders, this is an indication that the rubric needs to be improved for this question.

Make sure all graders contribute to this process and leave with the new rubric, confident that they can use it objectively.

Although this stage seems like adding another big time commitment to the process, it can save a lot of time further down the line; you will need to spend considerably less time double-checking grading, answering questions about interpreting the rubric and/or results etc., if you tighten up the rubric before grading begins.

During

5: Encourage open discussion among graders as they grade

There will likely be ambiguities that only arise when students complete assignments due to a lack of clarity in certain sections of the rubric, or due to unforeseen – but not technically incorrect – answers arising.

If graders communicate these with one another (over email, in focus groups, or in class planning time), they will help to devise objective solutions much faster. Flagging troublesome answers with one another will draw in feedback from other graders, and a majority vote may prove helpful in deciding how to grade such responses.

After

6: Compare mean grades and variation around these for each grader

There will probably be inherent variation in grades (and in the mean grade and the variation around this) in different sections. However, when section sizes are large, this should be relatively minimal.

If one section does have a significantly higher mean grade, it would be wise to review a handful of papers graded from that section (perhaps even passing these out to graders from other sections). If these different graders grade the papers a little lower, it might be that a standardised adjustment needs to be made to all papers. At the least, some further investigation should take place to make sure these students have not received artificially high grades.

7: Compare random papers from each grader

Even if mean grades are similar between sections, it is good practice to compare papers from different graders to make sure there is close agreement between them. Some variation is to be expected, but if different graders differ greatly in the grades they provide for each assignment, then further investigation is merited.

General Tips

A: Consider randomly assigning papers to graders

Making all assignments anonymous to the grader removes any subjectivity in how they will approach their grading. It can also pay to split assignments up from different sections, or have graders only grade assignments that come from sections they don’t teach.

B: Eliminate all ambiguity/subjectivity from rubrics

Try to provide clear, non-ambiguous scales for all questions, so that every mark can be accounted for. Try to avoid framing subjective scales and instead provide a quantifiable means of grading every answer. For example, try not to ask graders to award 1 - 3 marks based on an answer being poor – excellent; instead ask the graders to award 3 marks for an answer with no mistakes, 2 for one with one or two mistakes, and 1 for an answer with more than two mistakes, for example.

C: Advise graders to grade the same question back-to-back

When grading an assignment, it will save time and enhance objectivity if graders work through the same question sequentially on different students’ assignments, rather than working through each assignment from Question A to Question X. It is easier to memorize the specific grading criteria for a given question on the rubric than to work with different questions, one after another.