MET:Writing Formative Objective Assessments

From UBC Wiki
Test 3.jpg

Overview

Writing formative objective assessments requires a clear understanding of the [learning objectives and the ability to write effective feedback. Formative assessments coupled with feedback form a communication pathway between instructors and students that allow them to work cohesively towards improving both the student’s learning and the instructor’s teaching. A student’s strengths and weaknesses can be illustrated in completed assessments while whole-class test results can reveal areas that the instructor may have not taught clearly enough. At the same time, feedback helps to reinforce knowledge that the student has acquired correctly and to identify concepts that require more study. This back-and-forth process has been shown to improve students’ learning and is most effective when graded assignments are returned quickly to the student and include feedback (Chung, Shel, & Kaiser, 2006; Stull, Majerich, Bernacki, Varnum, & Ducette, 2011).

Some researchers found that assessment exerted a great influence on student achievement at highly-ranked universities (Snyder, 1971; Miller & Parlett, 1974 as cited in Gibbs & Simpson 2005). A short literature review by Gibbs & Simpson (2005) revealed that learning, achievement, student satisfaction and the quality of predictions for future performance were higher in courses where grades was based on a combination of coursework and examinations rather than examinations alone. For all of these reasons, instructors must write assessments with care, and using the best practices available to them.

On this page, suggestions are given for selecting the assessment types. As well, the importance of feedback is explained. Suggestions are given for writing multiple-choice questions used for developing critical thinking skills and other purposes. Suggestions for using objective assessments in NetQuiz Pro (Collegial Centre for Educational Materials Development, 2012) and commonly found in other assessment-writing applications are provided. NetQuiz Pro was singled out because it is a free program whose selection of assessments is more comprehensive than many other programs. Note that Essay questions are subjective assessments, but they have been included here nonetheless because they often appear along with many objective assessments.


Selecting the type of assessment

When creating an assessment, it is important to match the type of assessment to the nature of the course objectives. If a course objective is to create a song using digital tools, an assessment consisting only of multiple-choice questions will not be useful. Shank (2005) recommends determining whether a goal is declarative (requiring recall, descriptions, explanations…) or procedural (requiring creation, calculation…). Tests can be used to assess whether declarative goals are being met. They can also be used to assesse theoretical knowledge associated with procedural tasks or written portions of a performance. However, to evaluate someone’s performance skills globally, the learner must be allowed an opportunity to demonstrate his or her knowledge in a real application that is evaluated by observation (Shank, 2005).




General comments on feedback

Two comprehensive reviews cited by Gibbs & Simpson (2005) revealed that feedback had by far the greatest impact on student learning. Also, it is most effective when it is of high quality and the student reads it. It is highly recommended that the reader consult the article written by these authors before beginning to write feedback as it contains research-based theories on using feedback to the greatest positive effect on learning. Some of these theories are summed up by other authors: “In order to be effective, feedback on formative assessment needs to possess a number of qualities: it needs to be timely, constructive, motivational, personal, manageable and directly related to assessment criteria and learning outcomes (Race, 2006; Irons, 2008; Juwah et al., 2004; Race, 2001 as cited by Hatziapostolou & Paraskakis, 2010, p. 121).

Instructors need not only consider the content of their feedback, but also its format (Hatziapostolou & Paraskakis, 2010). While immediate high-quality feedback in any format is very beneficial to students, the automated sort may be too mechanical and may not be interpreted correctly. Hsieh & Cho (2011) found that feedback delivered by audio and video clips had a greater positive impact on learner outcome and satisfaction. Greater media richness allowed for clearer communication (video combined with whiteboard vs. video or audio alone). Feedback delivered in these ways was personalized and more natural, making it easier to interpret. Given that students have differences in many aspects (learning style, personality, communication skills…) individualized feedback fills students’ needs better than automated comments. It also serves to address deeper questions and misconceptions (Hsieh & Cho, 2011).

What is the affective impact of feedback? Some students see feedback, in combination with marks, as a measure of their worth. For some students, the implications for self-esteem are powerful. “Receiving feedback is inherently related to emotion (Higgins, 2000). This is especially true in the case of bad performance where students may feel embarrassment, guilt, anxiety, lack of confidence, confusion, discouragement etc. Therefore, in order to engage students in the feedback process, one must consider students’ feelings and tactics for stimulating motivation” (Hatziapostolou & Paraskakis, 2010, p.116). However, when feedback is given in the absence of marks, students are much more detached from and receptive to the comments. These factors must then be considered whatever the strategy instructors use to evaluate their students' work (Gibbs & Simpson, 2005).

Innovative methods for providing formative feedback for a variety of types of assessments that are beyond the scope of this article are described by Calhoun, A. W., Rider, E. A., Peterson, E., & Meyer, E. C. (2010), Chung, G. K. W. K., Shel T., & Kaiser W. J. (2006), Hatziapostolou, T. & Paraskakis, I. (2010) and Nix, I. & Wyllie, A. (2011).


Writing multiple-choice questions to evaluate and increase high-level thinking

Multiple-choice and other types of objective assessment used for learning purposes often serve to evaluate concepts that require little more skill than memorization. The reason may be that it takes much more time and skill to write questions that evaluate higher-level thinking. The effort would be well worth it for learners who need to develop higher learning skills. The information provided here is meant to provide basic knowledge on how to write multiple-choice questions (MCQ) for higher learning.

One important thing to consider when writing assessments for developing critical thinking is to use specific situations from the subject matter in question. “McPeck asserted that critical thinkers evaluate information in light of background knowledge, context, and reflective skepticism (McPeck, 1981, 1990) and postulated that it is impossible and incoherent to attempt to teach critical thinking in isolation from the skills being taught to students (McPeck, 1981). Further, McPeck noted that 'truly suggestive, and therefore useful, thinking skills tend to be limited to specific domains or narrower areas of application' (McPeck 1990a, p. 12) and that 'critical thinking is not a content-free general ability, nor is it a set of specific skills' (McPeck, 1990b)" (Morrison, 2001, p.17).

Question.png

As well as using specific subject matter as a context for writing multiple-choice questions to evaluate critical thinking skills, it has been suggested that one should apply four criteria:

  • “Include a rationale for each test item;
  • Write questions at the application or above cognitive level;
  • Require multilogical thinking to answer questions;
  • Require a high level of discrimination to choose from among plausible alternatives” (Morrison et al., 1996 as cited by Morrison et al., 2001, p.19).

Including a rationale for each test item equates to providing feedback. This feedback serves not only to increase the student’s learning, but also to get the student to reflect on why they chose a particular answer, which promotes critical thinking (Paul, 1993, as cited by Morrison et al., 2001, p.19). The feedback best serves these purposes when it explains why an item is right or wrong. The time spent on writing effective feedback decreases the amount of time an instructor needs to spend on content review after a test has been administered (Morrison et al., 2001).

According to Bloom (1956, as cited by Morrison et al., 2001, p.19) skills related to the first and second levels of his Taxonomy require nothing more than memorization and a restatement of the facts, respectively. Questions designed for assessing critical thinking skills need to deal with application and higher levels of thinking (Morrison et al., 2001).

When formulating questions for assessment, instructors need to pay close attention to the verbs they use. One verb that is used for evaluating analysis skills is “distinguish”. If a student is required to distinguish elements in an assessment, that word should not be used when presenting the same elements within the course. Otherwise, the student will simply be recalling information rather than using critical thinking skills to answer. Therefore, one must keep in mind the wording of the course content when preparing an assessment (Morrison et al., 2001).

Questions that develop higher learning skills must challenge multilogical thinking. This type of thinking is defined as “Thinking that requires knowledge of more than one fact to logically and systematically apply concepts to a clinical problem” (Paul, 1993, as cited by Morrison et al., 2001, p.19). This involves having students combine at least two concepts to answer a question (Morrison et al., 2001).

In multiple-choice questions, the distractors should be plausible alternatives to the correct answer. To achieve this, it helps to ask the student to determine the most important or highest priority among the alternatives, for example. Writing good distractors takes time, but the time invested will result in greater skill development for the learner (Morrison et al., 2001).

The reader is referred to Morrison et al., 2001, for numerous examples of questions dealing with higher thinking. They are all applied to nursing, but there are enough that one can determine how they might be adapted to a different field. Aiken (1982) also offers advice on writing MCQ for higher learning.


Combined assessments for higher learning

An alternative to Morrison et al. (2001)’s method is to follow-up a multiple-choice question with a short essay question. According to O’Loughlin & Osterlind (2007), a MCQ would be answered online and a justification for the answer would be written out on paper. It is also possible to have MCQ followed by essay-type questions, all done online. The reason for this assessment strategy is to allow those who guessed correctly to prove that they understand the answer to the question and didn’t simply guess. Those who would have chosen the wrong MCQ answer but who can explain the concept can obtain partial marks.

Eventually, perhaps all objective assessments in learning management systems and authoring tools will come with an option to explain an answer in an objective question. This may seem to defeat the purpose of using automatically-corrected questions, but they can be useful in the case of very important concepts. As well, the number of this type of blended question can be limited by the instructor to a manageable amount (O’Loughlin & Osterlind, 2007).


Uses for multiple-choice questions

Quizzes can be used to assess students’ knowledge following an assigned reading. Consider an instructor who wants students to read approximately twenty pages from their textbook before each class. She or he can assign the pages to read, ideally in a reading schedule provided at the start of the semester. The students are tested immediately upon entering the class and have less than five minutes to submit their answers. If the test can be done this quickly on computers, this would be the best option. Otherwise, the test can be done on paper slips that are collected and viewed later for correction (Moodle, 2012). Stull et al. (2011) recommend a similar use of pre-lecture online quizzes as they resulted in higher student achievement.


Types of objective assessments used in NetQuiz Pro

For images and basic information on the following types of assessments, please consult the user's guide About NetQuiz Pro (Collegial Centre for Educational Materials Development, 2006). Please note that the guide is outdated and does not contain descriptions for all types of assessments. For example, “Checkerboard matching” is called “Matching board” in the current version of the software (NetQuiz Pro 4). As well, “Classification” and “Highlight text” are types of assessment that have been added since the publication of the guide. At the bottom of the website of NetQuiz Pro 4 is an example of an assessment titled “Cinequiz” (Collegial Centre for Educational Materials Development, 2012). The reader is encouraged to explore this quiz as it puts to use many of NetQuiz Pro 4's functions.


Matching, Short answer and True/False

These types of questions are limited to evaluating knowledge and comprehension skills. As such, it is suggested that they be used at the beginning of an assessment, as a warm-up leading to more difficult questions, such as multiple-choice and essay questions.


Classification

In the Classification assessment, learners match text or image items to a maximum of six text or image categories. This type of assessment is automatically linked to level 4 of Bloom's Taxonomy (analysis).

Examples of use

  • In a history course, students may be asked to match the names of events to a map of the world or of a country.
  • In a math lesson, the categories may be the left side of different equations and the items may be different versions of the right side of those same equations. To show complex notations, it is recommended that the equations be created in an equation editor and that the instructor take screen captures of the left and right sides of the equations. These items can thus be inserted as images.
  • Pictures of different types of medication can be matched to the class they belong in. For example, ativan would be classified under benzodiazepine or sedative while acetylsalicylic acid would be placed under analgesic.
  • In an ESL course, students can match images of objects with an image of a room or situation in which they belong or are relevant.


Dictation

In the dictation assessment, the instructor provides instructions for the students and includes an audio or video recording. The student must play the clip and type the content of the message. Thus, Dictation is limited to the Knowledge level of Bloom’s taxonomy. That said, it is possible to vary the difficulty and the nature of the assessment by using shorter or longer passages that vary in complexity (many short sentences vs. one long one). To get full marks, the student must type every word correctly. Pictures and text can be added as hints. Geiven that this type of assessment is uncommon, it is recommended that students be given the opportunity to practice using it in non-graded assessments.

Examples of use

  • For a simple task, the recording can contain only a short sentence where the student is instructed to type one or two words. For example, a clip plays the sentence “The kangaroo is an animal found in Australia.” and is followed by instructions such as those below:

Play the audio/video clip and type the missing words in the following sentence:

The ____ is an animal found in Australia.

The student would simply have to type “kangaroo”.

This type of assessment is useful for children who are learning English or adults who are beginning to learn the language. To increase the level of difficulty, the student can be required to type out an entire complex passage.

  • To vary the nature of the activity, the instructor can record the passage in one language and have the student translate it into another language. For an added challenge, The instructor can use a clip of someone who is singing or speaking with an unusual accent.
  • This activity can be used in various subjects as a way to familiarize students with the use and spelling of new terminology. For example, students in an engineering course may be learning a new term like Quadrature Amplitude Modulation.
  • In a chemistry course, students could be asked to listen to the names of chemical compounds (such as aluminium trifluoride) and to type out the chemical formula AlF3. For this type of assessment, it is best to evaluate one word at a time, given that format of the student's answer must match the format of the correct one. For example, if three chemical formulae are each written on a different line, the student would have to write the answers that way to get get full marks. If the three answers are correct, but all written on one line, no marks are given.


Essay

For Essay questions, the instructor can insert a picture, audio, video or text to accompany the instructions. The instructor’s questions can correspond to any level of Bloom’s taxonomy, but it would be best to put the four highest levels (application, analysis, synthesis and evaluation) to practice.


Fill in the blanks

For this type of assessment, the instructor can have students fill spaces in a statement by choosing from a list of options. Depending on the use, this type of question can be used for all levels of Bloom’s Taxonomy.

Examples of use

  • The instructor can evaluate math concepts. An instructor could provide the following instructions: "Fill in the blanks with a combination of numbers that indicate an increase in temperature of 15 degrees." The students would have to select the numbers that make the statement conform to the instructions. For example:

"Yesterday, the temperature was -5/-6/-7 degrees Celsius, and today it is 10/20/30 degrees."

  • By including an audio clip, the instructor can test students' language skills. For example, for an audio clip of the following sentence: “The baseball player threw the ball” the instructor can have a fill-in-the-blanks sentence such as this: The baseball player through/threw the ball.
  • The teacher can embed an image such as a diagram, a symbol or a map and ask the student what is being repressented.
  • A short video clip of a medical emergency can be shown, and the student could be asked to determine the nature of the emergency (heart attack, seizure...), the recommended medical response or the type of medication that would be helpful.


Highlight text

In this type of assessment, the student is asked to highlight parts of a text using up to five colours (red, yellow, green, blue and purple). For easier tasks, the instructor can select only one colour in the assessment. For more challenging tasks, more colours can be added. Note that for a student to get full marks, he or she must only highlight the correct parts items. When correct and incorrect items are highlighted, the student gets no marks. It is important to remember that this type of assessment will not work with learners who are colour-blind. This type of assessment can touch on all levels of Bloom's taxonomy.

Examples of use

  • The Highlight text assessment can be used for having students identify different parts of speech using different colours. For example, verbs would be highlighted in red.
  • A student could be provided with a poem and asked to highlight all the positive terms in red and all the negative ones in blue.
  • In a math assessment, students could be asked to highlight all the units in yellow. A more difficult task would have the student highlight all volume units in yellow and all area units in red.
  • In a geography lesson, students could be given a passage and asked to highlight all African countries in red and all European countries in green.
  • In an ESL course, students could be shown an embedded text in his or her native language where one to five words are highlighted in different colours. The student would have to find the equivalent terms in an Enlgish translation of the text.


Identifying the parts of an image

In this type of assessment, the instructor uploads an image and creates labels that students must place at indicated areas. This assessment could be used to evaluate skills at all levels of Bloom's ta

Examples of use

  • The teacher can have students simply label the parts of a vegetable cell (biology), or the colours in a picture (kindergarten), or the objects in room (ESL).
  • In a physics lesson, a video could be used to show objects in motion. A snapshot from that video could be labelled according to the types of forces at play.
  • In an art lesson, the student might have to show where different pigements or techniques were used in a painting or sculpture. Alternatively, students could identify the architectural features of a building.
  • A student is shown a weather map and plays an audio clip explaining how the weather patterns on the image change into something else. The task could consist of indicating the location of certain phenemona resulting from the change.
  • In a lesson on lifesaving at a public swimming pool, the learner could be shown an image of children in and around the pool. The task would consist of placing labels such as “major incident 1”, “major incident 2” “minor incident” and “no immediate danger” or "priority 1", "priority 2", "priority 3" and "priority 4".


Matching Board

In this type of assessment, the instructor can have students match text or an image to another text or image. Content is shown on “cards” displayed in a rectangular area. Students can either reveal the cards as in the game of Concentration, or begin with all cards face-up, masking them as they make correct matches.

Given that the Matching Board assessment involves using short bits of information, it is best used to review a theme or a chapter unit. It would be preferable to give feedback only in a general manner once the assessment has been completed to avoid interrupting the flow of the activity. The Matching Board assessment is well-suited to the knowledge and comprehension levels of Bloom<s Taxonomy. Concepts used for matching can include those found in the table below and many others.

Examples of use

Text–text Text-image (image-text) Image-image
Cause Effect Portrait Name Worker Work place
Problem Solution Painting, sculpture Artist’s name Symbol Situation relevant to symbol
Date Event Map Location Picture of reactants in a chemical reaction Picture of products in a chemical reaction
Part of speech Example of part of speech Landform (mountain range, plain…) Name of landform Animal Habitat or ecosystem
Word or sentence in one language Word or sentence in another language (translation skills) Part of anatomy Name of anatomy part or associated system A variant of a specimen (red tourmaline) Another variant of a specimen (green tourmaline)
Name of song/poem or book Composer or author Diagram of a system or process Name of corresponding system or process Diagram of a polymer (polyethylene) Diagram of a monomer (ethylene)
Unit (cm, mL…) Dimension measured (length, volume…) Data chart Interpretation of chart Flag/cultural symbol Country or continent

Multiple-choice and Multiple answer

Multiple choice questions have been described above. As for Multiple answer questions, they can be used much in the same ways that MCQ are, so they would touch on all levels of Bloom's Taxonomy. However, it is imperative that it be made clear to the learner that more than one answer may be correct. Students must know in advance that they are dealing with multiple answer questions and not MCQ.

Sequencing

In Sequencing assessments, students can be asked to place up to fifteen text or image items on order. Sequencing activities relate to analysis, the fourth level of Bloom's Taxonomy.

Examples of use

  • In an art history course, students may be asked to place images of paintings in chronological order of production.
  • Students may be asked to sequence text and image items related to a medical response. For example, there would be different images of a patient going through four stages of a ten-step procedure. The remaining steps would be described with text.
  • Students may be asked to sequence images of the the steps of a chemical reaction mechanism.
  • In a course on physical geography, a student may be asked to sequence the steps of changing land formations using text or images.

See also

References

  • Aiken, L. R., (1982). Writing multiple-choice items to measure higher-order educational objectives. Educational and Psychological Measurement, 42,803-806. doi: 10.1177/001316448204200312
  • Calhoun, A. W., Rider, E. A., Peterson, E., & Meyer, E. C. (2010). Multi-rater feedback with gap analysis: An innovative means to assess communication skill and self-insight. Patient Education and Counseling, 80, 321–326. doi:10.1016/j.pec.2010.06.027
  • Chung, G. K. W. K., Shel T. & Kaiser W. J. (2006). An exploratory study of a novel online formative assessment and instructional tool to promote students’ circuit problem solving. The Journal of Technology, Learning, and Assessment, 5(6), 1-27 Retrieved from: http://ejournals.bc.edu/ojs/index.php/jtla/issue/view/184
  • Hatziapostolou, T. & Paraskakis, I. (2010). Enhancing the impact of formative feedback on student learning through an online feedback system. Electronic Journal of e-Learning, 8(2), 111 – 122. Retrieved from: www.ejel.org
  • Hsieh, P. J. & Cho, V. (2011). Comparing e-learning tools’ success: The case of instructor–student interactive vs. self-paced tools. Computers & Education, 57, 2025–2038. doi:10.1016/j.compedu.2011.05.002
  • Nix, I. & Wyllie, A. (2011). Exploring design features to enhance computer-based assessment: Learners’ views on using a confidence indicator tool and computer-based feedback. British Journal of Educational Technology, 42(1), 101–112. doi:10.1111/j.1467-8535.2009.00992.x
  • Stull, J. C., Majerich, D. M., Bernacki, M. L., Varnum, S. J. & Ducette, J. P. (2011).The effects of formative assessment pre-lecture online chapter quizzes and student-initiated inquiries to the instructor on academic achievement. Educational Research and Evaluation: An International Journal on Theory and Practice, 17(4), 253-262. doi.org/10.1080/13803611.2011.621756

Image credits

External links

How to write questions for objective assessments

Writing feedback

Writing assessments using Bloom’s taxonomy

General advice on writing assessments