Reimagining Assessments
Introduction
This Wiki page is intended to be an open resource for instructors who need to move from traditional face-to-face assessment to online assessments. The authors acknowledge that assessments may vary from discipline to discipline, and that not all learning outcomes can be assessed in a digital environment. The authors of this document are Sunita Chowrira,Fred Cutler, Jonathan Graves, Patrick Pennefather, and Amber Shaw.
The following considerations and suggestions apply to classes of all sizes. We considered classes above 30 to be large classes, since it is around that number that videoconferencing with the full group present visually becomes impractical.
General Considerations for Instructors
- In the move to remote, online assessment, instructors can do more than convert their ‘normal’ in-person assessments to online delivery. We recommend that instructors consider re-designing assessments, even very slightly, in order to: (i) use the assessment as a learning experience in its own right (Dann 2014); and (ii) to provide fair evaluation of the achievement of learning objectives in an online environment. In other words, changes to assessments should be directed towards maximizing student learning and how they can demonstrate it effectively online. Backwards Design can be a helpful course design model to achieve this.
- Taking things one step further, instructors can use this opportunity to explicitly map assessments, and even the separate components of those assessments, to desired learning outcomes. With that said, it is important to make the assessment-to-learning-objectives connection obvious to students on the syllabus, during the course, and on the assessments themselves.
- When designing assessment, keep in mind overall issues of accessibility (time zones, internet access, bandwidth, webcam/mic quality, and privacy/security/freedom in some countries).
- Design and delivery of assessment must consider the workloads of the instructor, support staff, TAs, and students. Always keep in mind the demands of students mastering material, learning skills, and dealing with multiple learning technologies in the context of a full course load
- Instructors should consider using frequent low-stakes assessments rather than end-of-term high-stakes assessments, while keeping workloads in mind. This applies across disciplines, including writing-based assessments, knowledge-based assessments, language learning, and skills learning.
- More attention may need to be paid to student progress, including more personal outreach conditional on assignment work and performance. UBC's pilot of the OnTask system may provide a useful mechanism.
- Understand privacy and copyright and legal issues in student production and online communication.
Unique Assessment Ideas
There are many alternative assessment strategies. This list is not meant to be exhaustive:
Problem-Solution (Problem-based learning)/Case Studies
Structured revision of paper/project
Assessment Design
Traditional Exams
| Traditional Exams | |
|---|---|
Overall ease of implementation C+ | |
| Key Challenges/Benefits | |
|
This refers to online versions of standard examinations, in which students answer a selection of questions in a written, numerical, or multiple choice format. Exams are intermediate in their ability to transfer online; while there are many technical solutions to support them, they pose challenges around accessibility and academic misconduct.
Context: Used across disciplines
Opportunities/Benefits
- Exam questions may already exist, either in test banks or in "offline" examination
- Typed text may be easier to grade and can take advantage of multimedia integration
- The existence of question pools and random administration allows for individualized assessment
- Technologies exist to help support conducting online examinations and maintain academic integrity
Challenges/Concerns
- Academic integrity concerns which do not have fool-proof technical solutions including:
- Prohibited collaboration
- Consulting prohibited materials
- Identity verification
- Creating exams which are robust to online assessment is more time consuming, and may require extensive modification of question types or formats
- Multimedia questions may require reworking or the mastery of new technologies (i.e. Canvas Quizzes, Crowdmark/Gradescope, WebWork, student-recorded video).
- For example: include math solutions, drawing a regression line through points, drawing, or identifying parts of an image.
- Multi-section, multi-instructor courses require re-consideration of the coordination of: the examination itself, academic integrity, technology, allocation of grading to instructors and TAs, and appeals.
Online Considerations
- Always consider examination (preparation and exam session) as a learning experience as well as for evaluation.
- When using individualized questions (e.g. Canvas formula questions or Question Groups), instructors need to ensure that each student is still being evaluated (i) on the same learning objectives and (ii) at the same degree of difficulty.
- Available tools (such as open book/take-home format, and permitted websites) needs to be carefully considered and as permissive as feasibly possible.
- In many situations, it may be appropriate to re-work assessment strategy to allow student collaboration.
- To limit misconduct, conduct more lower stakes assessments, fewer high stakes exams.
- Sharing grading/feedback between instructor and T.A.'s must be facilitated through technology.
- Possible Platforms:
- All platforms supported by central UBC LT Hub
- Canvas Quiz (with Lockdown Browser or Proctorio)
- CLAS (for image/video annotation, including student-recorded video for assessment)
- Crowdmark / Gradescope
- edX
- TopHat Test
- WebWork (math & stats)
- Respondus Quiz
Two-Stage Exams
Typically, an individual test immediately followed by a group test.
Context: Used across disciplines
| Two-stage exams | |
|---|---|
Overall ease of implementation B | |
| Key Challenges/Benefits | |
|
Opportunities/Benefits
- Benefits of peer instruction
- Can reduce stress
- Formative and summative
- Evidence of better learning impact (Gilley and Clarkston 2014)
- Can promote social interaction and social learning to substitute for in-person social interaction
Challenges/Concerns
- Timed exams with groups may be stressful for students and may raise concerns about fairness. Instructors should consider adding time for interpersonal and technical difficulties.
- Some platforms might require groups to submit an “assignment” as one may not be able to deploy a test to a group.
- Requires bandwidth and accessible technology for the peer instruction phase.
Online Considerations
- Requires video-conference breakout groups or chat rooms. Challenge of pre-assigning these and having students successfuly enter the 'rooms'.
- Timing of stages and groups should be flexible, especially for overseas students. If possible, create different time-zone groups.
- Students must test communication channels first -- ideally the instructor will give a practice two-stage exam to get everyone comfortable.
- Exams must be either locked down or open book to prevent one student from affecting the academic integrity of the group.
- Some students may have academic accommodations to consider.
- Possible Platforms:
- Canvas with Collaborate Ultra
- Canvas with Zoom or Google Meet
- Mattermost or Slack for text-based collaboration (video calls on paid plans)
- Crowdmark(paid at the Faculty or Department or Course level)
Collection and Portfolio Projects
Context: Collection projects are more popular in life science courses. Portfolio projects are common across disciplines.
Opportunities/Benefits
- Engagement with local environments
- Can be multimodal (photos, video, audio, data, creation)
| Collection & Portfolio Projects | |
|---|---|
Overall ease of implementation B+ | |
| Key Challenges/Benefits | |
|
- Individualized production makes it easier to ensure academic integrity
- Students can retain originals for ‘integrity audits’
Challenges/Concerns
- Specific academic integrity issues
- Students must be familiar with tech to present their content
- Technology mastery must be provided and a learning objective of the assessment
Online Considerations
- Can convert to fully digital presentation and submission
- Allows for multiple file types in same submission, with some annotation or audio narration
- Oral presentation may need to be asynchronous (for example, a narrated screencast)
- Multimodal projects may not be fully accessible (for example, for a screenreader)
- Student ‘gathering’ may not be possible for some students due to physical, geographic, or socio-economic differences
- Possible Platforms:
- UBC Blogs
- ubcarts.ca (Faculty of Arts only)
- Canvas Portfolios
- Portfolium (not supported at UBC)
- Student website: Wix, Squarespace, Wordpress, etc.
- Prezi
Performance
Context: Common in Music, Theatre and Film, Creative Writing, Education
Opportunities/Benefits
- Opportunity for iterative improvement of a single take performance
- Pre-recorded performance without a live audience may reduce nervousness associated with facing an audience and especially peers
- Asynchronous removes the logistics/delays
- Student reflection comments on own performance can really boost self-awareness and improvement in performance
Challenges/Concerns
| Performance | |
|---|---|
Overall ease of implementation B+ | |
| Key Challenges/Benefits | |
|
- Synchronous video logistics may be challenging for instructors with students performing in sequence
- No accompaniment/other performers. Very difficult for ensembles and groups, though different platforms may deliver better on this in the future
- Student ‘gathering’ may not be possible for some students due to physical, geographic, or socio-economic differences
Online Considerations
- More or less time for one-on-one lessons, contact, feedback, mentoring?
- Instructors need time to get comfortable with asynchronous self-recordings for assessment and feedback, including setting expectations for how much individual feedback is possible
- Pre-recorded performances can serve as accompaniment to other players
- Shared greenscreen using specific platforms (cdnstudio.ca)
- Temporarily move somewhat away from ensemble to solo- and monologue-driven assessment
- Assessment of technique over time [PF, pls clarify]
- Combine synchronous work with asynchronous feedback, especially in ensemble settings
- Possible Platforms:
- Zoom (synchronous meeting and recording)
- Loom (individual recording)
- Kaltura-Canvas
- CLAS (for video and audio time-specific annotation/comments/feedback by students and instructors)
- WeVu (public version of CLAS)
- Collaborate Ultra (for in-Canvas videoconferencing and student breakout rooms)
- Meet (Google alternative to Zoom - free as of June 2020)
- Skype
Psychomotor Skills
Context: Common in Nursing, Dentistry, Midwifery, Physiotherapy, Engineering, Fine art execution / design
Opportunities/Benefits
- Opportunity for iterative improvement of a performance through focussed performance, reflection, feedback, and trying again (Debourgh and Prion 2017)
- Self-recorded performance may reduce nervousness associated with facing instructors and peers
- Asynchronous recording and uploading to a course site removes the logistics/delays
- Student reflection comments on own performance can really boost self-awareness and improvement in performance
| Psychomotor Skills | |
|---|---|
Overall ease of implementation C- | |
| Key Challenges/Benefits | |
|
Challenges/Concerns
- Lack of equipment
- Safety
- Lack of immediate guided feedback and demonstration
- Synchronous video logistics may be challenging for instructors with students performing in sequence
- Group physical interaction is impossible
- Some students may have willing family members / roommates to role-play but others may not
Online Considerations
- Modified assessments. More use of simulated equipment/patients (e.g. teddy bears).
- Adaptation to available equipment or simulated equipment. Mailing out of kits to students.
- Online simulation from relatively low-fidelity to high-fidelity and AR/VR simulation
- Remove group-based activities involving equipment/patients/field locations
- Possible Platforms:
- Collaborate Ultra
- Zoom
- Kaltura-Canvas
- CLAS (for video and audio time-specific annotation/comments/feedback by students and instructors)
- WeVu (public version of CLAS)
Oral Language Testing
Context: Used across language learning courses
Opportunities/Benefits
- Students may have less anxiety than in person testing
- May be more efficient and with permissions can be recorded for more specific feedback
- Multiple attempts are possible if asynchronous
- More opportunity for repeated practice
- Student-to-student language practice is possible and can be used formatively or for assessment
Challenges/Concerns
| Online Language Testing | |
|---|---|
Overall ease of implementation A | |
| Key Challenges/Benefits | |
|
- Accuracy in listening through a student’s microphone and through the instructor’s headphones
- Need to substitute or get difference licensing for specialized software?
Online Considerations
- Technical recommendations may have to be in place to ensure that the reason for poor performance is not due to technical issues
- Headphones for proper assessment
- Demands an ideal testing environment (with less distractions) from students
- Instructor needs to provide guidance for assessment design, prep, scaffolding of asynchronous assessments. (Including integrity)
- Possible Platforms:
- Language learning platforms (DuoLingo, etc.)
- Collaborate Ultra
- Zoom
- Kaltura-Canvas
- CLAS (for video and audio time-specific annotation/comments/feedback by students and instructors)
- WeVu (public version of CLAS)
Group Projects
Context: Common across disciplines and may include assessments like essays, reports, briefs, story telling, podcasts, wikis, academic reading groups, blogs, case studies, screencasts, etc.
Opportunities/Benefits
- Meta learning objective can be the development of collaboration skills and the modalities and protocols of digital collaboration
- Iterative nature supports student interaction outside of class time
- Development of multimedia skills
| Group Projects | |
|---|---|
Overall ease of implementation B+ | |
| Key Challenges/Benefits | |
|
- Students can build an artifact of their learning to preserve and present to other audiences (see Collection Projects, above)
- Experience with different roles in a team
- Collaborative
- Can reach a larger audience
Challenges/Concerns
- Challenges of group collaboration
- Managing group interactions, equal contributions, accountability, responsibility
- Assessment of contributions to group work and to final product (should be considered separately)
- use iPeer
- require individuals journals / work logs to be graded
- Need a responsibility-enforcement mechanism like peer grading or team contracts
- Academic integrity
- Students may not be able to access various platforms including Google Docs or participate synchronously
- Instructor may need access to ongoing group work, which poses logistical challenges and mandates the use of one collaboration too for the convenience of the instructor (Microsoft Teams, Google Docs/Drive, UBC Blogs)
- If the projects are to be presented, sychronously or asynchronously, and for peer feedback, Canvas is not equipped for an instructor to re-share a submission to the whole class, so they will have to be posted to another platform
Online Considerations
- Accessibility of online materials/platforms
- Possible Platforms:
- Collaborate Ultra, Zoom, Skype, Messenger
- Canvas groups/discussions
- iPeer (for student evaluation of contributions of self and other group members)
- Google Docs
- Microsoft Teams
- UBC Wiki
- Kaltura-Canvas
- UBC Blogs or ubcarts.ca or websites (Wix, Squarespace, Wordpress, etc.) for presentation of work (Canvas not good for multiple files as one submission)
Presentations
Context: Used across disciplines
Individual
This may include student generated screencasts, synchronous or asynchronous audio/video, etc.
| Presentations | |
|---|---|
Overall ease of implementation A- | |
| Key Challenges/Benefits | |
|
Opportunities/Benefits
- Most preparation for presentational aspects can be assessed using specific rubrics. For example, students may receive credit for scaffolded aspects of a presentation (brainstorms, mind maps, outlines, scripts, visuals, etc.)
- Presentations can still be made synchronous with an audience and shared screen. Have students practice with professor or TAs first to ensure that it will work and the student presenters are comfortable with the technology.
- Pre-recorded presentations (single or multiple takes) can allow for increased reflection and iterative improvement. These asynchronous presentations can be shared in CLAS and other students can comment and ask questions.
Challenges/Concerns
- It may be difficult to have similar rubrics of performance when content is presented via a synchronous platform.
- Students must master the technology and may 'over-think' or 'over-produce' it.
- Accessibility to recording tools, bandwidth, etc.
Online Considerations
- Different types of rubrics may be needed for the synchronous presentation of content
- Possibility of pre-recorded presentations. (Could be screencast, animated, or could be traditional front-of-room.)
- Presentation techniques would need to be integrated as part of the course in order to guide students towards excelling in an online setting.
- Possible Platforms:
- Zoom
- Loom (for screencast recording)
- Camtasia-Kaltura-Canvas(for screencast recording)
- CLAS (for video and audio time-specific annotation/comments/feedback by students and instructors)
- WeVu (public version of CLAS)
Group
Opportunities/Benefits
- Most preparation for presentational aspects can be assessed using specific rubrics. For example, students may receive credit for scaffolded aspects of a presentation (brainstorms, mind maps, outlines, scripts, visuals, etc.)
- Presentations can still be made synchronous with an audience and shared screen. Have students practice with professor or TAs first to ensure that it will work and the student presenters are comfortable with the technology.
- Pre-recorded presentations (single or multiple takes) can allow for increased reflection and iterative improvement. These asynchronous presentations can be shared in CLAS and other students can comment and ask questions.
Challenges/Concerns
- It may be difficult to manage multiple presenters in an online environment
- It may be difficult to have similar rubrics of performance when content is presented via a synchronous platform.
- Students must master the technology and may 'over-think' or 'over-produce' it.
- Accessibility to recording tools, bandwidth, etc.
- Responsibility, contributions, accountability, grading fairness are all considerations -- see above under Group Work
Online Considerations
- Different types of rubrics may be needed for the synchronous presentation of content. (ex. articulation, being succinct, sticking to time, shared voice(s) in the presentation
- Transitioning between speakers (better) or the decision to have just one (not ideal)
- Presentation techniques would need to be integrated as part of the course in order to guide students towards excelling in an online setting
- Possible Platforms:
- Zoom (for multiple students presenting, sharing one screen, with all talking over it, and recorded for uploading to Kaltura-Canvas or CLAS)
- Loom (for single student screencast recording)
- Camtasia-Kaltura-Canvas(for single-student screencast recording)
- CLAS (for video and audio time-specific annotation/comments/feedback by students and instructors)
- WeVu (public version of CLAS)
Lab Work
Context: There are many different types of "Labs" across UBC courses. Some labs with equipment or interactions may not be possible remotely as opposed to labs that can be replicated in the online environment. Labs are commonly used in Science, Engineering, Computer Science, and Language courses
Opportunities/Benefits
- There may be some efficiency in terms of workflow
| Lab Work | |
|---|---|
Overall ease of implementation C- | |
| Key Challenges/Benefits | |
|
- There may be a need for more frequent status updates
- With simulated labs that can be done repeatedly, student mastery may improve due to more practice
Challenges/Concerns
- Lack of equipment will slow down or make work impossible
- Lack of computing power only available with university labs and not portable
- No way to conduct specific assessment of embodied hands-on manipulation (see also Psychomotor skills above)
- Academic integrity if results only are presented for assessment
Online Considerations
- Hardware and software investment to support the advancement of research may be necessary
- Big setup investment in Labster or VR.
- More flexible timelines would need to be established
- More flexible funding timelines would need to be adjusted and/or proposed
- Possible Platforms:
- Labster
- CLAS for student-recorded simulated lab work with feedback from instuctors
- UBC Skylight may provide guidance but nothing as of May 20, 2020
Class Participation
Context: Participation requirements vary from discipline to discipline and course to course. Participation is assessed in the service of multiple learning objectives (we hope). Participation may include student-student, student-instructor, student-T.A, student-content, break-out groups (synchronously or asynchronously), and more.
Opportunities/Benefits
- There are many different types of participation to assess
- Multimodal: Students may participate in various ways (modes) including posting writing, video, audio, pictures, movement, etc.
- Transparent: Make it clear to students what the purpose of the participation is and how it will be assessed
- Structured: Participation can be very structured online. For example, you may expect students to post on a discussion board at the end of each week. Stay consistent and provide feedback on participation
- Synchronous and asynchronous possibilities: Create opportunities for students to participate both synchronously and asynchronously. Some students who may be hesitant to participate in the moment, may contribute a great deal asynchronously
- Formative feedback can be built in to participation modules: Most platforms allow instructors to post feedback within modules that address common challenges. This is especially helpful for large online classes
Challenges/Concerns
| Class Participation | |
|---|---|
Overall ease of implementation B | |
| Key Challenges/Benefits | |
|
- Students may perceive participation as subjective
- Students may not be familiar with the online culture of participation
- Students may not value online participation as much
- Time-zones, technology, and accessibility may require adjustments in expectations or modalities
Online Considerations
- Online participation may “look” and “sound” different than face-to-face
- Students may find it more difficult to participate in an online setting
- Instructors may need to reconsider what participation looks like and adjust rubrics/assessment
- Instructors would need to define participation rules of play depending on the environment they are in. (ex. Put your hand up if you wish to speak)
- Instructors may need to learn/re-learn the art of elicitation
- Synchronous classrooms will need to use the hand-raising and particularly the chat functions very deliberately
- Consult best practices on chat moderation, use TAs as moderators
- Allow use of the chat only at instructor-dictated times and with focussed questions
- Practice student use of breakout rooms. Discussions in Canvas may be preferred to breakout rooms in Zoom or Collaborate Ultra because groups can be defined beforehand (randomly or opt-in or instructor-defined) and participation as text can be assessed later.
- Possible Platforms:
- Kaltura-Canvas (for pre-recorded participation)
- CLAS (for pre-recorded participation)
- Piazza (class-wide Q&A, integrated into Canvas)
- Collaborate Ultra (videoconferencing)
- Polling and Chat (can be copied and pasted into document or sheet to analyze)
- Recorded video could be reviewed for participation
- Zoom (videoconferencing, but note privacy considerations)
- Polling and Chat (can be copied and pasted into document or sheet to analyze)
- Recorded video could be reviewed for participation Guide from Harvard
- TopHat (free in Faculty of Arts courses), Socrative, Poll Everywhere, Kahoot and other Classroom Response Systems
- iPeer (for student evaluation of other students' participation)
- Discussion Boards (Canvas and other formats)
- Coggle(asynchronous participation in a 'mindmap')
- Tapestry
Project Based Learning and Capstone Projects
Context: These refer to a broad category of assessment, such a theses or directed studies. These can include both individual or group projects, community-engaged learning, and the assessment of ongoing work throughout the term. In general, these transfer well to an online environment.
Opportunities/Benefits
- Standard format and project outline transfer well to online environments, making it a good substitute for online exams.
- Online discussion and commentary allow deeper engagement with content and less direction
- Authentic experience and assessment (remote working)
- Students master tools of digital collaboration and presentation of work; essential for the modern workplace
Challenges/Concerns
| Project Based Learning | |
|---|---|
Overall ease of implementation A | |
| Key Challenges/Benefits | |
|
- Small-group F2F interactions are difficult to replicate asynchronously, so expectations around Group Work (see above) must be set
- Can be facilitated using tools like Collaborate Ultra.
- Time commitment for managing course is higher due to the expectation of constant feedback for multiple projects/groups, including grading the final projects.
- Manage this by scaffolding and structuring components to be handed-in, presented, or demonstrated along the way
- Use peer commentary or review for some of the feedback
- Format may pose barriers to student-faculty (or student-student) interaction, especially with accessibility issues (e.g. lack of webcam, time zones)
- In general, strong accessibility considerations due to the interactive and collaborative nature of the work
- Certain in-person activities may not translate (e.g. field trips) or equipment may not be available (e.g. graphics hardware, lab equipment, community access)
- All concerns of group work contributions and assessment (see above)
Online Considerations
- Project design requires firm structure and clear “on-boarding” for students, including scaffolded points during the project to keep students on track (and limit misconduct)
- Increased need for scheduled faculty-student and student-student interactions
- Consider alternative scheduling approaches. Weekly check-in meetings with groups, etc.
- Potential need for more resources for marking and feedback
- Alternatives need to be found for “hard-to-replicate” experiences or tools
- Possible platforms:
- see Group Work above