Documentation:CTLT Resources/Selected TL Topics Evaluation Tools

During the 2006-07 academic year, a committee co-chaired by Dr. Anna Kindler (Vice-Provost, Academic Affairs) and Dr. Joy Johnson (Nursing) revisited the ways student data are collected and used to improve teaching and learning at UBC. This committee’s objectives were to:

draft an updated policy statement on student evaluations of teaching, to be presented to Senate;
develop and present to Senate a set of best practice recommendations related to concurrent (“mid-term” or formative) evaluations;
derive and recommend options for implementing a modular approach to end-of-term evaluations.

There is considerable precedent at UBC for committee work on the use of student evaluations in teaching and learning improvement, and the committee was called upon this for their deliberations.

Some examples of their work are available:

Suggested Principles and Guidelines for the Peer Review of Teaching

Peer review assessments of teaching are used increasingly at UBC and the practice has been strongly endorsed (1996 Senate Report on the Evaluation of Teaching). Proper peer review assessment is an important complement to our widespread practice of student evaluations of teaching. If peer reviews of teaching become part of the ongoing activity of the university, part of the culture, then we think that constructive peer reviews will enhance the quality of learning at UBC. This document provides a brief rationale for peer assessment, as well as some suggestions for developing standards, criteria and approaches.

Rationale for a Peer Review Process

Following standard practice in university settings, the peer review of teaching is based on principles of peer review already used in assessments of grant applications and publications. Colleagues who are knowledgeable about content areas, and who are themselves accomplished teachers, can provide peer review teaching assessments that contribute to both the evaluation of teaching as well as the improvement of teaching.

The key difference between peer reviews of grant applications, publications, and the like, is that with the peer review of teaching the peers are not arms-length, they are immediate colleagues. For this reason we stress that the peer review of teaching is more useful as a form of constructive feedback for the improvement of teaching than it is as an evaluative tool in making career decisions (i.e., promotion, tenure). When using peer review for evaluative purposes, involving two independent reviewers enhances the evaluation.

Some Basic Principles

Reviews of teaching should parallel principles involved in peer assessment of research (e.g., professional, responsible, knowledgeable)
Peer reviewers must have appropriate preparation for this process and be knowledgeable about sound principles of teaching and learning
All colleagues must be aware of the criteria that are to be used in any peer review

Factors to Be Considered When Formulating a Peer Review Process

Key Institutional Issues

What constitutes good teaching at UBC?
See “Effective Teaching Principles and Practices,” UBC Senate (May 1999)

What do we, as an institution, accept as evidence of good teaching?
Mainly, two types of evidence are used:
- Student evaluations of teaching
- Peer review assessments of teaching

Do we have programs on the peer review assessment of teaching?
Yes, through the Centre for Teaching, Learning and Technology.

Are there standard procedures in place to present evidence from the evaluation of teaching?
Evidence of teaching is a required element of evaluation reports for promotion and tenure many units require evidence of teaching evaluation for merit decisions teaching dossiers typically include evidence of teaching evaluations.

Qualities That a Peer Review Should Reflect

The process must be fair, protecting both the individuals under review, and their units, from arbitrary and unprofessional outcomes;

The process must be explicit, where criteria for success are known and understood;

The process must be manageable, with resources for evaluating performance available within the unit (especially time resources) and peer reviewer preparation and consultation available;

Feedback from the process must be available to the individual under review and should be provided in writing (and perhaps verbally as well).

The standards used in the process should be consistent and complementary for local (unit level) interests, as well as for faculty and university needs and priorities;

The process should incorporate both information on student learning and the approaches and conduct of the individual's teaching.

Consider the breadth of activities to be measured

The following provides two related approaches to the peer review assessment of teaching, gleaned from lists developed at many institutions in North America and elsewhere.

Focus on: a careful analysis of the conception and execution of a course of study goals and objectives: consider course design; outlines; hand-outs; other printed/electronic material; reading lists; fit with other courses

Enactment and interaction: consider in-class observations; discussion with class; focus group discussion; discussion with teaching/lab assistants and/or co-teachers; individual interviews with students; interview with instructor
Results: consider evidence of student learning through examinations, assignments, reports, essays, projects

Focus on: a review similar to what one would do for assessing research

Clear goals – does the scholar state basic purposes clearly? Are objectives realistic and achievable? Are important questions in the field covered/illuminated/discussed?
Adequate preparation – does the scholar show an understanding of existing scholarship? Does the scholar have the necessary skills and knowledge to teach this course? Are all the necessary resources properly marshaled to do an effective job?
Appropriate methods – does the scholar use methods appropriate to the goals established? Are the methods effectively applied? Is there evidence on flexibility in modifying approaches as necessary?
Significant results – does the scholar achieve learning goals? Are additional areas for further learning exploration revealed? Are students learning?
Effective presentation and learning enhancement – does the scholar use a suitable style and effective organization to help students learn? Does the scholar interact with students with clarity and integrity?
Reflective critique – does the scholar critically evaluate his/her own work? Is an appropriate breadth of evaluation used to improve the quality of future work?

Other issues to consider

It is important to include the following, as appropriate, when undertaking peer review assessments:

Mentoring of students, via advising, employment, workshops, etc.
Supervision of students, in honours courses, directed studies, graduate supervision,
Curriculum development, department/faculty/university committees
Innovation in teaching, via TLEF, Web courses, field work, Problem Based Learning, etc.
Research and scholarship related to teaching
Multiple classroom visits

Common Questions on Student Evaluation of Teaching Forms

A goal of the UBC Senate ad hoc Committee on Teaching Quality, Effectiveness and Evaluation (1998/99) is to have the attached endorsed by Senate for use as a University-wide tool for obtaining undergraduate student feedback on teaching and learning. The form is designed so that Faculties (or Departments) could ask other questions they feel are essential. These questions are meant only as a common core set of questions, as opposed to a common form for use by all. The proposed questions build on questions that are now almost uniformly asked on most Faculty-specific student evaluation of teaching forms.

The committee was encouraged to design the form for three reasons. First, it provides at least one question in each important teaching domain, as determined by years of research on student feedback. Second, each of the questions reflects a learning-centred approach to teaching: students are asked to judge how the learning experience affected them as learners. Third, a common form allows academic units to make better informed judgments about their peers who teach in multiple faculties/programs (something that is occurring with greater frequency at UBC).

And finally, the form is generic and can be used across disciplines, supplemented by any additional questions, either general or discipline-specific, that the department or faculty deems appropriate. If used across disciplines, these twelve items can provide norm-based referencing of teaching. It can also begin to address faculty concerns about student publications of feedback (e.g., the AMS Yardstick), by offering one “yardstick,” rather than a number of department-specific forms with differing criteria.

Recommended Common Elements for Inclusion on All UBC Student Evaluation of Teaching Forms

This course: ________

is a requirement. Yes___ No___

(For the following questions, we recommend the use of a five point scale: 5 = strongly agree, 4 = mildly agree, 3 = neutral, 2 = mildly disagree, 1= strongly disagree)

has appropriate classroom and learning facilities

uses fair methods of evaluating my learning

Learning

The instructor helped me to learn the course material.
The instructor stimulated me to think about the subject.
The instructor made this subject interesting to me.
The instructor encouraged active participation by students

Individual Rapport

The instructor was welcoming when I sought help.
The instructor was available to provide assistance outside of class.

Examinations and Assignments

The assessed work, such as tests, quizzes, assignments and presentations, helped me to learn the course material.
The feedback on assessed work helped me to learn the course material

Overall

Overall, I found my instructor to be an effective teacher.

Please comment on:

what helped your learning?
what hindered your learning?
when you were most engaged in this course?
when you were least engaged in this course?
How you would change this course?

Effective Teaching Principles and Practices

The University of British Columbia is committed to maintaining the highest standards of teaching…. (Trek 2000, p. 6)

The current Agreement on Conditions of Appointment for Faculty (Section 4.01) states:

Candidates for appointment, reappointment, tenure or promotion…are judged principally on performance in both teaching and in scholarly activity.

One crucial step in attaining “the highest standards of teaching” is the placement of a high value on the teaching role, including in the tenure and promotion process, or in rewarding teaching through salary increments and prizes. To value teaching appropriately we require a clear set of principles and practices with which to guide the assessment of teaching activities.

The Senate ad hoc Committee on Teaching Quality, Effectiveness and Evaluation (1998/99) has developed just such a set of principles and examples of practice. They are generic in nature, and can be modified to suit specific disciplinary approaches to teaching and learning. Examples of each practice can be defined on a departmental basis. For example, those teaching in co-op programs will immediately be able to provide examples for the practice that “helps students relate their learning experience to the world outside the classroom” (Principle 5, Practice [b]).

The following seven principles for effective teaching, taken together with the exemplary practices, reflect many aspects of “the highest standards of [classroom] teaching” at UBC. Those standards are seen within the context of a university education, which is to enable students to learn and to apply knowledge, skills and perspectives in the larger community.

They are designed with classroom teaching in mind, and can serve as a model for departments and faculties wishing to design parallel sets of principles and practices for broader aspects of instruction. This can include: designing courses and curriculum, advising and mentoring undergraduate students, and supervising graduate students.

Principle 1: Sets Clear Goals and Intellectual Challenges for Student Learning

Exemplary practices:

demonstrates and shares a clear vision of intellectual goals and learning outcomes for the class
identifies key concepts or ideas in the field and helps students to understand and apply them
integrates current research and conceptual approaches into learning activities
identifies key steps in achieving learning goals
actively helps students to accomplish goals and meet challenges as defined in the course outline
sets high, yet reasonable, expectations of students’ learning

Principle 2: Employs Appropriate Teaching Methods and Strategies that Actively Involve Learners

Exemplary practices:

shows awareness in teaching activities, that learning is a process which transforms and changes learners
encourages appropriate student participation and organizes effective learning experiences to meet intellectual goals and learning outcomes, both in the classroom and (as possible) beyond
evaluates and assesses learning in a manner consistent with established goals and learning outcomes
integrates appropriate teaching methods and technologies, tailored to course goals and learning outcomes, and facilitates student participation
encourages and assists students to participate in self-directed learning activities

Principle 3: Communicates and Interacts Effectively with Students

Exemplary practices:

expresses goals, intended outcomes, and expectations clearly and effectively and discusses these with students
balances collaborative and individual student learning to reflect the course aims and outcomes
attends to classroom dynamics that enhance or inhibit learning
engenders enthusiasm and interest in subject matter
uses fair and reasonable methods of evaluating learning

Principle 4: Attends to Intellectual Growth of Students

Exemplary practices:

provides, and discusses with students, explicit criteria for assessing learning
acquires regular and varied feedback on students’ intellectual accomplishments
reviews students’ progress in achieving intellectual goals and learning outcomes
provides advanced learning opportunities for those students who seek them

Principle 5: Respects Diverse Talents and Learning Styles of Students

Exemplary practices:

promotes a stimulating learning environment
recognizes and accommodates different learning styles
demonstrates sensitivity to intellectual and cultural issues

Principle 6: Incorporates Learning Beyond the Classroom

Exemplary practices:

encourages appropriate student-faculty interaction
helps students connect their learning experience to the world outside the classroom (both within and outside of the University)
helps students to apply their learning in a variety of ways

Principle 7: Reflects On, Monitors and Improves Teaching Practices

Exemplary practices:

seeks regular student feedback on teaching effectiveness
reflects on teaching practice through creation of a teaching dossier or other self-reflection activity
seeks peer feedback to enhance teaching
regularly revises and updates course content, format, teaching strategies, and assignments
takes advantage of opportunities to enhance teaching by attending professional development activities

Taken together, intellectual goals and learning outcomes encompass, for our purposes, specific and concrete statements about what students can apply or use as a consequence of participating in a course. These goals and outcomes range from complex goals of understanding to specific abilities in using or applying knowledge.

Responding to Information from Evaluations of Teaching

How should Deans, Directors, and Heads respond when they learn about excellent teaching or unsatisfactory teaching? The following are suggestions for best practice:

Excellent Teaching

Inform individual faculty about their outstanding performance.
Nominate individuals for university level and national level teaching awards.
Ensure excellent teaching is properly weighted in tenure, promotion, and merit decisions.
Ensure students are aware of excellent teachers among the faculty.
Help in publicizing the recognition of teaching excellence.
Reward excellent teachers by providing opportunities to further enhance abilities.

Unsatisfactory Teaching

Inform individual faculty about their unsatisfactory performance.
Encourage and motivate faculty to improve their teaching.
Further develop good professional resources, including peer coaching, mentoring, teaching workshops, and review procedures.
Ensure unsatisfactory teachers are not reappointed or given tenure as teachers.
Ensure faculty who teach are not promoted or given merit awards if their teaching is unsatisfactory.

We are assuming here that teaching is evaluated in various ways, including student evaluations of teaching and peer review of teaching.