Documentation:Open Textbook Publishing Guide/Impact Data

Developing strategies and approaches to measuring the success of your open textbook project should be developed during the Project Plan stages. Collecting and analyzing this data will help you maintain accountability to the goals of your project and to understand whether your project is successful. Measuring success will depend on a few factors:

What constitutes success for your project and how will it be measured? Pedagogical innovation? Surveys, Interviews?
Who do you need to report to about the open textbook project? Funders, your institution, department?
What data is needed to indicate "success"? Adoption numbers, students impacted?
How do you need to use the data for your own portfolio? Tenure and promotion, merit?

Working through these questions will assist you in collecting the right metrics to report fully about the success of your project. This guide outlines the kinds of data that you can collect to frame discussions about your open textbook project and to ensure you have reached your goals.

Collecting Metrics

Example - Calculus Volume 3 Adoption

BCcampus Open Textbook Collection - Data collected for textbook adoption includes the institutions and courses where the textbook is used as well as the cost savings for students.
Open Stax - Data collected by use rather than adoption. The form is broken down into students, faculty, homeschooler, administrator, librarians, instructional designer, adjunct faculty, and other to create a more tailored data collection process.

One of the side benefits of self-publishing an open textbook is an automatic membership in the open education community for which it is written. As the author of an open textbook, you are in a unique position that not only allows you to pinpoint members of this community — by tracking instructors who use or adopt your book in the classroom — but also establishes you as an OER contributor. This section provides you with some strategies when collecting data about your textbook's use.

Adoption Data

Collecting adoption data can be a difficult process as it requires adopters to identify that they are using the text in their courses or instructional practices. Developing a feedback mechanism for your textbook that requests that adopters identify their use of the textbook is the easiest way to gather this data.

To begin, decide how you will track adoptions and what statistics you’ll collect. For instance, you might want track adoption trends by course level, type, term or semester.

Examples - Open Education Repository Data Collection
Open Education Repository	Data Collected by Object
UBC cIRcle	Views and downloads by country; Views and downloads by month and year; Total views and downloads
OER Commons	Numbers of times a resource is viewed; Number of times a resource is saved; Ratings out of 5 stars; Comments If you have a paid OER Commons Hub, there are additional metrics available.
Merlot	User ratings; Comments; Merlot Awards System; If you use the Content builder you can add Google Analytics for additional metrics.

Then identify the tool you will use to gather this data. For example, you can develop a Qualtrics form located as a link on the homepage of your textbook, in the front matter of the textbook, or in a metadata field of your open textbook record to encourage self-reporting.

Open Education Repository Data

Depending where you decide to host your open textbook you may be able to take advantage of the platform to gather analytics about your textbook's use. The data you can collect includes views (by year/month), downloads (by year/month), and user ratings and comments. While there may be many data options for you, you need to decide how you will use it and select what is most useful to your success measures.

Catalogue Data

If you have made your open textbook available through cIRcle, Summon (General Search), and/through the UBC Library catalogue (Books & Media), there may be some circulation and usage data that UBC Library can collect. To learn more about adding your textbook to UBC Library systems, review Developing a Release Plan.

Open Monograph Publisher Data

If you are using Pressbooks, there are metrics available for the web version of your textbook. You can find metrics in the dashboard menu labelled "WP-Piwik". The Piwik Analytics can tell you how many users have visited your book over the last 30 days, which pages were viewed, and many other pieces of information.

Evaluation Metrics

While usage and adoption statistics provide part of the story of your open textbooks success, understanding the impact on students beyond cost savings can provide broader insights on the value of tailoring an open textbook for pedagogical purposes. This kind of evaluation may include the following questions:

What does ‘impact’ look like if considering innovation in teaching and learning?
- A rise in grades
- Greater understanding of complex topic areas
- Increased engagement with the textbook and course content

For support in developing an evaluation plan that addresses the pedagogical value of your open textbook project, contact contact Lucas Wright, Senior Educational Strategist, CTLT.

Put your stats to work

UBC Tenure and Promotion

In 2017 UBC included language recognizing creation and adaptation of open educational resources (OER) in the institution's Guide to Reappointment, Promotion and Tenure Procedures at UBC. The guide lists the following among several examples of evidence of educational leadership:

Contributions to the practice and theory of teaching and learning literature, including publications in peer-reviewed and professional journals, conference publications, book chapters, textbooks and open education repositories/resources (p. 14).

In the UBC/UBC Faculty Association Collective Agreement, "educational leadership" is one of the areas (along with teaching and service) in which candidates in the educational leadership faculty stream are evaluated for appointment, reappointment, tenure, promotion, and merit. It is defined in that agreement as:

Activity taken at UBC and elsewhere to advance innovation in teaching and learning with impact beyond one’s classroom (Part 4.04).

When considering how to talk about OER creation or adaptation as part of your annual report or dossier for reappointment, tenure, promotion or merit, given the definition of educational leadership above you should work to show the impact of such work "beyond your classroom," whether at UBC or elsewhere.

Possible ways to report your open education activity in your annual report:

Reporting adoption statistics;

Keep records of personal communications that give praise for the work or talk about how it's being used;
Note any adaptations of the work by others;
Solicit feedback from peers or students on their perceptions of the quality and usefulness of the work;
Or other ways to show that the OER is being used and considered a valuable resource by others.

Adaption Statements

Adapted from the Measure of Success section in Institution as e-Textbook Publishers by JISC licensed under a CC BY licence.
Adapted from the The Rebus Guide to Publishing Open Textbooks (So Far) by Apurva Ashok and Zoe Wake Hyde is licensed under a Creative Commons Attribution 4.0 International License, except where otherwise noted.