Quality Control

MER Manual
Table of Contents
	Contributors; Lists; MER/Manual; About; Content; Navigation; How To; Design; Installation;

It is an essential for the success of the MER project that high quality is maintained throughout the resources that we offer. To accomplish this, the following elements are in place:

Contributors

Inside the UBC wiki, the Science space is special in the sense that only authorized users can edit content. To get access, talk to the Organizing committee.

Peer Review Process

All mathematical contributions are reviewed by another contributor. For question statements, hints, solutions and tags, this is achieved using flags.

First, a flag will call for contributors to add the relevant content (e.g. a CS flag calls for writing a solution).
Once the content is add to the satisfaction of the contributor(s), the content can be flagged for review (e.g. by changing the CS flag into a RS flag).
Another contributor, not involved in typing the content, will review the content for quality. The pages Statements, Hints, Solutions and Tags contain relevant information for this (what to look for, agreed standards, etc.).
The reviewer choses a matching quality flag (e.g. a QBS flag to indicate there are mistakes and more work to do in a solution or a QGS to indicate the solution satisfies good quality criteria).

Flags serve two purpose: indicate to the contributors where there is work to do and the nature of that work; and inform students of the quality of the information they are reading.

Flags

Flags are categories used by contributors to identify and request work to be performed on the Math Exam Resources wiki. There are flags for two different page-types on the wiki: exam status flags and question status flags.

There are four kinds of question flags:

Content flags which request contributors to add specific content to a page.
Review flags which request contributors to check pages for Quality control purposes.
Quality flags which is used by a reviewer to indicate the quality of the reviewed content.
Special flags for other purposes.

There are two kinds of exam flags:

In progress flags which make an exam available to students, even if it is incomplete.
Quality flags which indicate that an exam is completed.

Each flag is described in more details below.

Organization of flags

The content, review and quality flags used for exam questions are organized as follow: We identify four different types of mathematical content: Question Statements (Q), Hints (H), Solutions (S), and Tags (T). For each type of content, we have four status: a flag to request the input of the content (C), a flag to request the content to be reviewed (R), a flag for content reviewed as bad (QB) and a flag for content reviewed as good (QG). Using the convention to put the status first and the type of content after; you can now easily imagine the 4 by 4 matrix which describes the 16 flags listed below:

List of Flags
	C - Content request	R - Review request	QB - Quality is bad or insufficient	QQ - Quality is good
Q - Question Statement	CQ	RQ	QBQ	QGQ
H - Hint	CH	RH	QBH	QGH
S - Solution	CS	RS	QBS	QGS
T - Tag	CT	RT	QBT	QGT

How to add or change flags

Question-related flags are listed in the source of the main page of the corresponding question. Eg, the flags for Math 103 April 2012 Question 1(a) are found on the source of Science:Math_Exam_Resources/Courses/MATH103/April_2012/Question_1_(a).

Click "Edit" and you find

[[Category:MER QGQ flag]][[Category:MER QGH flag]][[Category:MER QGS flag]][[Category:MER CT flag]]

which indicates that the Question Statement, the Hint, and the Solution are all reviewed as good, while the Tag is flagged as missing content.

If you were to add a Tag, you would change the associated flag from "missing content" to "review requested", hence change

[[Category:MER CT flag]] to [[Category:MER RT flag]].

Exam related flags are listed in the source code of an exam page. For example, one this exam page, you can see

[[Category:MER Exam QG flag]]

Content flags

Content flags request contributors to keep adding content to a page. Whether you have a few minutes or hours ahead of you, this is where the chunk of the contributions are being made. Before adding content, have a look at the associated discussion page to be aware of previous concerns, demands or issues related to the page you are working on.

Here is the list of all adding flags and their specifics.

Note to new contributors: New blank pages are created with CQ, CH, CS, CT flags at the mother page of the question that contains the question, hint and solution subpages. To change the flag, say, CQ to RQ, go to the mother page to do so instead of the sub-page (i.e. question statement page) you're editing.

CQ - adding a question statement

The title of the page will already identify which course, exam and question is required. You can find pdf file with past exams on the department's website. Once a question statement has been fully written, change the CQ flag into RQ to request review.

CH - adding hints

Providing hints allows students to work on problems for themselves before reading a solution. Please read the Hints page for more information about providing useful hints. Once enough hints have been provided, change the CH flag into RH to request review.

CS - adding solutions

Worked out solutions should provide more than just the answer to an exam question, they should allow students to examine what skills are required to solve the problem and invite them to examine their strenghs and weaknesses. Please consult the Solutions page for more information about writing effective solutions. Once a solution has been provided, change the CS flag into RS to request review.

CT - adding tags

Tags help students identify the relevant skills and topics connected to an exam question. By adding tags, we create the possibility for instructors and students to search for all exam questions with that tag, for example related rates problems. Please consult the Tags page for a precise description of existing tags and ongoing discussions on the creation of new ones. Then change the CT flag into RT to request review.

Review flags

Review flags request contributors to review edited content on a page. This is an important task to guarantee excellence in our quality control. If you want to review content, please read the quality control section of the relevant pages (see below for details) to make sure you are aware of what the task entails. Please also have a look at the associated discussion page to be aware of previous concerns, demands or issues related to the page you are working on.

An important rule of reviewing is that you cannot review content that you added yourself. The purpose of the reviewing process is to ensure at least two contributors examined a given content to ensure high quality on this wiki. To signal that a contributor has reviewed some content, the corresponding review flag is removed and a quality flag is added (see Quality flags below).

Here is the list of all review flags and their specifics.

RQ - review a question statement

Make sure that an exam question actually is the correct one. If so, change RQ to QGQ, otherwise, change to QBQ. You can find pdf file with past exams on the department's website.

RH - review hints

Reviewing hints requires contributors to ensure they are effective and well scripted. If so, change the RH flag to QGH, otherwise, change to QBH. Please consult the Hints page for more information.

RS - review solutions

Reviewing solutions requires contributors to ensure they are effective and well scripted. If so, change the RS flag to QGS, otherwise, change to QBS. Please consult the Solutions page for more information.

RT - review tags

Reviewing hints requires contributors to ensure they match the standards and are well scripted. If so, change the RT flag to QGT, otherwise, change to QBT. Please consult the Tags page for more information.

Quality flags

Quality flags indicates to contributors and students the quality of a specific type of content. For each type of content (Statements, Hints, Solutions, Tags) we will have two kinds of quality flags: B for bad (with errors or mistakes) or incomplete content; and G for good and complete content. Quality flags are added by a reviewer and follow review flags.

Here is the list of all quality flags and their specifics.

QBQ - bad question statement

For Question statements which the reviewer decides to be bad or incomplete, following the guidelines described in the Statements page.

QGQ - good question statement

For Question statements which the reviewer decides to be good and complete, following the guidelines described in the Statements page.

QBH - bad hint

For Hints which the reviewer decides to be bad or incomplete, following the guidelines described in the Hints page.

QGH - good hint

For Hints which the reviewer decides to be good and complete, following the guidelines described in the Hints page.

QBS - bad solution

For Solutions which the reviewer decides to be bad or incomplete, following the guidelines described in the Solutions page.

QGS - good solution

For Solutions which the reviewer decides to be good and complete, following the guidelines described in the Solutions page.

QBT - bad tag

For Tags which the reviewer decides to be bad or inaccurate, following the guidelines described in the Tags page.

QGT - good tag

For Tags which the reviewer decides to be good, accurate, and complete, following the guidelines described in the Tags page.

Exam Flags

There are two kinds of exam status flags.

In progress

The Category:MER Exam IP flag is added to unfinished exams where there is a "sufficient"* level of material for students to use it, despite being incomplete. This flag will add the exam to our "exams in progress" counters.

*Use of this flag is up to the discretion of the tagger. One rule of thumb would be that about half the solutions are present, and of good quality, in order for the exam to be flagged as in-progress. Exams with little (or poor!) content added should not be flagged at all.

Complete exams

The Category:MER Exam QG flag is added to exams where all questions, hints, and solutions are complete and flagged as good quality. This will add the exam to our "exams finished" counter.