Course:KIN570/TOPICS/Journal Issues/Critiquing Research

Research Methods
KIN 570
Section:	001
Instructor:	Nicola Hodges
	Kristin MacLeod
Email:
Office:
Office Hours:
Class Schedule:
Classroom:
Important Course Pages
	Syllabus
	Lecture Notes
	Assignments
	Course Discussion

A Brief Background

When the first “modern journal”, Le Journal des Sçavans (simply a collection of scientific essays) was published in 1665, it didn’t include a peer review process. Peer review was finally implemented in 1731 to publish Medical Essays and Observation. Individuals who reviewed the essays submitted to this publication were people the editor deemed to be “most versed in these matters”. The peer review process has become an important aspect of modern scientific society. This process ensures significant issues, possible impacting important observations and conclusions of a study, are dealt with before publication in a journal. Peer review has been shown to increase the quality of manuscripts.

In it’s most basic form, a critique is a thorough evaluation of the scientific rationale, strengths and weaknesses, and quality and originality of a paper. It generally assesses five major categories: too much information, too little information, inaccurate information, misplaced information, and structural issues.

Importance

Critiquing articles is very important for any individual involved in research, as we consistently use former studies as springboards for our own questions. Knowledge of what limits and strengthens other studies can then be applied to our own work. It is a skill that requires a certain degree of patience, as it is easy to take an article at face value. Ultimately, critiquing is important for different reasons for different people. For example, to the reader of a paper, a thorough critique helps ensure that the proper information is taken from the text. Often this means taking results and conclusions with a grain of salt. We are either instantly suspicious or easily swayed by a new finding that seems to trump all other work done in a particular area. The importance therefore lies in our ability to interpret the strength of the research and the credibility of the drawn conclusions.

To an author, however, a good critique can mean improvements to methods, a stronger development of arguments, and overall a more impactful piece of work–all valuable experience for the next study they decide to do. Really, the critiquing process leads our understanding of science–for better or for worse; if a research article is deemed unfit for publication, we may miss out on worthwhile conclusions. To highlight this, consider the Krebs’ Cycle–an energy cycle now taught to pretty much everyone. When the author first submitted his article about the Krebs’ cycle to Nature–a very high impact journal–it was rejected mainly due to its representation of an unconventional idea. Of course, the author quietly bided his time and went on to win a Nobel Prize for this work. Evidently the critiquing process can strengthen a study, but it’s important to understand that it may limit what we know if someone decides a paper just isn’t good enough…

Issues and Common Mistakes

Bias is probably the biggest issue to tackle during a critique. It is defined as the systematic prejudice that prevents objective and accurate interpretation of a study. We all have our own biases in science (though we may try to deny it), which can have a large impact on how we view research. We are often more willing to decide a study is ridiculous rather than give something, which is contrary to our own views, merit. Other issues related to this include a reviewer’s conflict of interest, breaches of confidentiality, disclosure of (or failure to disclose) funding sources, and intellectual property rights

Other common mistakes include primarily focusing on one section of a paper (usually Methods in people “new” to the critiquing process), and an inability to separate major flaws from minor ones. Things like writing style or semantics may be frustrating to the reviewer, but they are ultimately small problems. A major flaw is something where the outcome could be altered by addressing that flaw (a fundamental design error, a poorly thought out research question, inappropriate interpretation of results, and so on)

Questions to ask

First Look

Before you even read the first page of an article, consider asking the following questions:

-Is the journal appropriate?

-Does the title of the article clearly convey the research (could it be misinterpreted)?

-Is the abstract consistent with the study? (To bear in mind as you read further.)

The content of the article should be appropriate to the journal–for instance, a physiological study should ring an alarm if you find it in a psychology journal. Sometimes a study can be multidisciplinary–just make sure you can figure out when. Journal titles should essentially be the “key point” of the study. A good example of this is the following: Assume there is a cross-sectional study comparing blood pressure in a human model based on high and low sodium intake. The title, “Effects of sodium intake on blood pressure in humans,” could lead a reader to mistakenly believe there was a dietary intervention involved in the study. Simply changing the sentence to, “Blood pressure in humans with low and high sodium intake,” clears up the confusion. The abstract should reflect the study, with the same important information reflected in both the abstract and the body of the text.

Introduction

-Is there a misleading summary?

-Do you get the point? Does the study combine both novelty and import?

-Is there an appropriate, directed hypothesis?

It’s important to consider whether you are only reading about one side of a story. A good article will show you were the literature is at the moment–including studies that both support and refute the current research question. Is the author presenting information that could influence a bias? Sometimes articles must be short, and there is obviously more room for a complete back-story when the word limit isn’t cut down. All the same, it is an important consideration; if the author isn’t presenting both sides…is s/he truly aware of current research? Furthermore, it is up to you to decide if this study is a new idea (helping to fill a gap in the literature) that is also important. Are you interested in a study that is simply a rehash of an earlier one, in different words? Finally, the hypothesis should be relevant to the background being presented; it shouldn’t hit you from left field.

Methods

-Is it the right subject population?

-Is the “obvious stuff” there? (Ethics information, randomized groups, control group)

-Is it the right design?

Most of us are probably more comfortable assessing methods than other sections of a paper. This may be because an issue here is the easiest to spot. For instance, if someone is doing an exercise test that is non-invasive and submaximal with mice to determine something regarding human physiology…it’s probably the wrong subject population. Failure to include information about ethics consent, how groups are set up, and other important factors (like the time frame of a training study) is likely to jump right off the page at a reader. All the same, one of the more difficult questions also arises with the methods section–is it the right design? There are multiple ways to address a question, so the answer to this may be harder to elucidate. If you can think of a way that their design would be more valid, and still appropriate to their research question, this should raise a red flag.

Results

-Are the results clear (or are they misleading)?

-Is data presented in an overly complex way?

-Is the data realistic (and possible)?

As people involved in research, we understand the various ways that results can be interpreted; it is essentially in the hands of the person with the data. That being said, you may wonder whether the author has included certain numbers/tests that should have been done. More pressingly, is the author tricking you into taking something from the results that isn’t really there? For instance, a graph with inappropriate scaling can make even the smallest differences in numbers appear significant. Unclear results could be considered a fatal flaw of a paper. Another consideration is whether the data is presented in an overly complex way. If you have spent 20 minutes trying to figure out a graph (particularly if the paper is in your research area), and still have no idea what it means, perhaps it is simply poor presentation on the author’s part. Another thing to be aware of is whether the data is realistic and possible, within the given parameters of the experiment. For example, if a two-week training study in trained individuals demonstrates a large increase in maximal oxygen uptake…be wary.

Discussion

-Do things match up?

-Can you explain the results in an alternative way?

-So, what?

One of the first things to check for when you’re critiquing an article is whether the hypothesis has been accepted or rejected. The author should give full disclosure about the outcome of his/her study, and comment on the results using other published work as back up or to explain discrepancies. In the most basic terms, everything that was introduced at the beginning of the paper should now lead in to a more detailed discussion, including data gathered in the present study. The best papers will attempt to explain why an unexpected result occurred, rather than ignore it. The limitations of a study are a particularly important source of information–was the author aware of them? Do they change how you feel about the results as a reader? If they are not included in the discussion, you could probably find a better paper elsewhere. A further task as the reader is whether the conclusions made are justified by the results. In some cases, you may feel that you would have drawn a different conclusion from the results, or that the author has been too broad with his/her conclusions. Continuously asking whether claims made are reasonable is therefore very important. Finally, there is the “so, what?” factor. Does the author make a point of why this study is necessary? How it has shed light on a previously unknown gap, or provided new clues? Do they have a sense of what further research could or should be done? These are queries that should be answered by a well-written paper.

The Extras

-Did they get paid?

-How could the paper be improved?

-Would your concerns about the study change the outcome?

Most of these points were touched on already, but they are important to reconsider at the end of the paper, where any conflicts of interest or monetary gains should be claimed. If you read a well-written paper that demonstrated convincingly that Gatorade is the best sports drink to have in a marathon, would your opinion about the results and conclusions change if you saw Gatorade sponsored the study? Most likely. Also ponder how the study could have been improved. If you can’t think of anything major, the study may be sound. If you are coming up with idea after idea, take the results with a grain of salt. This leads straight back to a point made previously–is there a concern you have about the study, which, if altered, could change the outcome of the research?

How critiquing benefits you

Knowing how to critique a paper better allows you to view your own research with an objective eye. If you are aware of the things others will be watching out for in a study, you can jump the gun and work out any weak points before you start. Furthermore, having someone else critique a draft of your work is potentially helpful to you as well. Others see what you don’t–which is the whole point of the peer-review process to begin with. But as a reader, being able to confidently critique a paper allows you to get the important information out of a paper, and ensures you can pick out what is “good science” and what isn’t. After all, if you’re basing your research on someone else’s work, you want to be sure they have presented a solid study.

Helpful tips for tricky papers

Sometimes an article is just difficult to read. Instead of spending hours trying to read through it, walk away! Taking a break from it will allow you to think over the part you have managed to read, and digest it before you go in for round two. Another good idea is to start asking yourself very general questions about the paper (Is the journal and title appropriate? What was the author trying to find? What did s/he hypothesize? ), before you dive in to the specifics. This way you can at least understand the “big picture” of the study. Finally, remember that it’s not a competition. If someone is doing work in your area, you may feel it is inferior to your own. It’s important to always stay objective; if they have done a good piece of work, ensure you take the good points from it, rather than condemn it as sub-par. In this vein, a study may have merit even if it is outside of your comfort zone. Remember not to let your own beliefs prejudice you against another’s research–especially if it’s a good study!

Being a reviewer

As mentioned in the Brief Background, the peer review process has become an important aspect of modern scientific society. Consequently, the ability to soundly and objectively critique a research article has become an almost mandatory skill for scientists. Question is: how does one gather such a skill? This question is especially relevant for the beginning scientist with little to no experience in reading and critiquing research articles. Luckily, in recent years there has been more focus on developing guidelines as to how to go about the review process. So, what are some of the elements needed to be or become a good reviewer? In this respect at least four factors can be pointed out. First, what is the motivation of the reviewer? The desire to progress science might be one of the main motivations of a good reviewer. Second, what is the scientific expertise of the reviewer? In other words, does the reviewer have enough knowledge about the topic addressed to give valuable critiques on the research article? Third, what attitude does the reviewer display towards the authors? A helpful attitude in the form of an insightful and articulate review can be of much value to the authors. Finally, although the time it takes to write a review might vary, writing a good review can be time consuming. The reviewer must realize this and consider whether or not s/he has the time to go through the whole process.

Further Reading and Helpful Links

You may find the following links helpful and concise as you approach the critiquing process:

Seals, D. R., and Tanaka, H. (2000) Manuscript Peer Review: A helpful checklist for students and novice referees. Advan Physiol Edu. 23: 52-58

http://advanphysiology.org/content/23/1/52.full.pdf

Hoppin, F. G. (2002) How I Review an Original Scientific Article. Am J Respir Crit Care Med. 166: 1019-1023

http://ajrccm.atsjournals.org/content/166/8/1019.full

Benos, D. J., Kirk, K.L., and Hall, J.E. (2003) How to Review a Paper. Advan in Physiol Edu. 27: 47-52

http://advan.physiology.org/content/27/2/47.full