Course:KIN570/TOPICS/Research Designs/Experimental and other designs
| Research Methods | |
|---|---|
 | |
| KIN 570 | |
| Section: | 001 |
| Instructor: | Dr Nicola Hodges |
| Beverley Larssen | |
| Coralie Riendeau | |
| Email: | |
| Office: | |
| Office Hours: | |
| Class Schedule: | |
| Classroom: | |
| Important Course Pages | |
| Syllabus | |
| Lecture Notes | |
| Assignments | |
| Course Discussion | |
Research Design
There are a number of research designs available to scientists interested the field of Kinesiology. It is important to note that Kinesiology is a wide area that includes research in physiology, biomechanics, socio-cultural sciences, and sport & exercise psychology. Researchers in the field use a number of research designs to answer their questions. It is important to note that the research design needs to be selected based on the specific question that one is trying to answer. The result one obtains from a study is only as good as the data collection method.
Qualitative vs. Quantitative Research Design
There are some important difference between qualitative research design and quantitative research design.
Qualitative research seeks to understand phenomena by looking at the experience of people. Qualitative researchers use two main methods of inquiry: observation and interview.
- Observation: Qualitative researchers usually observe a given person or group of people in their naturalistic environment. They can use a covert or overt observation method. When making observations, the qualitative researcher will typically refrain from intervening with the people being observed.
- Interview: Qualitative researchers use interviews to gain access to the subjective experience of their research participants. They are a good method to obtain in-dept data about participants’ actions, thoughts, and feelings.
Quantitative research generally seeks to understand phenomena by using statistical methods to analyze numerical data such as blood pressure, score on a motor learning task, and score on a perfectionism questionnaire. A number of research designs are used by quantitative researchers. Quantitative research designs can have different goals such as describing/summarizing, predicting, determining causal relationships, and explaining the mechanism responsible for causal relationships. The design should be matched with the aim of the study.
Non-Experimental Research Designs
Case Study
An in-depth descriptive study that focuses on one individual or group. Often the subjects of case studies have experienced rare/unusual circumstances. The experimenter records observations and then constructs an opinion-based narrative on the findings. Data is often explained in terms of established scientific principles.
Case study could be considered a qualitative research design.
| Advantages | Limitations |
|---|---|
| Important for development of theory in areas where little research has been conducted. | Does not generalize well to population (Threat to External Validity). |
Note: For those interested in reading a compilation of unique case studies, I encourage you to read The Man Who Mistook His Wife for a Hat, by Oliver Sacks.
Naturalistic Observation
This type of design is often used to observe the behaviours of a person or organism in its natural environment. It can lead to discovery of important behaviours and events that would not necessarily occur in laboratory setting.
Naturalistic observation can be a qualitative or quantitative research design. The orientation of the design will depend on the type of data collected.
- A qualitative researcher would take notes on what one saw, such as actions, atmosphere, and type of interventions. For example, the subjective feeling of how aggressive team A and team B were behaving in a basketball game won by team B.
- A quantitative researcher would favor a type of observation that includes recording the number of time a given person, or group, behave in a certain way. For example, number of instrumental aggressive behaviours committed by team A and team B respectively in a given basketball game.
| Advantages | Limitations |
|---|---|
| Has strong external validity. Allow to see how people behave in real world setting. | Difficult to make valid observations if subjects are aware of researchers. |
Survey & Questionnaire Designs
Population surveys are tools used to gather information on public opinion and assessing trends. Questions and response choices must be carefully designed to ensure that the questions relate back to the specific aims of the research. The education level of the target group must also be carefully considered when selecting or designing a questionnaire.
This type of research design is quantitative.The data obtain from surveys and questionnaires designs can be analyzed using correlations (see correlational design).
| Advantages | Limitations |
|---|---|
| Easy and inexpensive to distribute. | Confounds such as responder bias can give results that are not a true representation of the population. |
| Opinions can change on a daily basis which can effect the reliability of this design. | |
| Researchers must be careful to collect responses from a representative sample of the population of interest. |
Cohort Study
This design observes a group of individuals with a common experience/characteristic over a period of time. Assignment to groups is determined by pre-existing conditions (ex. Smokers vs. Non-smokers; Obsese vs. Healthy Body Weight). There are 2 types of cohort designs: Retrospective Chohort Study and Prospective Cohort Study. In the Retrospective design, researchers examine historical data of a given population to investigate the effects of the variable of interest. In the Prospective design, the study is ongoing and individuals’ status is plotted over a long period of time. It is important that to begin with, all members of the prospective cohort must be free of the trait being examined.
| Advantages | Limitations | |
|---|---|---|
| Retrospective Cohort | Fast and cost-effective. | Cannot eliminate confounds from historical data. |
| Prospective Cohort | Allows researcher to measure and control confounding variables. | Expensive and long duration. |
| Participants may drop out of the study (decreases external validity). |
Longitudinal Study
Longitudinal design looks at long-term effects in a human population sample. This kind of design can be employed for qualitative and quantitative designs.
- An example of a longitudinal qualitative study would be annual interviews with a small sample of elite athletes over a four years Olympic cycle to look at coping mechanisms used by athletes over time.
- An example of longitudinal quantitative study would be annual measures of self-esteem and sport engagement in a large adolescent sample from age 8 to age 14 to look at how these two variables covary over time.
| Advantages | Limitations |
|---|---|
| Multiple measures can be taken. | The experiment cannot be restarted easily as time is a key variable. |
| Allow to study the progression of some variables in the same individuals. | Participants may drop out of the study (decreases external validity). |
| Due to the long-time span of the study, some measures may become dated. |
Cross-Sectional Design
This design takes a "snapshot" of a population at one point in time. It is often used to investigate prevalence of a specific trait in a group of people, which will then lead to further investigation. Can be used to study group of peoples who have a pre-existing condition or share a characteristic (for example, study different age-groups, or study males and females).
| Advantages | Limitations |
|---|---|
| Easy way to perform preliminary work. Not time-sensitive like a longitudinal study. | Only provides a "snapshot". Results could differ at a different time. |
Correlational Study
This design is used to examine the relationship between at least two variables. For example, high score on variable X is associated to high score on variable Y (Positive Correlation), or high score on variable X is associated to low score on variable Y (Negative Correlation), or the level of variable X is not associated to the level of variable Y (No Correlation).
This design is quantitative because it uses statistical analysis to determine the presence or absence of relationship between the variables of interest.
| Advantages | Limitations |
|---|---|
| Can provide useful information about the degree to which 2 variables are related. | Cannot infer causation. |
Experimental Research Designs
Experimental research designs usually seek to discover causal relationships and determine the mechanisms that bring about these causal relationships. A number of designs can be used, some of the main experimental research designs will be reviewed in the following section. It is important to know that the two key methods used in order to infer strong causal relationships are random assignment and manipulation of the independent variable.
- Random assignment is a method used to equate the different experimental groups on a number of variables that could potentially influence the dependent variable, such as age, gender, fitness level, and education level. When a researcher is randomly assigning participants to experimental group (or treatment group) every participants have an equal chance of being assigned to either one of the group. Random assignment allows reducing the influence of individual differences.
- Manipulation of the independent variable by the experimenter allows inferring strong causal relationships because the only thing that differs between the groups is the different levels of the independent variable.
However, random assignment and manipulation of the independent variable are not always possible, which leads to a number of variant of the experimental research design, which are often called Quasi-experiment designs.
Simple Pretest-Posttest Design
A random sample of participants is assigned to one group. Compares pre-intervention performance to post-intervention performance.
| Advantages | Limitations |
|---|---|
| Comparison allows us to infer effectiveness of intervention. | Does not control for effect of pre-test (Practice Effects). |
| Measures are taken on the same people twice, which reduces the impact of individual differences on change scores. | Doesn't have a control group (low internal validity). |
Between Subjects Design
A sample of participants is randomly allocated to 1 of 3 different groups (see diagram on the right). All groups undergo a pretest and a posttest. Group 1 is exposed to the intended treatment (ex. watching expert model). Group 2 (negative scientific control group) is exposed to a condition that is known to have little or no effect on slap shot performance. Group 3 (positive scientific control group) is exposed to a condition that is known to have a moderate effect on performance. Since these control groups have highly reliable expected results, we can assume that if unexpected performance effects are seen in the controls, there may be something affecting our treatment group as well.
| Advantages | Limitations |
|---|---|
| Negative scientific control group can act to control for effect of pretest. | Not practical to conduct. Adding more groups uses more time and resources. |
| Different control groups strengthen internal validity of design. | |
| Allows for comparison across groups. |
Solomon 4 Group Design
This design offers a method to control for practice effects (or learning effect) as a result of completing the pretest. Group 1 receives the experimental treatment and Group 2 receives the negative scientific control treatment. Group 3 and 4 do not receive the pre-test measure but receive the same treatment as Group 1 and 2 respectivelly (as illustrated by the diagram).
| Advantages | Limitations |
|---|---|
| Controls for effect of pretest (Practice Effects). | Results are difficult to statistically analyze. |
| Different control groups strengthen internal validity of design. | |
| Allows for comparison across groups and conditions. |
Within Subjects Design (or Repeated Measures Design)
All groups experience all treatment conditions, controls and tests in the same temporal order.
| Advantages | Limitations |
|---|---|
| Scores are relative to individuals. Controls for individual differences. | Subjects are likely to experience practice effects or carry-over between conditions. |
Counter-balanced Within Subjects Design (or Counter-balanced Repeated Measures Design)
One way that experimenters can control for the practice effects of a classic repeated measures design (see above), is by counter-balancing the order of exposure to treatments. For example, if one group of participants are given a period of observational practice followed by a period of physical practice, a second group with the same number of participants can be exposed to the opposite order of treatment conditions.
| Advantages | Limitations |
|---|---|
| Controls for practice effects. | Final posttest values may be biased if there is an effect of order of condition. |
Quasi Experiment Research Designs
Differs from a (true) Experimental Design because assignments to groups are pre-determined by fixed criteria (i.e. no random allocation of participants to groups). Researchers have only partial control over the levels of their independent variables. A number of quasi-experiment research designs exist:
- Person-by-treatment quasi-experimental design: The researcher measure at least one independent variable (for example Gender) and manipulate at least one independent variable. The independent variable manipulated by the researcher is typically randomly assigned to participants.
- Natural group with experimental treatment research design: The researcher uses naturally occuring groups (for example Families or Sport teams) and assign them randomly to the different levels of the independent variable.
To increase confidence that the variable manipulated by the researcher is causing the observed effect controls groups or "patched" groups can be added. They help control for confounding variables such as age, socioeconomic status, etc.
| Advantages | Limitations |
|---|---|
| "Patches" can help control for confounds | You cannot always randomly allocate participants to groups → Threat to Internal validity |
Literature Review
This is a method of critically analyzing existing research in a given field. It brings similar studies together and can help identify gaps in the literature. 2 methods are meta-analyses and systematic reviews.
| Advantages | Limitations |
|---|---|
| Does not require further experimentation because data is already collected. | Potential for publication bias, if no work refuting hypotheses is published. |
| By bringing many small studies together, sample size is effectively increased. | Methods of sampled studies are not always comparable. |
Meta-Analysis
Data collected from existing studies are brought together and statistically analyzed as a group.
Systematic Review
This design reviews a selection of studies that asked similar research question. The results are compiled and critically analyzed to generate some narrative on important findings. A systematic review often will include some form of meta-analysis.
Glossary
- Control group: A group which is often not exposed to the treatment/manipulation so that it can later be compared to the treatment group.
- External validity: A study with strong external validity, generalizes well to the population of interest.
- Independent variable: The variable being controlled/manipulated by the experimenter.
- Internal validity: A study has high internal validity when we are confident that variables are related/or that one caused the other. One way to achieve this is by eliminating confounds by implementing controls.
- Practice effects: Practice/exposure to one condition influences performance in future conditions.
- Random assignment: Participants are randomly assigned to groups. A given participant has an equal probability of being assigned to any group in the experiment.
References
- Research Design. Retrieved from http://explorable.com/research-designs.html
- Pelham BW & Blanton H (2003). Ch 5-7. Conducting Research in Psychology.
- Pelham, B. W. & Blanton, H. (2003) Conducting Research in Psychology: Measuring the Weight of Smoke. Second edition, Canada: Thomson Wadsworth.
- Thomas, J. R., Nelson, J. K., & Silverman, S. J. (2005) Other descriptive research methods. In J. R., Thomas, J. K., Nelson, & S. J., Silverman (Second edition), Research Methods in Physical Activity (291-305). United States of America: Human Kinetics.
- Thomas, J. R., Nelson, J. K., & Silverman, S. J. (2005) Experimental and quasi-experimental research. In J. R., Thomas, J. K., Nelson, & S. J., Silverman (Second edition), Research Methods in Physical Activity (329-351). United States of America: Human Kinetics.
- Hastie, P. & Hay, P. (2012) Qualitative approaches, In K., Armour & D., Macdonald, Research methods in physical education and youth sport (79-94). Great Britain: Routhledge.