Course:KIN570/TOPICS/Research Designs/Validity issues in Research Designs

Research Methods
KIN 570
Section: 001
Instructor: Dr Nicola Hodges
Email:
Office:
Office Hours:
Class Schedule:
Classroom:
Important Course Pages
Syllabus
Lecture Notes
Assignments
Course Discussion

RELIABILITY

Definition = Repeatable / Consistent

Forms of reliability statistical names''
Observers (JUDGES) Inter-observer agreement / inter-rater reliability

Trained and independent

Observations (ITEMS) Internal consistency

All items should measure the same thing

Occasions Temporal Consistency/test-retest reliability
"Reliability is a necessary but not a sufficient condition for validity"[1]

VALIDITY

Validity has been defined as "the relative accuracy or correctness of the statement". [2] It is essential to make an experiment as valid as possible. There are numerous forms of validity (mentioned below) but each experiment does not have to be valid or have high validity in each of these forms.

Types of Validity

Types of Validity Definition
INTERNAL
How confident are we that the variables are related/or that one caused the other? Need to make sure confounds are eliminated
EXTERNAL
How well can we apply or generalize these data to different people or conditions? Do they represent reality?
CONSTRUCT
Are we measuring the correct thing/construct? Do our measures reflect the thing we wish to generalize to (i.e., good operational definitions)?
CONCEPTUAL
Does the experiment/manner of testing make sense given the theory? Do the research and hypotheses make sense and allow broad conclusions about a theory or idea?

Further Explanations and Examples

Internal Validity: Laboratory experiments tend to have high internal validity due to the ability to isolate and control numerous variables. John Stuart Mill stated 3 requirements for an experiment to ensure that the results obtained are due to the manipulated (independent) variable:

1. Covariation: changes in one variable must correspond with changes in the other
2. Temporal sequence: changes in one variable will not only cause changes in a second but will also occur before the second
3. Eliminate confounds: no other variable, other than the manipulated variable, causes changes in the measure variable

External Validity: passive observational studies are usually high in external validity because any change in the environment should not cause an unnatural reaction

Construct Validity: Requires a very good operational definition to ensure that the ways of measuring the variable are actually representative of the measured variable.

Conceptual Validity: Pelham & Blanton (2003)[3] explain an example of an experiment with low conceptual validity through examining the Cognitive Dissonance Theory: According to the Cognitive Dissonance Theory, if money is supposed to create positive motivation then an increase in money received should decrease the amount of negative connotation associated with a task. To test this theory, participants would be given either \$1 or \$100 to poke their eye with a sharp object. The experimenters' alternate hypothesis would be that participants who receive \$100 would be more likely to poke themselves in the eye with a sharp object compared to participants who receive \$1. Pelham & Blanton (2003) describe this as having low conceptual validity because the experiment is too extreme and is not relevant to the theory.

Validity Threats

There are potentially many instances where validity of your experiment or study could be threatened. The major threats are to do with 4 factors:

Look out for solutions
Differences between people Randomize/sampling strategies
Time changes people Control groups/multiple pretests
Issues with being studied Deception/Debrief/Blinded
Hidden/unintentional variables Randomization/Double blind

In class we spoke about one of these threats

Glossary

Confound = an additional variable (uncontrolled) that covaries with the independent variable. I.e. An uncontrolled variable that also elicits changes in the measured variable.
Sampling

REFERENCES

1. Pelham & Blanton (2003). Ch 3: Moving front fact to truth: Validity, Reliability and Measurement.
2. Pelham & Blanton (2003). Ch 3: Moving front fact to truth: Validity, Reliability and Measurement.
3. Pelham & Blanton (2003). Ch 3: Moving front fact to truth: Validity, Reliability and Measurement.
4. Bland, J.M & Altman, D.G., 1984. Some examples of regression towards the mean. British Medical Journal, 309, 780

Additional sources: Pelham, B.W. & Blanton, H. (2003). Conducting research in psychology: Measuring the weight of smoke. Belmont, CA: Thomson/Wadsworth. <Ch 3 and 4>
Martens, R (1973, june), People errors in people experiments, Quest, 20, 16-20.
Shephard, R.J. (2003). Regression to the mean: A threat to exercise science? Sports Medicine, 33(8), 575-584.