Course:KIN570/TOPICS/Research Designs/Validity issues in Research Designs

Research Methods
KIN 570
Section:	001
Instructor:	Dr Nicola Hodges
Email:
Office:
Office Hours:
Class Schedule:
Classroom:
Important Course Pages
	Syllabus
	Lecture Notes
	Assignments
	Course Discussion

RELIABILITY

Definition = Repeatable / Consistent

Forms of reliability	statistical names''
Observers (JUDGES)	Inter-observer agreement / inter-rater reliability Trained and independent
Observations (ITEMS)	Internal consistency All items should measure the same thing
Occasions	Temporal Consistency/test-retest reliability

"Reliability is a necessary but not a sufficient condition for validity"

VALIDITY

Validity has been defined as "the relative accuracy or correctness of the statement". It is essential to make an experiment as valid as possible. There are numerous forms of validity (mentioned below) but each experiment does not have to be valid or have high validity in each of these forms.

Types of Validity

Types of Validity	Definition
INTERNAL	How confident are we that the variables are related/or that one caused the other? Need to make sure confounds are eliminated
EXTERNAL	How well can we apply or generalize these data to different people or conditions? Do they represent reality?
CONSTRUCT	Are we measuring the correct thing/construct? Do our measures reflect the thing we wish to generalize to (i.e., good operational definitions)?
CONCEPTUAL	Does the experiment/manner of testing make sense given the theory? Do the research and hypotheses make sense and allow broad conclusions about a theory or idea?

Further Explanations and Examples

Internal Validity: Laboratory experiments tend to have high internal validity due to the ability to isolate and control numerous variables. John Stuart Mill stated 3 requirements for an experiment to ensure that the results obtained are due to the manipulated (independent) variable:

1. Covariation: changes in one variable must correspond with changes in the other
2. Temporal sequence: changes in one variable will not only cause changes in a second but will also occur before the second
3. Eliminate confounds: no other variable, other than the manipulated variable, causes changes in the measure variable

External Validity: passive observational studies are usually high in external validity because any change in the environment should not cause an unnatural reaction

Construct Validity: Requires a very good operational definition to ensure that the ways of measuring the variable are actually representative of the measured variable.

Conceptual Validity: Pelham & Blanton (2003) explain an example of an experiment with low conceptual validity through examining the Cognitive Dissonance Theory: According to the Cognitive Dissonance Theory, if money is supposed to create positive motivation then an increase in money received should decrease the amount of negative connotation associated with a task. To test this theory, participants would be given either $1 or $100 to poke their eye with a sharp object. The experimenters' alternate hypothesis would be that participants who receive $100 would be more likely to poke themselves in the eye with a sharp object compared to participants who receive $1. Pelham & Blanton (2003) describe this as having low conceptual validity because the experiment is too extreme and is not relevant to the theory.

Validity Threats

There are potentially many instances where validity of your experiment or study could be threatened. The major threats are to do with 4 factors:

Look out for	solutions
Differences between people	Randomize/sampling strategies
Time changes people	Control groups/multiple pretests
Issues with being studied	Deception/Debrief/Blinded
Hidden/unintentional variables	Randomization/Double blind

In class we spoke about one of these threats in terms of the phenomenon regression to the mean (RTM). This is a problem when there are a small number of observations and people are assigned to groups based on one score (which could be due to many factors including chance). People change and with repeated observations their "true" score approaches the mean. So, we expect that people who do really well or score highly, will, just due to RTM, score lower when tested the next time. Conversely, people who score low the first time, are more likely to show a higher score in subsequent testing just due to RTM.

Glossary

Confound = an additional variable (uncontrolled) that covaries with the independent variable. I.e. An uncontrolled variable that also elicits changes in the measured variable.
Sampling

Blind experiment

REFERENCES

Additional sources: Pelham, B.W. & Blanton, H. (2003). Conducting research in psychology: Measuring the weight of smoke. Belmont, CA: Thomson/Wadsworth. <Ch 3 and 4>
Martens, R (1973, june), People errors in people experiments, Quest, 20, 16-20.
Shephard, R.J. (2003). Regression to the mean: A threat to exercise science? Sports Medicine, 33(8), 575-584.