Course talk:PSYC305/2013ST2/ClassProject/5.2.1 Discussion - Scale Validity

"Internal validity refers to the degree to which changes in the dependent variable are due solely to the effect of changes in the independent variable"

Since this is a questionnaire and not an experiment, do we even have independent and dependent variables? Is internal validity the right phrase to be using here?

That's what I was thinking. This part: "...this method is quite vulnerable to error and bias, such as the participants' level of social desirability, carelessness, or faulty memory." seems to be more characteristic of response sets, which affect reliability, not validity. Does anyone think it would be okay to have a "scale reliability" section under this page? Or something of the sort.

Hi Mona, remember that if a scale isn't reliable then it also won't be valid. Nevertheless, having a separate section on reliability could be a good idea to make it more specific.

I just read through the Limitations section and I think someone has already touched on what I had in mind on the subject of reliability. Thanks for your feedback!

I haven't looked at the specific part, but perhaps it would go better here than in the limitations section. What do you and others think?

I read through it again and I think that it fits well in the limitation section. It makes sense and flows well with the organization of that page. I will perhaps add something to it, though.

Hi all, overall I think we have good reason to believe we have external validity for our scale. The questions look to be measuring what we want them to (face validity), seem to form a single factor (factorial validity), and don't correlate with most of the Big-5 traits (discriminant validity). While all of the points outlined in this section are true, after reading this section one is left with the view that our scale isn't valid, whereas, as I said, I think we have good reason to believe it is.

Hello! I added the internal validity threat: participant maturation to this page. Let me know if there is anything that seems off about the edit. I thought it was important to add it, however, since there is no controlled setting for when participants actually take the questionnaire...unlike certain studies where participants are in a room or what not without distractions while completing the questionnaire.

Here is a tiny section from the book - internet research methods, that covered basically what you said about maturation. Checkout P.42-45, questionnaire , see if you can find more information to add onto your part. http://books.google.ca/books?hl=zh-TW&id=r5kz94nV_pIC&q=when+materials+are+administered+via+a+computer+terminal+rather+than+in+person%2C+the+researcher+is+less+able+to+judge+the+extent+to+which+the+responses+are+sincere+and+genuine%2C+the+conditions+under+which+the+questionnaire+was+answered+and+the+state+of+the+participants+at+the+time+of+participation+%28for+example%2C+intoxicated%2C+distracted%2C+and+so+on%29+-+See+more+at%3A+http%3A%2F%2Fwww.restore.ac.uk%2Form%2Fquestionnaires%2Fquesprint2.htm%23sthash.FYBtoHtQ.dpuf#v=onepage&q=distracted&f=false

Hey Chelsea! I like the maturation paragraph. I edited some parts of the paragraph, so it flows better with the rest of the page. However, I wasn't sure what you meant by fatigue in the sentence of "causing temporary factors, such as fatigue to affect the way participants answered questions". Just wondering what kind of fatigue are you referring to?

Hi all, factorial validity is also something that could be mentioned here.

I'm kind of confused about what factorial validity is exactly?

It is a type of construct validity. Based on the correlations between the items in our final scale, there is evidence that the items are measuring a single construct. The results of the factor analysis are what indicates this.

http://books.google.ca/books?id=t3fskSDJnjAC&pg=PA447&dq=factorial+validity&hl=en&sa=X&ei=C6P8UcOtOYWXiALc14GQBw&ved=0CC0Q6AEwADgK#v=onepage&q=factorial%20validity&f=false

I think the GD scale may not apply to all cultures, because the scale was made by researchers from Western culture. Using one cultural thought and value to set as a standard for study is difficult to examine subject of other cultures. For example, the participants from other cultures may misinterpret the wording, value or the meaning of the questionnaire. Therefore, decrease the validity of the measuring scale.

That's right - do you know what type of validity you're talking about here?

Would this be low generalizability then?

also, I just realized there were 50 reported female participants and only 10 males, which would extremely limit the generalizability here.

I believe it would be external validity, as the results can only applied to a particular population (ie, Western, female).

I was wondering should the discussion of the lack of validity of the scale be discussed here or the limitation session, because I've seen it being covered there but i think they are more appropriate to be posted here. Meanwhile, i will separate this session into external validity and internal validity, feel free to add stuff in:)

^I agree, I added a sentence about face validity in the internal validity part and started a short intro that tells people why we are doing this section, feel free to edit and add more stuff in!

Whoever mentioned the W.E.I.R.D. thing, could you add a definition of that in there for clarification? Not quite sure what that is.

Hi, thanks for getting this page going!

I just wanted to confirm that specific gender diagnosticity studies by definition do not attempt to imply causation; GD studies furthermore only apply to a particular group at a particular time, and are not meant to make broad inferences about whole populations (Lippa, 1990, p. 1053). Perhaps we could make this clear in one or the other of these discussions of validity.

Sure, I will add in the definition of WEIRD :)

What you said about gender diagnosticity studies not implying causation/only applying to a particular group etc. sounds good to me. Should this be added to the external validity section?

sorry for the late reply, the type of validity is external, which reveal that the measure scale and the participants would cause the steady have low generalizabiliy.

study***

What does the correlation of our Gender Diagnosticity scale and the Big-5 scales tell us about the validity of our measure?

Our study: E: -.12, p = .39 N: -.11, p = .47 A: -.42, p < .01 C: -.23, p = .12 O: .06, p = .71

In past studies, Extroversion, Oppenness, and Conscientiousness have all proven to have minimal gender differences. Our study also shows minimal differences for Oppenness and Extraversion; there are slightly higher differences for Conscientiousness, however, but this is still in the expected direction (with females scoring slightly higher). Females scored higher on Neuroticism, as expected, even though the p value is too high to make any reliable inferences from this data. Additionally, the factor which has most reliably been scored higher for females in past research (Agreeableness) not only has the highest correlation of r = -.42, but also has a p < .01, making it both a noteworthy and significant gender difference. The fact that these results match up more or less with previous research (especially the Agreeableness component) helps to raise the validity of our questionnaire. However, I'm not quite sure what type of validity it is - convergent?

• edit: I added this part to the validity section, feel free to let me know or edit if you don't agree with it! I also took out part of the intro that was a reiteration of stuff said a bit later on in the section.

I've just been wondering... is this questionnaire supposed to just be able to differentiate between men and women, or is it supposed to be actually getting at gender-related *personality* differences? Because if it's the latter, I feel like some of the questions tap more into expected gender roles that society has imposed on us, rather than actual personality differences. for example, take likelihood of carrying a purse - this has one of the highest d-values of all the questions. However, do most men refuse to carry purses because of some inherent personality trait, or just because society would judge them if they did, and they're following prescribed gender roles? another question I'm wondering about is "frequency of wearing make-up."

Hi Katerina, the questionnaire is supposed to be the latter, by doing the former. By asking about things like purses, make-up, we are trying to assess an underlying gender-related personality difference. You are quite right that social factors play an extremely large role. While most people might conform to society's gender expectations, there are still some people who violate societal expectations for their gender though. It's these individual differences that are what researchers who have used Gender Diagnosticity scales in research are interested in.

The questionnaire does not address such answers though, such as whether they are just conforming to society's gender expectations or whether that is their true self, all we know is if they prefer doing it or not, but not what makes them do it. This makes the internal validity very low, or doesnt exist at all?

yea that's kind of along the lines of what I was thinking. This would make the internal validity potentially low, wouldn't it? If we don't even know if the various questions are tapping into the same concept...

I think it depends on which questions you're talking about. For example the make-up question if you ask about how much you like wearing make-up that might get closer to one's true self, whereas a question about how often you wear make-up might also be measuring the amount that you conform to society's gendered expectations of you. Perhaps you could take a look at the questions that we ended up with and see whether you think they are measuring an internal sense of self, amount one is adhering to society's expectations, or perhaps a mix of both. I think that would be an excellent thing to discuss here.

I think that, for most of the questions, the ones that are more observable (such as hobbies, having a one-night stand, or wearing make-up) are probably more heavily influenced by society, because it would be much easier for your culture to watch and judge you as to whether you are living up to its standards. Also, like you said, how the questions are phrased would also have a huge impact - asking the *chances* of having sex with a stranger, as opposed to asking the *actual* number of times it has happened, would get largely different responses (with the former being more influenced by internal factors, the latter being more influenced by society). Perhaps asking someone's wishes/preferences is a better method than asking about their actual behaviour in some domains? Like how you feel about wearing make-up versus actually doing it.