Sample Issues and Generalizability

This project could not be generalized to the general population because it was neither an example of random assignment nor random selection although its methodology could be repeated across different populations.

This project included respondents with different ethnicities, religious beliefs, education levels, ages, genders, and personality traits. However, the number of respondents was too low to represent the general population. There were 63 students overall who responded to the survey, and of these 63, 60 reported their genders. Of these 60, 10 were male and 50 were female. The reported age range was between the ages of 18-49. This small sample size was not sufficient to represent the whole population. Perhaps the greatest limitation was that there were 5 times as many females than males who filled out the survey; the relatively low number of males increased the possibility for statistical errors, rendering it low in “statistical power” ("Statistical Power", n.d.). Although there is no ‘magic number’ for statistical integrity, common sense dictates that such a low number of unevenly distributed respondents drastically reduced the likelihood that the results were representative of the population.

Similarly, the samples were not randomly selected. According to the online definition, randomization characterizes a process of selection in which each item of a set has an equal probability of being chosen. This is a key factor in research because it determines whether the results of the research can be generalized to a designated population. The fact that this study was done in a Personality Psychology Summer class of UBC and its participants consisted of only those in this particular class reduced the randomness and threatened the external validity of this research. Rather than having a sample that was representative of the general public, this correlational study only consisted of a part of the target population. Therefore, it may be difficult to generalize the results of this research project since there was no random sampling alongside only a small sample size and the non-response bias.

Additionally, recent research has suggested how social roles and the impact of social role experiences affect self-reported gender. As the subjects in the study were entirely college students, this limited the ability to generalize to people of differing socioeconomic statuses and ages, which highlighted confounding generational differences. As we age, our social work roles define us more presciently, as opposed to college-aged students who tend to define themselves in terms of personal attributes. Barnett & Hyde (2001) argued that self-reported gender traits are influenced by socially determined roles. Underlying this assumption is that family roles are more strongly identifiable to women than men and that work roles are more characteristic of men than women (Kasen,2006). However, the conclusion that work roles are more characteristic of masculine linked traits and that family roles contributes to feminine trait may be outdated , with women gaining more momentum in work roles, and men spending more time at home.

How does our social roles change our self-perceived notion of gender? The Gender Diagnosticity (GD) scale highlighted potential occupations of the respondents. However, it falls short in addressing the nature of the occupation of itself, failing to report the nature of involvement in the occupation or prestige in each position. Also, as university students, it is often difficult to have a full time job concurrently. Interestingly, findings have suggested that employment status and education is more closely associated to the degree to which women associate with either feminine-or masculine-linked traits than the degree to which they associate with their social roles. Additionally, Roberts (1997) suggested that there was a significant rise in a sense of agency in women who reported increased occupational prestige or increased participation in the workforce.

The questionnaire had few questions related to the demographic of people who are parents at present. This oversight could skew results, negatively affecting the ability to generalize.

Moreover, the fact that the study did not use a control group nor manipulated any variable means that we cannot propose that any of the relationships found between the variables were caused by one another. Therefore, we can only propose that the relationships show a correlation between the two variables not causation. As well, the questions, which were structured by the students, may have included errors or misinterpreted by the respondents. This hurts the credibility of the findings, and puts them to question. Additionally, it was easy to pinpoint which questions in the questionnaire were targeted towards the feminine side or the masculine side. For example, a question such as "How many hours do you look at yourself in the mirror?" is clearly targeted towards the feminine side, which could lead some participants to answer either more femininely or masculinely according to how they view themselves or how they wish to be viewed by others, rather than what is actually true. This may indicate a conflict among self-identity, self-presentation and the actual self.

The large variability between demographic differences, such as age, religion and ethnicity also causes limitations. As Lippa & Connelly (1990) argued: "gender related individual differences can be defined only in terms of the behaviours that differentiate men and women in a particular population in a particular culture during a particular historical era". However, our research data showed a wide range of age groups of participants from 18 to 49 years old. An 18 year-old individual, may have different values, may have experienced different cultural groups and may have been exposed to different environments or stressors, compared to a 49 year-old individual. Also culture background among respondents is quite different. According to our statistics, 63% of our participants were Asian, 10% classified themselves as "other" and only 27% were Caucasian. In a study meant to discover the relationship between gender and personality traits in a western cultural environment, the findings of our study are too weak to generalize to a larger population, as over half of the total participants are Asian. Because different cultures often have different customs, inhibition or expression of certain personality traits may develop which would challenge the certainty of the relationship between gender and personality traits.This is especially true when Asian culture is more collectivism and western culture is more individualism.

Additionally, our data showed a range of differences in religious beliefs. The influence between religious beliefs and personality is bidirectional. Individuals with certain characteristics may self-select religion based on their personal beliefs, while certain religious beliefs may influence and shape one's personality. For example, a male who believes in Daoism and a male who believes in Christianity may respond differently to a question regarding aggression due to their particular religions. Similar to the influence of cultural background, this bidirectional effect may increase the uncertainty within our findings.

Lastly, in this questionnaire, we did not have any questions related to sexual orientation. Previous studies found that part of the gay men population demonstrates feminine-orientation than heterosexual men do, while part of the lesbian women population shows masculine-orientation more than heterosexual women do(Dunne, Bailey, Kirk, & Martin, 2000; Haslam, 1997; Lippa, 2000; Pillard, 1991). For instance, when comparing the masculinity of butches (masculine lesbians) and flamers (gay men who are more feminine), it may be found that butches express a higher degree of masculine behaviours and cognition than normal female. If so, the results will affect the reliability of our Gender diagnosticity test. In addition, in one study, researchers have found that "tomboy" as a gender-social identity is associated with increased masculine-typed behaviour in girls and women, and can be a protective identity at the same time (Craig & LaCroix, 2011). Thus, when determining gender diagnosticity, researchers also need to consider the participants’ sexual orientation and experienced gender role in a big social enviroment.

Participant Bias

Since the questionnaire was created and given to university students in an upper-level psychology course, some participants may have already had some prior knowledge of gender-diagnoses that could have influenced their responses. Furthermore, psychology students were encouraged to participate in human subject pool (HSP) surveys to gain credits toward their courses, from which they could have also gotten expectations. When creating those questions, creators might artificially distinguish some thoughts and behaviors based on their existing knowledge or bias of gender difference. When creators change their roles to be participants, they might respond those “structured” questions explicitly. Thus, the self-report questionnaire might include implicit effect of gender stereotype, which may weaken the reliability and validity in the measure.

Also, when they were responding to questions, people may be biased and untruthful regarding sensitive behaviors and private details. Generally, people tend to have the bias of social desirability, which is a tendency to answer items in a way that is socially likeable (Larsen & Buss). By responding with this bias, people may produce a falsely positive impression of themselves by responding with what they believe to be socially accepted in the Western culture. Considering that the survey was to be completed at the respondent's leisure, ie. in the comfort and privacy of their own homes, it is less likely that they felt the need to be biased or untruthful. However, it is always important to be aware that biases may play a role in self-report surveys. Subjects that participated on multiple HSP surveys may have encountered a lot of these questions from prior experiences. Prior experiences may affect how subjects respond to these questions by creating false memory (Martha, 1997) and by increasing self-awareness. An example of self-awareness is when subjects encountered these questions for the first time, they tend to answer based on however they felt was true about them. The more of the same questions were encountered, the more the subjects felt self-aware and the more the answers become untruthful to satisfy the concept to be socially favorable.

Furthermore, there may exist a self-serving attributions among respondents. A self-serving attributions means participants of this study may unconsciously attribute positive outcome to self and negative outcome to other factors (Myers, Spencer & Jordan,2012). Similar to social desirability, on questions related to more positive aspect, participants may rate higher, while on questions more associated with negative aspect, participants may rate lower, based on their 'biased' understandings of self.

Due to the long length of the questionnaire, the participants might demonstrate carelessness when answering the questions; thus, they might not be motivated to answer the questions correctly and carefully. As a common limitation of self-reports, participants might lack accurate self-knowledge. Despite the fact that participants might know themselves best, they might have inaccurate representations of their own personal traits, qualities and desires. These inaccurate responses may not have been due to an intention to seem socially desirable but instead due to a hazy self-conception. For example, someone might not realize their own aggressiveness to the extent that an observer or unbiased physiological test would.

There is a possibility that those who completed the questionnaire have answered with a self-bias or have used response sets: acquiescence, extreme responding and/or social desirability. Use of an infrequency scale to catch invalid responses was applied through the question "I can walk on water". Responding to this item affirmatively suggests that the participant may be careless. But overall, the researchers of this study did not thoroughly screen the data for careless, partially random, or otherwise inattentive responses. As a result, it is highly plausible that the data acquired is low in reliability as well as validity. Any explicit self-report measure includes only those beliefs with which the subject has conscious cognizant awareness and as such detects attitudes that are controllable and intended. Internalized beliefs can also be accurately measured implicitly such that potentially controversial responses are not altered due to heightened awareness.

Experimenter Bias

A particular limitation to the study during the creation of the questionnaire is experimenter bias. Those involved in creating the questionnaire (experimenters) may have influenced the results in that firstly, they knew they were also going to be participants so experimenters may have subconsciously created questions that could easily be applied to themselves culturally (i.e Westernized, Educated, Industrialized, Rich and Democratic). Secondly, experimenters are humans as well and are equally subject to societal norms, views, and beliefs about gender. Therefore, while creating items on the questionnaire, experimenters make their own decisions as to what item should or should not belong and such decisions may be affected by the very topics being studied such as gender, personality, gender roles, societal norms, etc. This is an inevitable limitation to administrating questionnaires. Another interesting issue with this study is that the experimenters who created the questionnaire were also the participants who participated in the study.

Furthermore, though the demographic age of the sample varies from 18-49, most of the questions are geared towards those who have not yet entered the work force or have completed their post-secondary education. Questions orientations such are socialization habits, career choices and dating choices seem to be more geared towards those in the lower ages of the sample. Though these topics may hold some bearing with those who are at more advanced age, the emphasis seems to be, on the surface at least, for those of a younger age.

Method

One of the main limitations of this study was that the only procedure applied was a self-report. Self-reports are easy to distribute to large groups of people and they allow access to a wealth of information about an individual that others may not have access to, but as with all methods of research, the self-report method has its limitations.

As per the abstract, questions on the questionnaire were created by the class participants over a three week period in July 2013; the creators were also the participants. It is worth noting that the results may be affected to some degree by this dual role; perhaps there may be bias for answering one way or another given one’s participation or lack thereof in the creation of the questionnaire. While any effects of bias may be minimal, the possibility does exist. Certainly future administrations to control groups who had no input on the creation of the questionnaire are in order to assess the possible influence of bias. Moreover, no repeats of the questionnaire were attempted with the same group to confirm reliability. Because the questionnaire was only administered to the students of the PSYC305 class, the creators of the questionnaire were essentially the responders as well. This means that the participants were well aware of the main purpose of the study and may have unintentionally provided responses in a certain way to validate the effect of the questions. Furthermore, the subjects were also aware of the kind of questions available prior to answering them, giving subjects time to think over some of the questions and to answer them unrealistically by exaggerating or diluting the actual response.

Content

Questions may have been unclear and therefore misinterpreted by respondents. The questionnaire was created by students of the Psych 305A class, a group of psychology students who had no prior training in creating survey questions and who did so in the manner of creativity, interest, and volunteer. This means that the questionnaire may not have addressed all the purposes of the research project, possibly leading to undesired or unexpected results. Also, respondents may have misinterpreted the questions and responded incorrectly to the content of the questions, hence their answers may not have reflected the aspect of gender differences that was intended to be measured.

The structured format of this questionnaire may have forced respondents to answer in a way which did not represent personal views. Also, the respondents may not have found that the available responses were representative of their desired responses. For example, some questions did not provide a good range of answer choices such as "none of the above" or "not applicable".

Since the questionnaire did not provide an array of options in regards to gender identity, genderqueer, pangender, agender and genderfluid individuals would have to be forced into the gender binary, regardless of whether or not this actually represents their self-identity. In addition, this could potentially impact larger validity and generalizability of the research findings due to the fact that gender identity is directly linked with the sex of the respondent (while it might not be directly linked in reality).

Language

There were specific questions that were directed to participants with sufficient knowledge of North American culture and a high level of English proficiency. These questions might have be ineffectual at eliciting a proper response in participants unfamiliar with North American culture and norms. Similarly, participants with a lower level of English comprehension may not have understood the connotations of certain terms included in the questions.

Some of the questions in the survey contained words and phrases that carry negative connotations. For instance, expressions such as “one night stand,” “mood swings,” and “talking about people behind their backs” hold negative emotional associations for most people. Moreover, the words “cheating” and “aggressive” can also imply negative undertones. The inadvertent use of these words and expressions may have influenced the way participants responded and thus led to biased conclusions.