Difference between revisions of "Sandbox:Vantage Research Working Group"

From UBC Wiki
Jump to: navigation, search
m (Other considerations)
(Potential research methods)
Line 47: Line 47:
*data mining
*data mining
*discourse analysis
*student work
*student work

Revision as of 22:32, 9 March 2014

Vantage College Program Evaluation and Research Framework

What defines success for the college?

We will get input from Anji/James on this.

Possible research dimensions

Here are some dimensions we have identified:

Learning outcomes

  • disciplinary knowledge
    • using established concept inventories to evaluate pre-to-post knowledge and compare to UBC students.
  • English
    • using standardized assessment and analysis of student work to identify level of english (professional and every day).
  • learning competencies
    • using focus groups, interviews, and surveys, to evaluate use of learning strategies.
  • attitudes and motivation
    • using established surveys to evaluate attitudes towards the relevant disciplines
    • evaluate time investment in the different courses
  • Note: we should figure out ways to account for the difficulty of disentangling improvements in language competency from improvements in disciplinary knowledge or academic sophistication

Well being

  • satisfaction
  • belonging
  • happiness
  • rate of at-risk and distress feelings.


  • use
    • survey instructors about their use of different pedagogies
    • conduct classroom observations to evaluate use of a variety of pedagogies
  • impact
  • transition across pedagogies => or across courses?
  • mainstreaming to UBC
    • interview students about their use of Vantage pedagogies in FoA and FoS courses.
  • evaluation

Institutional Issues (UBC & Vantage)

  • registration / attrition
    • evaluate how many Vantage learners continue in UBC on year 2, and their performance in years 2-4.
  • diversity
  • scalability

Potential research methods

  • ethnography
  • tests
  • surveys
  • interviews
  • data mining
  • observations
  • discourse analysis
  • student work
  • journals
  • instructor survey
  • instructor interviews

Possible priorities for Vantage College program evaluation

(in no particular order at this point)

(Arts, including AEP)

  • The ways our curriculum, the way that we have designed it, serves or doesn't serve students as they transition into 2nd year.
  • Evidence of meta-awareness/discursive roles as realized in writing practices. Evaluating the use of specific technologies in the teaching approaches adopted Example of a specific research question for a project already underway elsewhere that we could possibly adapt (thanks for the example Jackie): meta-understandings of writing in the disciplines students take with them from year to year (to answer the question: what kinds of knowledge actually transfers about disciplinary discursive practices given the pedagogical orientation of our course? And, how might this knowledge shift as students become increasingly involved in their chosen majors/disciplines?). To do this, we will ask students in 3 different disciplines from different years in their programs and who have taken ASTU 150 to read an excerpt of scholarly writing (peer reviewed) and comment on it as they read (this is think aloud protocol).
  • Evaluating students’ progress in the specific disciplines compared with their colleagues in ‘UBC proper' Possible specific questions: Using first year GPA as an indicator, how do Vantage students compare with international students in respective departments? A secondary question would be to determine the factors that led to any differences assuming there are differences.
  • 'Analyzing the integration of LLED into the content courses including students’ feedback on this can be improved/changed.
  • Exploring the process and utility of curriculum design in partnership with a large publishing house (ie Pearson)
  • Student satisfaction with the program, and their general mental health and well being—as compared to direct-entry international students.
  • How well our students are doing in their second year non-Vantage courses—as compared to other international students.
  • Evaluation of the practical skill set of Vantage students before and after the Vantage program—as compared to direct-entry international students. Possible specific question: What trends in terms of progress in writing skills and ability to synthesize lecture/reading materials can be identified in the Vantage student cohort? How do these compare to progress in the development of similar skills in non-Vantage students? (example of method: we could compare reflexive learning logs/journals that are used in Vantage courses and 'identical' non-Vantage courses-- ie Jen P uses learning logs and will ask same questions for these in both Vantage comparative politics and non Vantage comparative politics courses).
  • Intercultural understanding before, during, and after their time in Vantage. Possible specific question: In what ways do students feel their time at Vantage has allowed them to understand/gain knowledge of different cultures? What Vantage activities in particular led to this? (example of method: survey or focus group)

(Science, including AEP)

  • What works or doesn't work about the model we have chosen: e.g. tutorials, enriched tutorials, mentorship, research project, the timetable/pacing wrt language acquisition. What do students value?
  • What learning gains can be measured in the first year: e.g. concept inventories. Is learning retention improved? What are the appropriate tools for measuring this?
  • How do we track progress of VC students? How does that compare to direct entry BSc students? In second year? Beyond? What accounts for this difference? What elements of the VC program work very well and could be transferred to BSc.?
  • To what extent do students supplement their learning with materials from their first language? Using peers from same linguistic/cultural background? What are the students' perceptions about their preferred/necessary supports and/or how they learn best?
  • What role does the students' first language play in their gaining understanding of course concepts?
  • Do VC students identify as BSc students, Vantage College, UBC? Do they see themselves as legitimate users of English? What is the best way to measure the language gains during the program (e.g. post program TOEFL, IELTS, other)
  • How do we assess whether VC students are meeting the departmental requirements? How does the language component connect to this? Can this data be used to improve common exams? Interviewing students about interpreting exam questions.

Possible individual research questions

(Arts, including AEP)

  • meta-understanding of disciplinary discourses and the ways this meta-understanding informs their ability to take on the work they do as students/novice participants in scholarly research communities.
  • meta-awareness/discursive roles as realized in scholarly writing practices
  • I have designed my assessment to be spread widely over the term rather than the traditional midterm-paper-final exam in order to be fit the scaffolded/constructivist approach to teaching; hence I would be interested in developing a way of analyzing students performances and opinions on this method of assessment compared to traditional approaches - Evaluate how regular assessment and review in content courses impact on students overall performance? Data will be gathered from Vantage college students and from international students taking Geog 121/122 in the Department of Geography.
  • What are possible strategies and approaches needed in order to adapt current first year Geography courses for an diverse international student population? This is the current paper I am working on with Alfredo where we are applying a CLIL approach combined with innovative methods of delivery. This will tie in to my previous question in a longitudinal study where students performance is evaluated based on the course adaptations.
  • What benefits and support do students perceive from the use of Learning Catalytics in their content courses? This study would be done through the use of a variety of methods including a survey, focus groups and interviews. It could be set up with students so that it is conducted at 3 different stages throughout the term in order to analyze change of perspective with the use of the tool.
  • Mental health and well being amongst international students.
  • Intercultural issues related to mental health discourse.
  • Evaluation of SoTL-based teaching method effectiveness in international student populations.
  • Evaluation of flexible learning and flipped classroom methodologies in international student populations.
  • Innovations in simulations in the classroom—critiques and best practice
  • The impact of service learning in relation to goal/value of global citizenship
  • Student health and well beings and impacts on success at university

(Science, including AEP)

  • Using Chinese version of FCI to gauge concept understanding in first language vs in English.
  • What supports/PD/training/support staff do the teaching staff and instructors require to transition to the Vantage context? What is/isn't working? Should co-teaching be implemented for new faculty? What strategies are required in order to successfully adapt pedagogy? What are the minimum PD requirements before joining the VC program?
  • Looking at all the teaching roles and finding out what the perceptions are in terms of support required, training, etc.
  • What is the students' well being beyond VC, e.g. second year stress levels vs first year? Is this connected to learning networks? How can we/are we engaging Vantage alumni?
  • When using pre-reading resources, what is the impact on learning outcomes of using first language or English resources in group work. How can we best use group work in the VC context (e.g. self-selecting, randomized, teams)?

Other considerations

  • How will dissemination occur?
  • How will attribution be handled for collaborative research?
  • Will there be a common Vantage College research dissemination location? What will it contain?
  • How will this be resourced (financially and in terms of human resources)?
  • Ethics procedures for using student work/writing as data?
  • Strategies/policies for making sure our students aren't 'over-researched'?
  • Are we perhaps also going about setting our priorities in a bit of a problematic way: Is it possible that we need to have a preliminary literature review done, or at least an annotated bibliography to help us see what information/research already exists, identify gaps in the literature and think about what value added research we can do? We don't want to duplicate what is already out there (some research will just be there to serve particular/specific for Vantage but we should still be thinking about targeting our research to fill real gaps in the literature).