Course:COGS200/2017W1/Group22

The Effect of an ITS on Language Acquisition

Introduction

An Intelligent Tutoring System, or ITS, is a system designed to contain "computer programs that model learners’ psychological states to provide individualized instruction" (Ma, p.901). ITS was created in order to "help learners acquire domain-specific, cognitive and metacognitive knowledge" in diverse areas such as physics, languages, mathematics, medicine and so on (Ma, p.901). ITS have been implemented in both educational and professional settings, though not limited to the two. A typical ITS is built to provide individualized, one-on-one feedback and tutoring.

The conceptual components of an ITS, according to Intelligent Learning Systems and Learning Outcomes: A Meta-Analysis, are as follows:

1. "An interface that communicates with the learner by presenting and receiving information. Often constrained to the subject domain (e.g., algebra), the interface determines the moves the learner can make in solving problems, seeking information or responding to questions.
2. A domain model that represents the knowledge the student is intended to learn. The model is a set of logical propositions, production rules, natural language statements, or any suitable knowledge representation format.
3. A student model that represents relevant aspects of the student’s knowledge determined by the student’s responses to questions or other interactions with the interface. Although the student model may be a subset or “overlay” of the domain model, in some ITS the student model represents common misconceptions or other “bugs” in the student’s knowledge.
4. A tutor model that represents instructional strategies such as offering a hint when the student is unable to generate a correct response or assigning a problem that requires knowledge only slightly beyond the current student model (Ma, et al. 3). "

Hypothesis

Our project is based on ITS (Intelligent Tutoring Systems) and its role on educational development and language acquisition. We propose an experiment involving two groups of students. The control group will be the group of students being taught by a tutor, while the experimental group will be students in the same age range being taught the same subject by an intelligent tutoring system. We hypothesize that the group of participants who are taught a language by an ITS will have a better understanding of the language (both verbally and written) than the other group of participants who are taught by a teacher of a language (in this case French).

We have found plenty of relevant background research articles to support our hypothesis which are mostly based on Bayesian Networks which help to manage diverse difficulties and struggles with "student-modelling system" in ITS (Conati, p.371).

Relevance

Given the rapid advancement of modern technology, we believe our project carries significant importance in the age of developing artificial intelligence. As more and more technological inventions are integrated into everyday life, we believe this project is able to contribute to the future of learning. We can expect to see more technology-based learning in educational settings in the near future; reasons for this range from student convenience to simply changing times. In effect, we believe it would be useful to collect data on how students learn when using an ITS. We specifically chose language acquisition so as to be able to contrast it with existing research conducted on other, more science-oriented subjects, namely physics and math.

It also plays a role in the educational system as ITS can be the essential part of student's life all over the world in the future. ITS differs from a standard system of learning because the software can replace the human tutor, the system itself is the tutor for the learners, it can provide with correct answers and hints to get the right answer. The system not just teaches a student, it also learns while teaching. With every new student ITS is able to adapt to every student, so that each student is able to give some new information to ITS for further prevention of the same mistakes which the system can make.

Background Research

There has been a lot of research done in the field of Intelligent Tutoring Systems. The research papers we found support our hypothesis that the students taught by ITS will perform better than the students taught in classrooms. The American Psychological Association also supports the fact that ITSs that often presented interfaces with which students interacted throughout the learning activity taught the subject more effectively. By tracking individual student's move at each step, an intelligent tutoring system can build a more detailed student model and also provide hints and feedback at the step-level, not only upon completion of an activity. To further this support they refer to the studies done by Wenting Ma, Adesobe, Nespit and Liu. In this study, they mention the advantage of ITS over traditional classroom instruction. We also referred to work done by Silvia Schiaffino, Patricio Garcia, Analia Amandi on eTeacher - an ITS system - providing personalized assistance to students. The agent has been successfully evaluated with real students and the results obtained were promising. We will be using these references to add assisting and interactive features to our ITS model. We will be creating individual profiles for student and they will be tracked & updated all the time to provide feedback, help, hints at each step. As regards to Bayesian networks, we will refer to work done by Andes (Gertner & VanLehn, 2000) and SE-Coach (Conati, Gertner, & VanLehn, 2002). They use this technique to model students’ knowledge in physics and the precision of Bayesian model at detecting students’ learning styles was evaluated in their experiments. The results can be found in "eTeacher: Providing personalized assistance to e-learning students" (2007).

Methods

Approach

In order to test our hypothesis, our proposed experiment tests whether or not an ITS is indeed a more effective method of teaching a language to a speaker than a teacher of a language. We will recruit monolingual English speakers who are around the same age and will divide them into two groups: one group will be learning French from a regular teacher and the other will learn from an intelligent tutoring system which focuses on teaching foreign language such as French. Over the course of a twelve-week period, which is equal to a semester at the university, students will learn how to use the basics of French grammar, composition, reading and oral practice as well. It will be the Intermediate level of French same as the course FREN111 taught at UBC. The regular teacher is going to give students a standard textbook with various exercises on grammar rules, the teacher will show how to pronounce words and sentences without accent, there also will be labs where students have to do tests on material covered in class. As for the experimental group, the students taught by ITS are going to learn through online by the system itself.

Our group proposes that ITS for learning a foreign language can be created on the basis of such softwares as Rosetta Stone which is a "Computer Assisted Language Learning (CALL)" for students (Ikonta, p.350). According to the article called "The Impact of Rosetta Stone (CALL) Software on ESL Students' Proficiency in English Language" written by Nonye R. Ikonta it is stated that CALL "is a growing sector" which has "high potentials" to make learning of languages much easier (p.350). The Rosetta Stone contains diverse activities, so that students can practice in listening, reading, writing and other skills for learning a foreign language. With further development such CALL softwares can become more like Intelligent Systems which can provide not only with relevant exercises but also with an actual tutoring. Intelligent tutoring is an existing system for many subject matters or domains such as physics or maths. However, all of ITS rely on the same software base and on similar algorithms, that is why we would like to refer to one specific system for better understanding of ITS itself, this system is described in the article of Christina Conati called "Using Bayesian Networks to Manage Uncertainty in Student Modelling".

This system is called Andes and it is "an intelligent tutoring system designed to help students to learn Newtonian mechanics on university-level" (p.372). Andes not only provides "tailored support during problem-solving" but also "helps students study examples effectively" (p.372). One of the main advantages of Andes is that this ITS "provides a comprehensible solution to the assignment of credit problem for both knowledge tracing and plan recognition" and Andes does not reduce "student initiative" (p.372). By using "Bayesian networks" in ITS, Andes is able to workout solutions to both problems at the same time, so that "assessment" influences "plan recognition", and vice versa; that is why "these mutual influences are mathematically sound" (p.372). Another advantage of Andes is that it also does "predictions of student actions" which helps to gain better results because some subjects require several various interpretations of one solution and it is more reliable to have several of them in the software (p.373).

The article states that the system (ITS) should be able to do “assessment” before the start of teaching process because the system needs to know what exactly the student does know (Conati, par.1). The ITS also needs to get the “degree of mastery of domain knowledge” of the student which leads us to “knowledge tracing” technique used by the system (Conati, par.1). Another important aspect of any ITS is the “plan recognition”, technique to get some knowledge on what the student wants to learn. So, the ITS should do both of them in order to provide excellent learning process for our experimental group. There are two existing "technological types" of computer tutoring: the first one is based on providing "hints on answers" and feedback, this system is usually called as Computer Aided-Instruction (CAI); the second one is "characterized by giving students an electronic form, natural language dialogue, simulated instrument panel, or other user interface that allows them to enter the steps required for solving the problem" which is usually called as ITS and is used in our experimental group (VanLehn, p.198).

At the end of the 12-week period the students of both groups get assessed in a written and oral form. We are going to collect the results and make our own evidence on ITS depending on the results we will get.

Participants

Our experiment will consist of a group of 60 undergraduate students. The participants will be selected randomly from the university population and will have to meet the requirements of having a minimal background in French, which we are defining as at least having taken French in secondary school. Any fully fluent French speakers shall not be allowed to participate in this study, as people who are already capable of speaking French fluently would not be beneficial in terms of determining if the proposed ITS for French is an effective method for teaching a new language to an individual. In addition, to account for a possible third-party variable, this study will be exclusive to students who do not suffer from any form of test anxiety. All participants who participate in this study will receive extra credit, an extra two percent to their overall grade in the course (ie. FREN 101) at the end of the term.

Materials

Prior to participating in the experiment, each participant shall receive an informed consent form of the possible risks and benefits of this study, as well as a small summary of what the goals of our research are. All participants will also be provided with the necessary materials such as books and other resources in order to complete assignments and homework which shall be distributed over the course of the twelve week period. We shall also require funding for our ITS, which shall function as a replacement teacher, tasked with teaching our experimental group aspects of the written and verbal components of the French language. Also, we shall require a teacher of the French language who is willing to dedicate themselves to teaching a group of participants in our control group over the course of a twelve week period. Lastly, we shall need access to two classrooms in order to provide an optimal learning environment for both groups of participants, over the period of the twelve week study.

Experimental Design

Procedure

We shall begin our experiment by first recruiting 60 undergraduate students from a university population. In order to be eligible to participate in our experiment, participants must meet at least two requirements. The first being that a participant must have at least a minimal background in French, for our purposes we shall define minimal background as having taken French in secondary school. Secondly, participants cannot be fully fluent native speakers of the French language, as there would be no point in seeing if our proposed ITS is capable of teaching a language to an already native speaker of the French language. We shall distribute a proficiency test to possible participants in order to gauge their knowledge of the French language, which will include both verbal and written aspects of the language. We shall then proceed to provide our chosen participants with an informed consent form outlining the possible risks and benefits of this study, the duration of the study, and a brief summary of what we the researchers are going to be looking at. We plan to run our experiment over the course of 12 weeks and within that time the participants shall be split evenly into two different groups. The first group of 30 participants shall serve as our control group and will be instructed on the grammar and spoken aspects of the French language by a qualified French instructor in a classroom setting. The second group of 30 participants shall serve as our experimental group and will also be taught the exact same curriculum as the first group, except that they will all receive instruction from an ITS in a classroom setting as well. Both groups will attend class regularly, at least 3 times a week, for a period of an hour each day and will be taught various aspects of the grammar and verbal aspects of the French language. While receiving instruction, participants will be allowed to ask questions and will participate in a variety of activities in class in order to enhance their learning. As well, each group will be assigned homework and will be tested regularly on the material they are learning in order to see how over the course of the twelve weeks whether or not there is a noticeable improvement in the overall learning of each individual participant. At the end of the twelve week period, both groups will be assessed by means of a written and verbal exam in order to evaluate the effectiveness of both teaching methods. Upon completion of these assessments, both groups of participants will be debriefed by all of the researchers about the purpose of our study and then will be dismissed. The findings of our study will be calculated and then we shall proceed to run two more trials with a new group of participants, replicating the original experiment step by step. At the very end of the three trials, we shall observe the data that has been collected and see if there any noticeable differences in the test scores of both groups. Upon sufficient interpretation of our results, we shall report our results to our fellow faculty members, making sure to respect the privacy and confidentiality of our participants.

Discussion

Interdisciplinary Nature

The COGS fields involved in this project are computer science, psychology, and linguistics. By integrating educational psychology and artificial intelligence, we are able to gain a foresight of the extent to which intelligent systems contribute to the future of learning. The linguistics aspect enables us to map how a specific demographic picks up on foreign language if taught by an intelligent system, as compared to existing studies involving different learning methods.

Second Language Acquisition

Though there is still current debate about how exactly language is learned, second language acquisition, especially in adults, is believed to be a much lengthier process as compared to picking up on native language. The difficulties are mostly with phonemes. The main reason behind this comes with the theory of critical periods: "The plasticity of the brain allows children to acquire the language well before puberty, but after puberty, with the maturation of the brain as well as the formation of the partial side, adults may lose the physiological advantage in the second language acquisition, and therefore it is difficult to reach the level of the mother tongue (Den et al.,4)." However, as per Semantic Theory, for the second-language learner, the acquisition of meaning is arguably the most important task. That involves Lexical, Structural, Semantic, pragmatic and many other types of meanings. As meaning is heart of a language, all the different meanings contribute to the acquisition of meaning resulting in the integrated second language possession. All of these factors will be taken into account when creating the assessment to extrapolate the necessary data.

Language is best learned with verbal and auditory simulation. For this reason, it might be said that an ITS is not the best option as a tutor when attempting to acquire a second language. However, due to the collective result of studies showing ITS taught groups as having outperformed the control groups, we believe that in this study as well, the experimental group will outperform the control group.

Educational Psychology

Collaborative Learning

Recent research in educational psychology has shown that students learn better when in a more collaborative setting. According to one particular study on collaborative learning conducted by Zhining Qin, David Johnson, and Roger Johnson, it was found that teams performed better than individuals across all ages and types. (Barron, Darling-Hammond 1). In addition, "cooperative group work benefits students in social and behavioral areas as well, including improvement in student self-concept, social interaction, time on task, and positive feelings toward peers. Researchers say these social and self-concept measures were related to academic outcomes and that low-income students, urban students, and minority students benefited even more from cooperative group work, a finding repeated over several decades (Barron, Darling-Hammond,1)."

It is a result of these studies that our proposed experiment specifically includes collaborative learning. This differs from the traditional style of teaching, where students are assessed largely through individual standardized tests. Although our project does involve regular testing in monthly intervals so as to examine progress, students are to be taught in groups in this study. Doing so would allow the students to optimize higher-level thinking as well as promote communication and leadership skills.

In addition, we believe collaborative learning would help account for the lack of face-to-face interaction in learning through an ITS as opposed to a traditional tutor. Students in the experimental group would not have the same person-to-person connection with their tutor that students in the control group would have, for the simple reason that an ITS is not a person. So as to bridge the gap, learning in small-scale groups would allow students to promote face-to-face interaction with their peers, and further develop social skills.

Anticipated Results

We predict that the groups of students tutored by an ITS will produce higher test results as compared to the control group.

There are many controversies around this topic of difference between regular tutoring and Intelligent Tutoring System. For better prediction of our results we can refer to the evidence found in several peer-reviewed articles. According to VanLehn in his article called "The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems" regular tutors "do not seem to infer an assessment of their tutee" which can definitely involve "misconceptions, bugs, or false beliefs" (p.199). Another concern with human tutors is that "their assessment of the student’s mastery of correct knowledge" is used "to regulate how fast they move through the curriculum script" which leads to probable "lack a deep, misconception-based assessment of the student" (VanLehn, p.199). Most computer tutors "use more individualized methods for selecting tasks" rather than human tutors (p.199). There is also one more hypothesis which states "that human tutoring allows mixed initiative dialogues" and "the student can ask questions or change the topic" which is impossible for ITS (p.199). However, "analyses of human tutorial dialogues have found...[that] students take the initiative [to ask]" more than in class but "the frequency is still low" (p.199). Furthermore, it is a common sense that "human tutors usually have much broader and deeper knowledge of the subject matter (domain) than computer tutors" as humans can discuss on related topics with their students which is impossible for the system as it can only follow the algorithm (p.199). Most computer tutors do know about the domain but they cannot give you some specific insight into one exact question. However, it is stated in the article that such conversations between tutors and tutees can seldom occur as students are not typically tend to go further with questions (p.200). It can be concluded that ITS have several general advantages over the standard tutoring system with regular human teachers.

There is also evidence from the experiment made for testing Andes which included two groups of students (Conati, p.402). The control group included 162 people while the experimental had 173 students who were taught by Andes (p.402). At the end all participants took a midterm test to assess the difference between knowledge gained by regular teacher and knowledge gained by ITS (p.402). The participants "who used Andes, performed about a 1/3 of a letter grade on average than students who did not use Andes", and these results were "encouraging" for evidence that with ITS average performance of students is better (p.402).

Given that this study is to be carried out in a higher education setting and involving a first-year level course, the subject pool is anticipated to consist of students in the 18-20 age range. However, it is entirely conceivable that students falling outside of this range pass the screening for the study. In particular, if any such students are significantly older than the average student, they would have a more difficult time with second language acquisition (Deng, Zou,3). The implications behind this is that he or she is likely to produce lower assessment scores as compared to their peers, which would then affect the data collected. However, we believe this to make a negligible difference, considering the large majority of undergrad students fall under the expected age range.

Evaluation

The initial phase is of 12 weeks but a major limitation of this study is the sample sizes. However, we will carry out more experiments with more students to validate the initial findings. The more students and more groups we evaluate, we will get better understanding of outcomes.

We plan on carrying out at minimum three trials of the experiment. Overall, this should take roughly 36 weeks, considering that each trial lasts the duration of a twelve week semester. Each trial is to consist of a new batch of students; any student retaking the course is not eligible to participate in the study. Throughout the experiment, all groups of students will be tested in regular intervals on what they have learned so far. The 'test' here would not actually contribute to the students' grades; it is more so to track their progress at the end of each month. The legitimate evaluation will take place at the conclusion of the last trial. Both groups will also be administered the exact same test to assess their understanding of the material they have just been taught. The test will include both an oral portion as well as a written portion. Afterwards, the results will be calculated and compare to see if there are any substantial differences between the two groups. Although standardized testing has received its' fair share of criticism in terms of truly testing students' understanding, we believe it is effective enough in its' purpose, especially in such a setting where students will have had years of experience in standardized testing.

At the end of the study, the assessment tests will be gathered, graphed, and analyzed to determine whether or not the ITS group really did produce higher test results.

Finance

The high developmental cost of an ITS provides financial setbacks to this project. Combined with the fact that research surrounding ITS is also costly, in a practical setting this proposal is not likely to come into effect, even despite its viable contributions. The main issue that this raises is replicability of the study. Although the procedure and general setting of this study would be simple enough to generate, building or even acquiring an ITS would prove to be very costly.

Conclusion

Over the span of this project, our group learned a lot about the multidisciplinary fields of Cognitive Systems

Insights

The supporting research suggest that the more interactive the ITS is, the better it is for student learning. Also, assisting students at each step- level, giving them encouraging feedback and hints on how they can improve, will help a lot. This will be done through guiding students at an individual level and for that we thought of using the student profile idea from previous experiments.

ITS can be used for students with learning disability and/or who need personalized way of learning. Learning at the individual level will benefit many students who have difficulty getting that attention in classroom style of learning.

This ITS may prove to be helpful if implemented as a TA as well. It will help students learning some concepts at their own pace and might address some questions/concerns that's not been answered in the classroom. It will be available online so that they can access it at any time they need it. As this ITS will have individual student profiles and their track record, it will be easier for ITS to pay detailed attention to student's request/queries. As noted in previous studies, "ITS was also associated with greater achievement regardless of whether it was used as the principal means of instruction, as an integral part of classroom instruction, to support in-class activities such as laboratory exercises, for supplementary after-class instruction, or as part of assigned homework (Ma,12)." Even teachers can get some benefits by ITS as they can track and evaluate a student's progress through their profiles.

Limitations

An ITS is considered to have two main limitations when compared to a typical tutor. The first thing is in relation to dialogue; verbal communication is not at its optimal. An ITS, in gerneral, is not able to communicate with someone in the same manner as humans are with each other.

The second issue, which also ties into communication, is emotion. For example, in expressing approval of a students' work, an ITS response would be quite mechanical. According to studies, social factors play an important role in learning.

These issues are addressed in the Collaborative Learning section of the proposal. Although a group-oriented learning environment would not account for all the shortcomings of an ITS, it would still fill in some of the gaps that would otherwise be hard to bridge.

References

Anderson, John R., et al. (1995). Cognitive Tutors: Lessons Learned. The Journal of the Learning Sciences, 4.2, 167-207

Barron, B., Darling-Hammond, L. (2008) “Powerful Learning: Studies Show Deep Understanding Derives from Collaborative Methods.” Edutopia, 8 Oct. 2008, www.edutopia.org/inquiry-project-learning-research.

Conati, C., Gertner, A., Vanlehn, K. (2002). Using Bayesian Networks to Manage Uncertainty in Student Modeling. ProQuest, 12

Deng, F., Zou, Q. (2016). A Study on Whether the Adults' Second Language Acquisition Is Easy or Not-From the Perspective of Children's Native Language Acquisition. Theory and Practice in Language Studies, 6.4, 776

Ikonta, Nanye R., Ugonna, Nwannediuto C. (2015). The Impact of Rosetta Stone (CALL) Software on ESL Students' Proficiency in English Language. ProQuest, 8, 349-361

Naser, S. (2009). Evaluating The Effectiveness of the CPP-Tutor an Intelligent Tutoring System for Students Learning to Program in C++. Journal of Applied Sciences Research, 5.1, 109-114

Ma, Wenting, et al. (2014). Intelligent Tutoring Systems and Learning Outcomes: A Meta-Analysis. Journal of Educational Psychology, 106, 901-918

Schiaffino S., Garcia P., Amandi A. (2008). eTeacher: Providing personalized assistance to e-learning students. Science Direct, 51, 1744-1754

VanLehn, K. (2011). The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems. Taylor & Francis Online, 46, 191-221