Course:COGS200/2017W1/Group9Project

From UBC Wiki

Cem Ozcan, Student Number 43987163

Ethan Landon, Student Number 52608163

Marlowe Dirkson, Student Number 64199136

Nicholas Chin, Student Number 54101167

Free logo maker

This project proposes to build upon existing technology to design a software that will facilitate a more accurate diagnosis of people suffering from depression. In a world where mental disorders are becoming increasingly prevalent, current apps and clinical methods suggest that it remains a tricky task to determine whether people are depressed due to the different severities and types of depression that exist, the overlapping symptoms of mental illnesses, the uniqueness of the individual, etc.[1] Therefore, whether conducted by humans or computers, improvements could be made in the area of monitoring people on a more regular and inconspicuous basis. In this proposal, we aim to combine research in the fields of psychology, linguistics and computer science to catalyze production of a back-end operating software that will longitudinally monitor users’ physiological, psychological and linguistic characteristics/irregularities and upload the collected information to a mass database. The software will also provide a front-end verbal human-computer interactive medium for users to communicate with, which together with the longitudinal monitoring, will be able to predict the onset of, or detect key symptoms of depression. This software does not attempt to emulate or replace a clinical psychologist. It simply serves to provide quantitative, reliable and accurate information in additon to the qualitative information self-reported by patients as an improvement to current methods.

Introduction

Because current methods rely heavily on the interaction between clinical psychologists and their patients, perhaps one of the largest limitations to the former is the use of dominantly qualitative methods that depends largely on subjects' report and clinical opinion, which risks a range of subjective biases. Such an approach may prove helpful at times, but is susceptible to untruthful or exaggerated reporting by the patient and aggravated by the lack of quantitative data. By employing the fields of psychology, linguistics and computer science within the dimension of cognitive systems, this project proposes to develop the software Pix.P to improve the accuracy of depression diagnosis through the longitudinal monitoring/recording of users visual (facial), auditory, tactile, and physiological features and activity. A mass corpus of information will also be recorded to aggregate the signals of certain symptoms that will enable more accurate future diagnoses. As such, this program is expected to rely heavily on quantitative data, but will also preserve the qualitative aspect of methodology employed by leading clinical psychologists to facilitate an accurate diagnosis or predict the onset of depression. It must be noted that Pix P. is not going to diagnose depression explicitly, but merely improve the accuracy of diagnosis through the collection of quantitative and qualitative data across an extended period of time to aid medical practitioners, researchers and psychologists with making a diagnosis of depression for individual users/patients and further improve the software.

Methods

Back-end Function

Physiological

There are multiple factors on how physiological factors affect depression. The main five factors are; Blood Pressure, Heart Rate, Sleep-wake Schedule, Energy Level, and Mental Stress.

1. Blood Pressure: In the current research, there has been a correlation between depression and hypertension.[2] The research explains how screening for depression could be made cost-effective by adding Blood Pressure as a viable element.


2. Heart rate: Scientists know that depression increases the risk of dying.[3] However there is a deficit in the research gap on how depression increases this risk. Carney et al. did a research on how low heart rate variability in depressed patients contributes to high mortality after heart attack.The researchers concluded that "Depressed patients were nearly three times as likely to die during a low heart rate variability."


3. Sleep-wake schedule An inability to sleep (or insomnia) is one of the key signs of clinical depression. Another sign of clinical depression is sleeping too much or oversleeping.[4]


4. Energy Level: Exercise and depression has been studied for many years. A new research in 2013 states that exerise helps relieve anxiety thus leading to relief in depression.[5]


5. Mental Stress Stress is a crucial indicator in someones anxiety levels. It is mainly difficult to measure stress with regards to many elements, such as lifestyle, occupation, attitude and diet. There is a stress questionnaire named "Stress 360" created by American Institute of Stress. By this questionnaire it is possible to have a unit for outcome.

The main importance of these elements is the accessibility of data from these categories. All of these elements can be measured by every day smart technology and thus makes the data collection quite easy. Except blood pressure, all of the above can be measured by a smartwatch and the data can be stored in large amounts in a remote server. It is important to note that there is a consequence that arises with the availability of the data. Most of these conditions could also rely on or be effected by other illnesses/diseases. The other important factor is that physiological factors vary by age group so all of the recorded measurements will be correlated according to the participant's age. 

This shows us that again, Pix P. is not a diagnostic tool. It is a assistant to the diagnosis. The most important concept to comprehend is that correlation is not causation.

Psychological (Facial Recognition)

Through the application of modern face mapping technology Pix P. will be able to scan for changes in a person’s facial features over a long period of time. Currently Apple’s FaceID technology, built into the Iphone X, uses 30,000 infrared dots to create a 3D depth map of a person’s face[6]. These facial maps can be used with the facial action coding system (FACS), which detects and codes the minute facial changes of individual facial muscles, to analyze the quantitative differences in a person’s face and the likelihood that they are depressed.


This process is applicable to most people due to the neurocultural theory which postulates that there is a universality to emotional expression, i.e., human expressions are developed worldwide to convey the same emotions. Paul Ekman theorized that there exists six basic patterns of expression. These are happiness, anger, disgust, fear, sadness, and surprise. Combining these six expressions with FACS, he linked certain facial muscles to being strong indicators for certain emotional states. The face can be split into two separate regions, upper and lower, and the disgusted and happy faces show significantly higher reliance on the mouth region, while the angry, fearful and sad faces show stronger reliance on the eyes, for at least one of the face models[7].


A study on the relationship between change over time in the severity of depression symptoms and facial expression found that there exists a correlation between certain muscle, or action units in FACS, and the severity of the depression. Through longitudinal analysis of patients undergoing treatment they could find that many action units, specifically around the mouth and eye regions, had significant changes in activity as the severity level of the depression changed.


Action units

While there was a high severity of depression they found significantly lower AU 12 activity, significantly higher AU 14 activity, significantly more AU 14 activity while smiling, less overall AU 15 activity, and more AU 10 activity while smiling[8]. This combined with Paul Ekman’s charting of emotions and the linked action units shows an effect on which emotions seem to be less prevalent strictly from a quantitative analysis.


Through a backend process this data collected over a period will be turned into a group of factors which contribute to the overall score which will be shown to users indicating the likelihood that they are depressed. The individual factors will also be accessible by medical professionals who can use the specialized data to track the progression of symptoms along with treatment.

Psychological (Aero-tactile integration)

Just as facial and visual information provide information to a person's emotional states, introducing the tactile modality can do so as well. According to researchers Thomas F. Quatieri and Nicolas Malyska from MIT, their research indicates aspiration increases with overall depression severity. This is largely due to the motor retardation in many depressed subjects, which reduces laryngeal muscle tension hence resulting in a "more open, turbulant glottis."[9] Drawing on the observation that many languages uses 'aspiration' to convey basic lexical contrasts, Pix P. aims to measure the frequency of which users 'aspirate' to help better predict and detect symptoms of depression.


Psychological (Auditory)

Vowel Sounds

Characteristic signs of depression in linguistic patterns, when identified, can help doctors diagnose depression more accurately. For example, certain vowel sounds are associated with depressive symptoms; machine learning algorithms can find them. To that end, University of Southern California researchers developed an AI tool known as SimSensei which detects signs of depression and post-traumatic stress disorder through the analysis of frequencies of vowel sounds. The researchers gathered large data sets from depressed and non-depressed subjects, which clustered based on average values. With their findings, they were able to compare the speech patterns of depressed people against those of “normal” people.[10] A finding that showed up as characteristic of depressives and PTSD sufferers was a reduced vowel space.[11] This characteristic shows up clearly when measured, but would be impossible for a physician to perceive. AI allows for the diagnosis, quantification and monitoring of signs and symptoms of depression.

Shimmer and jitter

Shimmer and jitter are measures of acoustic variation; shimmer is a measure of amplitude; jitter of frequency. Both are the period-to-period variation in glottal pulse when speaking. According to researchers Quatieri and Malyska, shimmer increases with increasing overall depression severity, as well as the increase of Psycho-motor Retardation as a sub-symptom.[12] On the other hand, they report no correlation between severity and jittering; due to difficulties in measuring jittering in the presence of strong aspiration, a typical characteristic exhibited amongst depressed subjects.[13] Adopting their method to measure this quantity, Pix P. will approximate glottal pulse times using Praat, which, is one of the many speech-signal processing tools that can be used.

Pitch and energy

Quatieri and Malyska's research indicate that pitch variance decreases with increasing depression severity. In agreement with previous studies, they found that the depressed voice is more 'monotonous'; pitch variation decreases when a person becomes depressed so their speech sounds flatter.[14] Volume also drops overall; the depressed person does not “energize” their delivery with louder volume even when they need to be heard, don’t convey emphasis when they need to emphasize, and don’t convey urgency when there is urgency. Pitch periods and volume information will also be processed using Praat as the software is able to differentiate between noise when aspirating and sounds projected by the user.

Pauses

Speech pauses are longer and more frequent than normal so there’s more starts and stops, making delivery sounding laborious (can be described as herky-jerky). Mundt, Vogel, Feltner, and Lenderking found that there was a strong correlation between the number of pauses in language and the severity of depression. As the severity of depression increased the number of pauses, total time spent paused, and the amount of pause variability all increased significantly[15]. This can be recorded through simple voice recording software which will identify and tally the amount of pauses in daily speech without actually recording any content.

Tension

Too much tension in the vocal cords gives a tense-sounding voice; too much relaxation gives a breathy voice. Either of these tension extremes indicates a problem as both have been linked to depression and suicide risk. Depressed people may also have a slight slurring of speech - a result of tongues and breath becoming uncoordinated. These could be recorded through a simple audio recording using the mobile phone's microphone which would then have the vocal qualities compared to average samples to test whether there is a difference.

Semantics (Linguistics)

A Russian study of 201 subjects (124 clinically diagnosed with mild depression and 77 "healthy control" participants) found that mildly depressed subjects used simpler sentences, (i.e., fewer complex, compound or compound-complex sentences), more words, more narration than reasoning, more ellipses, more repetitions, more self-centered speech, and more verbs in the past tense than did mentally healthy people.[16] They also fuzzy their meaning by using comparatively a lot of pronouns, especially personal pronouns.[17] They seldom speak altruistically or with self-realization. These “abnormal” language characteristics can help AI diagnose mild depression and distinguish between it and "normal sadness". Participants with normal sadness (i.e., situational sadness, such as grieving the loss of a loved one) did many of the same things, but used more present tense and said more about altruism, self-realization, and social status compared to the clinically depressed subjects.[18]

General practitioners determining whether a patient is depressed or experiencing a normal sadness response must pay careful attention to keywords and semantics. It is very hard for doctors to spot the difference between mild depression and sadness. The validity and reliability of depression diagnosis is questionable because of the subjective experience of a doctor in consultation with a patient. This is known both inside and outside the psychiatric community. It will take a more objective method like longitudinal data collection by AI to fortify validity and reliability. Therefore, such prospects indicate promising results by using linguistic pattern analysis in Pix P over time.

Front-end Function

Human-computer interaction

In the front-end paradigm, users of Pix P. will be able to have a verbal conversation and screen-based interaction with a machine-learning Chatbot in order to preserve the qualitative aspect of information-gathering used by current methods before diagnosing a user. This includes the bot being programmed with certain qualitative capacities:

Standardized Questionaries

Although their tools may vary, clinical psychologists rely on a distinct number of standardized screening questions to determine depressive symptoms. Typical questions range from motivation, sleep patterns, suicidal thoughts, energy levels, mood, anxiety etc.[19] Two of the most common standardized screening tools include the Hamilton Depression Rating Scale (HAMD), the Montgomery-Asberg Depression Rating Scale (MADRS) and nine-item Patient Health Questionnaire (PHQ-9).[20] In accordance with its procedures, upon completion of the questionnaires, the machine will answer accordingly to the answers recorded to the data corpus. It will then identify patient characteristics and compare strategies for treatment accordingly. Ideally, the Chatbot will only require users to answer these questionnaires from time to time to track users' progress, or if the back-end system detects any form of abnormal visual, auditory, linguistic or psychological activity.

QPR suicide prevention

The Question, Persuade, and Refer Institute provides proven suicide prevention training. When conversing with the Chatbot, the bot will be programmed with the QPR protocol to recognize the warning signs of a suicide crisis and how to question, persuade, and refer someone to help.[21] In order to perform the QPR procedure, the bot will ask a series of questions to engage the user, and quickly transfer the line to a trained medical specialist. Although signs may not be obvious, drawing upon previously collected external data (based on similar patterns/results in the data collection) and user's recorded physiological, linguistic and psychological data will also aid in detecting suicidal tendencies.

An 'Intelligent' Machine

The most important reason for using a Chatbot is for it to mimic the 'intelligence' and presence a psychologist. With the ability for it to have agency (in this case, to have a goal and proceed autonomously towards that goal), this method will prove most useful as this intelligent agent is able to adapt accordingly to users' interaction and situation. Each agent proceeds towards its goal through sense-think-act cycles.[22] The first step of the process is for the machine to sense the surrounding environment (in this case, detecting users' emotion, tone, mood) by listening and processing the sentences users' input. Next, the thinking process comprises of converting, storing, updating user information and ultimately, making a decision based on that knowledge.[23] This will be done by developing scripts in order to interpret user answers to facilitate a conversational (back-to-back answer and respond communication). Finally, converting the decision into an action will be performed by recommending courses of action to users.

Developing an 'intelligent' Chatbot

However, it is to be noted there are many uncertainties and limitations that will initially exist due to the complexity and variation of depressive symptoms. Therefore, we propose the implementation of three phases for a smooth transition from weak to strong AI; where the machine ultimately becomes completely independent. The first phase of development is the beta stage, which will involve text messaging style interactions between real doctors/psychologists and patients. In line with the emergence of 'online consultation' services, this will be done to grasp the vast and complex nature of interactions. Upon attaining greater data sets and patterns, the processed data will then enable the transition to Phase 2, the preliminary stage, where while interactions will take place between users and computers, the latter's autonomous responses will be administered online by trained specialists. Ideally, when the machine's network of information becomes developed enough, Pix P will be function independently with minimal external administration except user-computer communication.

Discussion

Target Audience

The target audience of Pix P. is anybody with concerns of depression with access to a smartphone. Ultimately, Pix P. is intended for accuracy in diagnosis and accessibility for all. Pix P. is also intended to be used as a companion medical tool for medical professionals in diagnosing and tracking the progression of their patients.

Diagnosis

Users can request results at anytime, however, insufficient data will not yield results and is only available after a long enough period for the machine to confidently make a decision (varies depending on user's results). Users will not have access to the quantitative data collected on the back-end side of Pix P. Instead, the program will only output recommended steps of actions for the user to take. After much evaluation, we have decided to not reveal any raw or processed percentages or statistics collected from user activity. This decision was made due to numbers having definite affects on people. Our software is not designed on diagnostic principles, rather it is designed upon the ideals of assistance to the user and medical practitioner.

What it is

Pix P. is the incorporation of multiple different quantitative measures and a learning algorithm to maintain qualitative methods with the end goal of providing a tool to help increase the accuracy of depression diagnosis. It will be able to provide not only an initial gateway into dealing with a person's depression, but also as a medical tool for the extended monitoring of users' auditory, visual, linguistic, and physiological activity to aid medical professionals. Users would have access to the chat assistant as well as, ultimately, recommended steps of action to take according to Pix P's procession of results. On the other hand, medical professionals will have access to the more specific data collected, that is not shown to the user, to help in assessing the accuracy of diagnosis through the analysis and study of the results, and to further improve Pix P.

What it isn’t

Pix P. is not a all-encompassing device; despite its use of artificial intelligence, mass data collection, and its eclectic approach, the software recognizes the limitations of making an accurate diagnosis due to anomalies and the complex nature of depression. It does however, increase accuracy through the simplification and categorization of patterns. Therefore, Pix P. is neither an actual diagnosis of depression or a replacement for a trained medical professional in the treatment of depression symptoms. It is neither a replacement for traditional therapy, but serves more as an addition to the toolkit of approaches to mental health; as a person's only source of information as to whether they are in need of help. Finally, while Pix P. tries to emulate human features to obtain users' attention and honesty when

Evaluation of why it is the best method

In computer science literature, chat bots were one of the first problems tackled under AI and popularized because of the Turing test. Even these days, awards like the Loebner Prize are given for passing the Turing test (more precisely, it is given for being the most human like). While Pix. P will be designed to emulate basic human-like features, it emphasizes on the development of 'intelligence' in order to possess a therapeutic significance by impersonating a therapist. This impersonation and anonymity is believed to draw legitimate and honest user responses to important questions to aid in the diagnosis.

Significance of a Quantitative and Qualitative Approach

Pix P. also relies heavily on the back-and-forth constant data collection and pooling from large data corpus to evaluate users' mental health. An extensive network of data will provide opportunities to extract distinct quantitative patterns and trends. This quantitative, subtle collection of users' information, coupled with the Chatbot's capacity to obtain qualitative answers from users on their state of mind will prove to enable researchers to gain valuable insight and increase the accuracy of diagnosis.

Significance as a Longitudinal Study

Pix P. achieves what clinical psychologists and their methods do not; it monitors user activity on a regular, 24/7 hour basis. The approach will therefore be not constrained to the subjectivity of user verbal responses when questioned in a clinical setting, and is able to monitor their activity thoroughly without the omission of important changes. The only limitation is the absence of the machine with users.

Why it is Important

This is important due to the fact that depression affects 5% of the world population and is quite treatable even though only 35% get treatment.[24] Pix P. could provide a gateway for people to tackle their mental health problems without the associated stigma. It could also prove useful to get help for people without the financial capability to go see medical professionals. This program would allow for extended data collection in a discipline which has a lot of self-report at brief meetings separated by possibly lengthy periods of time. It will also add long term quantitative biological and psychological data for professionals to assist in diagnoses.

How to evaluate the performance of software

Thorough analysis of the individual back-end components accuracy with overview by a specialist to ensure quality of data collected. Then analyzing the correlation between the program given scores of severity of depression compared to medical professionals diagnoses over periods of time for the accuracy of the software's diagnoses.

Limitations

The entire existence of Pix P. revolves around the possession of a smartphone. Although this may not affect 'urbanites' in developed countries to a great extent, accessibility remains a huge issue for minorities, discriminated groups, and in general, people with no technology and smartphones.

Another crucial limitation Pix P. experiences when trying to detect depressive symptoms is, due to the complex nature and variety of depressive states, it is subjected to anomalies despite access to an extensive collection of recorded data. Although correlations still remain positive/negative and patterns remain, facilitating more accurate diagnosis for exceptional users/victims may prove difficult despite a quantitative approach.

Another important limitation of Pix P. is the data it collects. The data Pix P. incorporates from all fields has a background and an effect in depression,however, correlation does not definitely and necessarily mean causation. It is possible that other third variables like personal illnesses could effect the system due to the close nature of some of these elements. This problem is not a vital one when paired with a medical practitioner, because a practitioner could tell if the data collected was caused by an abnormality or by depression. Hence, the human input is still required.

Finally, a major obstacle to consider when developing Pix P. is the distinct characteristics of depression unique to different cultures. Cross-corpus generalization in depression have shown to be difficult to investigate; evidence of a wide range of effects for major depression has been found consistently across cultures and countries.[25] Although there exists extensive studies of depressive states unique to one's culture, it is an element this project has yet to take into full consideration when implementing its usage worldwide. As it stands, there remain a lot of questions left to be answered when discussing the epidemiology of depression across cultures.

Afterthoughts

How much time needed

Three years of constant development time with a five man team.

Funding

$500,000

- $150,000 per year for five developers for three years

- $50,000 for various licenses, fees, and deals with medical companies.

Insights

We learned how involved the process of even just coming up with a proposal idea is, let alone actually starting any sort of development or research process. There are also a lot of varying studies out there that address many of the individual components of what we researched, but none have attempted to combine them all which we feel could be very useful as future research. There are a lot of individual moving parts to these large scale initiatives and to even get started on this project would take quite a long time so we understand more of why development for a lot of new ideas can proceed quite slowly.

References

  1. ↑ Diagnostic And Statistical Manual of Mental Disorders : DSM-5. Arlington, VA :American Psychiatric Publishing, 2013.
  2. ↑ Abreu-Silva, Erlon, and Alexandre Todeschini. "Depression and its Relation with Uncontrolled Hypertension and Increased Cardiovascular Risk." Current Hypertension Reviews 10, no. 1 (2014): 8-13. doi:10.2174/157340211001141111144533.
  3. ↑ Batty, G. David, Tom C. Russ, Emmanuel Stamatakis, and Mika Kivimäki. "Psychological distress in relation to site specific cancer mortality: pooling of unpublished data from 16 prospective cohort studies." Bmj, 2017. doi:10.1136/bmj.j108.
  4. ↑ NHS Choices. Accessed November 27, 2017. https://www.nhs.uk/conditions/clinical-depression/symptoms/.
  5. ↑ Howale, D. D., Bathija, D. A., Gupta, D. S., & Dr D P Pandit Dr D P Pandit. (2012). Http://theglobaljournals.com/ijsr/file.php?val=February_2014_1391505377_aa754_130.pdf. International Journal of Scientific Research, 3(2), 400-403. doi:10.15373/22778179/feb2014/131
  6. ↑ "About Face ID advanced technology." Apple Support. November 05, 2017. Accessed November 26, 2017. https://support.apple.com/en-ca/HT208108.
  7. ↑ Wegrzyn, Martin, Maria Vogt, Berna Kireclioglu, Julia Schneider, and Johanna Kissler. "Mapping the emotional face. How individual face parts contribute to successful emotion recognition." Plos One12, no. 5 (May 11, 2017). Accessed November 27, 2017. doi:10.1371/journal.pone.0177239.
  8. ↑ Girard, Jeffrey M., Jeffrey F. Cohn, Mohammad H. Mahoor, Seyedmohammad Mavadati, and Dean P. Rosenwald. "Social risk and depression: Evidence from manual and automatic facial expression analysis." 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 2013, 1-8. Accessed November 26, 2017. doi:10.1109/fg.2013.6553748.
  9. ↑ Quatieri, Thomas F., and Nicolas Malyska. "Vocal-Source Biomarkers for Depression: A Link to Psychomotor Activity." https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/2012_09_09_MalyskaN_Interspeech_FP.pdf.
  10. ↑ Nield, David. "Scientists Can Now Diagnose Depression Just by Listening to Your Voice." ScienceAlert. July 11, 2016. Accessed November 28, 2017. https://www.sciencealert.com/this-computer-program-can-tell-when-someone-s-depressed-by-their-speech-patterns.
  11. ↑ Ibid.
  12. ↑ <Quatieri, Thomas F., and Nicolas Malyska. "Vocal-Source Biomarkers for Depression: A Link to Psychomotor Activity." https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/2012_09_09_MalyskaN_Interspeech_FP.pdf.
  13. ↑ Ibid.
  14. ↑ Quatieri, Thomas F., and Nicolas Malyska. "Vocal-Source Biomarkers for Depression: A Link to Psychomotor Activity." https://www.ll.mit.edu/mission/cybersec/publications/publication-files/full_papers/2012_09_09_MalyskaN_Interspeech_FP.pdf.
  15. ↑ Mundt, James C., Adam P. Vogel, Douglas E. Feltner, and William R. Lenderking. "Vocal Acoustic Biomarkers of Depression Severity and Treatment Response." Biological Psychiatry 72, no. 7 (2012): 580-87. doi:10.1016/j.biopsych.2012.03.015.
  16. ↑ Brauser, Deborah. "Language Patterns May Help Diagnose Depression" October 07, 2013 https://www.medscape.com/viewarticle/812151#vp_2
  17. ↑ Ibid.
  18. ↑ Ibid.
  19. ↑ Hamilton M: A rating scale for depression. Journal of Neurology, Neurosurgery and Psychiatry 23:56-62, 1960
  20. ↑ Ilades, Chris, MD. "5 Questions Doctors Ask When Screening for Depression." EverydayHealth.com. June 22, 2011. Accessed November 24, 2017. https://www.everydayhealth.com/depression/5-questions-doctors-ask-when-screening-for-depression.aspx.
  21. ↑ "QPR Institute | Practical and Proven Suicide Prevention Training." QPR Institute | Practical and Proven Suicide Prevention Training QPR Institute. Accessed November 28, 2017. https://www.qprinstitute.com/research-theory.
  22. ↑ "The Disruptive Chat Bots Sizing up real opportunities for business." https://www2.deloitte.com/content/dam/Deloitte/ie/Documents/ie-dispruptive-chat-bots.pdf.
  23. ↑ Viswanathan, Vaisagh. "How To Make A Chatbot Intelligent? – Chatbots Magazine." Chatbots Magazine. February 06, 2017. Accessed November 28, 2017. https://chatbotsmagazine.com/how-to-make-a-chatbot-intelligent-a232dc367aed.
  24. ↑ American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision, Washington, DC, American Psychiatric Association, 2000.
  25. ↑ Alghowinem, Sharifa, Roland Goecke, Julien Epps, Michael Wagner, and Jeffrey Cohn. "Cross-Cultural Depression Recognition from Vocal Biomarkers." Interspeech 2016, 2016. doi:10.21437/interspeech.2016-1339.

Diagnostic And Statistical Manual of Mental Disorders : DSM-5. Arlington, VA :American Psychiatric Publishing, 2013.

Retrieved from "https://wiki.ubc.ca/index.php?title=Course:COGS200/2017W1/Group9Project&oldid=483411"