Introduction

Emerging Media Lab
About EML
A collaborative space for UBC faculty, students and staff for exploration of emerging technologies and development of innovative tools and solutions. This wiki is for public-facing documentation for our projects and procedures.
Subpages
	Main EML wiki page; Public EML website; EML Projects;

Background

In healthcare, nurse-patient interaction is of paramount importance. Student Nurse Practitioners’ (NP) understanding of theoretical concepts could be improved with a tool that models clinical decision-making through automation. Through virtual reality (VR) technology, students will be able to participate in realistic learning experiences and enhance their communication and clinical skills in a safe and controlled environment.

“Our goal was to transform the education of future Nurse Practitioners by merging the cutting-edge realms of virtual reality and large-language models. We wanted to create a ground-breaking learning environment where students can hone their communication, diagnostic and clinical-decision making skills through realistic, interactive simulations. This project isn’t just about teaching; it’s about immersing students in an authentic clinical experience, where every virtual patient encounter is a step closer to excellence in patient care. We are committed to bridging the gap between theory and practice, ensuring our Nurse Practitioners are not only knowledgeable but also profoundly skilled in the art of healing.” - Ashley Scott, Principal Investigator

An OSCE (Objective Structured Clinical Examination) is a performance examination which evaluates student NPs on their ability to interact with and diagnose/manage a specific patient’s medical concern. Student Nurse Practitioners describe the experience as a nerve-wracking and stressful process, with great fear of the unknown as they never know what to expect. They expressed sentiments that it is hard to practice for these OSCEs as the only way is to learn by doing the real thing, as traditional peer-to-peer or online practice methods fall short. Real-world instructional hazards like time constraints, scheduling conflicts with practice partners, as well as the lack of realism from existing web-based solutions are mitigated by the usage of this tool.

Prior to coming to EML, existing web-based solutions which were primitive in its animations and visual design were investigated. However, the tools were not customized for an OSCE setting, and those that were, included a text-based simulation which involves the user having to talk to or type out their responses to a written chatbot-like interface. The text-based chat interface lacked realism, as in actual OSCE, the student Nurse Practitioner would not be typing on a keyboard to communicate with the patient. Hence, the interactions in existing solutions lack realism and similarity to the actual OSCE, failing to prepare the students effectively for their examinations. Moreover, there are limited VR tools for simulating the OSCE experience for student Nurse Practitioners.

Hence, the challenge was to develop a realistic VR solution by leveraging realistic MetaHuman avatars, facial and body animations, and a Large Language Model (LLM) to enable contextually appropriate conversations between the student NP and the patient.

Objective

The project aims to develop a VR application that utilizes an LLM, and realistic 3D modeling/animation techniques to create an immersive learning experience. Nurse Practitioner students can engage in interactive consultations with virtual patient avatars, all the while mitigating real-world instructional hazards.

These avatars will exhibit and describe real signs and symptoms, allowing students to practice their clinical skills in various simulated patient care situations. Ultimately, it is hoped that integrating VR technology into pedagogical practices used in nursing programs will aid students’ transition from fragmented textual knowledge to an interactive learning experience.

Format and Versioning

This project uses Unreal Engine 5.5, Meta Quest Virtual Reality headsets, various motion capture techniques, MetaHumans for realistic avatar animations, accurate lip-syncing. Integration with LLMs enables dynamic, lifelike conversations with virtual patients, moving beyond pre-configured responses whilst still allowing core concepts to be controlled to ensure parity between users.

Primary Features

Simulation

The tutorial and simulation will include two avatars, an avatar performing the role of examiner, observing the student’s performance during their virtual OSCE and providing post-simulation feedback, and a virtual patient avatar. The virtual patient avatar is programmed to give realistic variable responses of pertinent content, allowing multiple students to engage in practice consultations mirroring real life. This design ensures consistency in training and evaluation, while preparing students for both clinical practice and licensing board OSCEs.

Our current prototype consists of six different OSCE scenarios where the student NP can conduct a focused history and brief counseling of patients with different symptoms. The patient avatar describes realistic signs and symptoms when prompted, and matches the traditional OSCE experience, providing a realistic training ground for the student NP. Additionally, the student is provided with the virtual avatar's scenario brief and instructions. The examinee is then able to practice conversing with the patient avatar and perform their OSCE while being actively evaluated. At the end of the simulation, students will receive a personalized results screen, scoring them on how well they followed the OSCE structure. Hence, by utilizing our tool, they gain confidence in preparation for their OSCE.

We produced a significant increase in the authenticity of interactions and practice by utilizing a realistic clinical virtual reality environment, purchased through Fab. By harnessing MetaHuman Creator, we designed seven realistic avatars, an examiner avatar, and six patient avatars, which the user would primarily interact and communicate with. This allowed the avatars to resemble real people, exhibiting lifelike appearances, facial expressions, and body language, through realistic facial and body animation.

An LLM, was used to enable dynamic and contextually appropriate discussions between the students and the patient avatars, which would help facilitate the scenario to resemble an OSCE. This is a key feature of this project, as other clinically focused software currently available lacks this aspect of authenticity and realism to an OSCE.

FUNCTIONALITIES

Here is a table of all the existing features that were proposed at the start of term and what was achieved.

No.	Task	Priority	Status
F1	Realistic MetaHuman avatars	Must have	Complete
F2	Fine-tuned prompts for conversation flow	Must have	Complete
F3	Realistic clinic environment setting	Must have	Complete
F4	Realistic facial and body animations	Must have	Complete
F5	Consistent & stable user experience	Must have	Complete
F6	Accurate OSCE rubric evaluation	Must have	Complete
F7	Essential user interface components and navigation menus	Must have	Complete
F8	6 available scenarios	Must have	Complete
F9	Accurate MetaHuman Lip-Sync	Nice to have	Complete
F10	Examiner Animations	Nice to have	Complete

Tech Stack and Development Overview

Technical Components

Here is a table with all the tools used in the project's design and development.

Tool	Description
Unreal Engine	Game Engine used for the development of this project
Runtime Audio Importer	Plugin used for importing speech audio at runtime
MetaHuman	Highly realistic virtual humans created for use in Unreal Engine
VaRest	Plugin used by the evaluation system to obtain results
Runtime MetaHuman Lip Sync	Plugin used to facilitate runtime MetaHuman lip-sync
DXL AI Server	Custom AI middleware for facilitating the use of an LLM in NP
OpenAI gpt-4o-mini LLM	Facilitate speech-to-speech system and OSCE results scoring algorithm
Threepeat Picker	Open-source free animation picker tool
UE_DXL	In-house plugin for facilitating the connection between NP and DXL AI server
Figma	Design software used for wireframing and prototyping UX/UI designs
Miro	Mind map visualization software used for architecture diagrams and UX related work
Trello	Project management software used for team and project organization
Qualtrics	Survey software used for gathering quantitative user research findings

In fall 2023, MetaHuman Creator was leveraged to bring to life a patient and examiner avatar. Convai was used to provide speech-to-speech functionality, alongside body and face animations for our OSCE patient avatar. For the examiner, we utilized Unreal Engine’s Text to Speech sub engine, with its own animation blueprints for body and face that sourced animations from Mixamo, Livelink, and Nvidia Audio2Face. The environment used was purchased from the Epic Games Marketplace and created by Blue Dot Studios. At this point of the project, there were no functioning user interfaces, but simple buttons were used as placeholders. VR actions implemented included an interactable laser pointer for pressing buttons, and a button for talking into the microphone. To implement the OSCE scenario into the patient MetaHuman, we made use of Convai’s knowledge bank feature.

At our internal showcase in December, we demoed a prototype that featured user-patient avatar interaction facilitated by Convai on Meta Quest 3. This prototype had much of the functionality already working, as the user would be able to speak to the patient avatar and the scenario / conversation would resemble an OSCE, as the patient avatar had knowledge relevant to the OSCE facilitated by Convai. The demo prototype also featured a tutorial with an examiner avatar that was able to orient the user to the scenario.

During the second term, we took Convai completely out of the project, and implemented speech-to-speech using an EML in-house server (DXL AI Server) for making API requests to OpenAI. The role of the OSCE patient was prompt engineered in the configuration setting of the LLM’s role. Transcription of user voice input was implemented using functions from Runtime Speech Recognizer. The animations were redesigned to better function with the new speech-to-speech feature. The Unreal Engine text-to-speech sub engine was also taken out due to its robotic voice. We generated audio files for the script of the examiner beforehand to be played during the tutorial instead of utilizing OpenAI. User interfaces were created using widgets and controlled using a switcher for the tutorial, and a server initializing button was created to start the OSCE examination.

We also developed a scoring system that allows users to get instant feedback on their performance during the examination. To do this we effectively added a few more components into VRPawn that allowed it to record all of the text that was spoken by a nurse practitioner and then get OpenAI to score how well the practitioner did based on a rubric. Moreover, we also implemented a user interface for displaying the user's score and performance.

Nurse Practitioner was iterated on in Fall 2024 to expand the number of available scenario to six. The patient response was improved upon by refining the OpenAI structure. Furthermore, the UI for choosing a scenario was implemented, and the results were expanded to include a breakdown by category.

During the second term, The AI was updated from a single big prompt into a structured output API, enabling better control over general behavior and enabling fine-tuning of each question. Meta-Human animations were added to make the avatars more realistic and less robotic for the user to interact with. Lastly, the UI was iterated upon to improve readability and user intuition.

Nurse Practitioner returned to the EML for one last iteration during the summer of 2025. The S2025 team was focused on addressing the numerous challenges identified in the previous iterations of NP. The team began by performing a complete rewrite of the codebase since upgrading the previous codebase was not feasible due to the aforementioned issues. Moreover, the rewrite was done in a newer Unreal Engine version (5.5), and involved heavy optimizations. One of these optimizations included refining the previous terms' prompt engineering by adding jail-breaking prevention as well as controlling how much the MetaHuman avatar reveals when speaking to the user. This allows the user to ask more questions, which is aligned with the OSCE rubric, as our evaluation system only evaluates what the user asks and not what the responses are from the avatars. Furthermore, minor UI updates were made for consistency throughout the project. In terms of the animation and MetaHuman design, the upgrade to the newer Unreal Engine version enabled us to create more responsive and performant MetaHumans with hand-keyed animations. Additionally, we seamlessly incorporated MetaHuman lip-syncing into the project, allowing for a more immersive learning experience.

Design Overview

User personas and character builds were created in Figma during the fall term to learn from target users’ needs and pain points. User flows and user journey maps were created in here and here to map out the OSCE and tutorial scenarios in the prototype. Character builds for the patient and examiner avatar helped visualize their role in the OSCE simulation.

MetaHuman Creator was leveraged in the design of the patient avatar. Unreal Engine Editor was used to create the initial start menu for the prototype that was demoed at internal showcase in December. Tools like ShapesXR and Bezi were explored but ultimately not considered for the needs of the project at the time, and Figma was used instead for the design of all UI/UX design assets.

Design Assets

The UI was designed using Figma. This included high fidelity wireframes that are in various stages of implementation (indicated on the file) and prototype flows of how a prototype would go.

A welcome flow consisting of a loading screen and start menu were designed.
Menus (pause, tutorial)
Scenario brief UI
Choosing Scenario UI
UI for the OSCE score summary
UI for score category breakdown
Control UI
Timer assets
Visual indicators for listening and conversational AI interactions
Visual indicators for usability and greater ease of use user interactions

User Research

We started a user research initiative in early February, reaching out to current Nurse Practitioner students. After collecting consent forms, we conducted two user interviews with two current Nurse Practitioner students and collected responses to a usability questionnaire.

The primary goals of the initiative are as follows:

User Experience Evaluation: To understand and assess how intuitive and engaging the VR tool prototype is for nurse practitioner students. This includes navigation, interface design, and overall user satisfaction.
Subject Matter Expertise: Leverage the domain expertise of student nurse practitioners to gain insights into how well the VR tool aligns with actual clinical practices and educational standards.
Future Improvement Identification: Compile feedback and insights to guide the development of future iterations of the prototype, with an emphasis on improving user experience.

The structure of each interview aimed to reflect the goals of the initiative. For the full procedure, please refer to the appendix. Each interview was split into two sections – the first being a more typical user interview format, and the second being a usability test of the prototype demo at the time. After the usability test, there was an exit interview to get immediate feedback on the demo. There was also a usability questionnaire that individuals filled out asynchronously within a day.

In the user interview format, we started out with background questions that aimed to leverage the domain knowledge of student nurse practitioners to better inform the future design and development of the OSCE simulation in the prototype. The next stage aimed to assess the technical experience students have with virtual reality and any accessibility concerns (e.g. history of motion sickness, visual impairment) so that we could be aware of them if they come up in the demo. If students indicated that they had less experience, the user interview practitioner would provide them with more guidance and instructions in using the demo. This would prove helpful in figuring out a key insight – which was that users would need guidance in basics like setting up a headset or how to hold controllers, which was later translated to a VR guide.

Then, we headed into the usability test of the prototype demo. The student was asked to interact with the prototype and converse with the patient avatar as if they were conducting an OSCE. The interviewer could give the patient direction if they needed help, which proved useful in figuring out which areas or tasks the users struggled with. Afterwards, the interviewer asked the participant some closing questions and directed the participant to fill out a survey asynchronously within 24 hours. The survey consisted of questions based entirely on the System Usability Scale, which is a tool used to measure the usability of a system (like an app prototype).

The prototype demo at the time consisted of a tutorial and the scenario demo itself, which took place in a hospital room setting. Since then, the current prototype takes place in a clinical setting.

Insights

A complete list of insights attained from the user research initiative can be found here: User Research Findings Summary

Some of the most key insights were immediately translated into user stories and then into high fidelity UI designs that aimed to “fix” the pain point. These included the insight that feedback was needed in communication, “As a user, I would like an indicator that the AI avatar registered the question that I asked.” This was translated into UI designs for a listening, processing, and error message popup indicator. Additionally, an average SUS score of 67.5 (barely below the average SUS score of 68) was achieved, but it is worth noting that the SUS score is usually measured on finished applications.

Setup Guide

Instructions related to Nurse Practitioner's initial setup and development can be found via GitHub repository This and handover documentation, is not currently available open access. For requests about those sources or other comprehensive information contact the Lab.

Known Issues

This section covers a series of common issues (and potential solutions) that users may face when using Nurse Practitioner.

1. No Response

If you have spoken to the patient, and have not received a response back, it may be due to the unavailability of one of NP’s cloud-hosted services:

OpenAI

Since NP uses the services (chat, speech synthesis, and transcription) provided by OpenAI, it could be affected by any outages in OpenAI’s services. Since this is outside of the control of those involved with this project, we recommend checking OpenAI's status page until the issues are resolved.

DXL AI Server

This server acts as a “middleman” between NP and the Large Language Model provider (such as OpenAI). If you think all other services are running normally, reach out to EML administration or a supervisor for help.

2. Delayed Responses

Sometimes, the patients may take longer and longer to respond. There are various reasons why this could be happening. Naturally, since information has to move around multiple servers, and back to your device, some latency is expected. If the responses are taking too long, please restart the NP application and try again.

3. Non-functional Evaluation

There are two primary reasons why you may be experiencing this issue. First, this may be an issue with OpenAI’s services (as mentioned above). In this case, you should refer to their status page, and follow their updates. Second, it may be due to a lack of an API key in NP’s files. Please ensure that the key is correctly installed by referring to the GitHub page or the handover document.

4. Off-topic / Wrong Responses

Given that NP utilizes large language models, some unpredictability is expected to occur occasionally. The patient may provide wrong or unrelated answers. This is unfortunately, a limitation of large language models that is beyond NP’s control. If you are experiencing this, we recommend restarting the scenario, and trying again.

Challenges

Overall, there were not many significant challenges faced by the Summer 2025 team, aside from some runtime audio handling issues in the beginning.

Future Plans

1. Aliasing issues

Currently, due to the graphical limitations of Unreal Engine VR applications, the game suffers from a medium-high degree of aliasing. Current anti-aliasing options provide significant drawbacks (e.g. severe ghosting) that outweigh the benefits. While we have taken some measures to reduce the aliasing (such as combining low quality hair with high quality MetaHuman bodies), the effect is still noticeable, especially in reflective objects. We recommend that future teams investigate various current anti-aliasing techniques and find a solution to the problem.

2. Migration to future Unreal Engine versions

Since newer Unreal Engine versions can bring about improvements to various technologies (e.g. MetaHuman design), we recommend that future teams work towards migrating the project to the latest stable release of Unreal Engine at the time.

3. Patient Logs

NP evaluates the users based on their speech transcriptions throughout the OSCE examination (and not the patient’s speech). Sometimes, the patient might give away extra information that may prompt the user to not specifically ask more specific questions. In a real OSCE examination, the examiner may elect to give points to the examinee since this information was provided by the patient. Since our evaluation system is unaware of the patient’s transcriptions, it could be beneficial to include the full interaction for a more concise and accurate evaluation system.

4. Lighting

Due to the lighting design in the current hospital room, some shadows may look out of place, and as a result, some object may seem like they are floating. Future developers should aim to further tune the room lighting to create a more realistic environment.

5. Random Eye Movements

To maintain an immersive experience, we recommend that future developers add random eye movements (e.g., looking to the side) to the patients.

6. Animation Refinement

Currently, the sitting (idle and speaking) animations for this project suffer from a minor design flaw. The patients’ feet appear as if they are resting on the air instead of “dangling” from the bed. Future designers should change these animations to enhance the realism of the project.

Team