AAC Simulator is an Emerging Media Lab project for EPSE 411 students to experience immersive use of Augmentative and Alternative Communication.

Introduction

Emerging Media Lab
About EML
A collaborative space for UBC faculty, students and staff for exploration of emerging technologies and development of innovative tools and solutions. This wiki is for public-facing documentation for our projects and procedures.
Subpages
	Main EML wiki page; Public EML website; EML Resources; EML Onboarding; EML Offboarding; EML Operations; EML Projects;

Background

For Assignment #2 in EPSE 411 (Augmentative and Alternative Communication for Individuals with Severe Speech and/or Physical Impairments), students need to create communication boards which allow for non-verbal communication during a meal. As part of their research for making the communication board, students need to look into how preliterate Augmentative and Alternative Communication (AAC) users experience communication, the purpose of the exercise is to give the students a sense of the challenges in predicting needed vocabulary.

AAC tools range from low tech to high tech supports for non-vocal communication with a variety of users (eg. autistic people, stroke patients, etc.), communication needs (eg. pre-literate, post-literate), and purposes (eg. social, information exchange) [CITATION NEEDED]. AAC Boards are boards with rows of (often labelled) symbols that a user can point to to construct and express meaning. [IMAGE NEEDED]

Objective

The objective is to create an immersive replacement for the assignment using emergent technologies where students can empathetically and practically experience the need for a communication board.

The overall goal is to make them better educators, communicators, and communication partners to users of AAC.

Our priorities are accessibility, immersion, empathy, and flexibility of experience.

Format

The project consists of two main modules: (1) an AAC Board maker and hub, and (2) the virtual environment. The (1) AAC Board is being developed in Drupal (designed in Glitch.io and Figma). The default format of the (2) virtual environment is a web build of a unity project, to make the project accessible to students. The virtual environment is also being developed to support a VR Headset for full immersion. The two will be linked to create a relatively seamless user flow.

Primary Features

AAC Board Site

The AAC Board Site is made with Drupal, specifically Drupal version [10]. This site currently lives on the EMLx server with the url to the site as https://aac.emlx.ca/ This site is meant to be used in tandem with the VR environment. The supposed workflow is that students would fulfill part of the board making requirement of this assignment on this site, then they would be brought to the VR environment via the site after the completion of a board suited for the scenario of a conversation they are asked to complete. Therefore, this site only includes the tools students need in order to successfully complete the creation of their board within their given context.

For access to use or manage the site, one must sign up. For more complex moderations on this site, one must be assigned a role of an administrator on the site. For more information about roles and other basic knowledge on Drupal, please refer to this documentation.

Authentication & Data Collection

Making Board Contents inside Board Type Content

Authentication is taken care of with Drupal. All users are asked to either create or provide their credentials prior to entering the site. Once the user has signed up or signed in, they are immediately activated as a member of this site. Note: All users who sign up/sign in are assigned an authenticated user role. To change specific users' roles, please refer to this documentation.

Data collection in the case of this site would be collecting the boards and the board contents that students create. Note: Each authenticated users are only able to view, edit, and delete their own board. The content types that are created for users to create their board are: Board and Board Contents.

Board is the content type that compiles referenced entities from Board Contents content type. Board has a restriction of 32 entities since the assignment calls for the filling of 32 grids as completion. Users will only need to be creating board type contents when they want to create a board for their assignment.

Board Content is the content type where users fill in each grid element by giving an image/icon of their choosing a vocabulary or phrase to use during their conversation.

For more information about what content types are on Drupal, please refer to this documentation.

Method 1

Assignment Instructions

The assignment instructions are modified and listed currently on this nodeon the current Drupal site. This node is built from a basic page content type. In order to modify the instructions, one must be logged in as an administrator or a content editor to edit the node. For more information on how to edit any node as an administrator/content editor, please refer to the Board Editing section below.

Interlocutor Configuration

This feature is not yet implemented. Please refer to the Future Directions section below for more elaboration.

Method 2

Board Modification/Management

There are two ways to view, edit and delete a board content as a user:

Select View My Boards on the menu bar > Select the desired content to modify > Select the desired option after the accordion expands.
Select View My Boards on the menu bar > Select the desired content to modify > Select View under the desired content to modify > Select any other desired action towards modification.

Board Usage

This feature is not yet implemented. Please refer to the Future Directions section below for more elaboration.

Starting screen from the POV of the user. When video chat is on, the user's face will appear where the white square is placed.

Virtual Environment

Once a board is developed, the long term goal would be have the user test out their board in a unity environment. At the moment the website and unity applications are separate, so at the moment the user is able to test the environment with a default board.

Starting page when app starts

Features

As for August 31, 2022

Run a 2 way video call (with channel authencation for multiple separate rooms) including audio and video. (2 way video feed is buggy)
User movement and navigation with WASD keys.
Addition of webpages in the scene with an interactive keyboard.
UI elements including Quit and Start tutorial.
Object interaction with left mouse click.
Inclusion of 3D avatar models
Zoom onto AAC board.

Challenges

One of the issues we found was how to integrate the communication for the 3D environment?

Several solutions were discussed

AI partner with generated verbal response.
- Ideal for practicing alone but the issue is generating responses to continue or start a conversation.
AI partner with default responses
- Easy to implement verbal wise
- Provides ground work for above
- Not useful in long term but alright for demo purposes
Video call where 2 people within the app are able to call
- Complications include setting up a proper video call platform and server for real time connection

Ultimately, we decided that for this semester to keep in spirit with the assignment and have the 2 people converse within the app.

Future Directions

AAC Board Site

Example Interface of Interlocutor Configuration

Interlocutor Configuration

First, let's start off with the definition of the interlocutor. The Interlocutor in the case of this project is the avatar/communication partner that users will be conversing with in the VR environment once their board is made. Students may not always choose to complete the conversation part of this assignment with a familiar communication partner. Therefore, the team has ideas to give the students the same choice they would have in real life by giving them the choice to choose between a familiar or an unfamiliar partner to communicate with using the board they make. This configuration would be ideal to implement before the users begin making their board as they get to take this into consideration when they are coming up with strategies to create their board.

Board Making Features

Some additional and helpful features to have while users create their boards are:

Color coding each grid/box on their board
Rotate and use the board in landscape or portrait orientation
Customize text positions relative to the image
- Text on top or bottom of the image inside the grid/box

These features would likely have to be implemented while editing/making a board.

Board Usage and Interaction in VR Environment

Once the board is completed, the same board that a user made should be expected to appear in the VR environment. The idea for using the board within the VR environment is that there would be some interactions/indications for the user that they have selected a grid/box to communicate to their communication partner within the environment. Additionally, the interaction/indication would be check marks appearing on the grids/boxes that users have used to converse with their communication partner and all of the other grids/boxes that were not used will be circled in the end to indicate that the grids/boxes they did not use. This is to further implement the requirement from the assignment itself.

Implement User Test Feedback

The major recommendations/feedback from user testing to incorporate are the following:

Replace “Access Denied” with instructions for the user to create an account and login in order to access the website.
- Can also consider including a hyperlink to the login/account creation pages in case user experience technical difficulties with the “Log in” button disappearing
Reconsider how buttons are labelled to better indicate their functionalities.
- For example, buttons for saving the board contents and the board itself could be more specifically labelled
Include reminders for saving boards since autosave cannot be implemented

After the tests were conducted, the team decided that providing default icons was not necessary for this project. However, in case this feature gets reused, here are recommendations to consider:

Be mindful of where the default icons are displayed
- The version of the Drupal site used for the test had the default icons displayed at the very bottom of the page where users create a board. To minimize confusion, future implementations should consider alternative placements, such as the top of the page, or integrating this section with the “Board Contents” section. Alternatively, this section could be included as a separate page.

Provide detailed instructions for how default icons can be reused
- Due to limitations with Drupal, using default icons can be an unintuitive workflow and not many changes can be made to alleviate this. As a result, detailed instructions may be necessary to ensure users understand how this feature can be used.

Implementation of Interface Design

The rest of the team has come up with a visual design for the overall product. If it is desired to align the visual design of the overall product, please refer to the existing design for implementation.

AAC environment

Tutorial Integration

The team decided that rather than having a menu with instructions, it would be more intuitive to have a tutorial that can run you through the basic functions of AAC since function wise there are not a lot of instructions to memorize. The tutorial would hypothetically be accessed through the UI element on the top right of the screen which the user can click on at any time with the mouse.

Game server

After development, the team has determined that a gamer server is the best way to include immersion was it will allow both people using the app to see what the other person is doing in real time thus increasing immersion. This will also serve the purpose of having both users see the AAC board and what the other is pointing towards.

AAC user board upload and interaction

As a key feature, developers would have to link the Webviewer to the Drupal site link where the board is made. Once the board is uploaded it must be configured in unity to allow selection of certain squares thus keeping it in a same size format every time.

Entering into the environment with VR

VR was initially proposed to increase user immersion. The functionality between the desktop and the VR will have to be adjusted for controllers over keyboard and mouse.

Integration of AI partner & animations

To test out the board, users may like the option to practice by themselves. Thus autogenerated verbal responses and animations are necessary.

Statistics with AAC environment

This feature is useful for keeping track of what squares are being selected and how long the user spends in the environment. These options will likely be put in the starting page and when the user quits will see the results.

Customization of Avatars

Ready Player Me is an avatar generator. For this project, if possible would allow custom avatar generation so that the models can be used within the app.

Code & development details

Unity version -> 2021.3.5f1

The Github link is: https://github.com/ubcemergingmedialab/AAC-Unity

3rd Party Assets

Mixamo, Ready Player me, and Plask.ai -> For 3D models/avatars (Are uploaded with the git repo)
Unity 2021.3.5f1 -> build environment
- Assets/Packages (uploaded from Unity asset store and imported via package manager)
  - 3D WebView for Windows and macOS (Web Browser) -> upload webviewer pages (access to teams asset )
  - Oculus Integration -> for VR integration
  - Complete Classic Style Kitchen -> 3d models (access to teams asset)
  - Agora Video SDK for Unity -> for video call (free asset)
Agora.io -> account for video call

GitHub Branch summary

Branch status

Active -> Recent updates and in active development
Not active -> Serves a purpose but not updated frequently
Dead -> Branch is not and will not be updated. Should be removed.

Github branch summary - https://github.com/ubcemergingmedialab/AAC-Unity
Branch	Purpose	Last modified date as of 8/31/2022	Active status	Modified by	Build/Demo ready?
Main	Branch where official builds with minimum bugs go to	July 11, 2022	not active	Michelle Huynh	no
Menu	UI development for appearance and layout	July 15, 2022	not active	Michelle Huynh and Rae Jourad	no
develop_v2021	Current build for integrating primary functions	August 31, 2022	active	Michelle Huynh	yes
develop_version2021	Version testing	July 11, 2022	dead	Michelle Huynh	no
develop	Version testing	July 11, 2022	dead	Michelle Huynh	no
models	model development and animation troubleshooting	August 2, 2022	not active	Michelle Huynh	no
rae-test	push test	rae	dead	Rae Jourard	no

Unity key files/ folders overview

Below is a tree outlining the file of the project and identifying key scripts for developers.

Assets -> Where you will access all the files you need for this project

Agora Engine -> This folder will show up when you import Agora Video SDK for Unity. Contains Demos and scripts that you can reference.
Complete_Classic_Style_Kitchen -> This folder shows up when you import Complete Classic Style Kitchen. Note before you push that this folder should have spaces replaced by underscores (this is to prevent the folder from getting pushed onto github).
Oculus -> Folder appears when Oculus Integration is imported (will be relevant when integrating VR)
People -> Contains 3D avatars (the folders within people contain texture/material files) which can be dragged onto the scene
Resources -> Keep these files but don't need modifications.
Scenes -> Where you will be focusing your work on.
- Kitchen -> The kitchen scene where VR users would be directed to. (needs development)
- Kitchen_Desktop -> The kitchen scene where desktop users would be directed to. (present in current build)
- new & sample_scene -> could delete but is there to bug test specific scripts occasionally.
- SceneHome_modified -> The starting page of the application and where users will be putting their credentials. (run this page first to enable video call).
Scripts -> Where important scripts that you made/modified will be stored.
- Button_Handler_Modified -> Used for the Join button in SceneHome_modified.
- DragObject -> Script is added to objects that can be moved around the scene with a mouse.
- Game Object Camera POV -> Attached to Camera that will follow designated object in a relative position. (attached to main camera in Kitchen_Desktop)
- Highlight -> Script that when attached to object will highlight when mouse hovers over object. (not currently being used in scenes)
- Keyboard_movement -> Contains script to move the player character (in this case Player in Kitchen_Desktop via WASD).
- MouseCamera -> When attached to camera will allow the player's camera to follow the mouse. (attached to main camera in Kitchen_Desktop)
- OculusWebViewDemo -> Instantiates the webviewer pages. When the player is moving the webpages will stay at a relative distance. (found in main_scripts in Kitchen_Desktop).
- StartPage -> Used in GameController Object for SceneHome-Modified.
- Video_desktop Demo -> Is called when join is pressed in SceneHome-Modified through StartPage script. Starts the video call.

TextMesh Pro -> Handles text in canvas elements (don't delete)
Vuplex -> Imported folder from 3D WebView for Windows and macOS (Web Browser)
XR -> Default asset for enabling XR

Packages -> do not delete but project wise do not need to touch.

First Time Setup Guide

Running a build

With a premade build open the folder and run AAC_testing.exe.

Put the channel ID and Temporary Token given from the lab coordinator in the boxes, then join to the enter the video call.

Setting up Virtual environment

If you would like to download the repository into your computer, these are the instructions to get it setup.

** Note all the instructions are located in the file Installation and Setup for AAC.pdf in the Github repository that is downloaded.

To run VR side of AAC Simulator we need to download the git repository via git clone.

https://github.com/ubcemergingmedialab/AAC-Unity [1]

2. Open Unity Hub and download the project version of the AAC simulator onto your computer.

3. Add the project into Unity Hub and open it through safe mode if the computer requests it.

4. Download the following packages via package manager and then import them. (You would need to have these assets via Unity Asset Store or a file to download via microsoft teams in the AAC teams Unity asset folders)

3D WebView for Windows and macOS (Web Browser)
Oculus Integration
Complete Classic Style Kitchen
- After import double click Built-in and agree to import the assets
Agora Video SDK for Unity
- After import go to Build Settings and add Kitchen_Desktop and Scene Home-Modified

5. Press play to run the application

Obtain channel ID and Temporary Token from lab coordinator and fill in the boxes
Press join to enter the video call

Making a new build

For developers in build settings add the scenes present in the build (in this case Kitchen_Desktop and Scene Home-Modified).

Save the build.

User identification for Video Call

Agora.io is a video call service that requires an app ID, channel name, and a temporary token. To obtain the identification go to the agora site and login (ask lab coordinator). https://www.agora.io/en/

Navigate to project management and go to AAC's project and click on Config.

Scroll down to Features and click the link for Generate temp RTC token , insert your desired channel name, and generate token.

On this page you can copy all the credentials needed to enter the site.

Known Issues and bugs to be fixed

- WASD movement can result in player passing through objects (need collider adjustment for player)

- WASD movement is global instead of relative to mouse

- Video is not 2 way at the moment

- App ID and credentials need to be up to date (tokens expire after a day).

- Webviewer might crash due to positioning of player when moving.

- Animations are not integrated for walking and turning

Poster link

Upload a pdf link of your poster to the wiki. An editable version of the document is preferred.

Development Team

Principal Investigator(s)

Tammy McEachern Hughes

Adjunct Professor

Gabrielle Coombs

Educational Technologist at Education Technology Support