Introduction

Emerging Media Lab
About EML
A collaborative space for UBC faculty, students and staff for exploration of emerging technologies and development of innovative tools and solutions. This wiki is for public-facing documentation for our projects and procedures.
Subpages
	Main EML wiki page; Public EML website; EML Resources; EML Onboarding; EML Offboarding; EML Operations; EML Projects;

Background

In the rapidly evolving educational landscape, the integration of technology into post-secondary environments has become increasingly important. Advancements in Natural Language Processing and Generation presents new opportunities to enhance the student experience in online courses.

A specific area of interest are courses with significant engagement components such as class discussions and supplementary reading. In these activities, students are often passive, having not completed, or not understood the provided academic texts.

This project aims to develop an improved method for interaction by using large language models in an educational context. Currently, most chatbots, AI tutors, and educational assistants follow a “Question and Answer” format, which relies on the student’s input to start and maintain the conversation. This model is not a proper solution to student passiveness.

The Emerging Media Lab (EML) will therefore leverage generative AI to design an Intelligent Assistant - OCELIA - that will support first year Geography students in their engagement with the academic reading materials for the course GEOG 121.

Objective

The central objective of this project is to create an accessible, conversational experience that extends beyond the standard prompt-answer interaction, facilitates deeper understanding, and encourages first year Geography students to critically engage with the course materials, including academic journal articles and books. Key indicators of the desired behavior are as follows: ability to understand and process complex academic texts; respond to student queries related to the reading material; encourage critical thinking by asking probing questions; and provide explanations and clarifications where necessary, while maintaining a focus on promoting a student’s independent thought.

The EML team has addressed these objectives by:

Creating a responsive, accessible user interface that allows users to easily switch between separate discussions for each week’s dedicated reading materials
Encoding AI behavior and robust guardrails that opens the discussion, then sustains the conversation with the student by asking follow-up questions.

Format and Versioning

OCELIA (EML Prototype)

To Do

Primary Features

Overview

The application consists of an interactive chat interface. The student is given the option, via drop-down menu, to select a weekly reading to discuss. The agent first gauges the student's familiarity with the topic, then engages them with probing questions designed to scaffold their understanding and promote critical thinking about the course material.

This project uses the OpenAI API to interpret user inputs and generate dynamic, context-sensitive responses tailored to the student's knowledge level and contributions.

The following section lists features that were proposed and accomplished throughout the term:

Custom, web accessible AI chat interface

A web-accessible chat interface was developed. The current interface consists of:

Responsive design, to accommodate different screen sizes and/or mobile formats
A main chat component
An interactive toolbar that displays the current discussion topic (weekly reading) and allows the user to switch between different weekly topics.
A sidebar with placeholder buttons for information, warning, and settings components

There are secondary interface components and functions that are currently prototypes, but not implemented due to time constraints. These include:

Instructions/Tutorial
Privacy Information
Contact Information
Help
Export Chat Log
User Onboarding

AI Tutor Chat Interaction

Through the chat component on the web interface, users can send conversational messages or questions to the assistant and receive AI-generated responses.

In terms of AI behavior, OCELIA currently acts as an educational assistant. To encourage students to scaffold knowledge and formulate their own answers, OCELIA is first prompted to ask about their familiarity with the discussion topic. Based on the student’s response, OCELIA will adjust its conversation, asking questions related to class discussions, probing questions, and following up with questions related to the reading to start and sustain the conversation.

Course Material Knowledge Bank

The relevant weekly readings, academic journals, syllabus and course materials were successfully uploaded to the project, allowing OCELIA to successfully refer to, discuss, and respond to questions about the course material with the appropriate context.

Implementation of useful text interaction between user and AI and testing of AI behaviour

Based on preliminary tests and feedback, OCELIA seems to produce the desired behavior. As mentioned, it asks probing questions to scaffold knowledge, sustain discussion, and keep the user on topic. However, OCELIA should undergo formal user testing in the future to fine-tune, identify and resolve behavioral weaknesses in edge cases and hallucination risks.

Design Methods

User Research

We wanted to discover user motivations, needs, and attitudes towards course chatbots, course readings, and class engagement. We conducted two semi-structured user interviews to this end.

Some key insights:

Users value the ability to pinpoint key takeaways succinctly and understandably
Reading and hearing the content in a different way is the most impactful part of discussions, and can help the student get unstuck, or improve their understanding
The strength of course readings is that they ground course concepts in read world examples
Readings are often perceived as supplemental or nonessential, leading to a lack of motivation
Readings can be long and convoluted, making them difficult to get through and understand

Discovery/Design

Based on by literature about AI Dialogue and user interview data,

Opportunities for OCELIA were identified as follows:

Creating a tool that actively sustained discussion through probing questions
Preventing information overload by scaffolding knowledge, focusing on key points, and simplifying concepts from the reading
Creating opportunities for students to describe concepts in their own words, and receive dynamic feedback

User Stories

As a student, I want a tool that incrementally helps me explain key concepts and construct answers in my own words so that I can develop a robust understanding of them.

As a student, I want to engage in discussions that expose me to the key concepts from different perspectives and in worked examples to help me digest and understand the material.

As a user, I want to be able to engage in discussions that allow me to pinpoint and understand the main takeaways of a reading.

As a student, I want a tool that will simplify and propose different ways of interacting with long and convoluted material so that I can feel better about doing course readings and reduce information overload.

As a student, I want a tool that connects course readings back to course learning objectives and lecture concepts so that I can visualize the bigger picture of the taught material.

These user needs and functionality requirements were outlined in the creation of a persona of the target user, and a user flow of the experience.

Prototyped Design Features

User Onboarding

A landing page which introduces the product, provides privacy information, and collects the user’s name and desired discussion topic.

“Currently Discussing” Menu

Displays which week’s reading is currently being discussed. From this menu, users can also switch out the discussion topic.

“Export Chat” Button

Allows users to export the chat log.

Privacy Icon

Displays a tooltip for privacy information.

Sidebar

It consists of 3 buttons, each displaying a tooltip or menu with respective information for: Information about OCELIA, Contact Information for feedback/bug reports, and Settings.

Privacy Pop-up

A central dismissible pop up informing the user that their information and responses are secure, which ensures visibility and understanding.

Chat window

Includes text bubbles, and text inputs.

Quick Reply Bubbles

Reply suggestions that offer the user an easy way to reply, to encourage continued participation.

Generation Tools

Below each AI response, these buttons allow the user to give feedback about the response, regenerate the response, simplify the response, or read it aloud.

Development

Technical Components

The core framework used to construct the application was Blazor, provided as a part of Microsoft's ASP.NET Core 8 Runtime. Blazor supports both client and server-side rendering through clever use of WebAssembly and WebSockets. For this prototype, only WebAssembly rendering was targeted for means of smaller project size, faster/easier development, and better dependency management. The opportunity to migrate to both client/server side rendering in the future is still present.

The MudBlazor component library was chosen to implement the project designs. In addition, the MudBlazor.Markdown library was used to display agent-emitted markdown in it's formatted representation.

Tool	Description
ASP.NET Core 8	Blazor-containing framework used to build the application
MudBlazor	Blazor component library to assist in design implementation
MudBlazor.Markdown	Markdown component library for MudBlazor to display agent response
OpenAI Assistants API	Used for agent implementation with respect to designated behaviour and targeted readings/documents
Figma	Design software for creating high-fidelity wireframes and prototypes

Project Structure

The project's structure is large, with many intermediate files. Relevant files and their paths with respect to project root are shown and described below.

.

├── Layout

│ ├── CenteredPopup.razor

│ ├── CenteredPopupDialog.razor

│ ├── NavMenu.razor

│ └── MainLayout.razor

└── ...

The 'Layout' directory serves to contain components that are used in the construction of the home page. Specifically:

CenteredPopup.razor dictates the setup and use of a centred message dialog, as well as the logic behind the title, message, and close callback;

CenteredPopupDialog.razor manages the popup window element itself, as well as the opening and closing logic;

NavMenu.razor covers the navigation/information bar at the left hand side of the screen;

and MainLayout.razor organises all prior components in a responsive manner.

.

├── Pages

│ └── Home.razor

└── ...

As there is only one page, the 'Pages' directory holds a single file called 'Home.razor' to represent the chat session/interface with the AI. When this page is reloaded, the chat session will begin anew and the previous conversation will be discarded.

.

├── Services

│ ├── IWebSocketService.cs

│ └── WebSocketService.cs

└── ...

The AI is accessed through EML's OpenAI websocket server, which requires a websocket connection to persist for the duration of a conversation. All chat messages exchanged are stored as history for said specific conversation (no data is saved). The 'IWebSocketService.cs' file describes an interface for communicating over WebSockets; the 'WebSocketService.cs' file contains an implementation of this interface, simply supporting behaviour to connect to some address, send a message to said address, process some event on message receive, and disconnect.

.

├── Program.cs

└── ...

The 'Program.cs' file defines an entry point to build the project; this would be a refactoring point when migrating from Blazor WASM to Blazor Server/Server+WASM.

.

├── wwwroot

│ ├── favicon.png

│ ├── ubc.png

│ ├── index.html

│ ├── css

│ │ ├── app.css

│ │ └── bootstrap

│ │ └── ...

│ └── ...

└── ...

The 'wwwroot' directory contains page content and layouts, including bootstrapped layouts. The 'favicon.png' file defines the image displayed in the web browsers' tab (can be changed to anything), whereas the 'ubc.png' file represents the UBC logo displayed on the NavMenu component. The 'index.html' file links all necessary user and library stylesheets and exposes JavaScript bindings to library functions. Furthermore, the 'scrollToEnd' function used in the '.razor' files to smoothly scroll an element into view is also defined in 'index.html'. With regards to CSS, more complex settings (such as media queries and keyframes) and other user defined classes are defined in 'app.css', with non-user generated CSS stored in 'bootstrap'.

Github Branches

Our repository can be found here: https://github.com/ubcemergingmedialab/24-3003-OCELIA

Please refer to the table below for a description of all the branches in the repository.

Active Branches

Branch Name	Description
master	main branch

Issues/Bugs

Internet Disconnect

When the internet disconnects, the WebSocket connection to the AI will crash; it will produce an error, and the page will need to be reloaded. This is not a huge deal, as the user likely has bigger problems if the internet is down, but all conversation history will be lost and must be restarted upon reload. In the future, to potentially combat this problem, cookies containing the last context and user message may be stored on the user's browser, to which the application could ask if they wish to resume on the previous topic.

Markdown Large Font Size

Agent emitted markdown is sometimes rendered with larger than usual markdown headers (the response titles/sections), which could be caused by either the agent itself, or the markdown library used. It is recommended to explore potential constraints on maximum header size in the MudBlazor.Markdown documentation.

First Time Setup + General Usage

Getting Started

Clone this Repository
Open the solution in Visual Studio
Run the project with the HTTPS profile to start a development instance in your browser

Building

Navigate to the project directory
To build, open a shell, and enter dotnet build
The application can instead be both built and run with one command, dotnet run

Challenges

Other than the noted issues/bugs, the team faced no issues with the current implementation of this project.

Future Plans

The team suggests recommendations for future development of this project:

Long Term Memory

The current application stores no data, as each session is effectively stateless on the user's side. When a page is opened, a new connection is established to a websocket, to which the websocket will manage context of exchanged messages. In the event that long-term memory (cross-conversational) is desired, the application would need to uniquely identify users to present their previous context. This could be implemented using a login system and user-specific storage for previous conversations (through use of some external database).

User Experience Improvements

Rigorous usability testing with students enrolled in online courses should be prioritised in order to gauge the effectiveness of the tool, the appropriateness of the agent's behavior, and improvements to be made for the graphical user interface.

Additionally, the team suggests that nest steps should implement the proposed user interface designs - specifically, the addition of quick replies, generation tools, and accessibility functions. Minor changes would include adding onboarding screens and completing website copy for privacy information and the like.

AI Behaviour Improvements

The agent is built on the OpenAI Assistant API, which allows for the specification of behaviour through prompt engineering and document embeddings. The current AI behaviour is dictated by the prompt it is provided, to which it is instructed to send a preliminary greeting to the user, as well as a question to gauge their skill level. Weeks can be selected for context, and additional information can be provided to fine-tune AI goals. Ideally, the documents uploaded would all have associated learning goals; these goals would assist the AI in asking leading questions to prod the user towards their current objective.

Poster

As the title suggests

Team Members

Principal Investigators

Dr. Siobhan Wittig McPhee

Associate Professor of Teaching

Department of Geography

University of British Columbia

Michael Jerowski

Subject Matter Expert

Team

Mariane Olivan, Project Lead, UI/UX Designer (May 2024- August 2024)

Work Learn at the Emerging Media Lab at UBC

Undergraduate in Bachelor of Science in Cognitive Systems

University of British Columbia

Walker Rout, Developer ((May 2024- August 2024)

Work Learn at the Emerging Media Lab at UBC

Undergraduate in Bachelor of Science in Cognitive Systems

University of British Columbia

FAQ

As the title suggests