Introduction

Emerging Media Lab
About EML
A collaborative space for UBC faculty, students and staff for exploration of emerging technologies and development of innovative tools and solutions. This wiki is for public-facing documentation for our projects and procedures.
Subpages
	Main EML wiki page; Public EML website; EML Resources; EML Onboarding; EML Offboarding; EML Operations; EML Projects;

Background

Vancouver neighborhoods such as Kitsilano and Strathcona were home to many poets that experimented with the medium throughout the 60s and 70s. Judith Copithorne was one such figure who used illustration to create whimsical concrete poems (poems where the words resemble a specific shape on the page).

Emerging technologies – specifically Virtual Reality (VR) and Large Language Models (LLMs) – present new opportunities to experiment with the visual representation of poetry in a three-dimensional way.

This project intends to recontextualize poetry as an art form that can exist in a tangible space. The Emerging Media Lab aims to create an educational VR experience for Creative Writing students that pays homage to the experimental work of Judith Copithorne and allows users to create procedural poetry.

Objective

The primary objective of this project is exploring the implementation of “concrete poetry” in a VR space, creating a unique and interactive learning experience for Creative Writing students by engaging with new forms of media.

The team’s main challenge is to create a developmental framework and a design direction that explores concrete poetry and its relation to space, both in terms of 3D space and the poet’s physical location.

The EML project team has addressed, and surpassed, these objectives through its rendition of the Poetry Funhouse – an experience where users will collect randomized words and compose poetry in a three-dimensional space. This is an auditory and visual experience that experiments with time, space, and the relationship between a poem and its environment.

Format and Versioning

Procedural Poetry Funhouse (EML Prototype)

This project uses Unreal Engine 5.3.2, Meta Quest Virtual Reality Headsets, and integration with LLMs (OpenAI).

Primary Features

Here is a table of the proposed features for this term and their level of completion:

No.	Task	Priority	Status
F1	Design direction and concept	Must have	Complete
F2	VR framework for implementation	Must have	Complete
F3	Immersive external VR environment	Nice to have	Complete
F4	Activity rooms	Nice to have	Partially Complete
F5	Ability for users to collect word blocks	Nice to have	Complete
F6	Ability for users to compose poems	Nice to have	Complete
F7	Audio layer	Nice to have	Complete
F8	Tutorial	Nice to have	Complete
F9	User Interface	Nice to have	Partially Complete
F10	AI Sentiment Analysis	Nice to have	Complete

Feature Descriptions

F1. Design Concept and Direction

The Funhouse is a virtual environment based on the Arbutus Grocery building that takes inspiration from Dadaism, whimsy and Meow Wolf to create an experimental and whimsical environment.

The Funhouse is composed of rooms the user will go through to create a poem. In each room, users will be faced with choices that affect the poem they write. The user will encounter these environments in the following sequence:

Onboarding screens: The user is onboarded and introduced to the project concept through a series of virtual panels explaining the project’s historical context.
Outside Arbutus Grocery: Modelled after the actual neighborhood of Arbutus Grocery, the user will spawn in this environment in front of the Arbutus Grocery building and learn the movement controls before moving to the Lobby.
Lobby/Frothing Room: The Lobby is an environment symbolizing Arbutus Grocery’s change from a grocery store into a coffee shop. It sets the tone for the experience and guides the user to the first activity room.
Activity Rooms: These rooms use certain parameters to shape the output of the user’s poem. It either constrains or provides outputs based on the rules of each room. For the details of each room, see F4.
Studio: The studio is the space where users will compose their poems and observe how their poem changes the environment based on AI sentiment analysis. Users will visit the Studio after each activity. The Studio contains its own features, such as the Common Word Shelf, and the Audio Selection.

F2. VR Framework for Implementation

The framework utilizes Unreal Engine 5.3.2 as the rendering engine and Meta Quest SDK for input handling. Features such as player controls, level generation, and game mechanics are developed in Unreal Engine Blueprints and C++. Data handling is through importing .csv files and functions as data tables within the engine.

For Large Language Model (LLM) integration, the Funhouse uses Unreal Engine's WebSocket Client in C++ to establish a communication protocol for built AI assistants using OpenAI's Assistants API.

F3. Immersive VR Environments

Environments are modeled in Blender and integrated in Unreal Engine. The post-processing effects used are toon shaders made by ymt3d's Stylized Post Process shaders. These are tweaked based on the goal of the experience.

A significant portion of 3D assets were provided from free models on SketchFab and TurboSquid.

Various audio tracks were also created. Please see F7 for details.

F4. Activity Rooms

Beyond the Lobby and the Studio, this project focused on experiential learning, with users learning by creating visual poetry. These rooms contain activities that experiment with randomness, where following the same procedure can create immensely different results.

The design concepts for three activity rooms are listed below:

Word Hunt Maze: In the word hunt maze, users will navigate a randomly generated maze while collecting word blocks. This dynamic maze visually replicates the chaos of ideas scattered throughout a labyrinthian mind. The maze’s size is set to 16x16, meaning there are over different possibilities for the generated maze.

Arcade: Inspired by the n+7 poem activity, users should be able to gamble away their collected words in slot or gacha machines. In turn, they will receive new, random words, that might change the meaning of their poem.

Un-Blackout Room: In the Un-Blackout room, users will enter a dark room containing their poem. They will have a flashlight with low battery and must find as many of their words as possible before the power runs out. This activity combines the concept of blackout poems and Slice and Dice.

At the time of handoff from the EML, the Word Hunt Maze is fully implemented. Arcade and Un-Blackout Room are conceptualized but not implemented.

F5. Ability to Collect Word Blocks

Words are represented as 3D objects called Word Magnets in the Poetry Funhouse. This can be seen in the Word Hunt maze and the Studio. Users can collect these words and store them in their ‘inventory’, or Word Bank. The Word Bank is a word wheel that pops up as a 3D widget from the left controller, allowing users to select words from their inventory.

If the user has a word magnet in their hands, they can either place it in the space or return it to their word bank.

F6. Ability to Compose Poems

In the Studio room, the user can use words from their word bank and move or place them in any direction, location, and angle, allowing them to compose a poem in a three-dimensional space.

F7. Audio Layer

Each level or room has a different corresponding audio track. These tracks combine spoken word excerpts from poets Lionel Kearns and Judith Copithorne with instrumental music. All tracks and recordings used the software Audacity to provide an old-timey radio feel. For the vocal tracks, a filter-curve EQ was applied with the audio’s low ends and high ends cut out. Additionally, compression under the ‘Podcast/Radio’ preset was applied.

There is also a music player in the Studio where users can revisit these audio tracks or choose a public domain BGM to control the mood of their poem creation phase.

F8. Tutorial

Users are onboarded through a series of UI widgets that provide context about the project’s inspirations: concrete poetry, the Vancouver poetry scene, and Judith Copithorne. Users are then introduced to the activities that will take place in the experience – mainly, collecting words and composing their poem.

F9. User Interface

The team has designed User Interface widgets and components that guide the user through the experience. This includes the user tutorial, signs throughout the environment, tooltips, and menus. Radios are present in the levels to provide audio visual descriptions of the instructions. The Settings menu has not been implemented yet.

F10. AI Sentiment Analysis

The AI sentiment analysis is to evaluate user poems. Currently, it is being used in the Studio room to evaluate the 3D poem and produce an output that changes the color tint of the environment. The set of words in the space is then analyzed based on their location and distance with other magnets to establish the poem’s structure and tone.

Design Methods

The team used personas and user flows to guide the design of the application in Figma. With the given design inspirations provided, the team created room concepts, style guides, and high-fidelity prototypes of the experience in Figma. Blender and Unreal engine were used to create custom assets and design the in-game environments.

Development

Technical Components

The following table describes the software used for design and development:

Tool	Description
Unreal Engine	Game engine, version 5.3.2
Blender	3D modelling software to design custom assets
Meta Quest 3	Virtual Reality headset used during development
Figma	Design and prototyping software used to design wireframes and UI components
Audacty	Audio tracks edited with audacity
ChatGPT	GPT4 and GPT4o were used for development, https://chatgpt.com/g/g-f52QYAJK1-unreal-engine-5-expert Unreal Engine specific GPT.

Project Structure

Github Branches

Our repository can be found here:

Please refer to the table below for a description of all the branches in the repository.

Active Branches

Branch Name	Description

Stale Branches

Branch	Purpose	Last Modified Date	Modified By	Build/Demo Ready?

Issues/Bugs

Auto-Jump Bug

The teleportation movement sometimes causes the player to teleport in a higher z-value. While we did implement an auto-jump feature, this bug does not use that function but still produces an auto-jump effect. Note that the auto-jump feature can be tracked based on the sound effect implemented with it.

Word magnet Position and Scaling Issues

When dropping words outside the Studio level, they land too close to the player camera.

Since the Lobby level also is given the chance to drag and drop words, words get rescaled inappropriately to different sizes.

When dragging words using the line trace feature, there are times where the word stops being dragged unintendedly.

When two or more words are dragged at the same time, the only way to separate them is to collect all of them again in the word bank.

Off Zones and Out of Bounds

When the player goes out of bounds, there are times where they do not respawn as intended. Players can also clip through walls and floors if they spam multiple teleportation or movement inputs.

Sentiment Analysis Overload

Players that produce multiple actions in poem creation might overload the Web Socket module of the server, causing late feedback from the AI. A stop gap such as a deloader can be used to mitigate this.

Hover State Bug

The onboarding screens have a bug that makes the button hover states trigger at the wrong times

Stuck in Open Sesame

When snapping to the Open Sesame area, users can sometimes get “soft locked” in an uncomfortable position (i.e. too close to the words). They cannot teleport out.

First Time Setup + General Usage

Running the Project (from Unreal Engine)

Setting up Unreal Engine

Below are instructions on how to install Unreal Engine 5.

Visit Download Unreal Engine - Unreal Engine and install the Epic Games Launcher
After following the steps to install the launcher, select Unreal Engine.
Go to the Library Tab and press the “+” Icon. Choose Unreal Engine 5.3.2
In Folder, choose where to install unreal engine. It is recommended not to change this if you have the space on your primary drive.
In path, click on options, and unselect the Android, IOS, and Linux platforms. This will save you about 22 GB in installation.
Let the install happen, and then run. The first time you run Unreal Engine, it will take a while so be patient!

Plugin Installation

Please install the 3D Text plugin to be able to work on 3D words. Ensure it is installed as this first as this is the main mechanic used on two of the three main levels. To install:

Navigate to Edit>Plugins
Search Text 3D and install it.

Setting up Oculus

Visit the Meta Quest website and scroll down to Quest 3 and click on download software.
After pressing download software, follow through with the instructions to install it.
On your mobile phone as well, install the Oculus App.
Turn on the Quest 3 and login to your account both on the headset and the Oculus pc app
Connect the headset over AirLink (same network) or via Cable to the computer Note : make sure the USB-C to USB-C cable is 2 way data capable
Open the oculus Rift app through the quick setting menu on the headset
When game is deployed, the headset should automatically open up the VR game

Project Launch in VR

Copy the entire Poetry project from an SSD/Perforce onto a local machine
Connect a quest 2 or quest 3 to desktop and launch the meta quest link on the headset and the desktop.
Find the Poetry.uproject file inside the Poetry directory copied in step 1. Right click and go to: More Options->Launch Game
After ensuring the headset is properly connected, Click on the button “Launch in VR”
To exit the program, click "Esc" on the keyboard.

Running the Project (Packaged Build)

Setting up Oculus

Visit the Meta Quest website and scroll down to Quest 3 and click on download software.
After pressing download software, follow through with the instructions to install it.
On your mobile phone as well, install the Oculus App.
Turn on the Quest 3 and login to your account both on the headset and the Oculus pc app
Connect the headset over AirLink (same network) or via Cable to the computer Note : make sure the USB-C to USB-C cable is 2 way data capable
Open the oculus Rift app through the quick setting menu on the headset
When game is deployed, the headset should automatically open up the VR game

Project Launch in VR

Run Poetry.exe

Challenges

To do

Future Plans

The team suggests recommendations for future development of this project:

Mandatory Lobby Tutorials

Currently, the Lobby tutorial is set up in a way that users must enter trigger boxes in order to engage with the tutorials. A consideration for these areas is to have a “blocking” mechanic to force new players to engage with these tutorials in a linear manner. With this suggestion, there should also be a way for experienced users to skip this tutorial entirely.

Adding Interactive Easter Eggs

Due to the lobby having the most asset heavy level (as activity rooms such as maze and studio focuses on gameplay rather than styling), it is suggested to add easter eggs or other features in this area. These may include:

Gallery of created poems in the studio by past players

NPCs

Minigames that allow players to familiarize themselves with various game mechanics.

Development of Activity Rooms

-- The team has created conceptual designs for the implementation of future activity rooms.

Expanded Maze Door Functionality

The are four doors in the lobby that lead to the maze room. The door that is entered determines which post processing volume to be set. An initial, but unimplemented idea is for the doors to also determine the general mood of the Word Magnets that spawn in the chosen maze.

Word Magnet Rotation

The word magnets rotation is currently in a fixed state based on the random rotation value provided by the word spawner, making most words not facing the direction of the user’s FOV. It was suggested to have a dynamic rotation value instead so that the words would always face the user.

It is also suggested to have the word magnet’s color value be based on what color it will contribute to produce in the Sentiment Analysis’ output.

Due to heavily stylized postprocessing volumes, it was suggested that the word magnets should not be affected by the styling to accentuate its appearance. This can be done by creating or setting an unlit material for it.

Word Bank and Input Mappings

Currently, the word bank can only be spawned on the left hand of the user. It is suggested that it should be spawned with both hands.

The Common word Shelf and Music player blueprints also rely on input mappings heavily on the left controller. This is because the right controller is used to move away from their collision boxes.

Exporting Poems

There is a prototype that can export poems created by the user in ReadWriter.cpp (text exporter and importer) and CSVUpdater.cpp (CSV/Data Table manager). By connecting the poem created in the studio, these can export a *.txt file with an address mapped in the CSV file connected to a data table. A consideration of this feature would be for users to save their created poems outside the game. Since this is just a text file, fonts and stylized structures are currently not considered.

Sentiment Analysis

An opportunity for future research is to look into how else one can use the LLM integration and AI feedback to manipulate the Studio environment in ways besides just the color settings.

Additionally, adding more categories for words in the word bank would mean that the AI will be able to produce more variation of feedback. These can be adjusted from word_bank_1 data table.

User Interface

There are a few unimplemented features from the high-fidelity Figma prototype that should be implemented:

Add Settings menu, which includes pause, restart, and game settings. The UI panel has been designed in Figma. See Section J.1

Add a skip button during the onboarding sequence

Add the “Settings” and “Exit” buttons to the starting screen

Minor suggested changes to improve user experience include:

Adding a pointer indicator. This will be especially useful for the player during the onboarding.

Adding audio – music and/or voice acting - during the onboarding section

Usability testing

The team recommends that formal usability testing should be conducted, especially in the following areas:

Intuitive controller input mappings

Clarity of instructions/contextual information

Usability in the poem creation process

Poster

As the title suggests

Team Members

Principal Investigators

Dr. Bronwen Tate

Associate Professor of Teaching

School of Creative Writing

University of British Columbia

Jen Moss

Lecturer

School of Creative Writing

University of British Columbia

A.E. Osworth

Lecturer

School of Creative Writing

University of British Columbia

Ray Clark

Subject Matter Expert

Team

Mariane Olivan, Project Lead, UI/UX Designer (May 2024- August 2024)

Work Learn at the Emerging Media Lab at UBC

Undergraduate in Bachelor of Science in Cognitive Systems

University of British Columbia

Walker Rout, Developer (May 2024- August 2024)

Work Learn at the Emerging Media Lab at UBC

Undergraduate in Bachelor of Science in Cognitive Systems

University of British Columbia

Jiho Kim, UI/UX Designer (May 2024- August 2024)

Work Learn at the Emerging Media Lab at UBC

Undergraduate in Bachelor of Science in Cognitive Systems

University of British Columbia

Samia Sajid, UI/UX Designer (May 2024- June 2024)

Work Learn at the Emerging Media Lab at UBC

University of British Columbia

James Edralin, Developer (May 2024- August 2024)

Work Learn at the Emerging Media Lab at UBC

Undergraduate in Bachelor of Science in Computer Science

University of British Columbia

Julien Roy, Developer (May 2024- August 2024)

Work Learn at the Emerging Media Lab at UBC

Undergraduate in Bachelor of Science in Computer Science

University of British Columbia

Ameya Goel, UI/UX Designer (May 2024- August 2024)

Work Learn at the Emerging Media Lab at UBC

Undergraduate in Bachelor of Science in Computer Science

University of British Columbia

Asset Credits

"Antique Book Set" (https://skfb.ly/oFYn6) by FrodoUndead is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Antique Desk" (https://skfb.ly/o8Ptu) by Matthew Collings is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Antique Mantel Clock" (https://skfb.ly/oAoBA) by Matthew Collings is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Assetsville Town” (https://www.unrealengine.com/marketplace/en-US/product/assetsville-town) by Assetville is licensed under Epic Content License Agreement (https://www.unrealengine.com/en-US/eula/content)

"Rendered Coffee Cup" (https://skfb.ly/o7DMZ) by cosmic.arin is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Bankers Lamp” (https://skfb.ly/orsMy) by Zgon is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Burger King Paper Bag" (https://skfb.ly/6Xvrs) by RPSebb is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Bed low poly" (https://skfb.ly/oJnsY) by santifaster12 is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Catastroffee - Cat Paws - VR Hands” (https://skfb.ly/oIH7o) by Lena Pauly is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Ceiling Light” (https://skfb.ly/6WXO9) by Heliona is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“CHAIR” (https://skfb.ly/WuNM) by uVv is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“☕coffee cups low poly☕” (https://skfb.ly/oISw6) by gulvtaepper is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Coffee Cup” (https://skfb.ly/6vRtV) by Lasse Harm is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Coffee Kiosk” (https://skfb.ly/69KJI) by joelandy is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Coffee Tables Four Legs with Rattan I Low-Poly" (https://skfb.ly/oqGRo) by Pascal Garten is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Coffee Menu”(https://skfb.ly/oyGxA) by Zende licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Coffee and donuts” (https://skfb.ly/6AXsU) by Ivan Dnistrian under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Commercial Coffee Machine” (https://skfb.ly/6G7Mv) by M.Reslan licensed under Creative Commons Attribution

"Dusty bookshelfs" (https://skfb.ly/orv9K) by Meanphrog is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Edith Finch: House and Common Areas” (https://www.unrealengine.com/marketplace/en-US/product/ef-house) by Epic Games is licensed under Epic Content License Agreement (https://www.unrealengine.com/en-US/eula/content)

“Floor Lamp” (https://skfb.ly/otBoo) by Agha.Najam is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

“Grovery Trolley” (https://skfb.ly/oqRDS) by korep.daneh is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Marlboro Gold" (https://skfb.ly/orZE6) by Tiago Lopes is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Messy Tack Board" (https://skfb.ly/6TLMA) by M.Reslan is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Mid century design armchair oneseater cushion" (https://skfb.ly/oQrJu) by fuchs&bär is licensed under Creative Commons Attribution-NonCommercial (http://creativecommons.org/licenses/by-nc/4.0/).

"Scattered Papers" (https://skfb.ly/opFKv) by aysenaz is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Old vintage mid century Sideboard fully PBR" (https://skfb.ly/oMuMJ) by lightpixel is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Mid century sofa danish design cosy blanket" (https://skfb.ly/oPPIU) by fuchs&bär is licensed under Creative Commons Attribution-NonCommercial (http://creativecommons.org/licenses/by-nc/4.0/).

"Soviet Cupboard" (https://skfb.ly/oCEvv) by sergeilihandristov is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Vintage Stereo Hi-Fi Stack w/ Speakers" (https://skfb.ly/6ADNG) by Glowbox 3D is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Vintage Refrigerator" (https://skfb.ly/oOFtQ) by 3D_Films is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Vintage Wooden Table" (https://skfb.ly/oButK) by Poring is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Table Vintage" (https://skfb.ly/oPTXZ) by LeanAzuos is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

"Wooden Bookcases with Books" (https://skfb.ly/oJpQJ) by Matthew Collings is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).

License

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.