Documentation:MindCraft

Emerging Media Lab
About EML
A collaborative space for UBC faculty, students and staff for exploration of emerging technologies and development of innovative tools and solutions. This wiki is for public-facing documentation for our projects and procedures.
Subpages
	Main EML wiki page Public EML website EML Resources EML Onboarding EML Offboarding EML Operations EML Projects

Introduction

Mindcraft is an experimental EML project that explores ways of generating personalized virtual environments that respond to the user's brainwaves. The project combines room-scale Virtual Reality with Brain-Computer Interfaces (BCI). BCI are systems that read signals generated by the user’s brain and translate them into inputs for a computer program. We use the Muse headband, a consumer-grade wearable EEG device, to record electrical activity from the the wearer’s brain and send it to the program that changes the virtual environment in response to that activity.

First Time Setup Guide

Requirements

• VR-capable PC
• Windows 10

Muse Direct Installation

In order to get the full Mindcraft experience, you need a Muse headband (the EML has one of those). When you set up Mindcraft on your machine for the first time, follow the steps outlined below:

1. For the headband to be able to send data to Unity, you will need to install Muse Direct. This app can be downloaded for free through the Windows Store.
2. Once Muse Direct is installed, open it and you should see a screen like this (you will probably not see any devices in the list):

1. Click the 'Add' button next to 'Output To'. Set the nickname to 'Unity' and the destination to 'OSC UDP'.
2. You will now see a menu like this one:

• • Change the IP to 127.0.0.1 - this is a shortcut for the local IP, since the computer that receives Muse data is the same computer as the one that has Mindcraft running.
  • Change the Port to 5000 - this is a bit arbitrary - if you're running Mindcraft from Unity, it doesn't matter what this port is as long as it matches the Input Port of the OSC script. If you're running from the build, the port will be set to 5000, so just leave it at that.
  • For the Prefix, choose the Custom Static Text option and set it to /muse - it is important to have the slash in front so that the message is recognized as a valid OSC message, and addresses are set to start with '/muse' by default.
  • For Output Data and Output Algorithm, check the EEG Data and Absolute Band Powers boxes as these are the values that Mindcraft currently uses - you could also just check all the boxes to be futureproof.

In the end, your settings should look like this:

Running Mindcraft With Muse Direct

Once you have the Muse Direct output set up, you will need to follow the steps below to run it:

1. Click on the Muse headband you're curently wearing in the list of devices to select it
2. Uncheck the first two outputs and check the output you created so that it looks like this:

1. Click on the Bluetooth button next to your headset name to start connecting
2. Check the 'Info' tab to see if you're connected or not - if you are, you can now run Mindcraft! 😎

Primary Features....

Brain-painted cubes

• The environment has EEG-responsive cubes scattered around
• More cubes can be spawned by pressing a button in the environment or by pressing the grip on the right controller
• When the user picks up a cube, 4 of its faces will be painted with dynamically generated spectrograms for each of the 4 sensors on the headband
• The texture is saved when the user releases the cube

Modifying terrain with your mind

• Pressing the touchpad on the left controller will modify the terrain at the location the controller points to
• The direction (raising or lowering) and rate of change of the terrain are based on the relative alpha power averaged over the 4 sensors of the headband

Changing sky colour with your mind

• The skybox colour will gradually change based on the relative alpha power averaged over the 4 sensors of the headband

Changing background music with your mind

• The background music will speed up or slow down based on the average alpha to average theta ratio

Switching environments

• The experience contains a main menu where the user can switch to one of the implemented environments (currently only one is implemented)

Basic VR features

• Teleporting
• Player body physics
• Grabbing and throwing objects

Lessons learned

• When incorporating external hardware (e.g biometric sensors) it's important that there are as few connectivity issues as possible; alternately, the experience should be enjoyable even without sensor feedback

Known Issues

Muse Direct Issues

• Muse Direct only works on recent updates of Windows 10 (no Mac or Linux support)
• There seem to be persistent Bluetooth connection problems with the Muse Direct app. However, if you lose connection, the following steps usually help reconnect:
  • Close Muse Direct.
  • Go to your computer's Settings --> Bluetooth & Other Devices.
  • Choose your Muse headband in the list of paired devices (it will be called Muse-{4-letter code}) and click 'Remove device'.
  • Turn Bluetooth off and on.
  • Restart Muse Direct and try reconnecting.

Development Team

• Adam Herr
• Asia Schmok
• Yana Pertels