Documentation:24-1002 Augmented Reality Assisted Standard Operating Procedures (AR-A-SOP)
Introduction
| Emerging Media Lab | |
|---|---|
| |
| About EML | |
| A collaborative space for UBC faculty, students and staff for exploration of emerging technologies and development of innovative tools and solutions. This wiki is for public-facing documentation for our projects and procedures. | |
| Subpages | |
Augmented Reality Assisted Standard Operation Procedures (AR-A-SOP) aims to enhance student learning, safety, and confidence when operating machinery at UBC’s Centre for Advanced Wood Processing (CAWP) using Augmented Reality (AR). By leveraging the Apple Vision Pro headset, the project creates an interactive AR training program to guide students through the process of replacing the sawblade on the Martin T75 sliding table saw. This tool provides step-by-step, visually guided instructions that complement in-person demonstrations and allow for self-paced, hands-on practice, improving learning retention and providing a reusable training resource.
Primary Features
- Uses the Vision Pro's spatial capabilities to identify the physical sliding table saw and align digital content precisely to real-world components.
- Provides a full, guided experience for replacing a saw blade, including detailed visual instructions overlaid directly on the machine
- We support several modalities of instruction, from text and video, to augmented reality-based indicators
- Guards are implemented so that the user does not skip over any instructions during the repair
Intro: The user is taught basic controls for the Apple Vision Pro including eye movement, pinching/dragging, and clapping.
Calibrate: The user is instructed to look at the logo on the machine to calibrate the program.
Repair: The user is guided step-by-step through the process of replacing the sawblade on the Martin T75 sliding table saw. Digital overlays and visual cues highlight each task, from safely disengaging the machine and removing the existing blade to installing the new one and reassembling the components. Throughout the process, the system provides text and video instructions.
Design
The project's interface is designed using SwiftUI, ensuring a seamless and native experience on the Apple Vision. All user interface components are built with standard visionOS interface conventions. The immersive UI features interactive buttons, inline and embedded video playback, textual explanations, and virtual instructional entities. Each step is presented within a spatially anchored window, which users are free to move around and manipulate based on their active needs.
Development
This augmented-reality assisted tutorial was developed for the Apple Vision Pro using Xcode 16.2 beta, targeting visionOS 2.0. The application leverages SwiftUI for declarative interface design, RealityKIt for rendering 3D content in an immersive environment, and ARKit's image tracking capabilities to detect a "Martin" logo on the physical machine, establishing a spatial anchor that defines the coordinate system used to introduce "instructional entities" such as visual overlays, 3D markers, contextual indicators. We similarly use ARKit for hand tracking, enabling the system to detect user hand gestures and collisions with virtual objects to ensure a smooth tutorial experience.
User Research
A round of user tests was conducted at the CAWP at UBC. The goal of these user tests was to determine usability issues associated with the Apple Vision Pro and its unique learning curve. Overwhelmingly participants reported that the application would be applicable in a classroom setting. Accessibility concerns around eyesight is a valid issue, as many participants struggled to see clearly, even post-calibration. Every single participant* in our user test completed the tutorial; the majority did so in less than the allotted time frame. Participant feedback indicated that we should place greater emphasis on standard Apple Vision navigation and manipulation commands.
*One participant had to redo the user test, as they needed to put contact lenses in; they also successfully finished the tutorial.
Next Steps
- The application was designed to be extensible; it is almost effortless to support other machines in the future
- Tutorial text should change based on a user's selection of language
- Voice commands could aid in eliminating the overhead of manually changing between steps in the tutorial, particularly when users are directly interacting with the machine
Code
- We use Xcode's documentation generation to present the application's doc-comments in a clean and navigable fashion
File:ARASOP Xcode doccarchive.zip
Libraries
- Utilised John Haney's HandGesture repository for gesture recognition
First Time Setup Guide
Getting Started
1. Make an admin account on some Mac
- You will need somewhere around ~30GB available space to install Xcode with Apple Vision Pro (AVP) component
2. Navigate to Safari, open https://developer.apple.com/xcode/resources, and download Xcode 16.2 from the Mac App Store
- Sign into the UBC Studios developer account, use the AVP to complete 2FA (the target account is signed into the AVP)
3. Open Xcode application (use cmd + spacebar and search Xcode)
- In the initial "Download and Install" popup, select the 'visionOS 2.2' component to install (or whatever version is available)
- We recommend disabling "Predictive Code Completion Model" as well...
4. Then click "Download & Install"
- Enter the account's admin password if prompted
- If you see a 'What's New in Xcode' window after install finishes, just click continue
5. Clone GitHub repository at https://github.com/ubceml/24-1002-arasop by clicking the 'Clone Git Repository' and signing into GitHub using your account user name and a personal access token with access to the repository (you may need to create a new personal access token (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)
- Once signed in, select path to store cloned files and continue
- If you know how to clone a repository using git CLI feel free to do so instead, you just need to double click the xcodeproj file to open the application with Xcode...
Building
1. Open the project from the cloned repository by clicking the on xcodeproj file
2. Turn on the AVP, navigate to Settings > General > Remote Devices and remove any computers listed
- The AVP Remote Devices settings window must be open to pair with the computer
3. Connect to same wifi network on AVP and Mac
4. In Xcode, navigate to Window > Devices & Simulator, and click "Pair" on the visible apple vision pro that pops up (ensure the AVP is on and is opened to the Remote Devices screen)
5. Enter the code that pops up on the AVP into the Xcode dialog that opens on the mac
6. Once connected, the computer may cache some files on the apple vision for a bit, ensure that the AVP is on and unlocked (ideally you should wear it while it sets itself up...)
- If the Xcode Devices & Simulators window says "Apple Vision Pro needs to be prepared for development", just wait it should disappear
7. Select "Apple Vision Pro" as the build target at the top of Xcode, to the right of the run button (make sure you dont select the simulator, you are selecting the build target to be the AVP you just connected..)
8. Attempt to build the project by clicking the Run button (or pressing cmd+b)
9. The build should fail with errors. Click on the errors at the top of Xcode to go to the error page. You should see "No Accounts: Add a nfew account in Accounts settings" and "No profiles for 'ARA-SOP' were found: Xcode couldnt find any iOS Appe Development provisioning profiles matching 'ARA-SOP'"
10. Then double click the above error messages to navigate to their locations, you should end up in the Signing & Capabilities page, you must enter a value for the 'Team' field under 'Signing'. If there are no existing 'Team's, then click 'Add an account' and sign in with the same apple id you used to download Xcode. You may need to do 2FA on the AVP, allowing the sign in and providing a code to enter into Xcode. **At the top of 'Signing', ENSURE 'Automatically manage signing' is ticked true**
11. Build again with the 'Run' button (ensure you are still on the same network for both AVP and Mac computer running Xcode)
Extraneous bugs
- Any problems with 'Bundle Identifier', in the 'Signing & Capabilities' page, if you have problems, just change the name/bunder identifier to a value you have not used previously (ie... if bundle identifier was 'ARA-SOP', try changing it to 'ARASOP', aka remove the hyphen)
- Old builds expire, if you see 'ARA-SOP is no longer available' on the AVP when you click on the app, just rebuild
- When moving between computers or building on new/different devices to build the project **ensure you delete the previous instance of ARA-SOP from the AVP homescreen**. If you encounter bugs building, this may be the issue. (ie... if you want to build/compile the project using a laptop, you should delete a desktop's build before you attempt to click 'Run' on the laptop...)
- If you build and it says you need to trust something, you may need to 'Trust' an account/certificate in the AVPs Settings > General > VPN & Device Management
Poster
File:AR-A-SOP Showcase Poster.pdf
Development Team
Principal Investigator
Jörn Dettmer
Associate Professor of Teaching
UBC Faculty of Forestry
University of British Columbia
Current Team
Walker Rout, Project Lead, Software Developer (September 2024 - April 2025)
Work Learn at the Emerging Media Lab at UBC
Undergraduate in Bachelor of Science in Cognitive Systems
University of British Columbia
Kai Zhang, Software Developer (September 2024 - April 2025)
Work Learn at the Emerging Media Lab at UBC
Undergraduate in Bachelor of Science in Computer Science
University of British Columbia
Bibliography
- Apple visionOS Documentation, https://developer.apple.com/documentation/visionos/
- John Haney, Hand Gestures, https://github.com/johnhaney/HandGesture
License
MIT License
Copyright (c) 2025 Jörn Dettmer
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
