Documentation:23-3002 Tapestry Tool ML
| Emerging Media Lab | |
|---|---|
| |
| About EML | |
| A collaborative space for UBC faculty, students and staff for exploration of emerging technologies and development of innovative tools and solutions. This wiki is for public-facing documentation for our projects and procedures. | |
| Subpages | |
Emerging Media Lab Project Documentation
Introduction
Background
Tapestry Tool is an open-source online learning platform that allows users (such as students and instructors) to collaborate in the development of interactive, associative, and multi-modal content. Tapestry encourages a non-linear approach to learning as tapestries are formed of connecting nodes, which represent ideas through text, images, videos, or H5P content like an embedded quiz, presentation, flashcards, and more. Essentially, it takes the form similar to an interactive, non-linear visual mind map.
It is the brainchild of principal investigator Dr. Steven Barnes and is supported by the University of British Columbia (through the TLEF grant) and the Microsoft Accessibility Team. Currently, it's used by students, TAs and instructors to collaborate on multimodal, interactive content in an effective learning visualization.
For more on this, check out the Tapestry Tool website: https://www.home.tapestry-tool.com/
Objective
The objective of the project here at EML is to develop a ML (machine learning) integrated version of this platform that allows for a Tapestry (web of interconnected content) to be automatically generated based on a given topic, prompt or PDF. It is important to note that the goal of automatic tapestry generation is not to produce a perfect Tapestry, but one that encourages editing and pruning.
Our project introduces a completely novel approach to mind mapping tools by allowing for the automatic generation of a Tapestry. The Tapestry is reminiscent of a visual mind map and as such, an autogenerating mind map tool has exciting implications for education. Students and instructors alike will now be able to generate a text based Tapestry by prompting the tool or uploading a PDF (e.g. the abstract of a research article, first couple pages of their lecture slides, etc.).
Format and Versioning
Tapestry Tool is a Wordpress plugin that uses Vue, PHP, JavaScript and HTML/CSS. It is only accessible through Wordpress (however, one of our prototypes is also accessible as a website - prior to use, please reference the User Manual.)
Tapestry Tool ML (Our prototype)
Our prototype for our project is based on the Tapestry Tool 2.0 Version, the Wordpress plugin. This is the open source version of Tapestry.
For demos and to showcase our prototype, it is hosted on a website.
This iteration of Tapestry is designed and developed by our EML team.
Tapestry Tool 3.0 Version
Tapestry Tool 3.0 is the latest iteration of Tapestry and is also hosted on Wordpress as a plugin. It is not open source but free to use for UBC community members.
This iteration of Tapestry is developed by Dr. Steven Barnes' team.
Tapestry Tool 3.0 ML
Tapestry Tool 3.0 ML is the future ML integrated version of Tapestry. It will include some aspects of the Tapestry Tool 2.0 ML prototype that we created in terms of some UI changes and features, but will mostly be a different tool.
This iteration of Tapestry is developed by Dr. Steven Barnes' team, but the ML integration is developed by our EML team and some UI features are designed by our EML team as well.
Primary Features
Prompt to Tapestry generation
The prompt to Tapestry generation feature is based on the original concept of automatically generating a tapestry, or web of interconnected content, when given a topic.
From a prompt
The user is able to prompt the tool with text up to 2048 characters. This can take the form of inputting a topic of their choice, e.g. "Role of machine learning in the workplace", multiple paragraphs from an article, or the abstract of a research article, etc. Although the model is trained on information prior to 2021, we've found that it does well with extracting information from recent (2023) news articles and summarizing it.
From learning outcomes
The integration of learning outcomes into the prompting tool was a priority due to the support it would give instructors.
Hence, the user is able to prompt the Tapestry with a learning outcome that they want to achieve through the generated Tapestry. This will follow a sentence format as follows: (e.g. "By the end of this Tapestry, I would like the student to have an understanding of <topic>.")
PDF to Tapestry generation
The user is able to upload or drag and drop a PDF for the tool to generate a Tapestry based off it.
There are some limitations - this feature usually works best if the pdf is just 1-3 pages due to the limitations with text input into ChatGPT. The PDF should be primarily text based content, and ideally all of it should be text based.
Strength of relevancy connections
As the Tapestry is composed of nodes and their connections to each other, the idea of exploring the relevancy of the autogenerated connections to each other was proposed. We were able to automatically generate connections from grandchildren nodes to each other.
And by using the Open AI API to compare the similarity of two texts, these nodes would then be sorted by the strength of their relevancy to one another. Each connection between grandchild nodes that was automatically generated represents a comparison value. The color of the connection represents that value. Lower comparison or correlation values would be represented in teal, medium in light blue, and the strongest would be represented in navy blue.
Other features
Our ML prototype also consists of features carried over from Tapestry Tool 2.0, including the ability to create your own Tapestry from scratch. It includes all the functionality of 2.0, including the ability to edit nodes and add in multimodal H5P content, change node colors, view the nodes in different layouts thanks to a slider, etc.
Methods
Design
The UI/UX design of the prototype was created in Figma, and Miro was used to formulate user flow. Assets were used from previous designs of Tapestry Tool, but the main components of the primary features of Tapestry Tool ML were designed from scratch.
User Research
As gaining a better understanding of our prototype and audience is extremely crucial to our project, we conducted two rounds of user testing and interviews.
June 2023
Our first round of user research in June was more centered on user interviews and getting a sense of our user group and their thoughts on our initial prototype. Our objectives for this round were as follows:
- Gain insights on what users of varying levels of experience with Tapestry Tool expect from a ML-integrated Tapestry Tool.
- Observe and understand how participants interact with our Figma prototype.
- Conduct A/B testing to understand how participants respond to node relevancy connections.
- Observe and understand how participants interact with our current prototype of the ML Tapestry Tool.
Our user group was composed of students, RAs (research advisors), and professors at the University of British Columbia. The students had never interacted with Tapestry before, however, the RAs and professors had.
We also conducted a round of usability questionnaires with questions from the System Usability Scale (SUS) and USE (Usefulness, Satisfaction and Ease of) scales.
Through this initiative, we received helpful feedback that guided our next steps in terms of pinpointing features that our users would like, UI changes for better user experience, etc.
August 2023
The goal of our second round of user research was to conduct usability testing on our MVP. Our objectives for this round were as follows:
- Observe and understand how participants interact with our current prototype of the ML Tapestry Tool.
- Understand how participants interact with the prompting feature and identify any usability issues.
- Understand how participants interact with the PDF feature and identify any usability issues.
- Understand the impact of our new features and UI changes (e.g. strength of relevancy, highlighted nodes, etc.)
Our participants were composed of all students and 6 out of the 7 interviews were conducted in person (1 was conducted over Zoom by proxy). Participants took an exit questionnaire afterwards composed of questions from the System Usability Scale (SUS) and USE (Usefulness, Satisfaction and Ease of) scales, in order to measure the usability and user experience of our platform.
Our SUS score with outliers was 78.9, which is an A- and the score without outliers is 84.2, which is an A+ and the highest percentile. This was a remarkable improvement, as in June, our SUS score was 67 (below average). This was an improvement of 12 percentage points.
Our average USE score was 5.87 (out of 7). We scored 5.64 on Usefulness (the only section measured last time - the score was 4.65 in June), 5.71 on Ease of Use, 6.25 on Ease of Learning, and 5.89 on Satisfaction.
We also gained valuable insights on which elements to include based on the users' interactions with the usability tasks and were able to make immediate UI changes.
Development
We used Vue, JavaScript, Python and HTML/CSS. The versioning we are working with is Tapestry 2.0
Machine Learning (ML) Integration
The OpenAI API was used to produce the text to populate each node. The "gpt-3.5-turbo" model was used.
The following API calls, excluding a call to summarize root node text for larger user inputs, will be made to the OpenAI Chat Completions endpoint for each tapestry generated. The following is a breakdown of the calls made:
- Generating child node text as a single string separated by the "~" character, then splitting the string up and populating the nodes programmatically.
- Generating grandchild node text as a single string separated by the "~" character, then splitting the string up and populating the nodes programmatically.
- Comparing the text within each grandchild node and producing a comparison value between 0-100 to represent the strength between the nodes (0 being unrelated and 100 being strongly related). The comparison values are generated as a single string separated by the "~" character (e.g. "23~46~12~9~34~100~98~67"), then split up and used to determine whether a link will be generated between two nodes. If the comparison value is above an arbitrary threshold of 70, then a link would be created between two nodes.
Tapestry Tool API
The Tapestry Tool API allows developers to perform a variety of actions to create, delete, and edit a Tapestry programmatically. This API was used to generate nodes with certain properties, including color, text, and position. The API was also used to generate links between nodes that had related text.
Github Branches
Our repository can be found here: https://github.com/ubcemergingmedialab/23-3002-Tapestry .
Please refer to the table below for a description of all the branches in the repository.
Active Branches
| Branch Name | Description |
|---|---|
| main | The default branch that has the most up-to-date stable code. |
| ui-changes | The branch containing functional code with UI changes made before final round of user interviews (August 2023). |
| generating-tapestries | The branch containing code for the original function for automatically generating tapestries. |
| link-connection | The branch containing code for generating links between sibling nodes based on their correlation value. |
| remote-site | The branch containing code relevant to the remote website, tapestry.emlx.ca. |
Stale Branches
Most of the stale branches have been deleted.
| Branch | Purpose | Last Modified Date | Modified By | Build/Demo Ready? |
|---|---|---|---|---|
| front-end | Front end formatting of the welcome page | June 2, 2023 | Victoria Lim | No |
Issues/Bugs
These are the issues/bugs across both the local prototype and the website prototype that we are aware of and working on.
- Tapestry glitches on first use when submitting a prompt: If there is a glitch when you submit your first prompt using the "Write a Prompt" feature, simply delete the root node and try again.
- When clicked, node jumps to the top right corner of the screen: This is a glitch that sometime occurs when your are viewing the Tapestry and clicking on a node, especially if you're in a Zoom ratio (e.g. 75% instead of the default 100%)
Specific to the website prototype:
- The "PDF to Tapestry generation" feature is not currently working.
- The images for the prompt feature might not load.
UI Bugs:
- Text in nodes will occasionally spill out of the node.
- Text in nodes will not show up when you click into the node. To troubleshoot this, click "Edit" and under "Content Type", select "Text" and it will pop up.
First Time Setup + General Usage
The following set of instructions is for the demo prototype that works locally, not the remote website. Instructions for the remote website can be found in the User Manual below.
Building
First, ensure that you are familiar with and have installed Git and Github and Visual Studio Code (the code editor we use), as well as Node.js and Vue Language Features plugin. No specific versioning of Visual Studio Code is required.
- Download the Tapestry Tool 2.0 WordPress Plugin and follow the installation instructions found here: https://github.com/tapestry-tool/tapestry-wp
- Download this Tapestry ML project to a local folder and replace the Tapestry Tool src folder with the file from this repository (tapestry-wp > templates > vue > src). For the most updated version of the project, ensure that you are in the main branch. (GitHub link: https://github.com/ubcemergingmedialab/23-3002-Tapestry)
- Make sure to make the following edits to the code (Ctrl+F "USER-EDITS" in the tapestryml.js and LOtapestryml.js files to see all places that require changes):
- OpenAI API Key: You can get a key for free by creating an account on the OpenAI website and generating your own secret key. Replace the string in the relevant places in the code.
- Tapestry ID and endpoint link replacements: Log in to Wordpress and create a new tapestry page (ensure the Tapestry Tool 2.0 Wordpress Plugin: https://github.com/tapestry-tool/tapestry-wp has been downloaded and src file has been replaced with the code from this repository first). Then right click and select "Inspect", then "Network". Refresh the page and look for the only item in the "Name" list with a single number. Enter that value in the tapestry_id variable and save the file.
- Make sure to make the following edits to the code (Ctrl+F "USER-EDITS" in the tapestryml.js and LOtapestryml.js files to see all places that require changes):
- Open up a command prompt or terminal window and run the following:
- npm run build
- npm start
- Log into Wordpress and open a Tapestry page linked to the code.
Dependencies
- Tapestry Tool 2.0 Wordpress Plugin: https://github.com/tapestry-tool/tapestry-wp\
- Vue Language Features (Volar) Plugin (v1.8.8): https://marketplace.visualstudio.com/items?itemName=Vue.volar
Troubleshooting
- If running tapestry on localhost, you must run "npm start" in a command prompt/terminal and have MAMP open. Ensure both the Apache and MySQL servers are running.
- If you are having trouble deleting a Tapestry due to the error "Node MetaId Invalid", go into MAMP > Open WebStart Page > Tools > PHPMyAdmin > tapestry > wp_posts (contains nodes and tapestry objects) or wp_postmeta (contains information on the content and relation between nodes and tapestries). The error message indicates that a node from wp_posts was unable to find its corresponding metacontent stored in wp_postmeta. This may be resolved by adding a new entry with the missing ID.
User Manual
For users who would like to access Tapestry without having to download the plugin locally and go through set up, you can use our website here: https://tapestry.emlx.ca/wp/tapestry/tapestry-new/#/tapestry-ml?depth=0
Note that this website will only generate one Tapestry at a time, so if someone else is using the website, you might have to wait for it to be cleared.
This website prototype also has several bugs that is addressed in Issues/Bugs above.
Challenges
Open AI API Integration
During the development process, we encountered challenges related to accessing the OpenAI API. To utilize the prototype, an API key from the OpenAI API was required. However, we encountered initial limitations on the number of API calls we could make. Fortunately, we managed to address this problem by optimizing our approach and reducing the overall number of API calls needed. This experience underscored the importance of considering the popularity of the API and the potential demand it might experience. At one point, we also faced difficulties in accessing the API due to high concurrent usage by multiple individuals.
Web Server and FTP
Our team faced certain difficulties when attempting to upload files to the website, particularly when using the first server. We encountered issues with permissions that led to complications in uploading the updated code files. However, after transitioning to a different server, we were able to successfully resolve the problem.
Future Plans
Improvements to current features
PDF to Tapestry generation
Currently, the Tapestry tool is only able to take in small PDFs of 1-3 pages. In the future, ideally, it would be able to take in PDFs that are larger, e.g. a complete slide deck of a lecture. To be able to generate a Tapestry based off lecture slides would be an incredible asset to the student learning experience, especially for visual learners.
Prompt to Tapestry generation
Currently, the Tapestry tool is limited by a character limit of 2048 characters due to ChatGPT's limitations on input size. It would be great to increase that character limit so that we can input more detailed prompts if need be.
Improvements to Tapestry generation
Automatic generation of nodes from a selected node
This would involve selecting a node, e.g. a grandchild node "Magnolias", and prompting the system to automatically generate nodes relevant to it. For example, the television series "Sweet Magnolias" or "Magnolia: 1999 Film" would be two nodes that pop up. This would allow learners to dive deeper into and gain more inspiration for specific topics.
Generating Tapestry based off a link
Another feature we can add to the Tapestry generation process is to generate a Tapestry based off a weblink, e.g. the link to this Wiki article. This would be similar to our current PDF to Tapestry feature.
Automatic generation of multimodal content
Perhaps most ambitious, is the idea for the Tapestry to automatically generate multimodal content in addition to text. Currently, users can only manually add a node with multimodal content like a quiz, presentation, video, etc. It would be incredibly exciting for it to be able to automatically generate multimodal content within the Tapestry.
Editing of node links
Currently, users are able to add new nodes to the already generated Tapestry. Links of the default navy blue connection are generated with these new nodes but it would be interesting to have the user be able to edit the node connection and select which type of connection it is rather than have it be automatically generated (e.g. user can choose connection strength).
Poster
.jpg)
Team Members
Principal Investigator
Dr. Steven Barnes
Professor of Teaching and Director
Undergraduate Program in Neuroscience
University of British Columbia
Current Team
Victoria Lim, Project Lead, UI/UX Designer (May 2023-Present)
Work Learn at the Emerging Media Lab at UBC
Undergraduate in Bachelor of Science in Cognitive Systems
University of British Columbia
Sophia Yang, Developer (May 2023-Present)
Work Learn at the Emerging Media Lab at UBC
Undergraduate in Bachelor of Science in Cognitive Systems
University of British Columbia
FAQ
As the title suggests
License
MIT License
Copyright (c) 2023 University of British Columbia
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Last edit: August 29, 2023 by Victoria Lim
|
