Assignment 5 Report

Share this post:

1. Introduction / Overview

The application depends on VR technology to help university students to better understand fire emergency evacuation in the simulation accommodation. The students should improve their knowledge with fire safety and emergency evacuation when they first time check in the accommodation. Assignment 4 designed that this application should display how to use the similar scale 3D model of extinguisher when they face a fire emergency. Also, this application can educate students how to put out the different kinds of fire, and the design includes the voice and grab and fire extinguishing and button interactions to achieve this education. In addition, students can follow the simulation evacuation route to have a safety escape experience to improve understanding safety and evacuation of the incident.

2. Technical Development

We have developed a Unity3D project targeting the Oculus Quest platform. Within our project, the scenes, 3D models, and sound effects were sourced from open online repositories, including the Unity3D Asset Store, Narakeet (an online text-to-audio transformation tool), FesliyanStudios, and NetEase Cloud Music. We integrated a simple quest system that we created into our project. The project encompasses around fifteen interactions, which cover sound effects, visualization effects, quest controls, and player operations. We employed straightforward yet effective techniques to achieve the goals set out in our fourth assignment. Our belief is that technology should cater to business requirements and not the other way around.

2.1 Quest System

Inspired by the YouTube tutorial titled "How To Make a Quest System in Unity | 2022 Tutorial" by Blackthornprod (2022), we crafted our quest system. It comprises two components: Quests Detail Description and Quests Menu.

Quests Detail Description

This component provides players with a comprehensive description of each quest, enabling them to grasp the objectives necessary for completion. We utilized Canvas to display a floating screen effect in front of players. This visualization appears at the game's onset or upon the completion of any quest. It disappears once players have perused all the descriptions and pressed the Y button on their left Oculus controller.

Quests Menu

Aimed at offering players a persistent visualization of the quest menu, this component aids players in swiftly determining their pending and completed quests. Using Canvas, we displayed the quest menu at the upper right-hand corner in front of the players. The menu employs color-coding: blue indicates unfinished quests, while green signifies completed ones. As players complete quests, the corresponding entry in the quest menu transitions from blue to green.

2.2 Filling the Water Tank

To enhance realism, players must fill a water tank before using it to extinguish a fire. This interaction necessitates players to hold the water tank and approach a water barrel in the scene.

We designed a Cylinder with dimensions matching the water tank's inner diameter but slightly shorter in height. This Cylinder, when adorned with a water simulation 3D asset, resembles floating water. Placed beneath the water tank GameObject, it gives the impression of a filled tank. By default, the Cylinder's Mesh Renderer is disabled.

The WaterFiller Script component, attached to the water tank, is programmed to detect proximity to the Water Barrel GameObject. Once the water tank nears the barrel, the script enables the Cylinder's Mesh Renderer, simulating a filled tank.

2.3 Extinguishing Fires

Players can combat two types of fires: standard and grease fires.

Standard Fire

A water-filled tank is essential to tackle a standard fire. We incorporated the WaterTankExtinguisher script into the water tank. This script, upon detecting the player's intent (pressing the A button on the right Oculus controller) and confirming the tank's filled status, extinguishes the fire.

Grease Fire

The logic for quelling a grease fire mirrors that of the standard fire but employs a fire extinguisher. The script ascertains that players are wielding the fire extinguisher before allowing the fire's extinguishment.

2.4 Door Operations

To amplify realism, doors within the game can be opened and closed. We attached the opencloseDoor script to door handles. Upon detecting player interaction, the script activates door animations, simulating opening and closing actions.

2.5 Sound Effects

To augment immersion, we integrated sound effects that play post-interaction. Each relevant script contains an audio clip parameter, which is played upon interaction trigger.

3. 3D Content

Our game environment predominantly consists of an Apartment 3D scene. We employed a fire simulation 3D model to represent materials ablaze and out-of-control fires. Players have access to a water tank 3D model, allowing them to fetch, fill, and use it against standard fires. Additionally, a fire extinguisher 3D model is available for players to combat grease fires. Our proprietary quest system is built using Canvas, and we've integrated a Capsule to depict a basic NPC responsible for initiating and concluding our final quest.

Apartment Kit

We use this 3D scene to provide players with an environment to complete all quests. This scene includes a kitchen, which we use to simulate a grease fire situation. It also features several floors and stairs, allowing us to simulate an evacuation scenario when the fire becomes uncontrollable.

Free Fire VFX - HDRP

We use this 3D model to simulate an object on fire.

PBR Water Collecting Tank

We utilize this 3D model to represent a water tank. Players can pick it up and fill it with water to extinguish a standard fire.

Fire Extinguisher

We utilize this 3D model to represent a water tank. Players can pick it up to extinguish a standard fire.

Simple Water Shader URP

We use this 3D model to emulate water. Combined with the water collecting tank 3D model, we can simulate a tank filled with water, offering players a more lifelike gaming experience.

Rusty Water Tank

We use this 3D model to represent a water source. Players naturally approach with an empty water tank and use this model to refill it.

Old Pans

We use this 3D model to depict a grease fire occurring in a kitchen oil pan.

Quest System

We employ Canvas to construct our unique quest system. This aids players in understanding the tasks they need to complete and tracks their progress throughout the game.

NPC

We have created two capsules to represent basic NPCs, providing players with a starting and ending point for evacuating the apartment.

4. Usability Testing

4.1 Design and Plan

4.1.1 testing scope

The aim of this application testing is to improve the usability of the application. The testing aims include the following scope.

1. The simulation environment is similar to the accommodation.

2. The extinguisher and fire 3D modules are similar with the real world

3. Grabbing extinguisher to put out the fire.

4. Using water to put out the fire

5. Interaction button to launch the fire emergency

6. The fire emergency simulation is similar with the real world

7. Following the evaluation route to have a safety escape experience.

8. Interaction button when students successfully escaped.

4.1.2. Testing tasks design

Task Timing:

Recording the start time when the user is instructed to begin a specific task or scenario.

Record the end time when the user successfully completes the task or scenario or when they indicate they have finished.

Calculate the time taken for each task or scenario.

Simulation environment similar to accommodation:

Task: Navigate through the application's virtual environment.

Scenario: Users should explore the virtual accommodation, including rooms, hallways, and common areas, to assess its similarity to real-world accommodation.

Extinguisher and fire 3D modules similar to real world:

Task: Locate the fire extinguisher and the fire source.

Scenario: Users must find the fire extinguisher and the source of the fire. This tests the realism and accuracy of the 3D models.

Grabbing extinguisher to put out the fire:

Task: Pick up the fire extinguisher and use it to extinguish the fire.

Scenario: Users should demonstrate the ability to pick up the virtual extinguisher and accurately use it to put out the fire, assessing the realism of the interaction.

Using water to put out the fire:

Task: Locate and use water to extinguish the fire.

Scenario: Users need to find a water source and use it to put out the fire. This tests the diversity of fire extinguishing methods.

Interaction button to launch fire emergency:

Task: Trigger a fire emergency using the interaction button.

Scenario: Users should initiate a fire emergency scenario by pressing the designated button, evaluating how intuitive and responsive this function is.

Fire emergency simulation similar to real world:

Task: experience the fire emergency scenario.

Scenario: Users should go through a fire emergency simulation, assessing the realism of the scenario and the effectiveness of the emergency response process.

Following the evaluation route for a safety escape experience:

Task: Navigate through the safety escape route.

Scenario: Users need to follow the designated escape route, which should mimic a real-world safety escape plan. Evaluate the ease of following this route.

Interaction button for successful escape:

Task: Use the interaction button to indicate a successful escape.

Scenario: After following the escape route and reaching safety, users should press the interaction button to indicate a successful escape, testing the clarity and usability of this feature.

4.1.3 Determine methods for data collection

Pre-experiment questions

Using questionnaires to collect information about participants' information and prior experience with similar applications, and expectations. Task completion time

Using the timer to record the time of completing each task or scenario. Task accuracy

Using observing method to record how accurately participants complete each task. For example, observers can note if they successfully put out a virtual fire or follow a designated escape route. Also, observer can document any errors and mistakes and difficulties participants encounter during the tasks, it can help developer to track the error.

Post-experiment questionnaires

The administrator will use a post-experiment questionnaire to gather feedback on the application's usability when the participants complete the testing.

4.2 Recruitment

As mentioned in the previous section of this report, in this VR fire safety simulation test, our goal was to evaluate and improve the usability of the application. To this end, we continued to design the questionnaire around the following test scope and target focuses:

Realism of the simulation environment: we wanted the simulation environment to be as similar as possible to a real accommodation in order to provide the user with a realistic experience.

Fire extinguishers and fire 3D module similar to real world: the game should be easy to understand and immersive to help users understand more intuitively how to play the game and learn the experience of having responded to a fire in a real situation.

Interactions are realistic and easy to use: users should be able to easily use fire extinguishers to put out fires, smoothly locate and use water sources, and use interactive buttons to activate fire emergencies.

4.2.1 Non-User Involvement in Data Collection Methods

Recruiting volunteers to assist with the testing and provide feedback will be more difficult as VR devices are not widely available. We decided to use a non-user participation method for data collection. This means that we design and develop without direct user feedback. Whilst this approach may reduce the accuracy of our understanding of user needs and expectations, it is a reasonable option given current device penetration rates.

Through interpersonal interactions, we worked with other groups we knew in the class to provide each other with critique material on each other's work.

In order to simulate feedback from real users, we created a few basic role-playing elements, profiles based on our research and understanding of the target user group, which allowed the testers to collect simulated data by substituting the qualities of the target audience while playing the game.

4.2.2 Role-playing elements

Based on the questionnaire data we collected, we categorised the volunteer types into three main roles for participants to play to ensure that the simulation was tested from a number of perspectives, and that the personality and background elements of the roles were randomly sampled.

Low ability freshmen

Role: A first-year student who is new to the university accommodation and he has little knowledge of fire safety.

Characteristics: The student is easily confused, and he may not know the location of fire extinguishers or emergency exits.

Goals: The goal is to evaluate how well the VR simulation guides a newcomer in a fire emergency.

The International Student:

Role: The new international student who is not familiar with local fire safety standards.

Characteristics: The student may need additional support and clear instructions because of cultural and language differences.

Goals: The goal is to test how well the application supports the needs of the international students.

The Anxious student

Role: The new student who is often anxious to navigate complex environments.

Characteristics: The student may struggle with anxiety during emergencies and has a weak sense of direction.

Goals: the goal is to assess if the VR simulation effectively supports someone who experiences anxiety.

Components of a Random Role Modelling Role:

· Interested in technology, but have never used VR equipment.

· Have never been exposed to VR but have a keen interest in fire safety.

· Has an in-depth knowledge of both fire safety and VR.

· Has attended fire safety training but is unfamiliar with VR.

· Does not know much about either VR technology or fire safety.

· Moderately knowledgeable about both VR and fire safety.

Through this approach we hoped to fully evaluate the effectiveness of the VR fire safety simulation and make any necessary improvements based on the feedback, in total we recruited 5 testers who were able to use the VR equipment to conduct gaming behaviours, each of which would play the role of a volunteer twice to conduct the test and conduct a questionnaire survey as well as a short verbal interview with the assistance of the developer's team after the game was finished, through this simulation approach The data will be collected from 10 volunteer tests.

4.3 Conduct the testing session

Testing Session preparation

The conductors prepare the environment, and they ensure that the VR equipment and scenarios are set up for testing, and the testing environment such as the accommodation lobby is ready for the testing session. Also, the developer team should provide training to the experiment conductor to ensure he understands his role and responsibilities, and this includes understanding the goals of the usability testing and how to work with role-playing participants.

Participant Introduction:

The experiment conductor introduces himself and explains the role in facilitating the testing.

Assign the role for participants:

Using questionnaires to collect information about participants' information. Assign the role-playing student participants their respective roles: the Low Ability Freshman, the International Student, or the Anxious Student. Provide them with background information on their roles.

Task Instructions:

The experiment conductor provides clear instructions and help for each task and scenario to the role-playing participants depending on their assigned roles. Emphasize the specific goals and objectives they should follow.

Time record:

The experiment conductor uses a timer to record the start time when the participant begins a task and the end time when the participant successfully completes the task. They also record and calculate the time taken for each task or scenario.

Observation and Error Tracking:

The experiment conductor closely observes the participants when they complete each task. He should record any errors and difficulties of task completion, and he should take care of the important tasks which are related to putting out fires and emergency responses.

Feedback data Collection:

The experiment conductor also uses post-experiment questionnaire to gather feedback on the application's usability of different participants testing.

Summary of Feedback on VR Fire Safety Simulation Experience Questionnaire

Total: 10 volunteers

1. Basic Information:

2. VR fire safety simulation experience:

3. Role Play:

4. overall rating of the VR fire safety simulation:

5. Suggestions or comments from the interviews: (dialogue interviews were conducted and recorded)

- Add more interactive elements.

- Provide multi-language options for international students.

- Smokey escape obscures vision.

- Improve the realism of the 3D models.

- Interesting, but would like to add more levels and gameplay.

4.4 Report the findings from the data After conducting usability testing with participants assigning various roles, the collected data have been processed to assess the application's effectiveness in achieving its training purpose. The following section summaries the findings using tables:

Task Timing

Tasks	Low ability freshmen	International Students	Anxious Students
Navigate through the application's virtual environment.	4 mins	3 mins	4.5 mins
Locate the fire extinguisher and the fire source.	2.5 mins	3 mins	2.5 mins
Pick up the fire extinguisher and use it to extinguish the fire.	0.5 mins	1.5 mins	1 min
Locate and use water to extinguish the fire.	2 mins	2 mins	2.5 mins
Trigger a fire emergency using the interaction button.	1 min	0.5 mins	1 min
Experience the fire emergency scenario.	4 mins	3.5 mins	4.5 mins
Navigate through the safety escape route.	3.5 mins	2.5 mins	4 mins
Use the interaction button to indicate a successful escape.	0.5 mins	0.5 mins	0.5 mins

Task accuracy

Tasks	Low ability freshmen	International Students	Anxious Students
Navigate through the application's virtual environment.	Success	Success	Success
Locate the fire extinguisher and the fire source.	Success	Success	Success
Pick up the fire extinguisher and use it to extinguish the fire.	Success	With help	Success
Locate and use water to extinguish the fire.	With help	With help	With help
Trigger a fire emergency using the interaction button.	Success	Success	Success
Experience the fire emergency scenario.	Success	Success	Success
Navigate through the safety escape route.	With help	Success	With help
Use the interaction button to indicate a successful escape.	Success	Success	Success

Post-experiment questionnaires (Scored from 1-5)

Criteria	Low ability freshmen	International Students	Anxious Students
User experience	4	5	3
Immersive experience	3	4	5
Effectiveness	4	4	5
User guidance	5	3	4

4.5 Analyse the findings

Aim 1: Ensuring immersive simulation environment

All participants found the simulation environment to be realistic, resembling UTAS accommodations to some extent. The setting, which includes an apartment with a lobby, different floors, and rooms, contributes to the immersive experience. However, some participants noted that the interior decoration is not entirely the same as real accommodations, potentially impacting the overall immersion. For instance, Low Ability Freshmen and Anxious Students encountered difficulties while navigating the safety escape route during Quest 3. This was due to the oversized fire object, which obstructed their view of the evacuation route.

Aim 2: Realistic of 3D models

All participants agreed that the 3D models used in the application, such as fire, fire extinguisher, bucket, and water, are somewhat aligned with real-world objects. However, some participants observed that the animations related to these models were not entirely realistic, leading to confusion in feedback after certain actions. For instance, all participants found it hard when tasked with "Locating and using water to extinguish the fire." The lack of an animation of water being stored in the bucket and the lack of an animation showing the fire being distinguished gradually were observed. Instead, the fire simply disappeared after using the extinguisher or water bucket.

Aim 3: Assessing interaction and engagement

The evaluation revealed that all participants required some form of tutorial or guidance on how to use the device, as none of them had prior experience with VR applications. This indicates the necessity of providing a proper tutorial on device usage, especially given that VR applications are not yet widely familiar to the general population. Following adequate guidance, most participants were able to interact with the application and found it engaging to use. However, the participant acting as an international student encountered difficulties during the task of "Picking up the fire extinguisher and using it to extinguish the fire." He found it challenging to read and understand all the information presented in the quest within a short timeframe. This suggests that the application could be enhanced to support diverse user needs by adding different language options.

5. Results of the Usability Testing

The findings from the usability testing sessions have provided valuable insights into the strengths and weaknesses of the VR application. Based on the analysis of the findings, a detailed plan is developed to address the problems found in the tests.

Aim 1

· Review and redesign the interior decoration to align more closely with the real-world UTAS accommodations.

· Revise the fire objects to be more realistic and not blocking views

· Conduct additional testing to ensure that participants can navigate the safety escape route without difficulties.

Aim 2

· Revise the animations to improve realism, providing a more gradual representation of fire extinguishing and water usage to make sure users are provided with proper feedback after actions.

· Ensure that feedback from testers is addressed by involving them in evaluating the animation improvements again.

Aim 3

· Develop a proper tutorial to guide users in using the VR device and the application effectively.

· Offer language options or translations to users with diverse language preferences, addressing the international student's feedback.

· Engage participants in reviewing the tutorial and quest instructions to ensure they are usable and useful.

· Conduct usability testing again with participants who have not previously used VR applications to validate the effectiveness of the tutorial and quest instructions.

References

Blackthornprod (2022) How to make a quest system in unity | 2022 tutorial, YouTube. Available at: https://www.youtube.com/watch?v=9sVZJTIIZws (Accessed: 17 October 2023).