A Guide to Running a XR Usability Study

This article is part four of the series that reviews the user testing conducted on Hubs by Mozilla, a social XR platform. Previous posts in this series have covered insights related to accessibility, user experience, and environmental design. The objective of this final post is to give an overview of how the Extended Mind and Mozilla collaborated to execute this study and make recommendations for best practices in user research on cross platform (2D and XR) devices.

PARTICIPANTS WILL MAKE OR BREAK THE STUDYResearch outcomes are driven by participant quality so plan to spend a lot of time up front recruiting. If you dont already have defined target users, pick a user profile and recruit against that. In this study, Jessica Outlaw and Tyesha Snow of The Extended Mind sought people who were tech savvy enough to use social media and communicate on smartphones daily, but did not require that they owned head-mounted displays (HMDs) at home.

The researchers approach was to recruit for the future user of Hubs by Mozilla, not the current user who might be an early adopter. Across the ten participants in the study, a broad range of professions were represented (3D artist, engineer, realtor, psychologist, and more), which in this case was ideal because Hubs exists as a standalone product. However, if Hubs were in an earlier stage where only concepts or wireframes could be shown to users, it would have been better to include people with VR expertise because they could more easily imagine the potential it.

In qualitative research, substantial insights can be generated from between six and twelve users. Beyond twelve users, there tends to be redundancy in the feedback, which doesnt justify the extra costs of recruiting and interviewing those folks. In general, there is more value in running two smaller studies of six people at different iterations of product development, rather than just one study with a larger sample size.

In this study, there were ten participants, who provided both diversity of viewpoints and enough consistency that strong themes emerged.The researchers wanted to test Hubs multi-user function by recruiting people to come in pairs. Having friends and romantic partners participate in the study allowed The Extended Mind to observe authentic interactions between people.

While many of them were new to XR and some were really impressed by the immersive nature of the VR headset, they were grounded in a real experience of talking with a close companionFor testing a social XR product, consider having people come in with someone they already know. Beyond increasing user comfort, there is another advantage in that it was more efficient for the researchers. They completed research with ten people in a single day, which is a lot in user testing.

SUMMARY OF RECRUITING RECOMMENDATIONSRecruit participants who represent the future target user of your product (identifying user profiles is often a separate research project in user-centered design).The farther along the product is in development, the less technologically sophisticated users need to be.You can achieve important insights with as few as six participants.

To test social products, consider bringing in people in pairs. This can also be efficient for the researchers.COLLECTING DATAIts important to make users feel welcome when they arrive.

Offer them water or snacks. Pay them an honorarium for their time. Give them payment before the interviews begin so that they know their payment is not conditional on them saying nice things about your product.

In fact, give them explicit permission to say negative things about the product. Participants tend to want to please researchers so let them know you want their honest feedback. Let them know up front that they can end the study, especially if they become uncomfortable or motion sick.

The Extended Mind asked people to sign a consent form for audio, video, and screen recording. All forms should give people the choice to opt out from recordings. In the Hubs by Mozilla study, the format of each interview session was:Welcome and pre-Hubs interview on how participants use technology (20 min)Use Hubs on 3 different devices (40 min)Closing interview on their impressions of Hubs (30 min)Pairs were together for the opening and closing interviews, but separated into different conference rooms for actual product usage.

Jessica and Tyesha each stayed with a participant at all times to observe their behaviors in Hubs and then aggregated their notes afterward.One point that was essential was to give people some experience with the Oculus Go before actually showing them Hubs. This was part of the welcome and pre-Hubs interview in this study.

Due to the nascent stage of VR, participants need extra time to learn about navigating the menus and controllers. Before people to arrive in any XR experience, people are going to need to have some familiarity with the device. As the prevalence of HMDs increases, taking time to give people an orientation will become less and less necessary.

In the meantime, setting a baseline is an important piece for users about where your experiences exist in the context of the devices ecosystem.SUMMARY OF DATA COLLECTION RECOMMENDATIONSPrioritize participant comfortSignal that you are interested in their genuine feedbacKAsk participants for consent to record themConduct pre-test and post-test interviews with participants to get the most insightsAllow time for people to get used to navigating menus and using the controller on new HMDs before testing your experience. GENERATING INSIGHTSOnce all the interviews have been completed, its time to start analyzing it the data.

It is important to come up with a narrative to describe the user experience. In this example, Hubs was found to be accessible, fun, good for close conversations, and participants experiences were influenced by the environmental design. Those themes emerged early on and were supported by multiple data points across participants.

Using peoples actual words is more impactful than paraphrasing them or just reporting your own observations due to the emotional impact of a first-person experience. For example,The duck makes a delightful sound. is more descriptive than writing Participant reported liking duck sound effects.

There are instances where people make similar statements but each used their own words, which helps bolster the overall point. For example, three different participants said they thought Hubs improved communication with their companion, but each had a different way of conveying it:[Hubs is] better than a phone call. Texting doesnt capture our full [expression]This makes it easier to talk because there are visual cues.

Attempt to weave together multiple quotes to support each of the themes from the research.User testing will uncover new uses of your product and people will likely spontaneously brainstorm new features they want and more. Expect that users will surprise you with their feedback.

You may have planned to test and iterate on the UI of a particular page, but learn in the research that the page isnt desirable and should be removed entirely.SUMMARY OF GENERATING INSIGHTS RECOMMENDATIONSDirect quotes that convey the emotion of the user in the moment are an important tool of qualitative researchPictures, videos, and screen captures can help tell the story of the users experiencesBe prepared to be surprised by user feedbackMozilla & The Extended Mind CollaborationIn this study, Mozilla partnered with The Extended Mind to conduct the research and deliver recommendations on how to improve the Hubs product. For the day of testing, two Hubs developers observed all research sessions and had the opportunity to ask the participants questions.

Having Mozilla team members onsite during testing let everyone sync up between test sessions and led to important revisions about how to re-phrase questions, which devices test on, and more.Due to Jessica and Tyesha being outside of the core Hubs team, they were closer to the user perspective and could take a more naturalistic approach to learning about the product. Their goals were to represent the user perspective across the entire project and provide strategic insights that the development team could apply.

This post has provided some background on the Hubs by Mozilla user research study and given recommendations on best practices for people who are interested in conducting their own XR research. Get in touch with contact@extendedmind.io with research questions and, also, try Hubs with a friend.

You can access it via mozilla. com/. This is the final article in a series that reviews user testing conducted on Mozillas social XR platform, Hubs.

Mozilla partnered with Jessica Outlaw and Tyesha Snow of The Extended Mind to validate that Hubs was accessible, safe, and scalable. The goal of the research was to generate insights about the user experience and deliver recommendations of how to improve the Hubs product. Links to the previous posts are below.

Part one on accessibilityPart two on the personal connections and playfulness of HubsPart three on how XR environments shape user behaviorI am planning a series of workshops to teach XR design and research. Subscribe to my newsletter to be notified of 2019 course offerings RELATED QUESTION How many volts has the output of a welding machine? To keep things simple, I'll talk about 2 welding machine types only; constant current and constant voltage.

Both types will take a high voltage, low amperage AC power supply and turn it into a low voltage-high amperage supply which is more suitable for welding.Why do we use them?Constant current -SMAWthis power supply type provides constant current or amperage Within reason, that is.

As you weld, the amperage fluctuates up and down a few amps, but for the sake of keeping it simple, let's say it stays constant. Voltage on the other hand does not. When you turn the machine on, hook up your ground and put a rod in the electrode holder nothing happens, but there is an open circuit voltage (OCV) present at the electrode.

OCV can be anywhere from 45 volts to 80 volts. Why?This voltage allows us to establish an arc when we scratch or tap the rod against the work surface, once the circuit closes and we start welding, the voltage drops down to, and can fluctuate between 15 and 40 volts.

As we weld however, we cannot hold our electrode an exact distance away from our work, nor can we travel at exactly the same speed; as our arc length increases and decreases, so does the the voltage.Constant Voltage -GMAWThis power supply works a little differently, it supplies a constant voltage Within reason. It's still taking our high voltage low amperage power and flipping it around, but our variables for setting our heat change a little.

While stick welding, we control our rod size and amperage, the voltage takes care of itself; with GMAW, we have shielding gas, wire size, wire feed speed (wfs), and voltage. The current or amperage is taken care of by how much wire we feed into the weld pool per minute. The voltage is set to balance the arc and provide the properties we want for transferring our filler metal, as the wire is fed into the pool, the voltage fluctuates.

Again, to keep things simple, lets say we have a voltage range of 1040 volts, when I set my current (wire feed speed), I'll use the voltage to balance the arc and provide me with the type of metal transfer called for in the weld procedure