Firefox Metro Usability Study Report - 2014 Q1 Work Week

Scope of The Study

Test a prototype and a live version of Firefox Metro with potential adopters

Usability Lab

  • [Location] SF Conference room as an informal lab
  • [Roles]One moderator and several remote and/or in-person observers
  • [Devices]Surface Pro 2, Ziggi USB document camera, Macbook Pro
  • [Tools]Silverback, BB FlashBack, Vidyo

Usability lab setup

Usability lab

Test plan

  1. Interview Windows 8 device and browser usage
  2. Watch Windows 8 system gestures
  3. Test first-run prototype
  4. Test live product
  5. Gather emotional feedback via product reaction cards

Participant Profiles

Eight participants

demographics

Ten Windows 8 devices

demographics

Time spent on desktop and metro Windows

time spent on metro

Familiarity with Windows 8 gestures

Familiarity

Opinions about using Windows 8

Likes and don't likes

Summary

  1. Participants who have touch enabled devices spend more average time on Windows 8 metro and more familiar with Windows 8 gestures.
  2. To people who have non-touch Windows devices, Windows 8 feels like a big adjustment that requires extra learning. People who have touch devices like the touch capabilities and think that’s more intuitive.

Findings

1. First-run tour

First-run screenshots Prototype web link

Read->Watch->Test is a typical pattern of how participants perceived the first-run tour

Feature introduction was more memorable than gesture education

what participants remembered

Summary - first run tour

First-run tour is informative for product introduction, not interactive enough for product education.

Iterated in-product on-boarding:

onboarding

2. Live version testing

tasks and pics

Half of the participants needed a hint to be able to launch Firefox metro.

Launch apps

6 of 8 participants figured out main gestures

swipes and search

“Back” was straightforward; “Plus” was new to some participants

On-screen buttons

“Star” and “Pin” are easy to understand by most participants

Star and Pin

Find on page is a relatively advanced feature that nearly half of the participants have not used before.

Find on page

Snap view is a hidden but memorable platform feature. Participants confused it with browser features.

likes and dislikes

Summary - live version testing

  1. On Windows 8.1, participants had difficult discovering Firefox Metro app.
  2. On-screen controls, icons were well accepted. “Plus” button was new to some participants. No one discovered the buttons can be moved up and down.
  3. Gestures for tile customization is easy to learn, but can be tricky to discover without help tips.
  4. Snap view is a hidden feature on Windows 8. Participants found it very cool after watching a demo. Many participants memorized and used the gesture later in the test.

3. Product reaction session

Firefox classic VS metro

  • More interactive
  • Designed for touch
  • More basic, minimized, fewer controls
  • 2 participants mentioned navigation is different and they need more time to get comfortable using metro firefox

7 of 8 participants may use Firefox Metro if it is installed on their device

adoption rate

2 of 5 participants recognized Firefox branding

brand recognition

Likes and dislikes

likes and dislikes

Product reaction cards

participant picking cards Product reaction card methods

Pick 5 to 6 words that are relevant to their feelings when using the product

participant photos holding cards

Words picked by all participants

all cards

By participants with touch devices

cards by touch participants

By participants with non-touch devices

cards by non-touch participants

Summary - product reaction

  1. Fresh, Simplistic, Flexible, Efficient are the overall reactions gathered from participants.
  2. Difficult, Frustrating are the negative reactions from non-touch device owners, mostly due to their unfamiliarity with Windows 8 touch environment.
  3. The product is recognized more as a native Windows 8 app than a Firefox app. More branding and eco-system level features will be needed to change that reaction.

Suggestions

Advanced on-boarding support

  • Support articles for getting the most of Firefox metro. Introduce features like customizing tiles, draggable on-screen buttons.
  • Make the first-run site or in-product tour always accessible

Increase app discoverability

Increase discoverability of Firefox Metro on Windows 8 through auto-launch, guided steps upon install, etc.

More Firefoxy

  • Support add ons, at least popular ones
  • Support private browsing
  • Apply a unified visual style of Firefox brand

Powerful tab support

  • A more intuitive tab system which supports switching, private browsing, synced tabs, all in one panel.
  • Dual windows

Thank you!

Yuan Wang (yuan@mozilla.com)

January 7th, 2014