Pillow - USABILITY STUDY
Background (May - Sept 2017)
Pillow is a startup that provides an innovative management solution for short-term rentals in multi-family apartment buildings. With Pillow’s foray into the multi-family residential realm, Design and Product were tasked with answering several questions:
- Do users understand the value proposition of Pillow?
- Are both onboarding and dashboard states intuitive?
Two usability studies took place in June and August 2017 with 15 participants.
- Planning (test plans, screeners, questionnaires, test scripts)
- Reporting & Presenting
Results & Recommendations
We conducted the studies within Pillow’s office utilizing both the QA environment and InVision prototypes.
1. Landing Page
- Program unclear
- Questions and confusion over pricing
From these findings, the team decided the landing page needed to be simplified. Less text. More white space. More visuals. A new infographic was designed to explain Pillow’s value proposition along with a section on pricing. A new video was included to provide success stories. And a new earnings slider was built to show how much money one could make...
Old Landing Page
New Landing Page
The team tested the new changes in August 2017, which showed a drastic improvement in program clarity. Users were able to identify the value proposition(s) of Pillow and showed interest in next steps.
Analytics data showed that once users finished onboarding, many left and never came back. The team sought to figure out "why."
- Onboarding experience different for someone familiar vs. unfamiliar with short-term rentals
- Narrative from onboarding to Airbnb listing creation (post-onboarding) not intuitive
While onboarding was simple, next steps weren't so clear (Airbnb linking and listing creation). Users wanted more knowledge or a "heads up" before continuing.
As a result of these findings, the team decided to create a more linear narrative by re-designing the onboarding flow to include Airbnb linking and listing creation. Additionally, the team realized existing hosts didn't need an introduction to hosting, while more education was needed for new hosts. Thus, a new flow was designed to accommodate multiple types of users (existing vs. new hosts) with optional how-to-guides for new hosts.
Users breezed through the new onboarding with minimal questions. As a result, the conversion rate from landing page to listing creation increased 600% (please note: data still trickling in and comes from a small base).
- Confusion over Airbnb listing creation
- Terminology and naming confusion
It was not clear who or how an Airbnb listing would be created (Pillow). Users were also confused what to do once their listing was live. Additionally, terminology was confusing. Users didn't understand what "Property Guide" or "Turnover Service" meant.
The new onboarding flow helped solve the confusion over listing creation. The team also realized listing customization was a big need to be incorporated. Additionally, "Guest Guide" replaced "Property Guide," which proved to be more intuitive. Finally, the name change for "Turnover Service" was decided to be a talking point in the future.
*Before and after dashboard screens left out due to confidentiality
WHAT I LEARNED
The power of screeners
...and where you recruit! When I first joined Pillow, the screener was composed primarily of 5 questions. We posted the screener to Craigslist, Facebook, and several job-searching platforms. We later learned that if you do decide to post on Craigslist, your screener better be damn good. When the applicants started rolling in, the recruiting coordinator was in charge of selecting and scheduling the participants...
Not the best idea. The first few participants that showed up weren't exactly our "target users." We quickly learned that we needed to strengthen the screener and have Product/Design select the participants vs. People Operations. I added an 10 additional questions and worked with Product/Design to handpick participants. Additionally, I experimented with recruiting by additionally posting to different platforms such as reddit, Eventbrite, Nextdoor, and Slack.
recruiting the right participants...is hard
From what I've been told, every researcher's career will have their fair share of participants gone bad. As frustrating as these can be, they are some of the best learning experiences. Recruiting the right participants is not so easy. I found that it takes effort in not just analyzing the results but trying to determine the type of participant they may be (eg. are they going to be talking a lot?). Luckily, you can do some LinkedIn and social media stalking to better inform your recruiting process.
In one case, we thought we found the right participant based on our criteria. However, he ended up spending a big chunk of time nitpicking the privacy terms and other small details regarding copy. It wasn't until after that we connected the dots. He was an Architectural Job Captain, a job that requires one to be extremely detail oriented. It made sense but didn't necessarily mimic the behavior of the "millennial" we were targeting.
PRODUCT VS. USER FEEDBACK
"I think the text is too small. I think the button color should be darker. I think it would be nice to have this feature..."
I quickly learned the difference between product and user feedback. Through several rounds of moderating and analyzing, I understood how to filter out "product feedback" and pay attention primarily to pain points, frustrations, and user expectations.
Talk less, observe more. Be comfortable with silence. But remind the participant to keep talking. To be honest, I wanted to help. I truly did (it "should" be intuitive!). But when you just let the participant feel at home and go with the flow...that's where you'll find the greatest insights.
The pain of no-shows! Having 2 no-shows in one day is definitely one of the most frustrating experiences. We realized it was our fault. We recruited too early (two weeks in advance) and realized people don't really know their plans 2 weeks in advance. We also didn't send reminders.
So we decided to recruit a week before testing, asked participants to send confirmation emails, and sent a reminder email one day before the test. This solved the problem, and we had 0 no-shows with our new recruiting plan.
MODERATING IS A SKILL
When you are sitting on the other end taking notes and watching an expert with decades of experience moderating, it looks easy and fun. I'd done my fair share of interviews in the past -- how different could moderating a usability study be?
Very different!I learned that you're not just interviewing someone, but you're trying to guide someone without asking leading questions while trying to be a fly on the wall! However, practice makes perfect. It took me several sessions before figuring it out. It's still a work in progress, but I gained a bigger appreciation for great moderators.