Why Qualitative Research Matters for Engineers

This article has to do with understanding user research from an engineering perspective. Whether you’re curious about the different types of research or want to learn about research from the perspective of a mobile engineer, read more below!

We’re #1, We’re #1

For the 6th time this back-to-school season, Remind has hit #1 in the App Store. As an iOS engineer, this is a thrilling time. I get to see our app light up in Fabric, watch our App Store rank climb to #1 over and over, and best yet, see tangibly that Remind is having an impact on people’s lives.

As App Store reviews pour in (mostly positive), I see one that hits a nerve:

We have downloaded the app, but in spite of all our efforts we are unable to set up an account. Once we get past one impossible step, we are stopped cold by yet another step with no way to move forward. We gave up after numerous tries.

…Oof. Getting caught up in the shuffle of back-to-school makes it easy to forget that users are not just numbers and flows.

I can point to the code that I wrote to help make the sign-up flow make sense. That said, I can’t get inside the brain of the person that wrote this review. While I can look at funnels and see where they probably dropped off, I have no conception of why they found something confusing. I have no direct experience using our product for our most common use cases. I’m also not a teacher, parent, student, or administrator, so I have no clear idea of what they expect.

This made me question— how do we understand our users better?

How Do We Understand Our Users Better?

Recently, I listened to an episode of Invisibilia, where they discussed implicit racial bias. Through interviews with Tony Greenwald, the creator of implicit bias research, they revealed that while someone might believe one thing about their views on race, their behavior might run totally contrary to their beliefs:

The idea of impression management is that what you’re doing when you respond to these things is image management… I’m not saying that subjects were actually deliberately deceiving and lying when they were doing this, but rather … that people are actually in the process of constructing this image of themselves. When it came to prejudice, people themselves couldn’t accurately report on who they were, and what was influencing their behavior.

Despite differences in focus between our field and the field of racial bias research, impression management reveals a lot about the methods we use to get to know our users. When we ask a direct question to a user, it is entirely possible that they are answering from the perspective of the person they would like to be, instead of the person they actually are. We simply cannot take their words entirely at face value.

What does this mean in the context of getting to know teachers, students, parents, and administrators? Well, we have two different ways we can find out what they need:

1. Quantitative Research

We look at data related to what users are reporting and determine correlations and behavioral patterns from what we see. Methods include:

• Logs and Dashboards. We often look at internal dashboards as a marker for our success, but this ignores why someone does a certain action.
• Surveys. We often ask our users questions about the product and their lives. This gives us insights into users perceptions of themselves and their surroundings. Good survey questions can also help us identify behaviors. The user will tell us what they think and perhaps we can figure out what they do, but we don’t always know exactly what problem they are trying to solve.

2. Qualitative Research

We go to the user, get to know them through conversation, present them with our technology, and ask them questions in context. This gets at our deepest problem— why does someone behave in a certain way?

Finding the Why

I decided to ask our Head of User Research, Ana, to find out more about option two.

Usability Sessions

At first, she invited me to sit in a few hours of usability sessions for our apps and website. In these sessions, she and Jordan, another user researcher, asked users to complete certain tasks, explain out loud their thought process, and watch them struggle and often get stuck. As someone normally writing the code for these screens, it’s the closest thing to Chinese water torture I’ve ever experienced; users take many actions you did not intend for them to do. Yet, from each session, we got to see how people used the product.

Contextual Interviews

After that, she invited me to come participate in Remind’s admin research in my hometown Chicago. Over the course of a weekend, I sat in three different interviews with Chicagoland administrators to understand what kind of tools they use day to day. In each interview, we got to tour each administrator’s office, get to know their workflow and find out intimately how they use the current technology at their disposal.

One elementary school principal at a lower income school keeps track of the concerns his parents have in a green notebook, and documents every issue meticulously when he greets their kids in the morning. Another administrator at a high-income school very rarely communicates with parents, leaving that responsibility to adviseries. These data contextualize how they might use a product in a different way than we expect.

Intercept Studies

That Saturday, Ana encouraged me to help with Intercept Research. That is, approaching strangers directly, asking if they were parents, and getting their feedback on something we worked on. Recently, we changed out the PDF we provide to teachers to send out to parents to get them onto Remind, and what we found out about the new version was striking. Through these conversations, we realized that most parents were not able to complete the instructions we created for them in the PDF. Many didn’t even know the name of the app they had instructions for. We had to do better.

You Betta Work

Building a product is hard, and building a product that matches a user’s workflow and expectations is even harder. That said, if we assume passively that our users love us because we hit #1 on the App Store, we lose. We must work really hard to make sure we understand our users, using data, interviews, and other forms of qualitative research.

Engineering isn’t only about building cool new things that we couldn’t before. It’s about identifying problems that people face and working to mitigate them. Remind presented me with an incredible opportunity to jump into this space, and it has taught me a valuable lesson. Qualitative research determines days of planning, weeks of engineering work, and can impact a company’s legacy for years. By jumping feet first into our user’s needs, we build the product they deserve.