User (re)Search
The Search before the Research
Once upon a time in a land far far away, there lived a UX’er who was conducting User Research. She was looking for a few youngsters who liked doing math homework. Of course it would help if they were iOS users, and awake while it was still day time in her timezone. But these are minor details. The designer opened her Macbook and switched it on. In the few seconds it takes the laptop to boot, she uttered a special spell — ‘Oh User Fairy! I wish for the ‘Real 6’’. And just like that, she had 6 Calendly notifications in her inbox with the Zoom links in place. Her wish was granted. Indeed they were perfect — all 6 of them. And she lived happily ever after!
Isn’t this a lovely fairy tale? Because well, it IS a fairy tale. In the land of User Research — as I was reminded recently — happy endings are much difficult to reach.
The Adventure Begins
Math Hero went live on May 9, 2021. And soon after, we started our research. The MVP was live, but we needed more info— both from an investigative point of view and to check the usability of our app. While we had plenty of quantitative data from Amplitude, we needed some qualitative data too.
We decided that it would be ideal if we could speak with some of our app’s users. They were after all our REAL USERS. All we had to do was get in touch with them somehow. So, we sent over a 100 emails to registered users who were using our app. We offered a fair incentive and asked — Would you be available for a quick chat? But we got no responses at all — sıfır, nada, zilch, shunya.
We were stumped! But we were not going to stop here. We decided to widen our pool. Maybe we didn’t target enough candidates. We were missing out on the unregistered users. What if, we could ask everyone who was using our app? So, we sent an in-app message.
All users of our app would see a pop-over when they opened the app. The pop-over then led to a screening questionnaire. The first time around while sending emails, we hadn’t expressly asked our users if they wanted to speak with us. This time around we tried to be as direct and clear about our intentions as possible.
Contrary to my expectations we got a lot of responses . I had assumed that users would hit close on anything that ‘pop-ed’ onto their screen. Maybe a few of them did hit close but a fair few answered.
We had a 182 responses before I turned off responding on our screening form. Surely we could get 6 users from the mix? These users knew our intentions from the start. They had filled out the screener, they had shared their email addresses and consented to talking with us. This was gonna work for sure. Right? Well of course I was wrong. We emailed the users who made it through our screener questionnaire, but we never heard back.
The app was growing from strength to strength but why couldn’t we get any of our real users to talk to us?
We pop-ed on our thinking hats and started brainstorming. What could we be doing wrong? Is there something we are missing? Is it us? Is it the email? We were trying to talk to the elusive Gen Z, the zoomers, the jet-setters, the discord-ers? If we wanted to infiltrate, we had to be one of them! We changed our approach once again. Maybe talking one-on-one was scaring off our relatively young user base? Maybe they simply didn’t check their email inboxes? So, we decided to connect with our users via Discord.
Our discord server is called the Math Hero Club. We envisioned it as a space where users answer each other’s questions, help each other out and also share their opinions about math education and our app.
Sure enough we gathered a huge user base. In a couple of days we reached 2k members. I was just looking for 6. This time we had cracked the code!I was certain I would be able to speak with a few of our users.
I tip-toed into the server. My first attempt to recruit a user blew up in my face. The user asked me ‘Are you a bot’ before blocking me. A teeny tiny set back compared to other issues we had faced so far.
I smartened up quickly. I had to be a little more subtle. I solved a few math problems which landed in our ‘Homework help’ channel before asking — Hey, could help me out. Sure enough the users said yes, they’d be more than happy to help. We made appointments — but those appointments fell through. There were plenty of excuses as well as some straightforward ghosting.
I wasn’t ready to give up. A few homework-helps later, I started doing written user interviews. We had experimented with a number of different approaches already, why not try a chat-interview. We also designed a survey and shared a poll question each day. While the polls worked fine, the chat-interviews were a flop.
I had my ‘Real’ users but the interaction was simply not rich enough.
The conversation didn’t flow as it should have. I got a few points here and there, but it was just not worthwhile. It was not helping us make progress. Our questions were still unanswered.
Though we had connected with our real users, we had made a big compromise regarding our research technique. This wasn’t giving us the kind of results we wanted. It was becoming clearer to us, that we had to choose a middle path of sorts.
Soon after, I conducted 3 guerilla interviews in Turkey. The interviewees were not our ‘real’ users at all. I had no expectations from these interviews. They were simply an experiment to see, what if?
These interviews were eye openers. I realized that the right research technique and the richness of data mattered a sliver more than getting the absolutely perfect user.
We couldn’t make compromises on our research technique, although we could replace our ‘real users’ with ‘representative users’.
We had robust persona based on our initial user research. We could recruit users based on these persona and then conduct moderated interviews, all we had to do was find the right recruitment service for us. We decided to work with userinterviews. With our screening questionnaire in place, all we had to do was ‘Start project’. And voila, we had 12 users fitting our profile. The number gradually grew as the days passed. And from among those candidates, we found our ‘representative 6.’
We conducted 6 investigative interviews as well as 6 usability tests. The insights from these studies are helping us shape the path of our app, Math Hero. We couldn’t speak with our ‘real’ user, but these insights are very ‘real’.
As designers and product owners, we may sometimes get caught up in the illusion of perfection. But more often than not, perfection interferes with progress. We’ve to learn to ‘make do’ to ‘make more’.
This was our little adventure in a nutshell. Want to cheer us on in our journey? Check out Math Hero here.
