Why I Don’t (Usually) Use Audience Surveys

I’m a big fan of user research. And to many nonprofits, user research often means surveys. But they’re really not my favorite way of collecting information. 

What’s Wrong With a Survey?

Surveys are a well proven research method. Well, yes. If you have a quite substantial research budget. I’ve used them in that case. But in the real world of a website or technology project, there’s a lot of downsides:

• It’s almost always a biased sample. Unless you do something really expensive and complicated, your data is by definition from people who chose to answer your survey. These are usually people with strong feelings about your organization. Unless you’re very careful about what you ask, this leads to skewed impressions about what’s important to your audience. The people who care deeply are an audience, but they’re unlikely to be your only audience. The quicker the survey and the less your audience needs to do to get to it, the less of a problem this is. 

• You can only ask a few questions. If you want people to actually fill out your survey, it has to be quite brief—just a minute or two to complete it, in most cases. And that really limits what you can find out.

• They can tell you “what” but not “why”. Surveys give you quantitative data, but they just aren’t good at giving you data to understand why— why is it important? What makes one thing more desirable than another to them?

• They’re not as easy to do as people think. By the time you write a survey, test it with a few people to make sure it’s interpreted correctly (which is important!), put it out in the world, and analyze the data, it takes as much time as a round of user interviews. 

Talk to People Instead

A set of user interviews is way better than a survey in most cases. Talk to five or so people in each audience. You’ll get a much more nuanced sense of not only what’s important but why it’s important. For more on conducting user interviews, see my article User Interviews on a Shoestring. 

But many organizations would ask: can five people be representative of your audience? They’re more representative than most surveys if you pick them to have varying levels of involvement with your organization. 

In general, you can tell that your interview data has reached some level of validity for predicting what other people would say if you start to keep hearing the same things over and over. If you’re getting bored, you’ve talked to enough people. If all five people say different things, then you need to talk to more people. But a survey doesn’t address this problem— because you’re only reaching the people excited to take your survey, you have more quantity, not more diversity of viewpoints. 

But.. Enter the Website Intercept Survey

But there is a type of survey I think is very useful: quick surveys that popup on a website to understand what people are hoping to do on the site. This is often called an “intercept” survey, as they intercept the user on your site. You can, for instance, show the survey when a user arrives at your site, to find out what they’re looking for. Or you can show it when they click to leave, to ask them what they wanted to find and how much difficulty they had finding it. The surveys are quick, to the point, and are asked in the moment that the user is doing something.

So in general: do interviews instead of surveys. But if you’re doing research about a website, consider augmenting your interviews with a quick website popup survey. Together, they’ll provide both the what and the why.