How (and why) to get started with quick-and-dirty surveys

Basic surveys are easy to learn and field. Learn to use them and your writing — and research — will be a whole lot better.

I learned surveys on the job. One year into my tenure at Forrester, the company decided to start a consumer survey business. In a previous project for the company, I’d invented a consumer segmentation called Technographics out of nothing but my own creative imagination. The company decided to base its first primary quantitative product on that.

Talk about high stakes. While I had decent statistical skills from work in college, I knew nothing about surveys and, frankly, there was no guarantee that we’d find that my segmentation actually matched actual consumer attitudes. Luckily, the survey experts we hired tested those attitudes, they actually did match my segmentation, and Technographics was off and running. (It’s still a Forrester product, 28 years later.)

Surveys are a lot easier now. You should learn about them, to add a quantitative dimension to your nonfiction writing and research work.

Surveys add credibility

Surveys add credibility. Charlene Li and I used them in our first book, and the resulting data on how consumers actually used social media was essential for proving out our insights about consumers’ online behavior.

Since then, I’ve used survey results in every book I’ve written. After leaving Forrester, I created, compiled, and analyzed a survey of business writers for Writing Without Bullshit and conducted an author survey for Build a Better Business Book. These survey results bolstered my books, demonstrating that I wasn’t just making stuff up. And they yielded insights that would otherwise have been unavailable — like the fact that only 33% of business writers felt their process for managing reviews was effective.

I’m now working with a major data company as a ghostwriter, and my survey experience has allowed me to suggest questions they can field to make our book more credible. I’m also working with partners to help to craft an updated business author survey right now.

Quick-and-dirty survey tips that will help you get started

If you’re a survey newbie, you can quickly gain experience that will give you opportunities like this. Here are few tips on starting out.

• Check out free or cheap survey tools. Survey professionals use sophisticated tools like SPSS, but if you want to get started more cheaply, there are many options. SurveyMonkey has a free trial and a $39 per month option. There are many other free survey toools available as well. (Google Surveys, which used to be available for free, has been discontinued.)
• Start with a question. Surveys are basically conversations with people, at scale. And like any business conversation, they should have a goal. Figure out first what you want to learn. For example, how do consumers think about home improvement projects? Or, how do marketers manage their suite of digital marketing tools?
• Source your sample. A survey sample should be a defined group of people with something in common (like homeowners in the Northeast US considering home improvements). You’ll need at least 150 complete responses to generate credible results. So, where will you get them? If you have a mailing list or followers on social media, podcasts, or a blog, that’s a great place to start. You can also buy samples of, say, consumers from survey companies, but it’s a lot cheaper to start with a group you already have access to. One option for sourcing a sample: consider partnering with a membership organization or conference that would want to survey its users. Survey tools even let you create different survey link URLs for different groups, so you can track which sources are generating the most valuable samples.
• Offer something of value to respondents. In professional surveys, respondents often participate in exchange for a promise to get the compilation report that you’ll be creating at the end of the project. For consumer surveys, rewards typically include a sweepstakes to get something of value, like an Amazon or American Express gift card.
• Start with a qualifying question. For example, “Are you considering a remodel valued at $15,000 or more in the next two years?” If someone fails to qualify, you can politely think them and branch them to the end of the survey. (Wasting people’s time only to find later that they’re not really in the sample group you’re seeking is not just inefficient, but rude.)
• Shape the survey experience. To avoid respondents giving up too quickly, start with simple questions that are easy to answer. Then move on to more complex questions. End with easy demographic questions, like age, gender, and income.
• Use question types wisely. For questions with multiple options, either choose to use multiple choice and ask respondents to pick one option, or ask them to check all that apply. (In the latter case, you can pipe the selected options into a second question and ask which option is the most important.) Or, create a series of statements and ask respondents to rate them on a scale of agree/disagree (a Likert scale). It’s also a good idea to ask some open-ended questions (for example, “What worries you about remodeling projects?”) to gather some illustrative quotes for your research.
• Use branching carefully. Surveys can include logic. For example, if someone says they’re planning on remodeling a bathroom, you can branch them to a set of questions specifically about bathroom remodeling, after which the respondents rejoin the rest of the sample answering more general questions. But it’s a good idea to limit such branching, or the logic of the survey will become too tangled to keep track of.
• Make sure the answers to your questions will show interesting variation. Don’t include questions that everyone will answer the same way; those give no new information. For example, there’s no point in a question like “Are you concerned about the quality of your upcoming remodeling work?” — because everyone will say yes. But you could get at more interesting information with a question like, “Which is your greatest concern about your upcoming project?” with answers including “Quality,” “Price,” “Finding a dependable contractor,” and “The inconvenience of living through a remodel.”
• Beware of “social desirability bias.” People lie, especially when it makes them look or feel better. For example, most people will agree with the statement, “I am concerned about the environment.” But you’ll get more information asking if they agree with a statement like “I’d be willing to pay 20% more for home improvements that are environmentally responsible.”
• Don’t ask convoluted questions. If there’s a hypothesis you want to test out, you may try to design a question to get at people’s mental processes. But most people aren’t that introspective. A cleverly designed set of simple questions can often reveal more than a complex question that respondents can’t easily puzzle out.
• Cut questions to keep survey length manageable. It’s reasonable to ask someone to spend five or ten minutes on a survey. But few will have the patience to spend 30 minutes. Once you’ve compiled all the questions, consider which ones you could cut without losing much information — and which are redundant with other questions. A short survey that omits some items of lesser interest is better than a long survey that respondents will give up on and fail to complete. Some types of questions, like ranking questions, require more effort to answer and will contribute to user fatigue; keep them to a minimum.
• Test before launching. Get a few people in your target market that you trust, and have them test out the survey. That’s one way to find the unexpected bugs like confusing wording and logic errors. As I can tell you from hard experience, it’s a lot less painful to fix survey issues before launching than to lament the mistakes you made when the survey is done.
• Turn the analysis into a slide presentation. Large-scale surveys can support sophisticated techniques like factor analysis and correlations. But even quick-and-dirty surveys can generate useful statistics. You can easily copy data displays from a product like SurveyMonkey and paste them directly into presentation software like PowerPoint. It’s also possible to use “filters” to compare subgroups. (That’s how I determined that authors are happier with their hybrid publishers than their traditional publishers — and that women are getting lower advances.) But keep in mind that slicing small samples into smaller subgroups generates questionable results due to sampling error; you may need a bigger sample to draw credible conclusions.

You won’t be an instant data professional. But . . .

True, your samples likely won’t be statistically representative. Your first shot at survey design may have some flaws, and your analysis may be limited.

But survey design and analysis is a great skill to develop, alongside writing, presenting, and analytical thinking.

You never know when it will come in handy for your next writing or research project.