Our Lady of Small Sample Sizes, hear my prayer

When we recently fielded the “AI and the Writing Profession” survey, I was delighted to be able to generate well over 1,400 responses. It’s not just bragging rights. Big samples are better in a lot of ways. I’ll explain why.

Use caution in interpreting surveys with small and biased samples

At Forrester Research, where I originated our consumer research product Technographics, we routinely surveyed tens of thousands of consumers. Not only that, we made sure that our samples were as close as possible to the census numbers for variables like gender, geographic location, marital status, income, and educational attainment. Results from surveys like that are rock-solid and dependable with small margins of error.

They’re also extremely expensive to conduct, since they require maintaining a demographically balanced panel.

Of course, it’s a lot easier to just field an online survey and attempt to attract lots of respondents. That’s why you’ll see so many survey results these days. These types of surveys are called “non-probability surveys” because they are not generally balanced or weighted back to any population norms.

You should be skeptical of non-probability surveys. First of all, a survey of 100 or 200 people can easily be misleading, distorted by just a few outlier responses. A random sample of 100 people has a margin of error of 10% (at the 95% confidence level), so any statistics quoted from a survey that small are quite suspect.

Also, many of the surveys you read have obvious biases. For example, Bookbub surveyed 1,200 authors about AI and writing. That’s a decent sample, but Bookbub focuses on self-published authors. So you’d expect the results to be biased towards self-published authors, rather than all authors.

Similarly, Substack surveyed its writer base about AI. But most writers are not on Substack. Its survey is best understood as representative of Substack writers, not writers in general.

There’s a lot of value in these surveys, but you should interpret the results with the biases in mind, and understand which populations they potentially represent.

If you’re interested in bias, samples, and interpretation of statistics, I recommend the book Fact Forward: The Perils of Bad Information and the Promise of a Data Savvy Society by Dan Gaylin, CEO of NORC, a large non-profit research organization that does many carefully balanced research surveys.

Large and varied samples enable deeper analyses

In our writer survey, we cast as wide a net as possible. We were supported by organizations that served speechwriters, ghostwriters, journalists, content marketers, and many other types of writers. That helped generate a large and varied sample. I cannot prove that the sample is representative of “writers” as a population, but it did hit a very broad collection of professional writers. There were more than 100 writers in many categories including content marketers, journalists, ghostwriters, fiction authors, and nonfiction authors, and nearly as many content editors, thought leadership writers, copy writers, speechwriters, and PR professionals.

It certainly helped that the topic interested so many people, who shared it with their professional colleagues.

A sample of 1,400 has a margin of error of 3% at the 95% confidence level. Since this is a nonprobability survey without a reference population, you can’t be sure that statistics are accurate within that amount, but it’s certainly better than a 10% margin of error.

But the interesting thing about a sample of this size is that you can compare slices within it and still have enough sample to draw conclusions.

For example, the survey reached 806 writers who were either freelance or worked for small agencies. That’s quite sufficient to draw conclusions from their answers about how AI affected demand and income. (In this group, 45% saw reduced demand and 40% reported reduced income.)

The survey reached 68% women and 27% men, so we could explore whether gender affected their attitudes about AI (it didn’t). It reached writers of a variety of different ages, so we could see if age was an important factor (it wasn’t). But we also could analyze whether writers of different incomes had different attitudes, and they did: the better compensated writers used AI more.

We were able to compare how different types of writers used AI. I can tell you that 84% of thought leadership writers use AI at least sometimes, while only 68% of ghostwriters and 44% of journalists do. There were only about 100 people in some of these subgroups, so you again get into sample size limitations, but I feel confident based on survey that there’s a big difference in how different types of writers treat AI.

A sample of this size also allows you to create your own segments. We divided respondents into advanced AI users, basic AI users, dabblers, and nonusers based on how often they used AI and how many tasks they did with it. Each of those subgroups has at least 170 respondents in it. That allowed us to compare AI use patterns, attitudes, and demographics across the AI use segments, which revealed fascinating patterns — specifically, that the nonusers are far more concerned about some AI problems than than the intense users are.

I recently received a question about AI users who’ve seen increased income due to AI. That’s 200 people. I can and will start analyzing that group — it’s a large enough minority to be worth of study.

At Forrester, where we had 50,000 respondents in some surveys, I would often ask the data folks to get me information on small subgroups, for example, people with diabetes, people in Missouri, or people with household incomes greater than $200,000. You can only do that with a big sample.

For some of those skinny slices, I would tell the data analysts that we needed to offer up a prayer to Our Lady of Small Sample Sizes. She often answered our prayers.

But that only worked because the full sample was so large. Our Lady of Small Sample Sizes is unlikely to heed your prayers if you only start with 150 non-random responses.