Amazon’s AI-generated review summaries are boring and useless

Amazon is creating summaries of the reviews on its site. Good summaries surface juicy details, they don’t just generate a homogeneous slurry — which is why Amazon’s summaries suck.

(Thanks to Phil Simon for bringing these to my attention.)

How to summarize

The problem with most people writing summaries is that they summarize. Averaging everything in a bunch of stuff you’re summarizing generates a boring and uninformative summary.

Instead, imagine that you’re creating a movie trailer. A movie trailer shows you a romantic or sexy moment, a couple of wisecracks, and an explosion: it hits the highlights, sharing the things that make a movie fun or distinctive. Similarly, a summary you write should show some examples of the best or most interesting elements of what you’re summarizing. Share a statistic. Explain something unexpected that you read. Use examples.

The purpose of a summary is to hit highlights, not to homogenize everything that’s being summarized into an unappetizing grey goo.

Amazon’s review summaries are boring and uninformative

The purpose of Amazon reviews is to tell you whether or not to buy the book (or other product).

If you’re too busy to read the reviews, look at the average star rating. That’s one number that tells you how much people liked the product, or average. Is that enough?

It isn’t. And that’s the challenge with Amazon summaries.

Here’s Amazon’s AI-generated summary of 128,000 reviews on James Clear’s Atomic Habits.

Customers find the book easy to understand and engaging. They say it’s packed with real, doable advice and actionable nuggets of wisdom. Readers also say the content is inspiring, less abstract, and on point. They appreciate the additional support materials and the lasting impact of the book.

First off, people who read books are “readers,” not “customers.” That already sets my teeth on edge.

But my bigger problem is that this summary could be written about any advice book. Would you buy the book based on this? “Easy to understand” and “engaging” and “packed with . . . nuggets of wisdom.” Sure, people like it, but are those nuggets the ones that you need? Will it have “lasting impact” for you? What sets it apart from other books?

If you read Amazon reviews the way I do, the first thing you do is look at the top reviews, the ones the most other people have voted for as useful. Here’s part of a popular review of the book, written by Mike Pritchard:

Atomic Habits by James Clear is one of those rare books that I immediately read twice in a row. It is filled with dozens of science-backed and actionable nuggets of wisdom. Do you want to improve any habits in your life? I heartily recommend Atomic Habits to you! We are all driven by our habits – many of which are unconscious. Below are 9 quotes and takeaways from this life-changing book:

1. “Getting 1 percent better every day counts for a lot in the long-run.” Atomic Habits explains why the little things you do every day matters. . . .

2. “You do not rise to the level of your goals. You fall to the level of your systems.” This is such an important point in the book. Oftentimes we focus on goals in our life, while neglecting to focus on the systems that help us achieve goals. . . .

3. “Every action you take is a vote for the type of person you wish to become. No single instance will transform your beliefs, but as the votes build up, so does the evidence of your new identity.” . . .

4. “How long does it actually take to form a new habit? You just need to get your reps in.” Atomic Habits answers the question of “how much time does it take to form a new habit” with a better answer of : X number of actions. Meaning, you may need to simply complete a new habit 100 times for it to stick, which could be done in 3 days or 3 weeks or 3 months, depending on the new habit. . . .

Thank you, Mike! Now I know something about the book. I personally don’t buy the 1% analogy — it’s unrealistic in practice — but the rest of the specifics here seem quite promising. These are the kind of principles I’d like to learn more about.

The other way I use Amazon reviews is to see what people complain about. (This is especially helpful for products that aren’t books, since I want to know whether they showed up broken, have poor documentation, or don’t work with a Google Pixel phone.)

So here’s part of a one-star review of Atomic Habits, titled “obvious drivel.”

This is a very simplistic book and here it is in a nutshell : Small habits done daily compound and add up. In other words if you eat cake today, no big deal, if you eat cake everyday you’re on your way to bad health. Wow that was earth shattering knowledge. Then it goes on to describe how to get rid of bad habits: one, try to make it hard for you to do them and to implement good habits try to make it easy for you to do them. And then each chapter will give you an idea of how to make it easy. Eg. Make it easy by your environment, ie. keep your gym shoes near your bed…..(wow such wisdom…..lol). Have friends who are similar minded to your habits, ie. you both like going to the gym etc.

Each chapter starts off with a short story about something connected to the habit discussed, which I find is a cheap way for an author to look “intellectual”. Eg, back in 1942 some Russians did experiments on bladdy blah .

I suspect the 5 stars are because its an easy read.

Really? That’s the worst you can say? As a business book expert, I can tell you that all the best business books are written this way. Good books are usually easy reads, because that’s what makes them effective. If that’s your idea of what’s wrong, you haven’t scared me off. Sounds like the book is worth buying.

What a good book review summary would look like

Here’s how Amazon summarizes 303 reviews (89% four or five stars) of my book Writing Without Bullshit.

Customers find the book practical and useful for clear writing. They also appreciate the recommendations and simple strategies.

That is useless. Completely generic. This describes every decent writing book every written.

If I were attempting to be objective and summarizing the reviews in a more human and interesting way, I would cite some of the specifics in the reviews, both good and bad. That summary might look like this.

Surprisingly witty for a writing book. One reader said this book cured their writer’s block. Readers enjoyed the critiques of terrible CEO emails and other bad writing. The ROAM (Readers, Objectives, Action and iMpression) framework will give you a structure for writing. The advice about avoiding passives and qualifiers is old hat. What the book says about SEO is contradictory, and some found the writing juvenile and too filled with pop references. A few objected to the “bullshit” in the title, although most found it completely appropriate given how much BS there is in most writing.

That’s a sampling, not a summary, and it’s a lot more interesting. Unlike Amazon’s AI-generated summaries, it’s unique to this book. By telling the potential buyer what the actual good and bad features are, this summary gives them a basis for comparison.

Beware of AI-generated summaries

AI is a great tool for generating summaries for your use. If you want to know what a mostly impenetrable science paper is about, or what happened last week in Trump’s trial, you can see the highlights in an AI summary.

But don’t publish them.

AI summaries have two problems. They’re sometimes wrong. And they’re boring. It’s the AI “accent” again — writing that’s too even and has no wit.

Great summaries pick out the highlights and focus on representative specifics. That makes them both useful and interesting. And for now, that’s another area where human writers are a lot better than AI writers.