Mistaken Metrics: Communicators Often Measure the Wrong Item

A good list for those who may wonder why companies, governments and public figures often appear to be communicating the wrong theme - and then repeating it. By Matt Wilson in Ragan.com:

"Experts in measuring communications effectiveness reveal the major blunders they see their peers making.

Measuring the impact of your message is imperative in communications, but what good is it if you’re doing it wrong?

“There are tons of mistakes communicators make in measurement,” says Angela Jeffrey, vice president of editorial research for the Institute for Public Relations.

Jeffrey and other experts told Ragan.com what they see as the most common and most detrimental communication measurement mistakes, along with tips for ensuring that the reader counts and that the survey data and sales figures actually mean something.

1. Not measuring enough.

Many communicators focus only on “counting the stuff we did,” says Ryan Williams, president of TWI Surveys. Or maybe they only take surveys or look at sales figures. “They’re not painting a full picture,” he says.

You’ve got to look at your outcomes—sales and conversions—in addition to your outputs—clippings, retweets, Facebook posts—Jeffrey says. “The real result of a PR campaign isn't how many clips you get; it's whether you moved the needle on outcomes like sales, survey scores, customer satisfaction,” she says.

2. Measuring the wrong thing.

Frankly, it doesn’t matter how many people read the CEO’s blog, says Katie Paine of KDPaine and Partners. That blog’s going to keep going no matter what. “Only measure what you can change,” Paine says. If you’re spending your time on metrics such as Twitter followers or Facebook “likes,” make sure those numbers influence sales and market share.

“You need to understand what matters to the business, and how you contribute to that effort,” she says. “Then you want to measure that, not implement some shiny new object that purports to measure success, but in fact is just a proxy for activity or hits or whatever other meaningless number is out there.”

3. Not looking at the qualitative.

You’ve got to look at the meat of what you’re counting, because search engines and online bots just look for keywords, not context. “It's better to use a combination of artificial intelligence and human oversight for maximum results,” Jeffrey says.

For example, Paine says a client of hers, Cleveland Federal Reserve Bank, known colloquially as “Cleveland Fed,” saw a spike in negative response last summer. “Turns out that a lot of Cleveland was ‘fed’ up with LeBron James,” she says.

Even when blog posts and articles really do talk about your organization, you have to look at what they say, how they characterize the organization and how prominently they were featured. “All clips are not equal,” Jeffrey says.

4. Not being specific enough.

Communicators spend a lot of time finding out how many followers they have, Williams says. But another question they should ask is, “What followers do you need?” If you’re trying to appeal to the entire state of Maine, he says, you’re casting too wide a net. In that same vein, if a boss asks you to create “awareness,” you should always ask yourself, “awareness of what?”

Paine says the same strategy applies to survey questions. Just plugging a few general questions into a survey tool isn’t enough.

“There is a right way and a wrong way to ask questions, and you need to test any question extensively to make sure you will get the answers you want,” she says.

For example, a recent survey she conducted for a nonprofit ended up with some puzzling results. “Fortunately, we had asked enough qualifying questions so we could determine that the results varied tremendously by geographic region and by familiarity with the cause,” Paine says.

5. Using faulty tools.

“Not using audited, third-party audience data from sources like SRDS, Arbitron, Nielsen, comScore, etc. can result in spurious quantitative results,” Jeffrey says.

When it comes to social media and blogs, there really aren’t any reliable tools to use, Paine says. Her advice? “Don’t even try.”

The best tools to use are sites such as Compete.com, which give you a general idea of how well-trafficked a site is. But even then, what you’re getting isn’t particularly reliable.

“If your mention appears on what in traditional media we would call an inside page and is written by Sexy Susie, who describes herself as a counselor, dating coach, trainer, and relationship blogger, Compete can’t tell you how many people see it because she writes on a subdomain of a subdomain,” she says.