Measurement is Critical, and Agencies Blame Clients for Failures

Holmes Report, December 12, 2005

Agency leaders believe effective measurement and evaluation—particularly as it relates to the ability to reach opinion leaders and change attitudes—is critical to the future success of the public relations business. But they give the PR industry low marks for its ability to measure effectively, and complain that clients are generally unwilling to pay for evaluation.

More than 100 PR agency principals responded to an online survey conducted by The Holmes Report in October and November, with their responses underscoring some of the problems that continue to make research and evaluation a major issue for the industry. In general, their responses suggested that an failure of commitment—rather than the absence of necessary tools and techniques—is behind the industry's poor performance.

"I've been having conversations about the importance of measurement and evaluation to the PR industry since I started writing about the business more than 20 years ago," says Paul Holmes, editor of The Holmes Report and author of the survey. "What's frustrating is that the conversations haven't changed much in all those years. This survey confirms that most people recognize the need for better measurement, and believe they have the tools to deliver it. But it's still not getting done."

When asked whether measurement and evaluation were critical to the future success of the public relations business, there was widespread agreement that it was. Agency principals said demonstrating the ability to change attitudes to their clients' companies, products and services was most important (9.06 on a scale of one to 10), followed closely by the ability to demonstrate that PR can change attitudes (9.04).

Perhaps somewhat surprisingly, demonstrating the ability to raise awareness of companies, products and services (8.89) was rated more important than the ability to change behaviors (8.8). Demonstrating the ability to reach a broad audience with corporate and product messages was rated least important (7.83).

But there was widespread agreement that the industry as a whole does a poor job when it comes to evaluation (3.90).

Agency leaders were most confident that the industry was doing a good job of demonstrating increased awareness (6.19) and broad reach (5.96). They were less impressed with the industry's ability to demonstrate that it could reach opinion leaders (5.35). And they were highly discouraged about its ability to demonstrate attitude change (4.91) and behavior change (4.33).

Interestingly, however, most respondents felt their own firms were doing a better job when it comes to evaluation (6.09).

They felt they did their best work demonstrating increased awareness (6.83) and showing broad reach (6.71). They also felt they did a good job of demonstrating their ability to reach an opinion leader audience (6.51). They were once again less confident in their ability to demonstrate attitude change (5.67) and behavior change (5.23).

Agency leaders believe the biggest obstacle to effective measurement is the unwillingness of clients to pay additional fees for evaluation (7.12). Many also felt that clients were too focused on counting press clips rather than measuring the impact of PR on business objectives (6.46) and that too many clients did not have clear business objectives (6.35).

All of those issues were seen as more important than the difficulty of isolating the impact of public relations from the impact of all the other activities—marketing, advertising and more—on sales and reputation (5.38) and the absence of the right measurement tools (5.27). Agency principals considered the resistance of their own people to rigorous measurement to be a negligible factor (3.79).

"Rightly or wrongly, public relations agency principals generally blame clients for the fact that measurement is not getting done," says Holmes. "At the very least, that suggests the need for a broader dialogue between agencies and their clients about the importance of research. But it might also require a change in attitude on the part of agency leaders, who need to realize that investing in measurement is the only way to guarantee that clients value the services agencies provide and commit to PR spending in good times and bad."

In general, public relations agencies are quite clear that they see paying for measurement as the client's responsibility. There was the strongest agreement with the suggestion that measurement should be paid for by the client and conducted by an independent research firm (7.00), but many also believed it was appropriate for measurement to be paid for by the client and conducted by the agency itself (6.27).

Both of those solutions were more popular than the suggestion that PR should be paid for by the agency and conducted by an independent firm (3.19) or that it should be paid for by the agency and conducted in-house (3.30).

And most agencies report that clients want their firms involved in the process. More than two-thirds (69 percent) said they had research capabilities in-house at their firms, and there was no widespread belief that clients wanted to do their own evaluation without agency involvement (3.95) or that clients wanted to use independent research firms without agency involvement (3.13).

Most agencies believed that between 3 and 10 percent of a client's public relations budget should be dedicated to evaluation. A third (33 percent) felt the number should be between 3 and 5 percent, while slightly more (45 percent) felt that 6 to 10 percent was a more appropriate amount. Only 7 percent felt less than 3 percent was appropriate, while 12 percent believed more than 10 percent was ideal, with 6 percent saying clients should spend more than 15 percent of their PR budget on evaluation.

The reality is quite different, however. In reality, the overwhelming majority of clients spend less than 5 percent of their budget on evaluation, according to 94 percent of respondents, while 38 percent of respondents say their clients spend only 1 percent of their budget on evaluation, and 13 percent say clients spend nothing at all. (A fortunate 2 percent of respondents say their clients spend an average of 15 percent or more on measurement.)

Moreover, clients appear to be spending what little they do spend measuring the wrong things.

Clients were believed to be most willing to pay for measurement that demonstrated increased awareness (5.08) and reaching opinion leaders (5.05). They were less willing to pay for measurement to demonstrate behavior change (4.96) or attitude change (4.78) and least willing to pay for measurement demonstrating broad reach (4.62).

Money is not the only obstacle to effective measurement and evaluation, however. Clients don't set the right objectives. Agencies gave their clients particularly poor marks when it came to setting marketing objectives (5.50) and business objectives (5.59) for their PR programs, and only slightly better marks (5.70) when it came to setting reputation or image-building objectives.

While there was some concern about the measurement tools available for meeting the evalution challenge, a slim majority said those tools were available.

Respondents were most confident that they had the tools they need to demonstrate increased awareness (6.98). They also felt they could demonstrate the ability to reach a broad audience (6.08). But they were less confident in the tools available to meet their most important objectives: changing attitudes (5.94) and reaching opinion leaders (5.89).

Not surprisingly, agency leaders were least confident in the tools available to change audience behaviors (5.28).

Most public relations firms do not use outside media measurement and analysis firms with any great regularity. Only 2 percent said they used outside firms to measure success on behalf of all their clients, with another 7 percent saying they use outside firms for most clients and another 26 percent saying they used outside firms on behalf of some clients. The remainder used outside firms for only a few clients, with 10 percent saying they never use outside firms.

Agencies were most likely to use firm to measure changes in attitude or perception (19 percent said they used outside firms this way for some or more clients) than to link PR to behavior change (9 percent).

Bacons was used more often than any other firm (by 71 percent of respondents), followed closely by Burrelles/Luce (69 percent). Delahaye Medialink (39 percent), Vocus (33 percent), PR Trak (29 percent), Biz360 (23 percent), and Carma (19 percent) were also used by a significant number of survey respondents.

Of the firms that had been used by at least 20 respondents, NOP World scored the highest marks for satisfaction (6.90), followed by PR Trak (6.13), and VNS (6.00).