The Case for Mass Social Media Measurement
Tuesday July 13th 2010, 8:38 am
Filed under: Gary Angel

Is ROI really the point?

By Breen Baker and Gary Angel (@GaryAngel)

We received a newsletter not too long ago sent by Adobe’s OmnitureTM, which asked the question “Can you quantify the ROI of social media?”

As a measurement company, our immediate thought was “sure you can.” But our second thought was “Is that really the point?” ROI has been the focus of the conversation in web analytics communities thus far and rightfully so. No revenue, no return, no business; the logic is infallible.

There is a school of thought about business that claims every decision a company makes should be based on a single concept – maximization of return on investment (ROI). Proponents of this view might admit that measurement costs would frequently make precise calculations impossible. But they’d argue that the measurement costs themselves should be factored into the decision and, along with some assessment of probabilities and risks, should drive an ROI-based decision about how much measurement is appropriate in determining other ROI calculations.

In the every-day world of practical business decisions, this view is either arrant nonsense or meaningless gobbledygook. But even on the theoretic level, it is nearly incoherent. It almost certainly falls into the category of theories that are “directly self-defeating.” Self-defeating theories are ones that, if you tried to follow them, would result in significantly worse performance on their own terms than if you use some other guide to action.

The simple truth is that businesses care about much more than just ROI. Almost all successful businesses become successful because they share a commitment to a product or a service or an experience that has little or nothing to do with ROI. All are committed to serving their community of customers and many highlight that goal with a similar commitment to their employees. They know that the best way to build a great business does not begin by thinking about how you’re going to extract wealth from the communities you serve. If that were true, the only businesses we’d have would be Casinos and class action Law Firms!

Economists think of the processes, relationships and commitments inside the company as the business culture. The broader extension of that would be the processes, relationships and commitments the business shares and makes with the outside world, which has been aptly named social capital.

It’s the idea of social capital that interests us here. Because while social capital isn’t the same thing as ROI, that doesn’t mean it isn’t measurable. And the vast and growing world of online social media has made measuring social capital easier and more important.

Measuring social capital is not a new concept. Through organizational surveys, questionnaires, personal interviews, and many other standard research tools businesses have tried to gauge customer sentiment, market awareness, brand loyalty, and much more.

All of these market indicators fit under the umbrella concept of social capital. With these indicators, and with measuring social capital in general, the goal is to measure the strength of connection between the business and the customer, community, country, etc. Why measure the strength of that connection? For the same reason you measure anything – so that you can understand how changes to your culture, your operations and your marketing are improving or damaging that connection – and to provide a guide for what types of changes can make a positive difference.

Every public-facing employee has the opportunity to strengthen or weaken these connections every day, and it is of course already a focus for organizations to maximize the effectiveness and efficiency of customer service through asking about and analyzing what works and what does not. But online there are no public-facing employees, save the virtual employee at the other end of some chat windows, and our ability to gain insight into those connections has been limited by our desire or ability to measure social interaction and connection on the web.

Our question is this: if measuring and building social capital is so important in the physical world, why are so few companies making an effort to measure and understand the effect of social capital online? A big part of the reason is that they are chasing after something else entirely – measuring ROI.

Measuring social capital is quite different than measuring ROI. And if you are hunting for techniques to measure something like ROI in the social space, not only are you likely missing the point, but you’ll probably find yourself chasing complex and expensive solutions that end up doing a poor job of answering the wrong question.

Some things can be easily tied to revenue. Coupons, promotions, and web site ecommerce conversions may have social implications that can and should be measured. But they also have direct revenue implications that are likely to be of primary concern. On the other hand, many marketing and branding activities have defied easy attribution to revenue; things like non commerce web site visits, corporate sponsorships and community events, none of which are easily measured in terms of revenue but may generate significant social capital.

How many analysts for an e-commerce site can say whether or not their website makes customers feel good after they visit? How many people have really tried to measure whether visiting the website raises brand awareness significantly? And how many analysts for informative websites could tell you how many communities and lives their site content has improved?

These are questions most of us would love to have answers for and I would guess most people would agree that if these numbers were good it would mean something real and important. But where and how do we begin to measure such tricky questions?

We begin the same way we would with any other customer analysis.

Answering “big” questions about the impact on brand awareness and brand value, connectedness and satisfaction of web and social activity isn’t going to get done with a single measurement.

Typically, this type of measurement involves a multi step process:

  • Find a means to accurately measure (even if expensive and one-time) the uber-measure
  • Create measurement around a set of sub-indicators that might be either causal or indicative
  • Correlate the sub-indicators with the uber-measure
  • Use a factor analysis to reduce the sub-indicator set and to identify combinations that represent key causal factors
  • Track the relevant sub-indicators on an ongoing basis as your KPIs

In terms of social capital, uber-measures are things like brand awareness, brand value, customer satisfaction, consumer trust and loyalty. Sub-indicators are likely to be things like social mentions, brand searches, web site visits, online satisfaction scores, relationship events (like registration), etc.

Some of these sub-indicators will, in turn, have finer grained levels of detail. Online site satisfaction is, itself, an uber-measure that may consist of many finer-grained judgments about the site and the company.

You’ll likely find that accurate assessment of the uber-measures requires professional primary research. Measurement of sub-indicators is likely to come from a whole grab bag of systems including web analytics tools, social monitoring tools, competitive landscaping tools and search monitoring tools. To bring all this together and do the requisite analysis is going to require a real statistical analysis package like SAS or SPSS.

Plenty of companies have taken the first step – some attempt to gauge the uber-measures they care most about. Plenty of companies have setup measurement around a variety of social and online metrics that are likely to be interesting sub-indicators. But almost no one has taken the intermediate and crucial steps of collecting these two items together and then analyzing them to create a measurement framework that might work on a going forward basis.

Broadening your focus from measuring the ROI of social media and online to measuring the social capital doesn’t mean giving up on hard-won advances in measuring the performance of web sites and campaigns. Coming, as we do, from a measurement company focus, that’s the last thing we’d like to see. But when it comes to measuring social media, understanding that ROI isn’t necessarily the focus can dramatically change your approach to measurement and open up powerful measurement directions that don’t directly yield an ROI but do yield powerful indications of how successful your efforts really are.

Sure, concerns about social capital are long term concerns, and in today’s economy long term concerns may seem like a low priority. But in any economy, finding ways to strengthen your connection to and relationship with your customers is a good thing.

As more and more commerce and more and more leisure time migrates to the internet and specifically to the social spaces on the internet, finding ways to measure the effectiveness of your online efforts becomes ever more important. What’s the best way to measure that effectiveness? We don’t think it’s a relentless focus on ROI. As stirring as it sounds, a focus on ROI over social capital in the social sphere will likely cause you to miss out on a deeper understanding of your actual effectiveness and to forgo the very real measurements that are obtainable around the retention and growth of your company’s social capital.

3 Comments so far
Leave a comment

Breen & Gary,

First off, let me say this was a great read!

I read this post last week, and the more I thought about it the more I really like this idea of “social capital”. Your point of “Every public-facing employee has the opportunity to strengthen or weaken these connections every day” for me is really the main point here.

People are watching what/how employees and those active in social media are doing and saying. Not just comments about their own company, but how they behave in the social media networks. And this behavior really impacts the perception of these companies.

There are many companies in Social media trying to do this, but surprisingly few have been successful.

I look forward to future posts exploring social capital even more.


Comment by Rudi Shumpert 07.22.10 @ 2:46 am

Great Article!

While this was written for specifically with social in mind, i feel this can be broadened across all of marketing with a sense of installing data driven cultures instead of that of faith based.

Social has continued to be a thorn in the side of web analysis based on the ever changing variables, the idea of an “uber measurement” which relies on several sub KPI’s is a great way to try and organize the ever changing data environment.

I look forward to reading more, Thanks!

Comment by Kyle Rudy 07.23.10 @ 10:57 am

Great article!! A recent report by Forrester proposes to use Balanced Scorecard approach in calculating social media ROI. Your thoughts are on the same lines and it makes perfect sense.

As per Forrester, the 4 balanced scorecard perspectives of social media ROI are:

1. Financial. Has revenue or profit increased or costs decreased?
2. Digital. Has the company enhanced its owned and earned digital assets?
3. Brand. Have consumer attitudes about the brand improved?
4. Risk Management. Is the organization better prepared to note and respond to attacks or
problems that affect reputation?

Comment by Tauqueer Ali 08.19.10 @ 7:19 am

Leave a comment (Your e-mail address will not be displayed)