By Phil Kemelor

I’ve just released this year’s Profile in Analytics research. Focusing on the relationship between Web governance and digital analytics, I drilled down on whether Web governance-oriented organizations really are “better” at analytics…Do they serve more stakeholders? Do they do higher value analytics work? Are they in alignment with senior leadership?

In doing the research, it was clear that analytics teams are now more common place and growing in size. Nearly 75 percent of those surveyed are part of an analytics team.  Teams of 5 and more now outnumber single person departments, 34 percent to nearly 27 percent. Most teams have between two to four people. Add to this that most digital analytics teams are working with between 1 to 3 third party agencies, and we can see that digital analytics is requiring management of many elements, as well as people who can carry out the tasks.

This raised a few questions:

• Is digital analytics growing in importance because it is recognized as having real benefits? Or is it simply viewed as a central location in the organization to manage ever increasing sources of digital data.
• Are those who work in digital analytics equipped with the management experience and leadership to promote the value of analytics at senior levels?
• Are analytics insights used within organizations?
• Are the analytics insights of high enough quality to be used?

One of my secondary agendas in looking at the influence of Web governance on analytics comes from my general skepticism whenever I hear an organization bill themselves as “data driven.” I’m always wondering:What does that really mean?

I know it does not mean cranking out dozens of reports to dozens of people and not hearing a word back. I also know it does not mean preparing ad hoc analysis based on a senior manager’s request for a unique visitor report to present to the board of directors.

Does a Web governance structure ensure that analytics is part of the organization’s “mainstream” Web operations and therefore potentially used with more purpose, and therefore actually “data-driven”?

I invite you to draw your own conclusions after reading the findings:

• Research Analysis: Does Web Governance Drive Digital Analytics?
• Research Results:  Survey Response Summaries
• Profiles in Analytics: Collection of Individual Responses, Shared Wisdom and Experiences

All the papers are available at:

http://www.semphonic.com/content/white-papers/profiles-in-analytics.aspx

By Gary Angel

Late last year I worked with Celebrus Technologies on a white paper entitled “The Future of Digital Measurement and Personalization.” I know that sounds a little bit grand, but I promise, it’s not the sort of empty, for-pay puffery you may have come to expect in our industry. First, I’m deeply concerned with the issues and arguments covered by the white paper: the right way to build a digital measurement infrastructure. So much so, in fact, that I’m working on a second, Semphonic only, white paper that extends the Celebrus Technologies effort with our research into the creation of an online data model for the warehouse. Second, I’m convinced that the Celebrus solution ought to be getting significant interest and traction in the marketplace. If you’re an enterprise committed to advanced digital measurement, you’d be remiss not to be at least considering Celebrus. Third, and I suppose this is what really matters, I believe the white paper makes an important case for a specific direction when it comes to building a digital measurement infrastructure.

My intention is to write two posts relative to the White paper – ones that largely mirror its structure. In this first post, I intend to layout some of the significant challenges in creating a really good digital measurement infrastructure. In the second post, I’ll explain why the Celebrus solution solves some of the biggest of those challenges and is a compelling direction for any organization wanting to build a measurement infrastructure that can support more advanced customer analytics, segmentation and real-time personalization.

While I hope to convey a good sense of the white paper in these two posts, I also intend to keep both rather short – precisely because the white paper exists and it is, after all, free. Here then, is a little sample, I hope tantalizing…

All last year I wrote about the evolution of Web into Customer analytics. If evolution in the natural world comes in fits and sudden dramatic starts, the same is surely true of technical advances. In 2011 we saw a dramatic acceleration in the use of data warehousing technologies to provide Customer Analytics even as traditional Web analytic tools worked hard to re-platform themselves to support deeper, faster, and better analysis at the customer level.

If you believe that data warehousing is the future of your digital measurement platform, then you should be thinking carefully about the nature of your measurement infrastructure. The vast majority of digital data warehouses that Semphonic worked with last year were sourced via a data-feed from a Web analytics solution. That’s an obvious choice since the work to create a good collection system via Web analytics tags has already been largely accomplished at most enterprises. But is it really the right direction?

In the white paper, I cover four areas of key concern when thinking about digital measurement infrastructure: Governance, Robust Data Collection, Data Model, and Real-time capability. In each case, sourcing from your existing Web analytics system presents real problems.

The problems of Web analytics tagging governance have become well-documented. Indeed, the problems have created a whole new industry of Tag Management Systems (TMS) that create a new layer of abstraction between the measurement system and the Website. I’m a big believer in TMS. Ensighten, Tealium, Tagman and Omniture’s new solution each have their respective advantages and disadvantages, but all provide significantly better governance than out-of-the-box Web analytics tagging. On the other hand, none of them entirely solve the single biggest problem in creating a digital measurement infrastructure – the necessity to pre-plan your information capture when designing a tag. Governance is improved because you move the tag creation function from IT to measurement. This is all to the good, but it doesn’t change the underlying dynamic. Someone still has to do all of the customization, and the customization has to happen before the measurement takes place. So a TMS only changes one piece of the problem. Measurement still has to be carefully planned. Customization still has to take place at the page level. You still lose anything you didn’t collect. That’s simply not ideal. What’s more, a TMS introduces an additional cost into the system – sometimes a fairly significant one. You’re paying more to collect the same information. If you’re committed to a Web analytics tool for digital customer analysis, then a TMS is the right way to go. Believe me, the improvement in governance is worth the cost. If you’re focused on the warehouse, however, a TMS may not be the best option.

This problem of pre-planned measurement and governance is all of a piece with the next issue – robust data collection. Web analytics tags are largely page-based. They have to be manually added to individual links, they require specific (and often rather arduous) customization to capture intra-page  based actions. Scrolls, internal search, DHTML, Ajax, and Forms are all completely mysterious to the traditional Web analytics one-per-page tag. Not only does this fundamental limitation make governance and tag creation far more difficult, it limits the collection of many of the most interesting customer remarketing data points. Did a customer start a form? Did they fill-in a field? Did they scroll? Which link did a customer click? It would be nice to get all this seamlessly, without effort, and without customization.

Of course, collecting information is the fundamental purpose of a digital measurement infrastructure. Nevertheless, how that information gets organized turns out to be a huge challenge. No issue in warehousing digital data has proven to be more difficult than creating a good data model of digital data. For all the attention paid to “big-data” technologies, it’s my belief that poor modelling cripples many more efforts than does query speed. You may not think of the Digital Data Model as a part of a collection infrastructure, but you’d be mistaken. True, the data model implicit in a typical Web analytics data feed is so poor as to be hardly a model at all; it’s built for interchange not usage. A really good collection infrastructure will house data in a good data model and the two are largely inseparable.

Finally, there is the issue of real-time. Few aspects of digital measurement are as poorly understood and misrepresented as the need and function for real-time measurement. Traditional tag-based Web analytics tools were often sold on the premise that real-time data collection and reporting was a significant and important advantage. In general, that’s simply not true. Very few business decisions and very few analytic tasks can or should be tackled in real-time. There are a few verticals and few problem sets where true real-time reporting is essential. For most of our clients, it’s simply unnecessary. So the 24 hour delay inherent in sourcing your data warehouse from a Web analytics data feed should be no big deal, right?

Wrong. Because if you want to use your digital data to drive personalization or re-marketing, time suddenly becomes much more important. Real-time decision-making is fundamental to good personalization and re-marketing because nothing, NOTHING, is more important than what a customer just did. While some people believe that this type of real-time decision-making is the sole purview of black-box tools, I strongly disagree. The best personalization opportunities will come from the integration of customer data with real-time data  using rule-based, analyst driven optimizations.

If I’m right, then sourcing your data warehouse from a Web analytics solutions will effectively preclude you from advanced personalization and remarketing. Even if that means little to you right now, I suggest that building your warehouse on a technology stack with such a critical limitation is short-sighted.

In short, if you’re committed to the warehouse as your solution for digital customer analytics and optimization, there are some pretty compelling reasons why sourcing that warehouse from your Web analytics solution isn’t ideal. It looks easy, it may be the right direction for a pilot, but it places some potentially crippling limitations on the long-term capabilities of your program.

In my next post, I’ll show how the Celebrus solution – a digital measurement infrastructure built specifically for the data warehouse – solves some of these challenges and explain why I think it ought be getting serious attention from any enterprise focused on digital data warehousing.

Or you can get the whole white paper here….

Answers to your questions!

By Scott Wilder, Gary Angel and Marshall Sponder

As usual I enjoyed the recent Social Media Measurement webinar – and it was great to have Marshall on as well. Tools always draw a crowd and this was no exception. Here’s the questions we got along with our joint answers…

Question: What tools are best for measuring social media ROI or business lift, with respect to advertising on Facebook, Twitter, Linkedin, etc

Marshall: There’s actually a new platform launching next week called Unified (UnifiedSocial.com – I will be at the launch) that promises to do something like that – I’ve seen the platform close up and I can tell you I am impressed.  It may be that 2012 will be a year where ROI will no longer be a totally elusive goal for social media.

Gary: This is far more difficult, I think, than people generally believe. The only easy path to ROI measurement is when user’s are either directly engaged in commerce on social sites (which is rare) or are directly clicking through to sites where they are engaged in commerce. In these cases, measurement is generally a straightforward application of existing Web analytics campaign tracking capabilities. Unfortunately, this isn’t often the case. In some cases, I’m not even sure that ROI is the proper path to measurement and where it is, I don’t think there is likely to be one answer or approach. If your Facebook advertising is directed toward increasing your Fanbase, you need to be able to measure the incremental value of Fan (and this won’t be one value by the way) to your marketing. Getting that measure takes a concerted research effort and won’t (in my opinion) be delivered by any single tool. I sometimes think that it might be better for organizations to – first glance – concentrate on the obvious optimizations points. It’s much easier to measure which campaigns generates engaged Fans and calculate their cost-efficiency in that respect. You can then optimize campaigns within the set of those targeted toward increasing your fanbase. It’s not ideal, but it is more practical.

Scott: In most cases, companies have to guestimate true ROI because of some of the limitations of the tools and also companies own infrastructure. I find it useful to create proxies – like determining cost estimates for certain activities, which in turn, would lead to a transaction.

Question: US cost is too high – example Engage121 is $1000 per month for first base level search – one profile with 3 seats.

Marshal: Well, as Gary pointed out, Engage121 is designed for a specific use case and type of client such as an airline or large franchised business with thousands of stores that each want a different response and editorial controls – think Dominos or Dunkin Donuts (though I think neither are Engage121 clients).  My point being, you can’t take the price of a platform in isolation from the use case and clients for whom it is designed and targeted to.  The Dominos and Dunkin’s of the world have plenty of money and need for this kind of platform – but if your looking for an “affordable point of entry” into Social Engagement- than go with HootSuite and be happy there are still some free platforms you can play with and get your feet wet.

Gary: Not every market is going to be served by a tool like Google Analytics – free and really good. I basically agree with Marshall here. One thing I will say that’s more general is that in my experience some pricing models are much worse than others for doing serious enterprise work. To do our kind of measurement (Semphonic) we need a pretty free hand to construct, test and use profiles of all sorts and we generally need quite a lot of them because all the interesting questions involve categorization. At the enterprise level, I’d much rather pay a significant lump sum for a pretty free hand with the data than have a pay-per-item model. Pay-per-item models tend to cripple analysis.

Question: Do you have preference for tools to measure public opinion about political candidates – public policy or litigation issues?

Marshall: Yes, I am working with one right now – 6Dgree.com – we are tracking two candidates in Rhode Island and breaking down their overlapping audiences – along with “persona” breakdowns of their twitter streams – here is what that looks like (I erased the names of the candidates because this is still in the very early exploratory stage of what works).

So far, the persona development breakdown looks impressive, as we can break it down by various sub dimensions and the founders at 6Dgree are very willing to pursue my suggestions, which really impresses me about them.  So yes, as of now, I believe 6Dgree might have a winning platform at an affordable price level that works for Twitter and Facebook.  Another is PeekAnalytics, but it’s not adapted specifically to Politics, yet.

6Dgree has done some interesting work with Australian Labor party around issues and produces a weekly portal report that breaks down tweets around several issues – I’m impressed with the solution, but of course, each campaign is slightly different and customization will always be a fact of life.

Question: What are the better tools for global internal scale? If any? Or just by world region?

Marshall: I like Comscore Media Metrix for world reporting – but that’s mostly panel based reporting -but it does a fairly extensive job of categorization of lifestyle and interest across channels, countries and technologies such as video, mobile and search.

Gary: Ditto Marshall. I like NMIncite for many larger markets. Alterian provides excellent language coverage.

Question: Do you believe the sampling of data should include statistical testing? Or how do you ensure your sampling is reflective of the entire population to provide confidence in the recommendations?

Marshall Well, Gary has a pretty good post on that, written recently, and I think, rather than speak to it, I’ll let Gary address it http://semphonic.blogs.com/semangel/2011/11/the-limits-of-machine-analysis.html

Gary: Thanks for the plug! Let me know if the several blogs I’ve written on the subject don’t fully answer the question! Social Media Measurement is an odd blend of attempts to get universal coverage and hidden samples – which makes a single approach challenging. You can use statistical testing to measure the variations in your samples and, where possible (it isn’t at all levels) that’s certainly advisable.

Question: When one wants to search and analyze Twitter postings and the topic is very low salience, so likely a very, very small percentage of Twitter mentions in U.S. in a given week, what are the best ways to maximize the amount of Twitter Firehose that you search to catch as many Twitter postings on your low salience topic as possible?

Gary: Depending on your method of access, you might want to start by talking with your vendor (if you’re using a vendor to make the initial data pulls). The initial pull is often tunable. This also speaks to your ability to capture the topic in all its forms. Traditional keyword research of the type often done for long-tail SEO can be useful. There is a range of tools appropriate for this – we’ve also just used scanning tools to pull the text off of sites (both client Websites, communities, and competitors) to try and build rich topic profiles. You can also take advantage of wildcards (in some tools) to scan from hash tags that include but are not limited to your topic. Hash tag references are often concatenations of the topic with other words and are nearly always pertinent. Sometimes, too, you have to be creative about what you’re looking for. If, for instance, you’re launching a product that is distinct, you can’t expect to identify potential influencers by targeting the obvious words – they generally won’t have any traction. So you have to look for analogs that might allow you to find and target a reasonably set of influencers.

Q: Any views on Netbase, which SAP just partnered with?

Marshall: Yes, it seems like a good partnership. Netbase does a pretty good job at NLP and creating structure and meaning around unstructured social data, and rather than SAP trying to build that (or buy Netbase, which is an option) they just partnered with them.

Scott: Netbase is doing some really interesting stuff, especially when it comes to Netnography (see www. Netnography.com). I think the partnership with SAP will be good because I know that the company is putting a lot of energy into understanding their own segmentation better. We are doing some work for them right now. SAP is also making a big push in mobile analytics and would probably pull Netbase into.

Question: Gary, perhaps you could ask each speaker to summarize which tool they think is strongest in each of the three key use cases you’ve outlined?

Marshall: Here’s a list of companies to consider

• For PR Effectiveness  - I’d say mPACT and Cision.
• For Consumer Sentiment – I would recommend be NetBase (in fact) for its NLP capabilities.
• For Social Campaign Effectiveness – Unified (once it launches)

Gary:

• For PR Effectiveness: NMIncite – though it does a poor job with identifying influencers the segmentation is excellent for tracking them.
• For Consumer Sentiment: Clarabridge and Crimson Hexagon – though we haven’t gotten to use Crimson Hexagon as much as we’d really like.
• For Social Campaign Effectiveness: This is a tough one. Most of the new management tools provide some integrated reporting – but I think that really good effectiveness measurement demands that level of reporting plus Web analytics, plus traditional listening configured for the purpose, and maybe CRM-based extracts at the individual level as well (we sometimes analyze Facebook campaigns by extracting all the individuals and looking at their pre/post behavior).

By Gary Angel

After I posted my blog on measuring and benchmarking overall Site Satisfaction, Marshall Sponder sent me this comment/question:

Another great post! Question – how does NetPromoter scores figure into the points? They are, after all, Survey based.

It’s a really interesting question, because I believe there are both similarities and differences relevant to my discussion. A quick recap – in my last post I argued that overall Site Satisfaction suffered the same issues as almost any other site-wide metric. Site-wide metrics – be they Conversion Rate or Revenue or Site Satisfaction – all confuse multiple factors together in a way that makes them almost useless and un-interpretable. This is contrary, of course, to the broad industry view of KPIs, but it’s a topic I’ve canvassed thoroughly in previous postsand I have yet to hear a convincing argument to the contrary. In addition to this problem common to nearly any site-wide variable, Survey data – when collected by traditional site intercept means – also suffers from a sampling problem. Because your site population varies with your marketing efforts, you’re mostly measuring shifts in the underlying population you’re attracting when you measure (or compare or trend) site-wide Satisfaction scores.

So what about NetPromoter?

On the whole, NetPromoter scores will suffer from pretty much the same problems. When you measure NetPromoter scores using site-intercept surveys, your likely measuring changes in your sample population not changes to your actual customer likelihood to recommend. So a trend or benchmark of NetPromoter scores is no better, in this respect, than Site Satisfaction.

However, there are a few differences. As I thought about Marshall’s question, I realized that in many respects my criticism of overall Site Satisfaction mirrors my criticism of Total Mention Counts in Social Media. In Total Mention Counts, you’re adding up fundamentally different things into a meaningless whole (mentions in the NY Times + Twitter Customer Support Mentions doesn’t equal an interesting Total Mentions). It’s similar with Site Satisfaction. Adding Site Satisfaction for Customer Support visits to Site Satisfaction for Pre-Purchase Visits to Site Satisfaction for Brand Visits doesn’t really add up to a meaningful number. The meaningful numbers are all at or beneath the Visit Type level.

NetPromoter scores, on the other hand, ARE coherent across both Visit Types and an entire population. Willingness to recommend is independent (to some extent) of whether you are buying, or getting support, or finding out about the brand. You’d still probably want to understand the impact of visitor and visit type on NetPromoter score but it’s not totally unreasonable to think about NetPromoter as an attribute of an entire population.

This also tells us something about what NetPromoter isn’t. It isn’t, for example, a good way to measure success by visit type. Site Satisfaction is actually much better for that.

Indeed, as I re-read my posts, I don’t want to leave the impression that I dislike Site Satisfaction as a metric or that I am opposed to online intercept surveys and their use. Not at all. Online surveys are incredibly valuable as is the Site Satisfaction question. You just have to understand and work effectively within the limitations imposed by the sampling method. In fact, I think Site Satisfaction is a better metric than NetPromoter if you’re intent is to measure visit-level site experiences. It gets at something much more specific and real. What Site Satisfaction isn’t, is a metric independent of those visit types in a way that makes it a plausible candidate for aggregation or site-wide benchmarking.

By Gary Angel

If there’s one question in Digital Measurement that I genuinely hate, it’s this: “How does my “x” rate compare to my competitors?” where “x” might be conversion rate, bounce rate, shopping cart abandon rate, or any other fairly important metric. I hate the question, because it is unanswerable. I’ve written before how movement in any single site-wide metric is un-interpretable in the sense of being action-guiding and, for similar reasons, it’s virtually impossible to meaningfully compare any single site-wide metric between two or more competitive sites. If a metric doesn’t mean anything when applied to your own site, why would you expect it to mean something when compared to another site?

If you are doing this, you’re misleading (I originally wrote “lying to” instead of “misleading” but perhaps not everyone actually knows better) your stakeholders.

It’s just that simple.

Your site-wide conversion rate (for example) is a function of your site design, your visitor population, your marketing program and your brand (and probably a bunch of other stuff too). It cannot be meaningfully compared to even the most accurate benchmark set at the site level.

Is it any different in the world of opinion research? Is survey data any more comparable than Conversion Rate?

As a site-wide metric, the answer is clearly no. Site-wide Satisfaction is a function of your site design, your visitor population, your marketing program, and your brand (and a bunch of other stuff too). It takes unified data collection, meaningful segmentation, and the ability to hold constant a large number of potentially influencing factors before any comparison can be done – and that comparison will NEVER, EVER be at the site-wide level.

So comparing your Overall Site Satisfaction to any competitive set, be it true competitors, industry leaders, world class Websites or Websites featuring clown pictures is just not useful. There simply is no learning you can take from a comparison of overall Site Satisfaction. Note that I’m not saying this because of the (very substantial) difficulties in building a valid competitive set. Those difficulties are real and legion – but my criticism of site-wide satisfaction holds even when we all agree that the benchmark set is perfect.

At a deeper level, however, survey data CAN help compare your performance to competitors in a way that behavioral numbers cannot. Survey data can indeed establish competitive benchmarks; however, their utility is entirely dependent on the type of sample you use for the job.

In the online world, the typical online survey sample is bound up with your site visit population - a population that is heavily driven by day-to-day changes in your marketing program. Under such circumstances, the one thing you can be sure of when you measure your total site satisfaction and compare it to other sites (using their intercept survey results) is that YOU ARE NOT MEASURING THE LIKELY SASTISFACTION OF A RANDOM CONSUMER VISITING EACH SITE AND THEN RECORDING THEIR SATISFACTION WITH THE EXPERIENCE.

So you cannot use a traditional site-based online intercept approach to the survey sample if you’re intent is to create a valid competitive benchmark.

To illustrate why this is so, consider an example of four companies that are exactly alike in their business and are, therefore, 100% comparable. Each company has five visit types. Satisfaction for these visit types range from 59% to 71%. Each site is completely identical in structure and design except for the name of the company and each visit type has identical satisfaction scores on every site.

I hope you’ll agree that this benchmark set is an absolutely implausible best-case in the real-world. Any problem with this benchmark is certainly not in the chosen competitive set.

Suppose, however, that the distribution of visit types to these identical sites is as follows:

Even with four perfectly identical sites, the average site satisfaction across all visitors ranges from 64% to 67% in our hypothetical example. In other words, even with four completely identical sites, with five completely identical sets of visit types, and the exact same satisfaction with every visit type on each site, a small difference in the distribution of visit types on the site can result in a significant variation in overall site success when measured across the entire visitor population.

Such a difference might easily result, for example, from differences in SEO programs where external site link building generates different page rankings and skews the visit types in one direction or another.

Any decision-maker, looking at Site Total Satisfaction comparison, will believe that Site 4 is worse than Site 2 – even though the sites are COMPLETELY IDENTICAL IN EVERY RESPECT INCLUDING CUSTOMER SATISFACTION BY VISIT TYPE AND CUSTOMER SATISFACTION ACROSS EVERY SINGLE MEANINGFUL VARIABLE.

I can’t help but think that showing a decision-maker this one number is a gross misrepresentation of reality.

This effect is not limited to visit type. It is true for every single measured variable and it is true for any effort to “trend” the data.

Imagine the implications for the real-world where “comparable” sites are actually dramatically different in marketing drives, structure, function and audience.

In no way does sampling your Website visitors constitute a single broad consumer population for ANY business and the concatenation of multiple Websites all with very different audience biases does not solve the problem. As I pointed out in my last post, when you gear up your PPC program, improve your SEO, or take down your Display advertising, you are changing the population you are sampling on your Website and changing your top-line numbers. Nine times out of ten, that’s what you’re measuring when your top-line Satisfaction scores change.

So the critical point about benchmark samples is that – to be valid – the numbers have to come from a single controlled sample that is independent of site marketing efforts. If your competitive set comes from numbers collected by other sites in the same fashion as you collect yours, then keep in mind that every member of your competitive set is changing their sample in ways that you simply cannot measure or understand. Adding your bad sample to their bad sample doesn’t make for a good sample. And comparing your bad sample results to a number produced by a good (independent) sample doesn’t solve the problem either. The limitations on your site sample make effective site comparison over time essentially impossible.

It’s true, of course, that in the example above we could – by drilling down into additional variables – show that the sites are identical in all but Visit Type distribution. To do this requires full access to the ENTIRE set of competitive data. The simple top-line comparison is meaningless. What’s more, only the extreme simplicity of my example makes it possible for drill-down on additional variables to easily succeed.

In the real-world, where large enterprises have hundreds or thousands of marketing programs going and constant web site changes underway, isolating a comparable population will be challenging even with full access to the underlying samples.

Do any of the enterprise scorecards that show a comparative satisfaction benchmark do that work? I think not.

Do you have identical visit types to your competitors? Identical demographic questions? Identical qualifying strategies? Identical survey branding? Identical visitor type categorizations? A controlled single sample?

A single, independent sample and a single, carefully crafted survey instrument are the minimum requirements for the job. Even then, you’re going to find it very challenging to answer the basic question you probably (should have) started with- given a typical consumer of some type, does your site experience for a specific task yield a higher or lower satisfaction than the competition?

If you’re using a sample based on traditional site intercept methods, a sample driven by your site marketing efforts, and you are relying on a single top-line Satisfaction metric to compare site performance, you might as well – like ancient Greeks before battle – be shaking bones on the sand to predict the future.

Small differences in the visitor population, it’s sourcing, or your sample can easily create the impression that you are doing better or worse than your competitive set; an impression that has absolutely no basis in reality.

To evaluate your business based on such numbers is madness.

It’s just another great example of a metric that looks oh so interesting but serves to hide, not reveal, the truth.

By Gary Angel

In one of my first posts on Social Media Measurement, I pointed out the hidden difficulties of sampling social data and the impact this has on the use of Social Media metrics for consumer and brand research. When you can’t pull a representative sample, your research results are biased and, in some cases, unusable. Because social media measurement has pretensions to providing universal coverage, most people don’t realize they are sampling the data and have paid little or no attention to the sampling issues there. That certainly isn’t true for online opinion research. Everybody understands that online opinion research is based on a sample. But how good a sample is it? Is the quality of our sample changing over time? And, finally, what are the limitations on the uses of online opinion research given the limitations of sampling?

These points become particularly important if you try to, as I’ve suggested you should, use online opinion research for more than site satisfaction tracking. When you’re trying to explore the drivers of consumer choice, understanding the “shape” of your sample is going to be critical.

Let me give an obvious and ubiquitous example. Suppose traffic to my web store (and subsequent conversion) is declining and I want to use opinion research to find out why. I can create a series of questions for visitors to my Website to find out why some people didn’t end up purchasing. But suppose the real problem is that a segment of my previous customers has simply stopped needing my product. If that’s true, I’ll never find it out from asking visitors to my Website. The consumer audience I’ve lost has no reason to visit the site and be sampled.

This most basic online sampling limitation (site visitors) may seem almost too obvious to note, but it’s implications are profound and more easily missed than you might think. With online survey research, you’re sample is always based on the population of site visitors. This makes online survey research inappropriate for brand awareness, brand tracking, and early-stage consideration research.

Here is a related and critical fact about online opinion research that is easily forgotten: as you gear up your PPC program, improve your SEO, add functionality or Content, or take down your Display advertising, you are changing the population you are sampling on your Website. In traditional opinion research, your sample and your marketing are entirely unrelated. That’s simply not the case when it comes to online surveys.

With that in mind, it’s fair ask how representative is an online sample of the population of site visitors? I’m really not sure. I see two very common attitudes about this question. Most common, perhaps, is a blithe assurance that’s it all fine. People who aren’t schooled in opinion research generally don’t appreciate how hard getting a good sample is and don’t really recognize how damaging a bad sample can be to most uses of the data. Almost as common is a deep but largely unsubstantiated concern that your online samples skew toward the “happy” and the “unhappy.”

If all you’re doing with your online opinion research is trending site satisfaction, this might not actually matter very much. As long as the skew is relatively constant (which is at least possible), then a sample skew toward strong emotional visitors may not matter. Of course, given the fact that your sample is influenced by your site marketing, there may be no stupider use of online opinion research than trending topline satisfaction scores!

Regardless, if you’re trying to use opinion research to understand customer drivers of choice, then oversampling the strongly opinionated population can lead you significantly astray.

There are several ways to check the quality of an online sample. The simplest one, a method I generally recommend, is to integrate the sample with your behavioral data (Web analytics) and measure the representativeness of your sample versus key behavioral metrics. Doing this will not only allow you to measure sample skew, it will allow you to oversample or weight your sample using behavioral metrics to study under-represented populations.

If you’re sampling a known population (such as registered customers), you may also be able to measure the sample against known exogenous data (such as customer demographics, relationships, account size, purchase history, call volume, etc.). When available, this method has some distinct advantages since it will allow you to measure sample skew and to potentially oversample or weight your sample against key consumer characteristics and that’s clearly better than using behavioral metrics.

I don’t think there is one right answer to the quality of online sampling. Not only will it vary based on offer methodology, choice of survey instrument, industry vertical, and strength of brand, it may change over time. In myWAA session in Philadelphia, several of us wondered about whether surveys have become too pervasive on the Web – damaging every organization’s ability to pull a representative sample. In the offline world, the dramatic growth of telemarketing made phone surveys much harder to do well – too many customers simply opt-out of the medium making it almost impossible to get a good sample.

In our experience here at Semphonic, most online surveys we’ve tested have done a fairly creditable job of sampling the “known” population. They have also mostly oversampled the behaviorally very engaged (likely translating into the most and least happy). You should not take either of these statements as a given. The use of anecdotal experience as a predictor of wider truths is a classic sampling error and one to which consumers of case studies are all too prone!  We haven’t studied this problem with anything like a large enough sample to be broadly predictive.

Test it for yourself – it’s the only way to be sure.

By Gary Angel

One really nice aspect of my recent series of posts around social media is that they’ve generated quite a few thoughtful and useful comments. Like every other blogger, I love comments. Next to someone ringing us up and asking Semphonic for a proposal because they “follow the blog”, it’s the best evidence I have that people are actually engaged. As an extra bonus, really good comments are grist for the mill – it’s much easier to write a conversation than a monologue (though perhaps less easy to manage). And these issues and questions around methodology in social media measurement interest me deeply, so it’s pleasing that my interest is not isolated.

In particular, I wanted to address this three part question/comment from Jason that I got this week. Here’s the comment:

I liked your post very much. A few points, if you could further elaborate: 1. “most social chatter has absolutely nothing to do with PR effectiveness”. How would you define PR effectiveness? Is it computable according to your definition? 2. “Those influencers have nothing to do with measuring traditional consumer awareness”. What about traditional consumer awareness. I am interested in the computational aspect of the term. 3. Don’t you believe there should be a holistic approach, in any case we are talking about a unified brand that we try to quantify. Thank you in advance.

Some of this presages issues that I hoped to talk about in more depth in the future, but I’ll take at least a first cut at them here. I’m going to start at the back of these comments and take one more whack at the concept of Total Mention Counts and measurement samples as a way of thinking about a holistic approach to brand measurement. I’m not really sure that this is what Jason has in mind, but I feel that while I’ve taken several whacks at the “Total Mentions” problem, I have yet to hit a clear home run.

Let me start by re-visiting the difference between traditional opinion research and PR brand monitoring. In traditional opinion research, the goal was to capture and understand consumer sentiment and attitudes. Questions like “how many people know our brand” and “what do people think about our brand” and “what would make people buy more of our brand” are all answerable with primary research. They are answerable not because you can talk to every consumer, but because you can sample a small but representative set of consumers and, from that sample, extrapolate with some confidence to the total population.

Building a representative sample for traditional research may sound simple but in fact, it isn’t. If you call people during the day, for example, you’re likely to reach mostly the unemployed, the retired or housewives. That’s likely not a representative sample for most businesses of their customer base. It may be that women are more likely to take a survey than men. Or that seniors are more likely to give you their time than young adults. There are a thousand different factors that can influence a sample and most commercial (as opposed to academic) research doesn’t attempt to do more than control for the most obvious biases.

But there is one bias that nearly all survey samples control for – they exclude anyone with any expertise in the industry. If you’ve ever been called (and I’m sure you have) for a survey, you’ll remember that there’s almost always a few qualifying questions. If the survey is for a health care provider, they want to make sure you don’t work for the client or their competitors. Likely, they want to make sure you don’t work in the industry at all!

Why? Because if you are in the industry, you know too much, you have too much bias, and your views are almost by definition different from the general consumer population. Nor are your opinions likely influencable by the normal channels of marketing. So you’re answers are going to distort the research, not make it better.

Keep this in mind – it isn’t just that traditional opinion research isn’t designed to study influencers, it’s specifically constructed to AVOID their opinions.

PR monitoring services, on the other hand, focus exclusively on one subset of people that are excluded from opinion research. PR monitoring services want to know what they influencers are saying. They want to know about mentions in the NY Times. They want to know whether you’re mentioned at expert Conferences.  They don’t care at all about Joe Consumer’s opinion of your service or drivers of choice. That stuff simply doesn’t matter when it comes to PR.

This is all clear, simple and straightforward when it comes to traditional research. No PR person would ever have thought to use random digit dialing to figure out whether he’s influencing key people. No survey designer would ever have bought of list of journalists and experts to target for his consumer opinion research (though I can see some merit in the idea if you’re planning a PR strategy).

With Social Media, there’s a misleading sense that these lines are erased. Blurred, perhaps is true, but they are by no means erased. Everybody is most certainly not an influencer. With blogging and twitter, there has been a vast increase in the amount of user generated content (UGC). But UGC is still readily bucketable into distinct types and categories that (should) mirror these previous distinctions.

Here’s a personal example. If you’re a Web analytics tool vendor then Gary Angel is an influencer. You’d be wrong to survey me if you’re goal is to understand what would make an enterprise buyer purchase your product. I’m not an enterprise buyer even though, heaven knows, I talk endlessly about Web analytics. In fact, my profligacy of output on the topic is generally hard evidence that I’m not a consumer – and this topic and posting density is a metric we use when we create automated samples of consumers and influencers.   On the other hand, if you’re Kayak.com or ESPN and your interested in consumer attitudes about travel or sports, I’m perfect. I’m not an influencer in those fields though I most certainly have a blog.

Or consider the Web analytics twittersphere. The vast majority of #measure tweets are generated by tiny number of devotees (nearly all with a strong interest in self or company promotion) who generate a vast amount of chatter amongst themselves. Certainly my own somewhat infrequent posts on Twitter fall in this category. In what sense is this interesting if you’re trying to understand what an enterprise  buyer of analytics might care about?  These are the LAST people you should be thinking about – far from measuring their attitudes you need to be going out of your way to exclude them from your sample. They are as unrepresentative a bunch as you could possibly imagine.

So yes, Social Media has created many more niche influencers. However, most of these influencers are topic specific. Within that topic, these influencers need to be treated as if they were traditional media or company spokespeople not consumers. That means you need to EXCLUDE their opinions if your goal is to understand your customers. Fortunately, that’s quite possible even in an automated fashion. By measuring their output and topic focus, you can nearly always identify a professional signature even when you don’t know who they are.

So let me sum up the argument here one last time (I hope). If your goal is to understand your consumers, then you need to be monitoring their conversation. By definition, experts and influencers ARE NOT your consumers. Their opinions are different from your true consumer population and they are vastly overrepresented in the opinion-sphere of social media. So unless you specifically exclude them from your consumer research, you aren’t measuring what you think you’re measuring and you aren’t measuring what you want to be measuring accurately.

The converse is equally true. Not everyone is an influencer worthy of PR attention. Kayak.com doesn’t need to send me their press releases just because I have a blog. It won’t help and I won’t read it. And if they are measuring the effectiveness of their PR, they don’t need to be sampling my blog.

The simple necessity to create a sample that is representative of your research target is NOT removed by Social Media. Far from it. You need to devote even more attention to this than you might in traditional research precisely because the bounds are blurred. When you random digit dial, the chances of influencing your research by oversampling influencers is quite small. With social media, this is most definitely NOT the case.

So Jason, I don’t know if I’ve really answered your last question because I’m not sure I interpreted it right, but I guess I’d sum it up this way. I’m most definitely not against a holistic view of the brand. At Semphonic, we routinely try to create just that in our Social Media reporting. But a holistic view of the brand doesn’t come from counting everything in a single bucket. It comes from careful sampling of different audiences each of which is important to a holistic view of the brand, but none of which should simply be added together. Creating a unified view of a brand might well encompass a report that showed how strong the brand was with influencers AND how strong the brand was with consumers, it would NEVER mix the two samples when generating those answers or even mix the two answers.

I hope that makes sense. This is probably my third or fourth cut at this point and I hope, at last, that I’ve gotten it right!

Now I’m going to skip back and tackle the first question (that middle question may just have to wait). How do you measure PR effectiveness and is it computable? The answer to this last part is yes – it’s computable if you have a good sample of PR influencers and that having a good sample of PR influencers is the critical ingredient in measuring PR effectiveness.

I think of PR effectiveness as quite distinct from viral marketing in that it targets a specific group of people (Influencers) and has two readily measurable goals: getting influencers to talk more about your topics and to talk more and more favorably about your brand.

A PR sample will surely include all traditional media sources as well as careful identification of influence-based online sources. This doesn’t include all blogs or all tweets – but only those generated by those who qualify as topic influencers.

I’ve shown these before, but I’m going to trot them out again. Here’s a PR report we created for a client that shows how topic focused potential influencers are:

The gist of this report is to help show which online sources are important for a given topic. The NY Times is probably influential across almost any topic. Online sources, however, are much more likely to be topically influential. If you’re interest is in ASP, then Blogs.MSDN.Com is authoritative. If you’re interested in Ajax, not so much.

We’ve refined this work recently into a generalized segmentation approach to social reporting. We identify individuals as influencers based on their posting frequency and topic specificity. This allows us to sample these influencers separately without prior identification (the list above was based on a pre-selected list). Equally important, the converse sample allows us to create a much cleaner set of true consumers.

Once you’ve identified a set of influencers, you can now track your mindshare and message share with them over time. Here’s an example:

The affinity graph is showing how much each source overlaps our client’s core topics. The Us vs. Them shows how much that source discusses our clients vs. their competitors. The shift measures how the Us vs. Them has trended in the past month. If you’re PR is working, you should be seeing shift – either with specific targets (as in this report) or with the overall set of influencers (depending on your tactics).

I think this is a perfect example of PR effectiveness measurement. Your goal with PR is to shift influencers to talk more about your topics and your brand. If you can effectively identify your target population, you have every opportunity to do just that.

I hope all this makes clear that while I have deep doubts about certain aspects of Social Media measurement I am confident that it has considerable value when used properly. Measurement of PR effectiveness is an entirely appropriate use of Social Media and an excellent example of a case where social media lets us measure a task far more effectively than was ever possible in the past!

I still have that post on survey research in my plans – but I don’t think I’ll be trotting it out for Christmas! Perhaps that will be the post to kick-off the New Years with…

Since I may not write again before the holidays, I’ll wish everyone a happy holiday right now.

From what was, for me, a troubled beginning, this has been a much better span of months than I had any right to expect. Business has been wonderful and interesting. Building Semphonic is more enjoyable and fulfilling now than at any time in the past 14 years.  I finally, after many years, left the City (as we title San Francisco) and made the short hop across the Golden Gate to Marin. Though I miss the burritos and the Asian food, we have a beautiful new home and I most certainly don’t miss the fog! Best of all, of course, my daughters have given me the endless charm of their last years of true childhood. Their enthusiasms, their music (so much better than my own), and even their Macbeth (from theatre this summer) were simple gifts that graced the year.

In my first post of 2011 I quoted the same famous lines that title this post. In work and life are we not always and forever going “once more into the breach”? It seems so to me.  I hope your year, too, has been good and that you’ll continue to keep company with me in 2012.

By Gary Angel

In what has become something of a tradition, Phil Kemelor (VP here at Semphonic) released this year’s edition of our Profiles in Analytics research. Profiles in Analytics is based on our own survey research of the Web analytics community. Each year has been fascinating and so, too, is the apparent evolution of the community. There are three parts to Profiles in Analytics – all based on the survey data. There is, first, the actual summarized survey results. The second piece, which gives the research it’s name, is a set of representative profiles based on the survey responses. The final piece is a write-up of the research that highlights some of the key trends and most interesting findings. All are available on the Semphonic web site.

It’s obvious from  a year-over-year comparison that there has been fairly significant growth in the size and maturity of digital measurement groups. Well, that should be obvious to most of us. What’s more interesting is the clear growth in maturity across a number of different fronts including Phil’s focus in the write-up: Web governance. But just how important is Web governance – does it allow organizations do more with less or is it, perhaps, an excuse to do less with more? The survey suggests some interesting answers. You can check out Phil’s write-up along with the survey results and the always interiguing profiles here.

I also wanted to address a few of the questions/comments that I’ve gotten in the last few weeks on prior posts. It’s  been incredibly busy here with the year-end rush to get work out the door before the holidays so I haven’t had boatloads of time but there were a couple of comments I wanted to address.

I’ll start with this comment from Jennifer Roberts at Collective Intellect with reference to my post on the futility of Total Mention Counts:

I think I understood where you were going with the whole futility of measuring mentions. Is the value of a NY Times mention vs a blog mention important? I think you definitely have a point but I’m wondering if it isn’t necessarily the mention but what it is that is expressed is what’s important. I think the initial analysis that identifies the ‘mention’ is only the starting pt., the next step is to extract meaning and that can mean different values to different companies. Retailers may be more interested in consumers expressing an intention to purchase; entertainment companies may focus their analysis on consumers expressing an intention to view, reducing the total volume numbers to an overall trending value. I don’t know that you can discount the ‘total mention count’ anymore than you can ‘total visits’; each provides some context for the more important metric.

On the whole, I don’t think Jennifer and I fundamentally disagree. We both think the most interesting questions are at a lower level than Total Mention Counts. But I’m more deeply skeptical about the value of the Total Mentions Count (as I am, to a lesser degree, about Total Visits - we at Semphonic strongly discourage Web reporting around total site metrics). Not every metric is worth reporting on just because it exists.

Let me give an example.

I could add together Entries and Exits on a Website and call it “Doorways“, and I can almost believe that if a tool reported this metric, that some analyst would use it. But “Doorways” isn’t interesting – it hides, not reveals, the interesting level of reporting. There’s nothing interesting about adding the Entries and Exits on a page. Similarly, I don’t see value in adding mention counts across fundamentally different channels. If you’re goal is to measure PR, then you need to focus on one kind of mention count. If you’re goal is to measure Consumer Sentiment, then you need to focus on a different kind of mention count. It’s not clear to me that when you add these two types of samples together that you’re counting anything more real than “Doorways.” I’m just not sure what I’m measuring when I use a statistic like “Total Mention Count” and I’m coming to think I’m not measuring anything at all.

I’ve also been thinking a lot more about my post on the changing role of Customer Satisfaction Surveys. In that post, I argued that most site intercept surveys were over-focused on the wrong sort of questions – one’s about site artifacts not customer needs and attitudes. I’m more convinced than ever that this is true, but I’ve also been thinking about some of the sampling problems inherent in online surveys -problems similar to one’s that I wrote about in the Social Media space.

I got this comment from Bryan Henn:

Gary, Great piece. It is important for any analyst to remember the age of social platforms, and the diverse landscape for user generated content. One thought I have shared with my team members has related to the idea that noise is often created around the volume of product sold, and the discounts that apply. Coupons, free offers, etc, plague the ProActiv market as you referenced, where as you are unlikely to type coupon for Ford 500 into Google and get many results. I read through the WCAI study and “Insights for Practitioners” but it seemed to lack this thought process. As you mentioned though, I think its best that practitioners continue to apply tested measurement and optimization strategies to an emerging segment such as social. If we can optimize conversions or attract engaged visitors through social channels, we are doing something right. Thanks.

Bryan’s point is dead-on. One of the most challenging aspects of doing Social Media measurement is separating out the fruits of your own (and competitive) marketing efforts from conversations that reflect un-channeled consumer conversation. This is really hard, but it’s a key part of building a good social sample – regardless of your intent. It’s interesting to measure the impact of your (and your competitors) marketing. It’s interesting to measure current consumer sentiment. But just as with Total Mention Counts across channel types, it’s not interesting to mash these two things together.

Reflecting on this point, however, I’ve realized that site intercept surveys are quite vulnerable to similar problems. There are some obvious limitations (as well as advantages) in the use of site intercept surveys to measure consumer sentiment, but there are hidden challenges here as well. I’ve been thinking about both and I’m hoping to discuss this in detail in my next post. I’ll foreshadow by saying that these problems place a premium on behavioral integration but also, rather unfortunately, make a mockery of any attempt to use online survey data for benchmarking purposes.

This whole issue of the right focus for online survey research is a deep one with all sorts of interesting offshoots. There are organizational and process issues here as well. David Harrod pointed that out in this comment:

I think the insight that using site csat tools for purposes beyond site csat is essential, and one you get to within 6 mos. of implementation. The interesting thing about both is that they rely on the same type of engagement between the product teams or business stakeholders and the analytics team.

I think that’s right on both fronts. There’s usually a gap between deploying online survey technology and understanding what it’s really for (though six months may be a “best-case” experience). Part of the reason for this gap is a communications disconnect between the survey owners and the people who could most take advantage of the data – a disconnect that effects many other aspects of Web analytics practice. In other words, one of the reasons why online survey’s are so uninteresting is that they aren’t designed, in most cases, by the right people!

By Greg Dowling

I was recently interviewed by Ritesh Gupta over at EyeForTravel.com and wanted to share the contents of it with my readers here as well.

Q: As an expert in mobile measurement and enterprise analytics, what do you recommend when it comes to making the most of mobile strategy especially in the travel sector? How should travel companies go about making the most of their investments in this arena?

A: When thinking about mobile strategy in the travel sector, the primary place to look for direction is within your existing site analytics. Start by looking at which specific mobile devices and operating systems are visiting your site – where are they coming from, what are they doing, and what content are they consuming. By understanding the mobile usage of your current site and the areas where users are succeeding or failing, along with what devices they are using, you will be able to inform your mobile product strategy greatly. Understanding what site content or features resonate with mobile users will allow organisations to address the primary need of their site visitors and help direct the mobile strategy. Determining your overall product suitability to a mobile offering and which devices or experiences you need to support will ultimately drive mobile strategy and identify where you need to focus your mobile efforts.

Q: It is highlighted that understanding how customers behave when using mobile devices is a major challenge. What are your observations and how do you think understanding behaviour is significant in assessing the overall measurement of any mobile product strategy and related initiatives?

A: Measuring mobile has historically been very hard to do given the fragmentation of devices, operating systems, and carrier restrictions. The majority of these hurdles have been eliminated over recent years with the advent of the smartphone and mobile application measurement SDKs. Previously, feature phones incapable of executing JavaScript posed a significant measurement problem as they relied heavily on manually (or server) generated image requests making measurement of mobile initiatives and device side experiences difficult. Current smartphones can execute JavaScript and device resident applications making traditional web measurement methods viable for mobile websites and applications. Device and operating system fragmentation still persists complicating measurement deployment and organisations need to focus on their current user technographic to understand which devices they need to support from a measurement perspective in order to ensure measurability.

Understanding mobile user behaviour is critical to evaluating and optimising mobile strategy. After all, you can’t improve what you don’t measure.

Q: The expectations and demands of smartphone customers are significantly higher than website visitors. What sort of benchmark should travel companies set for themselves when it comes to measurement of mobile web and app-related initiatives vis-à-vis any other component of digital strategy?

A: The mobile user expects immediate satisfaction and demands simple and clean interfaces. It is the nature of the mobile user. They are not ‘surfing’ (generally) and are attempting to complete a specific task when engaging with a mobile website or application. This can be booking a flight, hotel room, or dinner reservations as well as researching destination highlights but all of these have a singular focus – completing a task. For all intents and purposes, mobile users should be converting on these tasks at higher rates and engaging more frequently than traditional website visitors. Often this is not the case due to complicated checkout flows and of course this varies with the utility and appropriateness of the mobile experience. Organisations should focus mobile benchmarks at or above their traditional website rates and optimise the user experience and conversion flow of their mobile initiatives accordingly.

Q: How do you assess the maturity level of mobile application measurement framework at this juncture? What should travel companies learn when it comes to measurement?

A: Historically, mobile applications were developed and released without any inherit device side measurement and the primary metric was downloads. While downloads are an important metric, we have come to realise that user engagement along with recency, frequency, intensity and duration are more actionable metrics providing a holistic view of application success. Mobile application measurement has improved greatly over the last two years and releasing a mobile application today without measurement is heresy. Niche mobile application analytics vendors held a lock on this market until recently and in some cases still provide a superior solution, however all enterprise analytics vendors currently offer robust SDKs. The SDKs allow for the ability to measure all device side interactions (even when the device is offline) and by instrumenting your application with the appropriate level of tracking to capture metrics such as install date, app launches, daily usage, and key event success organisations will gain a greater understanding of actual application usage not just application downloads.

Q: Considering the varied utility and engagement level of mobile web and mobile apps, how should one approach measurement for both? What should one avoid in order to have unjustified expectations?

A: Mobile web measurement should align with fixed web measurement wherever possible. Existing measurement frameworks and implementation methodologies translate well to the mobile web environment and should be leveraged wherever possible as to not reinvent or double efforts. Technology frameworks aside, the mobile website experience is inherently different than the fixed web experience and organisations should avoid a wholesale migration of all of their content or site functionality to their mobile web experience. Screen size and input methods weigh heavily in mobile product strategy decisions and determining which elements and in what format to display these elements is crucial to mobile website success and ultimately mobile website usability.

Conversely, mobile application measurement shares little with traditional fixed web measurement and only the high level success frameworks and key metrics will translate well to this medium. Aside from the technical differences in implementation from mobile web measurement, application measurement focuses primarily on a subset of achievable tasks. Organisations should avoid porting all website content and functionality into their mobile application and focus on the key aspect or element of their product offering that would make the user want to engage with them in a mobile environment. Simple, direct, and facile task resolution should be the primary component of a mobile application.

Q: What do you make of the “Web Versus Application” debate in the mobile product strategy at this juncture? Would it be right to say that the pros and cons settled down now?

A: Mobile websites and mobile applications both have their appropriate place within a mobile product strategy – it is not an either/or debate. Ensuring the appropriate user experience is present in each is the true debate. All too often organisations will make a technology decision rather than a strategic decision in the form of “We need an iPhone app” or “We need to run an SMS campaign” without considering the product suitability to a mobile environment or why the customer would want to engage with their brand in the first place. At this point, a mobile website is a must and should be at the top of your mobile strategy list if you don’t have one. Once that is in place and you have gleaned actionable insights into how customers are interacting with your product offering then, and only then, should you craft a mobile application strategy.

Q: Where do you foresee mobile application measurement framework headed in the next 12 months?

A: Mobile application measurement needs to become much simpler to implement. A few vendors have succeeded in simplifying the development process by enabling automatic capture of device side events and variables but this needs to improve considerably for continued widespread adoption. Currently developers are expending tremendous effort to instrument measurement in their mobile applications significantly increasing time to market for new applications – this needs to change. Additionally, location based services are changing the way we visualise data once collected from mobile applications. Given the fixed nature of traditional web browsing this element has not had much attention until now. Being able to see where and when a customer is interacting with your mobile application creates a whole new dimension to the traditional marketing strategy. It is no longer about getting the right offer to the right person at the right time – but about getting it to them at the right “place”.

Q: What do you make of the mobile product strategies especially the usage of mobile apps in the travel sector?

A: The hospitality and travel industries have embraced mobile fairly well and have incorporated mobile strategies into their product marketing and service offerings adequately. However, some of the most innovative mobile product strategies I have seen recently in the travel sector have originated from capitalising on the “location aware” aspect of modern smartphones.

For example, Vail Resorts, using RFID technology, is able to tell when their customers are “on” or “off” mountain and provide appropriate messaging and content to them along with social network interaction opportunities. By capturing this location based data they are able to expand and optimise the on mountain experience for their customers. The ability to track your progress, vertical ascent, capture photos, share achievements with your social network, and have all this in your pocket is truly amazing!

I’m feeling a bit schizophrenic with my blog these days, bouncing between posts on digital database marketing,big data analytics, survey research (all our traditional milieu) and then back to Social Media Measurement. We’ve done so much interesting work in Social Media measurement recently and it’s a topic that I’ve hardly yet touched – so I feel compelled to continue that series. And yet, I hesitate to pull away from our main practice themes for any extended period of time. If you’re interested in the same range of topics that we at Semphonic are, it’s probably not much of a problem. If you’re interests are concentrated inside (or outside) of Social Media, then all this hopping around might be a bit annoying. I’m going to see if we can’t work out a solution – perhaps separate feeds by topic – since I fully expect to continue both themes.

Which is a long way around to saying that my topic today is a look at a classic digital measurement problem. Some of the most interesting work that we’ve done at Semphonic in the last few years has involved the integration of Call-Center and Web data for a couple of our largest clients. We rarely get a chance to talk about that work. It’s a shame that we generally aren’t free to say much about it, because it’s fascinating stuff involving warehouse integration, true customer-level analysis, and massive cost-savings – work that’s produced some of the largest ROIs that we’ve seen in our more than a decade of practice.

In this blog, I’m to show some sanitized data pulls that are representative of what we usually see in this type of analysis and explain some of the basic techniques and approaches. I’m going to draw from  work for  several different clients but all share a fairly similar structure – these are operational (non-marketing) Websites and Call-Centers. The goal here is not to sell product but to service customers and solve problems in the most efficient manner possible.

In this situation, the goal is generally call-avoidance. It’s much cheaper to service a customer via the Web than by call. A call for these clients will probably cost somewhere between $4-$7 to service. A Website visit – even with fairly generous cost-allocations – will cost only pennies.

Savings, however, are by no means guaranteed. A Website can actually generate more calls than it shifts if it isn’t well designed. Nor is it desirable to avoid every call. Some types of calls are difficult or even impossible to service on the Website and some customers are happier or more profitable when serviced in the Call-Center.

So it’s a good to start a Call-Center project with a “waterfall” analysis like this:

The idea in a Waterfall chart like this is simple. You start with the total universe of “All Calls”. You subtract out calls that are not applicable to anything on the Web – in this case about 1/3 of all calls. From the rest, you remove those that are deemed to have been unavoidable (here a very small percentage of the remaining calls).

This gives you your “Total Avoidable Universe” of calls. In this example, we’ve segmented these calls by Customer Type (the Gold/Red/Blue bars). The Customer Type is often critical to the call-avoidance strategy. For really high-value customers, we may have a strong bias toward keeping the high-touch, but more expensive Call-Center contact unless the customer really wants to self-service.

The next several break-outs refine these groups into those who use the Website and those who don’t. For non-users, we have a straightforward Activation Strategy – get these customers to try Web servicing when appropriate. For Web-users, we have a more challenging problem – figuring out why these visitors didn’t self-service online.

To understand that problem, we further divide this universe using Recency measures. There is always a population of lapsed Website users – and this population presents us with what is, essentially, a re-activation challenge.

For customers who don’t fall into this category, we like to further sub-segment by problem type. Understanding how many calls dealt with difficult to solve problems can help us understand how large the “slam-dunk” call avoidance opportunity is.

In this case, about 65% of calls are, in theory avoidable. Almost 70% of these avoidable calls represent an activation challenge since they came from customers who aren’t Web users. Of the remaining 30%, a heavy majority DO come from active Web users and about 1/3 are problems that are easily solved on the Web.

A diagram like this helps management understand the scope of the opportunity and understand where the biggest wins are. Here, it would probably make sense to target activation of non-Web Users (70% of avoidable calls) and easy problems of Active Users (10% of avoidable calls). The first is the largest opportunity and the second is the easiest win.

This type of table-set is also important in situations where an organization is debating the right mix of Call-Center and Web. Some organizations have strong bias toward self-service because of the cost-savings. Not everybody feels that way. There are organizations that pride themselves on the ability of their customers to talk directly to them. Fair enough. But here’s the rub. Some customers don’t want to do that. If you’re goal is really good customer service, it’s not enough to say you have great call-centers. Lots of customers (including me) prefer to service online when it’s possible. This type of Waterfall analysis with customer segmentation can help you identify customers whose needs aren’t met appropriately – whether that’s in the Call Center or on the Website.

To do this analysis requires a join between Call-Center data and Web data. That isn’t always possible. However, where you have Websites that have logged in capabilities, you can make this link. If you have customers logging in to track order status, to solve operational problems, or manage their relationship with you, then you have the ability to create this join for at least a subset of your Website traffic.

By joining Web data to Call Center data, we’re able to create interesting analysis at the individual customer or company level. That waterfall chart isn’t just for analysis. Using those segmentations, we can target the actual customers who fall into each of these buckets. That lets us target frequent call-center users that are inactive or have never used the Website and are not in the highest value customer segment. And that’s pretty much what we do. It’s a logical first step when building a call-avoidance program and we’ve had terrific success with it.

What about that segment of Active Website users with “easy” problems who somehow still ended up calling the Website?

Here’s a look at the time-lapse between the Website visit and the call to get a sense of how long you have to react to these types of situations:

Notice the absolute cliff at around 15 minutes after the last Website touch. Calls peak between 1 and 3 minutes after the last Website touch.

There are a couple of really interesting learnings in this view. First, it provides ample evidence of the relationship between the Web visit and the call. Where your (or your customer’s goal) is self-service, these sessions represent a clear failure. Second, it suggests that your window for call-avoidance is very short. You either have to fix the Website so that it solves the problem or you have to be able to react in something like real-time to send out an email or initiate a chat.

Waiting even an hour to address this problem will doom the effort. The vast majority of calls will long since have taken place.

This is just one of many cases where your time to response is critical in the online world. The need for (and role) of real-time and the demands of a real-time technology stack for decision-making are topics that I plan to tackle in more depth over the next month or so. This Website to Call-Center is a terrific illustration of the need part of the equation. In ecommerce, you rarely know how quickly a customer rolls off your Website and into another decision. But with Call-Center, the time-lapse between Web and Call is completely measurable.

A real customer-level join of Call-Center and Web data makes three analysis tracks possible. First, it allows you to separate out and individually identify customer-segments that SHOULD be self-servicing specific problems and aren’t. You can target these customer segments with activation or re-activation messaging. Second, analysis of Website to call sessions by call-type let’s you predict how impactful activations are likely to be on Call-Center volume and to calculate the actual savings from your Web effort. Finally, you can use the bridge between failed sessions and calls to identify Web usability problems. You can even target surveys at this population if the web behavioral data isn’t providing a clear picture.

You’re looking at the entire spectrum of analytics in those three efforts: from what amounts to traditional database marketing (albeit with a cost-avoidance twist) to predictive analytics to classic Website optimization.

Call-avoidance may not be the glamorous side of the Web. There isn’t a lot of scope for clever design or branding. But there are world-class returns on investment and a significant and obvious set of analysis paths that use data integrated at the customer level. To my mind, that makes it one of the best areas for an analytics department to focus on and one of the top integrations to consider when it comes to making warehouse and big data decisions in the online world.