Developing a Metric for Engagement

For several years the topic of measuring engagement has been important to marketers and still is, according to the recent study quoted in eMarketer (newsletter, January 26, 2010). The most desirable metrics can be interpreted as engagement—with your website or with your ads. As the chart suggests, the state of the art is still single metrics. Here’s another example that's interesting both for content and for metrics; the level of engagement with social media in various countries by comScore late last year. They use time on site and number of pages as the key—and separate—metrics. It seems to me there’s a need for a multivariate measure of engagement. I’ve looked around and have found two different approaches.

Business Week has taken a variety of steps to increase engagement with its readers, including hiring an engagement editor. Here’s an interesting summary of what they have done; I don’t find an update so far in 2010. Most germane to my point, they have developed a proprietary User Engagement Index. Here’s how they describe it:

We developed a proprietary set of metrics to help us both track, and make us accountable for, our goal of being the business and financial site with the deepest and most meaningful engagement of its users. The index is the ratio of our outputs to the world (the stories and blog posts we publish) to the world's inputs to us (perspectives on stories and blog posts from readers as well as their guest columns.)

It makes sense to me and it is an approach any business could sit down, think about, and adapt to its own product category and situation.

If you want a ready-made solution, I found that also. Dutch metrics supplier Nedstat has an engagement solution that’s based on basic website metrics. Web Metrics Demystified recently did a good post, which included this graphic. There are two basic components to the Nedstat approach. First, the user develops her own engagement algorithm; that follows the Business Week example. Engagement is not the same for every product category, every website. Perhaps even more important, it’s going to differ on the basis of your own communications strategy—what you are doing to try to encourage customer engagement.
Second, Nedstat has a solution they call Live Segmentation that allows you to choose a customer segment and calculate the engagement index for that segment. Both the engagement algorithm and the segmentation choice are said to be easily varied as the marketer considers better metrics or strategy options. Aurelie Pols’ post rightly points out that you can do this with traditional web metrics, but it may be easier to work with a metrics company that can guide you through the process.

What it comes down to is interesting. If you are serious about measuring engagement, you need to construct your own multivariate metric. You can DIY or you can engage a consultancy. Either way, it’s going to take some thinking, some work.

Another object lesson reinforcing the fact that none of this is easy!

Listening for Customer Understanding

I’ve expended a lot of energy trying to get marketers to really grasp the difference between marketing research and the analysis of behavioral data as avenues to customer insight. Sometimes I despair. We academics are partly at fault. Most marketing courses, especially the introductory ones, teach marketing research as the way (read that the only way) to learn about customers. We forget that there is behavioral data of many types, from many sources, that should be mined before enterprises spend time and money on marketing research.

Senior marketers have the same hang-up. Marketing research is what they were taught. They may feel comfortable with how it is done and the results it produces (or they don’t use it at all). But I thought an Ad Age headline (subscription required) earlier in the month captured the essence of the problem. When I used this slide in class last night, I got a lot of puzzled (disbelieving?) frowns. Let me see if I can restate the issue in terms of why companies need to listen as their first step in understanding customers in a way that leads to viable marketing and business strategies.

The ARF held another conference on listening yesterday and I looked this morning to see if any material had yet been posted. Not yet; I’ll keep looking but I did find a great quote in a blog post from ARF president Joel Rubinson. He says:

Listening is about hearing what people [say] rather than the marketer wants to talk about, (emphasis mine) and hearing it in people’s own words. It’s a window in the mind, heart and emotions of people, one you need to have your nose pressed up against continuously. Because things change…really fast…giving agile marketers great opportunities leaving traditional marketers wearing the WTF happened look on their faces.

As I looked, I found something else really good. I don’t know whether the ARF is happy having this report, clearly marked ‘confidential’ posted on the web, but for now at least, it’s there, so I’ll link to it. As all good researchers do, they start with an operational definition of listening:

“The study of naturally occurring conversations, behaviors, and signals, that may or may not be guided, that brings the voice of people’s lives in to the brand” (p 11)

They parse the definition in a useful fashion and go on to say a lot of important things about developing a listening strategy. You should read it for yourself. It’s long, but the main concepts take up only about 20 pages, followed by an incredible number of good short case studies, and ended with a discussion of technology and platforms. Let me leave it with the brand-related objectives that can be realized, in full or in part, by a well-crafted listening strategy. Their list is:

• Discover New Customers
• New Product Development and Innovation
• Improve Existing Products
• Maintain Sales Momentum
• Drive Brand Growth
• Re-brand or Re-position
• Tackle Public Policy Issues
• Manage Reputation
• Manage Brand Health
• Customer Care
• Increase Loyalty and Customer Value (pp 13-14)

I don’t think any of us can disagree with the desirability of any one of these objectives. That still begs the question of how to produce the best data to meet the selected objective(s). Try this conceptualization.



We have to listen to conversations that relate to our brand. That’s one kind of qualitative data. Behavioral data from everything to site visits to transactions is one type of quantitative data. Each of those types of data quickly becomes gigantic. Together they are mega-gigantic. For a brand of any size or reach, they have to be thoughtfully mined to come up with actionable insights.

Then the question is whether there is anything missing. In researchers’ terms, not just something interesting that we’d like to know, but data from which we can draw actionable insights. If the answer is ‘yes,’ we may need to do marketing research. It could be anything from a poll (or a series of them) on our website to a custom marketing research project. If it’s really important to a full picture of the customer, then do the marketing research. All the while behavioral and conversational data floods in. Also, the world moves on, which is another problem with pausing to do conventional marketing research.

That’s the nature of the challenge and it’s formidable. In the process of this investigation I’ve learned more about useful platforms that can help meet the challenge. More about that next week!

Are Behavioral Segmentation and Self-Segmentation the Same?

Last week an Ad Age article with the headline “The Death of Consumer Segmentation?” caught my eye. I consider segmentation a fundamental tenant of marketing, just as catchy headlines are a tenant of journalism. Anway, I read with interest. I’ve been part of some of those really big segmentation studies he talks about as well as many smaller ones. All produced value in an earlier era. In my opinion, their day has past.

Michael Fassnacht’s three key points were:

1. Static consumer segments have little value in a rapidly-changing environment
2. Consumers are never part of just one segment
3. By controlling the communications they receive, consumers have control over marketing activities.

The key take-away: big segmentation studies are out; consumer self-segmentation is in. He also uses an interesting term, “enabled self-segmentation;” I’m unclear whether he meant that to be synonymous with “self-identification,” which he refers to in the preceding paragraph. He ends with a section on how marketers can enable consumer self-segmentation, which is reasonable but I didn’t think went far enough.

I looked at Fassnacht’s own blog and found a couple of interesting posts. I’d recommend his reflections on the election (Binary Thinking, November 9, 2008) in which he rightly states that human beings are not one or the other in a given situation, they are highly nuanced. I looked further and found a recent post describing "micro analytics," which I though was important. The key steps, developed by Huayin Wang, are:

1. Treat UGC as the raw gold of data information. Find the right methodology to score the individual UGC pieces by relevance and relationship to each other. This kind of quantitative exercise will enable one to decipher patterns within the large universe of UGC, either on Flickr, YouTube, on blogs, etc.
2. Identify the right UGC content clusters to understand marketing opportunities. This will enable one to isolate potential opinion leaders within a certain content grouping as well as unveil unleveraged perception spaces for a particular brand.
3. Build a persuasion platform that uses the different attributes of each UGC element for an interactive program, all based on the principles of behavioral targeting.
4. Analyze the modified UGC landscape after a sufficient period of time to understand if the interactive marketing program has created any positive impact for the brand.

Wonder how many companies are doing something like that--not many I'll bet! It makes wonderful sense, and it's also based on the analysis approach used in the really big old marketing segmentation studies. It just uses behavioral data, not survey data--a huge step forward.

The situation is, as I see it, that the marketer can use this approach--self-segmentation uncovered by micro analytics--for analysis of content data collected from across the web, provided by unidentified subjects, similar to anonymous visitors on a website. Yes, you could link a Facebook behavior to a member (a person becomes a fan of a brand page, for example), but would it be worth the effort? I sincerely doubt it!

But there's an even simpler solution that few companies (except publisher sites with an array of newsletters) seem to be practicing. ASK THEM! In other situations, I've called that expanded permissioning. Ask registrants what they really want to get from you and how often. That would let them self-identify, at least to the extent you offer options/segments that are relevant to them.

Then take the next step. You might not be offering exactly what they want--chances are good that you are not. How do you ask them or analyze their activity on your site to find out what else they are looking for? The answer is--you set up ways to Listen--in ways I've talked about before!

That sets up a virtuous cycle in which the marketer first asks then uses analytics for deeper understanding. Time-consuming expensive traditional marketing research need not apply!

Identifying and Measuring Social Media Behaviors--Part 2

As I suggested yesterday, I’ve been struggling to organize the chaotic world of social media metrics into a comprehensible framework. While working on this I ran across thought-provoking material from comScore.












The report, “How Online Advertising Works: Wither the Click” is the result of a controlled study of exposure to online display advertising. Among other interesting findings, it makes the point that there is a long-time lift in website traffic from exposure that is not measured by mere click-through. What’s even more fascinating is that they found a smaller lift in traffic to competitor sites. When you stop to think about it though, isn’t that what shopping is all about? Do you necessarily buy in the first store you go into?

A follow-on post by comScore CEO Gian Fulgoni considers question of whether advertisers should buy ads on a CPM or a CPA basics. The answer continues to be “it depends” and it has implications for both publishers and advertisers.


This was all lead-up to preparing a presentation on social media metrics for my class tonight. I’ve got a start on a framework and some examples. There’s undoubtedly a lot more in both areas, but it’s only a 2-hour class!

Seriously, I've uploaded the presentation to Slideshare (you may need the new version of the Flash player) and I’d love your feedback. Do you think I’m on the right track? What would you add? delete? Do you have other great DIY sources for social media metrics?

Customers Rate Experiences

Forrester has released its 2008 customer experience index report, based on consumer ratings of their experiences at firms covered in the study. Bruce Tempkin has posted some data on his blog and has a link there to the full report.

Forrester has 3 basic experience criteria—usefulness, ease of use, and enjoyability. The report gives a brief overview of their methodology. There are interesting comments on the blog and in response to one, he has given a little more detail on the methodology.
















The results are interesting. Retailers and hotels rank highest of the industries studied. Health insurance and TV service providers are at the bottom. The large range of experience ratings given to ISPs is interesting.

In some ways, I’d say the top-performing firms are the usual suspects. Have you ever sat down in a comfy chair and browsed through some books at Barnes and Noble? The one I go to doesn’t have its own coffee shop; that would add even more to the experience. USAA is always near the top on satisfaction studies; one assumes that their superb customer service is a huge factor in the overall experience rating. When you look at other high-performing firms, they’ve worked hard on customer service, so it seems reasonable to me that the basic blocking and tackling matters. Then if you add a coffee shop or a pizza parlor on top, you can offer great customer experience. But you can’t buy great customer experience with only coffee or pizza, no matter how good they are! If customer service stinks, nothing else really matters.

It’s good to choose one or more of the high-performing firms to study and observe. For instance, there’s not a Cosco near me; I don’t shop there and was surprised by a student analysis of just how good their customer service was a few semesters ago. It also helps to follow one or more firms outside your own industry; that may open up new ideas.

Customer experience is the focus at the moment—on the web and off. It’s worth developing a vision and a strategy and devoting time and effort to offering great customer experience. It pays off, perhaps in sustainable competitive advantage.

Measuring Engagement

This could be a really short post. There is no commonly-accepted measure of engagement—bye, see you tomorrow!

Clearly that’s not very useful, so I’ll present some perspectives and approaches. According to the Economist Intelligence Unit in part 2 of their Beyond Loyalty report,

Some executives have decided that precise measurements may not be possible, and are trying to satisfy themselves with more general measures. “Quite often, the customer is satisfied, and if the customer satisfaction index goes up, that’s good enough,” says Mr Jennings of Reuters.

Interestingly enough, they point up the fallacy of the “we can’t measure it” approach in the paragraph immediately before:

Nearly half of our survey respondents say that the difficulty of measuring engagement is perhaps the biggest barrier to achieving greater levels of customer engagement. (both quotes page 9)

And I am often reminded of the quality management truism, “What gets measured, gets managed.” So ok, metrics are important, and some commercial measures are available. A search of comScore press releases turned up 45 releases on engagement and revealed metrics such as “visits” and “duration.” These are important site statistics, but are they a complete measure of engagement? Not as we’ve defined it.

About a year ago Jerimiah Owyang summarized a number of approaches to the engagement metric in an excellent blog post. It has lots of links and many interesting comments and it’s useful background leading up to a report released by Eric T. Peterson and others on September 7 of this year. This 54-page report is called “Measuring the Immeasurable: Visitor Engagement.”
Peterson and his colleagues go into great detail on the measurement issues and if you’re a metrics wonk—or if measuring engagement is mission critical to you—then you should read it all. I’ll summarize in lay terms, starting with Peterson’s conceptual definition:

Visitor Engagement is an estimate of the depth of visitor interaction against a clearly defined set of goals.

That’s a statement that incorporates behavior both on and off the website, and that’s important. As stated, though, it’s not measurable. Their computational definition is:

“Visitor Engagement is a function of the number of clicks (Ci), the visit duration (Di), the rate at which the visitor returns to the site over time (Ri), their overall loyalty to the site (Li), their measured awareness of the brand (Bi), their willingness to directly contribute feedback (Fi) and the likelihood that they will engage in specific activities on the site designed to increase awareness and create a lasting impression (Ii).”

Here is how they define the variables:
Click Depth Index: Captures the contribution of page and event views
Duration Index: Captures the contribution of time spent on site
Recency Index: Captures the visitor’s “visit velocity”—the rate at which visitors return to the web site over time
Brand Index: Captures the apparent awareness of the visitor of the brand, site, or product(s)
Feedback Index: Captures qualitative information including propensity to solicit additional information or supply
direct feedback
Interaction Index: Captures visitor interaction with content or functionality designed to increase level of Attention
the visitor is paying to the brand, site, or product(s)
Loyalty Index: Captures the level of long-term interaction the visitor has with the brand, site, or product(s)

The good news is that these all appear to be metrics that can be derived from or added to existing metrics programs. That’s also essential, because a good metric for engagement must be part of a comprehensive metrics effort.

My guess is that this metric or one like it will soon be available from metrics firms, although I can’t find any evidence of it yet.

Marketers need to decide whether engagement is an important part of their ongoing strategy. The first post in this series suggests that it should be. Engagement is part of all the models of new media strategy, including mine.

The major point of this series of three posts is that engagement is more than choosing “engaging media.” It represents the outcome of ongoing dialog with customers and the larger community around a brand. Making that work requires both commitment and a rational strategy. Are you working on it?

Part 1 here
Part 2 here

Drivers of Consumer Engagement

Yesterday I wrote about engagement—what it is and why it is important. The key take-away is that the combination of consumer and employee engagement improves bottom-line performance. I’ll concentrate on the consumer side and ask, “How do we go about producing consumer engagement?”

A superficial answer is that the new social media are interactive and engaging. That’s true, but it doesn’t provide guidance for marketing strategy. It’s not enough to just add some videos or start a company blog without having clear goals in mind and strategies for getting there. So the real question seems to be, “How do we get consumers to engage with our brand?” The chart from a recent Jupiter Research report distinguishes between interaction-based and communications-based engagement techniques and asks which ones marketers are using. That’s an interesting perspective, but it doesn’t address the strategy issue. Notice that their sample is “social marketers;” usage of these techniques is not this high among all marketers.

Last month I wrote about a CMO study of consumer experience with leading brands across several communications channels. That study focuses on the role of consistent, integrated content delivery across all channels—communications engagement in Jupiter terms. The most important part of the report (download here) may be the latter pages in which they give the items on which they rate the brand performance in each channel. Those items suggest performance benchmarks. However, that still doesn’t necessarily guide marketers in search of their own strategy, particularly which elements of social media will produce the most useful kind of engagement.

A study in 2006 by Carat for IAB in the UK is helpful. It focuses on mothers with small children choosing among several small car brands. The research on their information sources and decision processes summarized in this chart is pretty standard—multiple items (in this case “contact points”) factored into several engagement factors. Assuming typical marketing research, those factors have a significant impact on engagement with the brand. I’m not clear, though, how they measured brand engagement. There is an interesting Guide on this page, along with a series of brand engagement studies including this one, but I haven’t found the specific metric for engagement. I’m accepting it because they seem to be using “brand engagement” consistently, and I'm sure they must have defined it somewhere, perhaps in the IAB member content.

This kind of study (and they find the drivers to be different from one industry to another) illuminates the strategy development process. If the marketer knows the criteria (the factors, in this case) on which the consumer makes decisions, she can work to move the needle on one or more important factors. In this case, style is most important. Which experiential/ engagement techniques will move the needle on style? Video, perhaps? A customize your car, in this case around the needs of your children and resulting lifestyle? Fun is the second most important factor. How do you allow female customers to participate, even help to create, the “fun” part of the car experience? And since they want a feminine car, how do you deliver this to women and not to men?

I keep thinking that there is so much that marketers COULD do in the arena of social media. The strategy question is, “What SHOULD we do?” There is no one-size-fits all answer—this view of engagement makes that clear!

What Should Marketers Be Measuring?

Last week I missed a webinar conducted for Awareness Networks by Jerimiah Owyang of Forrester. He’s one of the most thoughtful observers of the social media scene, so I took some time over the weekend to catch up. (You can download both the webinar and slides from this page.) That also reminded me of a white paper from Coremetrics on measuring social media (download "Winning in a Web 2.0 World" from this page; they have a webinar by that title coming up soon) that’s been languishing in my files for too long. I thought I’d try to pull together some metrics issues from the two. They take quite different approaches.

Coremetrics focuses on Web 2.0 technologies, roughly divided into User-Generated Content and Rich Media. I’ve selected a few Coremetrics that are most relevant to the content of this blog. You can see the full list on pages 11 and 12 of their report.
Blogs are now considered “mainstream.” How do we know whether they are contributing to our program? First, traditional traffic measures, which are easy to obtain from the blog platforms or metrics programs. Who is adding content could help identify influentials. Since Coremetrics offers an integrated measurement platform, the paper discusses in some detail the issues of following visitors through multiple sessions in multiple media to achieve a conversion. That’s important, but more difficult than simple traffic measures. User reviews need similar measures of traffic and conversion paths. If you look carefully at the Social Networks entry, there are two issues here. One is traffic generated by social network sites/brand-related activity. That’s an entry metric that needs to be carefully filtered to understand traffic sources. The other is results of ads on social network sites which are generally best monitored by ad-specific landing pages.

This is helpful, but how does the marketer know where to start, which of the many techniques and channels to choose? Jerimiah Owyang gives a strategic overview.

His strategic objectives, which are pretty much in order based on a marketer learning curve are:
• Listening comes first. Spend some time hanging out where your customers hang and listen to what’s going on. Techniques of reputation management provide a formal programmatic approach. Tools include Google Alerts, blogs, discussion boards, Twitter and FriendFeed.
• Speaking comes next. We marketers are good at that. Just be sure you’re using traffic, audience and content measurement techniques to know what’s working.
• Energizing may be essentially the same as engaging your audience. Get them to use apps that deliver content and attract them to your site. Try spinning the wheel on Axiom’s PersoniX program site to see a B2B marketer engage potential customers. The easiest metrics are participation/interaction/click-through measures. Following what they do after the initial interaction is the subject of the Coremetrics whitepaper.
• Once you identify the activities your audience is interested in (not necessarily the same as the ones the marketer thinks they should be interested in), Support them. The recent post on the new Harley-Davidson social networking pages is a good example of support. Support also lends itself to traffic measures. Following those through to conversion requires more complex metrics.
• Then encourage them to contribute content, ideas, suggestions. Owyang calls this Embracing—draw your users into the community fold. Make them a part of the community. Give them a feeling of ownership by making them active participants. Traffic/participation measures are important here. Can you follow ideas/suggestions through to successful activities or products and measure the ROI from the activities or products?

On the surface these two approaches to metrics—technologies vs. objectives—look rather different. When you scratch below that surface, you see the same basic sets of metrics, which are essentially based on the techniques you use to achieve the objectives.

The take-away for marketers is not to be swayed by the siren song of technology. Good objectives always come first. Then choose technologies that:
1. Design marketing programs, using appropriate technologies, to achieve the chosen objectives
2. Provide measures of the degree to which objectives are being achieved.

Interesting that good marketing is the same, whichever channels are being used. Without good objectives, nothing else matters much.

Is It the End of Surveys?

You may have noticed an article in Ad Age last week, “The End of Consumer Surveys.” It is based on an ongoing program by the Advertising Research Foundation that aims—not to do away with consumer surveys—but instead to integrate conventional marketing research and social media metrics to give a clearer picture of how well brands are doing to get their message out. “Out” is meant in the broadest sense; not just how much is being spent on brand messages in conventional media but how well that is translating into (unpaid) online chatter favorable to the brand.

ARF is being supported in this effort by marketing research firm TNS and its Cymfony Division, which specializes in social media. TNS has a number of resources posted on their site. The ones I was especially taken with were a 2007 white paper “Making the Case for a Social Media Strategy,” (chart at right) and the ARF webcast ”Social Media Analysis for Consumer Insight: Validating and Enhancing Traditional Market Research Findings.” I’m going to hit some of the highlights and try to bring them together for you. The webcast runs about 40 minutes and is worth listening to in its entirety (access it from this page). There’s another in this series coming up on November 12 and other related webcasts on the ARF events page.

Using brands of HDTV as their basic case study, the researchers from TNS used what could be called marketing research triangulation to look at brand impact. They used:

• Conventional marketing panel research to understand conventional brand metrics like favorability
• They used projective techniques to uncover the attitudinal (“emotive”) dimensions that characterize brands in the category
• They used convention web metrics (e.g. visits to websites) and social media metrics (e.g., brand mentions in blogs) to measure online activity and chatter.

Two key metrics which they report—and which they find to be interrelated—are share of market and share of discussion.


One of the products of their analytics that I found most interesting was a list—ordered by “contact clout”—of various contacts points that affect consumers’ decisions about brands. Experience with the brand is really important—points 1, 2 and 4. The first mention of traditional media ranks 9th—articles in newspapers and magazines. This probably reflects the innovative and technological nature of the HDTV product. TV comes in at number 14, but remember the metric is “influences brand decisions” not “creates awareness.” Nevertheless, the preponderance of personal and in-store information sources is striking.


But the main theme of the ARF/TNS collaboration is the need to combine conventional marketing research and social media metrics. Their argument is that just looking at brand activities is not sufficient. It is also absolutely necessary that you look at what consumers are doing and saying. The tough part is bringing the two together. I’ve argued the point in a somewhat different way; use behavioral analysis—first—to find out what people are doing; then conventional marketing research adds value by finding out why. The TNS analysis adds to that argument by demonstrating that you can get some of the whys from online chatter.

Their recommendation for effectiveness includes two kinds of communication—marketer generated and consumer generated. The marketer needs to work at nurturing (“spinning”) social networks and generating real online dialog between consumers and marketers.

The ARF has long been known for serious scrutiny of marketing research issues. This is an important effort to move marketing analytics in a direction that is essential to marketing effectiveness in the Web 2.0 world.