I'm a partner in the advanced analytics group at Bain & Company, the global management consulting firm. My primary focus is on marketing analytics (bio). I've been writing here (views my own) about marketing, technology, e-business, and analytics since 2003 (blog name explained).

Email or follow me:


« September 21: Business Blogs Lunch Talk at KM Cluster | Main | New Gig »

June 17, 2005

OPA Breakfast Meeting: User Experience and Engagement

Opa_3_1On Wednesday, Jeffrey Rayport and I attended an Online Publisher's Association breakfast in Boston.  On the agenda:  presentation and panel discussion of really interesting new research from the Northwestern University Media Management Center on the links between user experience and consequent engagement with web sites.

[This write-up is based on my notes from the session.  Northwestern is apparently planning to wait until after their speaking tour to post their presentation on the Web.  I haven't vetted the accuracy of my notes with them yet.  If you have a different recollection, or you work for Northwestern and want to correct anything below, please comment at bottom.]

One conventional question many publishers ask is, "what can I do to drive traffic to my site?"  Principal metrics most people consider are things like hits and unique visitors per day.  But Tom McCauley, the research program's manager, noted that while publishers feel they have been making progress on these dimensions, user "engagement" with a site -- how much time they spend on it, what they do while they are there, and what they tell their friends about it  -- is an even more important driver of the economic results that site advertisers and sponsors care about, and has not improved as much.  Further, our current yardsticks may be deficient.  "Customer satisfaction measures are not enough," Tom noted.  "Satisfaction without engagement is worthless.  Satisfaction with engagement is priceless."  Why? "You really need to know what people *tell their friends* about you," said Tom.  "In fact, the only path to profitable growth may be to get your customers to be your sales people."

So, Northwestern's research posed the following questions:

1. how can we usefully describe the different experiences people have on publishers' sites?
2. how do these experiences relate to engagement?
3. of the experiences that drive engagement significantly, which ones are under-exploited, and what site attributes can be used to stimulate these experiences?

The methodology used for this research was very similar to that which Northwestern has used in the past for newspapers and magazines.  Northwestern first did a bunch of in-person, one-on-one interviews where they asked people to name their favorite web sites and talk in an unprompted way about their experiences on those sites, what they liked and didn't.  The sites varied widely across the following categories:

1. national news
2. local news
3. news aggregators
4. entertainment
5. special interest (e.g., health, cooking)
6. games
7. business

The comments from these conversations were first reduced to a set of 154 "statements".  Next, Northwestern conducted a more structured survey of 2,000 people and asked them to rate each statement for how effectively it describes their experiences on their favorite sites.  From this, the research further reduced/ grouped reactions to a set of 22 experiences:

1. "entertains & absorbs me"
2. "looks out for people like me"
3. "regular part of my day"
4. "my personal time-out"
5. "credible, safe place"
6. "connects me with others"
7. "touches me and expands my views"
8. "makes me smarter"
9. "turned on by ads"
10. "easy to use"
11. "helps and improves me"
12. "worth saving and sharing"
13. "tailored for me"
14. "guides me to other media"
15. "makes me feel I belong"
16. "a way to fill my time"
17. "something to talk about"
18. "my guilty pleasure"
19. "tries to persuade me"
20. "worries me"
21. "annoyed by the ads"

Photo_061505_003[Right, only 21.  I was scribbling fast and missed the 22nd.  But hey, not bad, since this isn't costing you anything.  If you've got the last one I'd like to hear from you.]

Question number two was, "how do these different experiences drive site usage?"  To answer this, Northwestern devised a Site Usage Measure (SUM), expressed on a 7-point scale, on which respondents would describe time spent and average number of sessions per day on different web sites they mentioned they use.  Then respondents were asked to assess, again on a 7-point scale, the degree to which they have each of the 22 experiences when they visit a particular site.


The resulting scatterplot and the associated regression analysis allows us to answer some interesting questions:

1. what is the effect on engagement with the site of increasing the likelihood of a user having a particular experience with the site?
2. which experiences provide highly-leveraged opportunities for differentiating the site?  (i.e., reported rarely but very engaging?)

Here's the raw picture -- only partial I'm afraid:


What's interesting? 

1. The range of engagement impact among experiences is roughly 2:1, with "entertains & absorbs me" at the top, and "tries to persuade me" at the bottom.  Really interesting feedback for web copy writers blowing hard about how robust and scalable and flexible their products are.  Show, don't tell.

2. Several experiences rated average or better on engagement, but were reported to be rarely perceived.  These could be opportunities for differentiation for sites that can stimulate them.  They included "regular part of my day", "my personal timeout", "worth saving and sharing", "turned on by ads" (not gonna touch that one), and "connects me with others".

Cool.  Three of the five speak to a sense of connectedness with the world in general and with others, while only two are about escapism (or worse).  Nice to see that on balance, the underutilized opportunities for making the most of the Web skew toward bringing people together rather than isolating them.

Tom noted that of course whatever experiences you seek to stimulate need to fit your site's goals and your organization's culture.  Within that envelope, what can you do to shape how people perceive your site?  He offered five categories:

1. content selection
2. story telling techniques
3. tools & features
4. site navigation
5. interaction with audience

(I find it interesting he didn't mention visual style...)

One big idea he presented:  the idea of varying navigation by experience instead of content.  Today most high-end sites provide two ways to find things: hierarchical navigation, and search.  Tom suggested that perhaps on arriving at a site, a user might be presented with the option of navigating, say, to relax vs. to do research. 

Tom then presented yet another interesting dimension to consider.  The SUM measure for web sites has cousins called RBS (Reader Behavioral Score) for newspapers and RUM (Reader Usage Measure) for magazines.  What's interesting is to consider what experiences drive higher scores on all three, versus what experiences drive higher scores uniquely for a particular medium:

1. Common across media:
a. regular part of my day
b. personal timeout
c. credible/ safe
d. touches me/ expands my views
e. makes me smarter
... and a few others I didn't get

2. Unique to online:
a. entertains & absorbs me
b. connects me with others
c. tailored for me
d. guides me to other media
e. way to fill my time
f. guilty pleasure
... and one I missed

3. Unique to newspapers:
a. people I know (i.e., through local editions)
b. good service & delivery
c. connect with writer
d. dining companion
e. awkward to handle
f. drowning in news (Tom noted the visual of newspapers stacking up probably reinforces this perception)

4. Unique to magazines:
a. grabs me visually
b. track celebs
c. reinforce my faith
... and a couple more

The panel discussion that followed was really thought-provoking.  Panelists included:

1. Mary Bermel, who runs (my notes) interactive marketing at HP (representing the advertiser perspective)
2. Greg Smith, from Carat Interactive (representing the agency perspective)
3. Michele Slack, from NYT Digital (representing the media view)
4. Tom

Some quotes (I'll skip certain attributions in case I'm misquoting or inaccurately paraphrasing):

"We don't control the brand story anymore by advertising in mass media.  What people are saying about brands in blogs and chat rooms is much more persuasive.  We're not in the advertising business anymore, we're in the communication or dialog business...We can no longer assert that a product is good, we have to make the case... No choice but to embrace the new medium... or fear it!" (CB note: shades of "Markets are Conversations")

Question to Michele: "How are you letting the audience into the conversation, given that advertisers fear being in those environments?"  Answer:  "'most-emailed articles' is a way for the audience to suggest what they find interesting to each other; ratings and reviews of such things as movies and restaurants; 'Kristof Responds' pseudo-blog..." ["Most e-mailed": yet another killer example of structured collaboration.]

"Sites that do ratings and reviews tend to drive leads..."

"Advertising on blogs is scarier, not ready for that yet..."

"But building better search is a high priority..."

"Also, experimenting with executive blogs..."

My favorite quote: "It's impossible to work in marketing today without knowing that the media is now under *consumer* control..."

"Pharma would love to engage with online communities, but can't  because of regulatory constraints..."

"Commerce clients [CB note -- I took this to mean retailers] don't care.  For them it's take it or leave it, they're very CPM focused..."

"Many clients are scared...they're not going to respond to (negative) commentary with an ad, but do we want to engage at all?"

Michele reports NYT Digital is now advertising on some of the bigger blogs, like Gizmodo for techie stuff, very successfully..."

What's next?   "Gaming... how can you be in this very engaging environment without being annoying?"

Advice: "Target passionate audiences, support with sponsorships and not ads..."

"Ads will have to change... the ad itself will have to engage" [CB: first thought to mind was of those shooting-gallery banner ads -- surely we can do better?  How about educational puzzles?]

"Need better behavioral tracking... don't yet know if frequent searches for 'Aruba' reflect real purchase intent, or just escapist behavior..."

Question from the audience: "How do 22 experiences and correlations with usage differ with demographics of respondents?"  Answer from Tom: "Online more homogeneous than newspaper and magazine responses"

Question from the audience: "As advertisers, how can we better now who is on the site and what they want to do?"  Answer: talk directly to the site sponsors and managers themselves.  Ask them what works for their audiences before you develop and ad or whatever."

Question from the audience:  "How can we better integrate ads with the site itself?"  [my answer, silently to myself:  the Binge-o-Matic is still the high-water mark for creativity here.]

Jeffrey is speaking at an OPA event in November.


The comments to this entry are closed.