One Marketing Analytics Myth? The Perfect KPI... published July 6, 2015
I lead Force Five Partners, a marketing analytics consulting firm (bio). I've been writing here about marketing, technology, e-business, and analytics since 2003 (blog name explained).
Email or follow me:
One Marketing Analytics Myth? The Perfect KPI... published July 6, 2015
Mary Meeker's annual Internet Trends report is out. It's a very helpful survey and synthesis of what's going on, as ever, all 164 pages of it. But for the past few years it's contained a bit of analysis that's bugged me.
Page 15 of the report (embedded below) is titled "Remain Optimistic About Mobile Ad Spend Growth... Print Remains Way Over-Indexed." The main chart on the page compares the percentage of time people spend in different media with the percentage of advertising budgets that are spent in those media. The assumption is that percentage of time and percentage of budget should roughly be equal for each medium. Thus Meeker concludes that if -- as is the case for mobile -- the percentage of user time spent is greater than budget going there, then more ad dollars (as a percent of total) will flow to that medium, and vice versa (hence her point about print).
I can think of demand-side, supply-side, and market maturity reasons that this equivalency thesis would break down, which also suggest directions for improving the analysis.
On the demand side, different media may have different mixes of people, with different demographic characteristics. For financial services advertisers, print users skew older -- and thus have more money, on average, making the potential value to advertisers of each minute of time spent by the average user there more valuable. Different media may also have different advertising engagement power. For example, in mobile, in either highly task-focused use cases or in distracted, skimming/ snacking ones, ads may be either invisible or intrusive, diminishing their relative impact (either in terms of direct interaction or view-through stimulation). By contrast, deeper lean-back-style engagement with TV, with more room for an ad to maneuver, might if the ad is good make a bigger impression. I wonder if there's also a reach premium at work. Advertisers like to find the most efficient medium, but they also need to reach a large enough number of folks to execute campaigns effectively. TV and print are more reach-oriented media, in general.
On the supply side, different media have different power distributions of the content they can offer, and different barriers to entry that can affect pricing. On TV and in print, prime ad spots are more limited, so simple supply and demand dynamics drive up prices for the best spots beyond what the equivalency idea might suggest.
In favor of Meeker's thesis, though representing another short term brake on it, is yet another factor she doesn't speak to directly. This is the relative maturity of the markets and buying processes for different media, and the experience of the participants in those markets. A more mature, well-trafficked market, with well-understood dynamics, and lots of liquidity (think the ability for agencies and media brokers to resell time in TV's spot markets, for example), will, at the margin, attract and retain dollars, in particular while the true value of different media remain elusive. (This of course is one reason why attribution analysis is so hot, as evidenced by Google's and AOL Platform's recent acquisitions in this space.) I say in favor, because as mobile ad markets mature over time, this disadvantage will erode.
So for advertisers, agency and media execs, entrepreneurs, and investors looking to play the arbitrage game at the edges of Meeker's observation, the question is, what adjustment factors for demand, supply, and market maturity would you apply this year and next? It's not an idle question: tons of advertisers' media plans and publishers' business plans ride on these assumptions about how much money is going to come to or go away from them, and Meeker's report is an influential input into these plans in many cases.
A tactical limitation of Meeker's analysis is that while she suggests the overall potential shift in relative allocation of ad dollars (her slide suggests a "~$30B+" digital advertising growth opportunity in the USA alone - up from $20B last year*), she doesn't suggest a timescale and trendline for the pace with which we'll get there. One way to come at this is to look at the last 3-4 annual presentations she's made, and see how the relationships she's observed have changed over time. Interestingly, in her 2013 report using 2012 data, on page 5, 12% of time is spent on mobile devices, and 3% of ad dollars are going there, for a 4x difference in percentages. In the 2014 report using 2013 data, 20% of time is spent on mobile, and 5% of media dollars are going there -- again, a 4x relationship.
So, if the equivalency zeitgeist is at work, for the moment it may be stuck in a phone booth. But in the end I'm reminded of the futurist Roy Amara's saying: "We tend to overestimate the effect of a technology in the short term and underestimate its effect in the long term." Plus let's not forget new technologies (Glass, Occulus Rift, both portable and large/immersive) that will further jumble relevant media categories in years to come.
I've written a second book. It's called Marketing and Sales Analytics: Proven Techniques and Powerful Applications From Industry Leaders (so named for SEO purposes). Pearson is publishing it (special thanks to Judah Phillips, author of Building A Digital Analytics Organization, for introducing me to Jeanne Glasser at Pearson). The ebook version will be available on May 23, and the print version will come out June 23.
The book examines how to focus, build, and manage analytics capabilities related to sales and marketing. It's aimed at C-level executives who are trying to take advantage of these capabilities, as well as other senior executives directly responsible for building and running these groups. It synthesizes interviews with 15 senior executives at a variety of firms across a number of industries, including Abbott, La-Z-Boy, HSN, Condé Nast, Harrah's, Aetna, The Hartford, Bed Bath & Beyond, Paramount Pictures, Wayfair, Harvard University, TIAA-CREF, Talbots, and Lenovo. My friend and former boss Bob Lord, author of Converge was kind enough to write the foreword.
I'm in the final editing stages. More to follow soon, including content, excerpts, nice things people have said about it, slideshows, articles, lunch talk...
So healthcare.gov launched, with problems. I'm trying to understand why, so I can apply some lessons in my professional life. Here are some ideas.
First, I think it helps to define some levels of the problem. I can think of four:
1. Strategic / policy level -- what challenges do the goals we set create? In this case, the objective, basically, is two-fold: first; reduce the costs of late-stage, high-cost uncompensated care by enrolling the people who ultimately use that (middle-aged poor folks and other unfortunates) in health insurance that will get them care earlier and reduce stress / improve outcomes (for them and for society) later; second; reduce the cost of this insurance through exchanges that drive competition. So, basically, bring a bunch of folks from, in many cases, the wrong side of the Digital Divide, and expose them to a bunch of eligibility- and choice-driven complexity (proof: need for "Navigators"). Hmm. (Cue the folks who say that's why we need a simple single-payor model, but the obvious response would be that it simply wasn't politically feasible. We need to play the cards we're dealt.)
2. Experience level -- In light of that need, let's examine what the government did do for each of the "Attract / Engage / Convert / Retain" phases of a Caveman User Experience. It did promote ACA -- arguably insufficiently or not creatively enough to distinguish itself from opposing signal levels it should have anticipated (one take here). But more problematically, from what I can tell, the program skips "Engage" and emphasizes "Convert": Healthcare.gov immediately asks you to "Apply Now" (see screenshot below, where "Apply Now" is prominently featured over "Learn More", even on the "Learn" tab of the site). This is technically problematic (see #3 below), but also experientially lots to ask for when you don't yet know what's behind the curtain.
3. Technical level -- Excellent piece in Washington Post by Timothy B. Lee. Basically, the system tries to do an eligibility check (for participation and subsidies) before sending you on to enrollment. Doing this requires checking a bunch of other government systems. The flowchart explains very clearly why this could be problematic. There are some front end problems as well, described in rawest form by some of the chatter on Reddit, but from what I've seen these are more superficial, a function of poor process / time management, and fixable.
Second, here are some things HHS might do differently:
1. Strategic level: Sounds like some segmentation of the potential user base would have suggested a much greater investment in explanation / education, in advance of registration. Since any responsible design effort starts with users and use cases, I'm sure they did this. But what came out the other end doesn't seem to reflect that. What bureaucratic or political considerations got in the way, and what can be revisited, to improve the result? Or, instead of allowing political hacks to infiltrate and dominate the ranks of engineers trying to design a service that works, why not embed competent technologists, perhaps drawn from the ranks of Chief Digital Officers, into the senior political ranks, to advise them on how to get things right online?
2. Experience level: Perhaps the first couple of levels of experience on healthcare.gov should have been explanatory? "Here's what to expect, here's how this works..." Maybe video (could have used YouTube!)? Maybe also ask a couple of quick anonymous questions to determine whether the eligibility / subsidy check would be relevant, to spare the load on that engine, before seeing what plans might be available, at what price? You could always re-ask / confirm that data later once the user's past the shopping /evaluation stage, before formally enrolling them into a plan. In ecommerce, we don't ask untargeted shoppers to enter discount codes until they're about to check out, right?
3. I see from viewing the page source they have Google Tag Manager running, so perhaps they also have Google Analytics running too, alongside whatever other things... Since they've open-sourced the front end code and their content on Github, maybe they could also share what they're learning via GA, so we could evaluate ideas for improving the site in the context of that data?
Postscript, November 1:
Food for thought (scroll to bottom). How does this happen? Software engineer friends, please weigh in!
The May 20th 2013 edition of The New Yorker has an article by Vogue writer Nathan Heller on Massive Online Open Courses (MOOCs) titled "Laptop U: Has the future of college moved online?" The author explores, or at least raises, a number of related questions. How (well) does the traditional offline learning experience transfer online? Is the online learning experience more or less effective than the traditional one? (By what standard? For what material? What is gained and lost?) What will MOOCs mean for different colleges and universities, and their faculties? How will the MOOC revolution be funded? (In particular, what revenue model will emerge?)
Having worked a lot in the sector, for both public and private university clients, developing everything from technology, to online-enabled programs themselves, to analytic approaches, and even on marketing and promotion, the article was a good prompt for me to try to boil out some ways to think about answering these questions.
The article focuses almost exclusively on Harvard and EdX, the 12-school joint venture through which it's pursuing MOOCs. Obviously this skews the evaluation. Heller writes:
Education is a curiously alchemical process. Its vicissitudes are hard to isolate. Why do some students retain what they learned in a course for years, while others lose it through the other ear over their summer breaks? Is the fact that Bill Gates and Mark Zuckerberg dropped out of Harvard to revolutionize the tech industry a sign that their Harvard educations worked, or that they failed? The answer matters, because the mechanism by which conveyed knowledge blooms into an education is the standard by which MOOCs will either enrich teaching in this country or deplete it.
For me, the first step to boiling things out is to define what we mean by -- and want from -- an "education". So, let's try to unpack why people go to college. In most cases, Reason One is that you need a degree to get any sort of decent job. Reason Two is to plug into a network of people -- fellow students, alumni, faculty -- that provide you a life-long community. Of course you need a professional community for that Job thing, but also because in an otherwise anomic society you need an archipelago to seed friendships, companionships, and self-definition (or at least, as scaffolding for your personal brand: as one junior I heard on a recent college visit put it memorably, "Being here is part of the personal narrative I'm building.") Reason Three -- firmly third -- is to get an "education" in the sense that Heller describes. (Apropos: check this recording of David Foster Wallace's 2005 commencement address at Kenyon College.)
Next, this hierarchy of needs then gives us a way to evaluate the prospects for MOOCs.
If organization X can produce graduates demonstrably better qualified (through objective testing, portfolios of work, and experience) to do job Y, at a lower cost, then it will thrive. If organization X can do this better and cheaper by offering and/or curating/ aggregating MOOCs, then MOOCs will thrive. If a MOOC can demonstrate an adequately superior result / contribution to the end outcome, and do it inexpensively enough to hold its place in the curriculum, and do it often enough that its edge becomes a self-fulfilling prophecy -- a brand, in other words -- then it will crowd out its competitors, as surely as one plant shuts out the sunlight to another. Anyone care to bet against Georgia Tech's new $7K Master's in Computer Science?
If a MOOC-mediated social experience can connect you to a Club You Want To Be A Member Of, you will pay for that. And if a Club That Would Have You As A Member can attract you to its clubhouse with MOOCs, then MOOCs will line the shelves of its bar. The winning MOOC cocktails will be the ones that best produce the desired social outcomes, with the greatest number of satisfying connections.
Finally, learning is as much about the frame of mind of the student as it is about the quality of the teacher. If through the MOOC the student is able to choose a better time to engage, and can manage better the pace of the delivery of the subject matter, then the MOOC wins.
Beyond general prospects, as you consider these principles, it becomes clear that it's less about whether MOOCs win, but which ones, for what and for whom, and how.
The more objective and standardized -- and thus measurable and comparable -- the learning outcome and the standard of achievement, the greater the potential for a MOOC to dominate. My program either works, or it doesn't.
If a MOOC facilitates the kinds of content exchanges that seed and stimulate offline social gatherings -- pitches to VCs, or mock interviewing, or poetry, or dance routines, or photography, or music, or historical tours, or bird-watching trips, or snowblower-maintenance workshops -- then it has a better chance of fulfilling the longings of its students for connection and belonging.
And, the more well-developed the surrounding Internet ecosystem (Wikipedia, discussion groups, Quora forums, and beyond) is around a topic, the less I need a Harvard professor, or even a Harvard grad student, to help me, however nuanced and alchemical the experience I miss might otherwise have been. The prospect of schlepping to class or office hours on a cold, rainy November night has a way of diluting the urge to be there live in case something serendipitous happens.
Understanding how MOOCs win then also becomes a clue to understanding potential revenue models.
If you can get accredited to offer a degree based in part or whole on MOOCs, you can charge for that degree, and gets students or the government to pay for it (Exhibit A: University of Phoenix). That's hard, but as a variant of this, you can get hired by an organization, or a syndicate of organizations you organize, to produce tailored degree programs -- think corporate training programs on steroids -- that use MOOCs to filter and train students. (Think "You, Student, pay for the 101-level stuff; if you pass you get a certificate and an invitation to attend the 201-level stuff that we fund; if you pass that we give you a job.")
Funding can come directly, or be subsidized by sponsors and advertisers, or both.
You can try to charge for content: if you produce a MOOC that someone else wants to include in a degree-based program, you can try to license it, in part or in whole.
You can make money via the service angle, the way self-publishing firms support authors, with a variety of best-practice based production services. Delivery might be offered via a freemium model -- the content might be free, but access to premium groups, with teaching assistant support, might come at a price. You can also promote MOOCs -- build awareness, drive distribution, even simply brand -- for a cut of the action, the way publishers and event promoters do.
Perhaps in the not-too-distant future we'll get the Academic Upfront, in which Universities front a semester's worth of classes in a MOOC, then pitch the class to sponsors, the way TV networks do today. Or, maybe the retail industry also offers a window into how MOOCs will be monetized. Today's retail environment is dominated by global brands (think professors as fashion designers) and big-box (plus Amazon) firms that dominate supply chains and distrubution networks. Together, Brands and Retailers effectively act as filters: we make assumptions that the products on their shelves are safe, effective, reasonably priced, acceptably stylish, well-supported. In exchange, we'll pay their markup. This logic sounds a cautionary note for many schools: boutiques can survive as part of or at the edges of the mega-retailers' ecosystems, but small-to-mid-size firms reselling commodities get crushed.
Of course, these are all generic, unoriginal (see Ecclesiastes 1:9) speculations. Successful revenue models will blend careful attention to segmenting target markets and working back from their needs, resources, and processes (certain models might be friendlier to budgets and purchasing mechanisms than others) with thoughtful in-the-wild testing of the ideas. Monolithic executions with Neolithic measurement plans ("Gee, the focus group loved it, I can't understand why no one's signing up for the paid version!") are unlikely to get very far. Instead, be sure to design with testability in mind (make content modular enough to package or offer a la carte, for example). Maybe even use Kickstarter as a lab for different models!
I just finished reading Converge, the new book on integrating technology, creativity, and media by Razorfish CEO Bob Lord and his colleague Ray Velez, the firm’s CTO. (Full disclosure: I’ve known Bob as a colleague, former boss, and friend for more than twenty years and I’m a proud Razorfish alum from a decade ago.)
Reflecting on the book I’m reminded of the novelist William Gibson’s famous comment in a 2003 Economist interview that “The future’s already here, it’s just not evenly distributed.” In this case, the near-perfect perch that two already-smart guys have on the Digital Revolution and its impact on global brands has provided them a view of a new reality most of the rest of us perceive only dimly.
So what is this emerging reality? Somewhere along the line in my business education I heard the phrase, “A brand is a promise.” Bob and Ray now say, “The brand is a service.” In virtually all businesses that touch end consumers, and extending well into relevant supply chains, information technology has now made it possible to turn what used to be communication media into elements of the actual fulfillment of whatever product or service the firm provides.
One example they point to is Tesco’s virtual store format, in which images of stocked store shelves are projected on the wall of, say, a train station, and commuters can snap the QR codes on the yogurt or quarts of milk displayed and have their order delivered to their homes by the time they arrive there: Tesco’s turned the billboard into your cupboard. Another example they cite is Audi City, the Kinnect-powered configurator experience through which you can explore and order the Audi of your dreams. As the authors say, “marketing is commerce, and commerce is marketing.”
But Bob and Ray don’t just describe, they also prescribe. I’ll leave you to read the specific suggestions, which aren’t necessarily new. What is fresh here is the compelling case they make for them; for example, their point-by-point case for leveraging the public cloud is very persuasive, even for the most security-conscious CIO. Also useful is their summary of the Agile method, and of how they’ve applied it for their clients.
Looking more deeply, the book isn’t just another surf on the zeitgeist, but is theoretically well-grounded. At one point early on, they say, “The villain in this book is the silo.” On reading this (nicely turned phrase), I was reminded of the “experience curve” business strategy concept I learned at Bain & Company many years ago. The experience curve, based on the idea that the more you make and sell of something, the better you (should) get at it, describes a fairly predictable mathematical relationship between experience and cost, and therefore between relative market share and profit margins. One of the ways you can maximize experience is through functional specialization, which of course has the side effect of encouraging the development of organizational silos. A hidden assumption in this strategy is that customer needs and associated attention spans stay pinned down and stable long enough to achieve experience-driven profitable ways to serve them. But in today’s super-fragmented, hyper-connected, kaleidoscopic marketplace, this assumption breaks down, and the way to compete shifts from capturing experience through specialization, to generating experience “at-bats” through speedy iteration, innovation, and execution. And this latter competitive mode relies more on the kind of cross-disciplinary integration that Bob and Ray describe so richly.
The book is a quick, engaging read, full of good stories drawn from their extensive experiences with blue-chip brands and interesting upstarts, and with some useful bits of historical analysis that frame their arguments well (in particular, I Iiked their exposition of the television upfront). But maybe the best thing I can say about it is that it encouraged me to push harder and faster to stay in front of the future that’s already here. Or, as a friend says, “We gotta get with the ‘90’s, they’re almost over!”
I've written a short book. It's called "Pragmalytics: Practical Approaches to Marketing Analytics in the Digital Age". It's a collection and synthesis of some of the things I've learned over the last several years about how to take better advantage of data (Big and little) to make better marketing decisions, and to get better returns on your investments in this area.
The main point of the book is the need for orchestration. I see too much of the focus today on "If we build It (the Big Data Machine, with some data scientist high priests to look after it), good things will happen." My experience has been that you need to get "ecosystemic conditions" in balance to get value. You need to agree on where to focus. You need to get access to the data. You need to have the operational flexibility to act on any insights. And, you need to cultivate an "analytic marketer" mindset in your broader marketing team that blends perspectives, rather than cultivating an elite but blinkered cadre of "marketing analysts". Over the next few weeks, I'll further outline some of what's in the book in a few posts here on my blog.
I'm really grateful to the folks who were kind enough to help me with the book. The list includes: Mike Bernstein, Tip Clifton, Susan Ellerin, Ann Hackett, Perry Hewitt, Jeff Hupe, Ben Kline, Janelle Leonard, Sam Mawn-Mahlau, Bob Neuhaus, Judah Phillips, Trish Gorman Clifford, Rob Schmults, Michelle Seaton, Tad Staley, and my business partner, Jamie Schein. As I said in the book, if you like any of it, they get credit for salvaging it. The rest -- including several bits that even on the thousandth reading still aren't as clear as they should be, plus a couple of typos I need to fix -- are entirely my responsibility.
I'm also grateful to the wonderful firms and colleagues and clients I've had the good fortune to work for and with. I've named the ones I can, but in general have erred on the side of respecting their privacy and confidentiality where the work isn't otherwise in the public domain. To all of them: Thank You!
This field is evolving quickly in some ways, but there are also some timeless principles that apply to it. So, there are bits of the book that I'm sure won't age well (including some that are already obsolete), but others that I hope might. While I'm not one of those coveted Data Scientists by training, I'm deep into this stuff on a regular basis at whatever level is necessary to get a positive return from the effort. So if you're looking for a book on selecting an appropriate regression technique, or tuning Hadoop, you won't find that here, but if you're looking for a book about how to keep all the balls in the air (and in your brain), it might be useful to you. It's purposefully short -- about half the length of a typical business book. My mental model was to make it about as thick as "The Elements of Style", since that's something I use a lot (though you probably won't think so!). Plus, it's organized so you can jump in anywhere and snack as you wish, since this stuff can be toxic in large doses.
In writing it amidst all the Big Data craziness, I was reminded of Gandhi's saying (paraphrased) "First they ignore you... then they fight you, then you win." Having been in the world of marketing analytics now for a while, it seems appropriate to say that "First they ignore you, then they hype you, then you blend in." We're now in the "hype" phase. Not a day goes by without some big piece in the media about Big Data or Data Scientists (who now have hit the highly symbolic "$300k" salary benchmark -- and last time we saw it, in the middle part of the last decade in the online ad sales world, was a sell signal BTW). "Pragmalytics" is more about the "blend in" phase, when all this "cool" stuff is more a part of the furniture that needs to work in harmony with the rest of the operation to make a difference.
"Pragmalytics" is available via Amazon (among other places). If you read it please do me a favor and rate and review it, or even better, please get in touch if you have questions or suggestions for improving it. FWIW, any earnings from it will go to Nashoba Learning Group (a school for kids with autism and related disorders).
Where it makes sense, I'd be very pleased to come talk to you and your colleagues about the ideas in the book and how to apply them, and possibly to explore working together. Also, in a triumph of Hope over Experience, my next book (starting now) will be a collection and synthesis of interviews with other senior marketing executives trying to put Big Data to work. So if you would be interested in sharing some experiences, or know folks who would, I'd love to talk.
About the cover: it's meant to convey the harmonious convergence of "Mars", "Venus", and "Earth" mindsets: that is, a blend of analytic acuity, creativity and communication ability, and practicality and results-orientation that we try to bring to our work. Fellow nerds will appreciate that it's a Cumulative Distribution Function where the exponent is, in a nod to an example in the book, 1.007.
Facebook's Sponsored Stories feature is one of the ad targeting horses the firm's counting on to pull it out of its current valuation morass (read this, via @bussgang).
Sponsored Stories is a virality-enhancing mechanism (no jokes please, that was an "a" not an "i") that allows Facebook advertisers to increase the reach of Facebook users' interactions with the advertisers' brands on Facebook (Likes, Check-ins, etc.). It does this by increasing the number of a user's Facebook friends who see such engagements with the advertisers' brands beyond the limited number who would, under normal application of the Facebook news feed algorithm, see those endorsements.
Many users are outraged that this unholy Son-Of-Beacon feature violates their privacy, to the point that they sue-and-settle (or try to, oops).
What they are missing perhaps is the opportunity to "surf" an advertiser's Sponsored Stories investment to amplify their own self-promotion or mere narcissism.
Consider the following simple example. Starbucks is / has been using this ad program. Let's say I go to Starbucks and "check in" on Facebook. Juiced by Sponsored Stories (within the additional impressions Starbucks has paid for), all of my Facebook friends browsing their news feeds will see I've checked in at Starbucks (and presumably feel all verklempt about a brand that could attract such a valued friend).
Now, what if I, savvy small business person, comment in my check in that I'm "at Starbucks, discussing my <link>NEW BOOK</link> with friends!" I've pulled off the social media equivalent of pasting my bumper sticker on Starbucks' billboard.
I need to look more closely into this, but as I understand it, the Sponsored Stories feature can't today prevent users from including negative feedback in their brand engagements, where such flexibility is provided for. So if they can't handle the negative yet, it may still be that they can't prevent more general forms of off-brand messaging.
I'm sure others have considered this and other possibilities. Comments very welcome! Meanwhile, I'm off to Starbucks to discuss my upcoming NEW BOOK.