http://bit.ly/watbs-econtmag, published October 3
I lead Force Five Partners, a marketing analytics consulting firm (bio). I've been writing here about marketing, technology, e-business, and analytics since 2003 (blog name explained).
Email or follow me:
http://bit.ly/watbs-econtmag, published October 3
http://bit.ly/htmyap-cmo, published September 26
A year ago, while doing research for my new book Marketing and Sales Analytics I wrote a post titled "Organizing For #Analytics -- Seven Considerations". I argued that you should "think organization last, not first", and presented, well, seven considerations that would shape the structure you end up with.
But I made an important mistake in writing that post. Enquiring minds still want the boxes. So, based on the 15 case studies I presented in the book, as well as other experiences over the past decade with even more firms, this post offers a "reference structure" and suggests how it might be tweaked under different circumstances.
In constructing an analytic organization, you've got two objectives to serve that should guide you. One is answering questions. The other is answering them better the next time.
To answer questions, once again you've got two things to balance. One is "business intimacy" -- how close you need to be to the business you're serving to understand what's going on, so your analysis is grounded in that reality. By what's going on, I mean things like terminology, the particular patterns and quirks in the data the business generates, and the options for acting on insights that the operational infrastructure, people, and process of the business make possible.
Balanced against business intimacy is "analytic capability". This means having a critical mass of skills for supporting decisions appropriately. If a decision requires a sophisticated model using lots of data, for example, having a bunch of business-intimate MBAs who last ran a regression a few years ago in business school using Excel might not cut it. So you might need one or more specialists who can do that sort of thing. Or, if it requires specialized knowledge about, say, how digital analytics work, you might need folks who know about things like tag management systems, or inferred matching across devices for multi-screen tracking.
In implementing these two requirements, you then face -- wait for it -- another two choices. One is how much to distribute into the field, or into business units, or aligned to functions, versus how much to centralize. The second will be how much to separate analysis from reporting. With respect to the former, a lot depends on how different or similar the needs in each business / field location are (the more different, the closer you want support to them, and the less leverage there is from centralization anyway). As for the latter, My thinking and research has suggested that separate "report generation" groups are a bad idea -- they encourage conveying data rather than insight, and thus the development of a second-class citizenry in analytics groups; they disconnect report generation from report usage, resulting in the proverbial pallets of unused reports that consultants on cost-hunts feast on; and, they discourage automating reporting (because most folks don't like to work themselves out of a job).
To answer questions better the next time, you need some folks thinking about everything from whether there are better techniques and associated tools, to whether there's more useful data, and how to better organize and provision the data you've already got. There are going to be -- yes, of course -- two options for how to proceed here. One is a "rapid prototyping" approach, and the other is a more traditional "waterfall" method.
Wrapped around all this is "governance", only with a capital "G". In most cases, governance is applied to data, but not to the prioritizations of the questions and decisions that analysis will answer and support. Capital G governance also sets analytic priorities, and recruits, develops, and evaluates senior analytic leadership.
So, here's your org chart:
OK, now that you got what you came for, here's the price: You have to consider the design above not an immutable multi-year reality, but rather a delicate, dynamic compromise that balances the objectives and tensions mentioned above. Where a group should live, as the footnotes suggest, and how relatively large each group becomes, should be a function of a constantly evolving portfolio of opportunities and challenges your governance process should determine. Conceiving of this not as a static chart prevents a pattern too often repeated -- a guardrail to guardrail "centralize versus distribute", "locate in the business versus locate in IT (or finance)" swing, where each successive move trashes the logic of the last. But that's a topic for another post...
What's your chart look like? Why? How often has it / does it evolve?
Via the WSJ, Emarketer's latest forecast has mobile ad spending eclipsing newspapers and radio (individually, not together, yet):
Ad spending on smartphones and tablets will this year eclipse radio, magazines, and newspapers, according to research firm eMarketer. Spending on mobile, which includes smartphones and tablets, will jump 83% in 2014 to nearly $18 billion, eMarketer predicts. Newspapers and radio, on the other hand, will draw nearly $17 and $15.5 billion respectively.
And yet, you will not be able to get full value from your investment, because the Revolution will not be Mobilized... until we can track and properly attribute the impact of advertising in this channel.
By various estimates, mobile now accounts for 30% of web traffic, but only 10% of ad revenues. Some, like Mary Meeker, make the case that this means mobile advertising spending still has lots of room to go. Here's my critique of her argument a couple of months ago. In that post I described demand- and supply-side factors that inhibit the "equivalency thesis" she outlines. One of these is understanding the properly attributed influence of mobile in a customer experience.
A couple of weeks ago I listened to Visual IQ's Solutions Consulting lead Andy Dubickas crisply describe three approaches for tracking a user across the desktop-mobile divide, for purposes of attribution analysis:
If you're not ready for this, you can at minimum set up your mobile ad buy with some variation in it -- waves, spikes, whatever -- that allow you to see, either with your naked eye or through modeling, the incremental direct and indirect effects of your investment. In fact, there's no reason the tighter-tracked approaches described above can't peacefully co-exist with this simple approach, even if it's just a cross-check.
(Apologies to Gil Scott-Heron)
Warning: due to author sloth, this post is dated. Most of you have likely moved on with your lives. Me, my Heart Is Full for los albicelestes, despite a disappointing outcome, but I still have a lingering case of futbolytics to get over.
Fivethirtyeight.com contributor Benjamin Morris had a fascinating article a couple of weeks ago that examined Argentine footballer Lionel Messi's play in recent years. Until this year's World Cup of course, Messi had come under some criticism for, against the backdrop of his ethereal play for Barcelona, under-delivering for Argentina in recent years. The article -- warning, 4600+ words long, with charts and videos and hyperlinks and footnotes, as in "Ask not, Bill Simmons, for whom the bell tolls..." -- explains persuasively that Messi is an outlier among outliers, even in his performances for the Argentine national team.
The analysis of his shooting is fascinating enough. But what really caught my attention, was the analysis of his passing and other influences on the game. In particular, here's a fascinating chart from Morris' article that makes the point neatly.
My friends at the multi-channel marketing attribution analytics firm Visual IQ are fond of a soccer metaphor to explain what they do. "Giving all the conversion credit to the last-touched marketing channel is like giving Mario Götze all the credit for his goal in the 113th minute of the World Cup final," they say (with a zesty cruelty that borders on the sociopathic).
In the sales world, it's a holy grail to get to this kind of dynamic, or even an understanding of where on Morris' chart members of your team would be. It can be a thorny path to get there though, because unlike in Messi's world, a helping hand in sales can be harder to observe, and even if you try to measure it, often it can be (will be) gamed mercilessly and unhelpfully.
One way you can track this sort of thing is through online and offline knowledge sharing by members of your team. Winning proposals, presentations, good answers to FAQs, and then views of these by others can all be tracked in relatively painless, game-free ways. A number of years ago when I worked at ArsDigita, we worked with Siemens to build ShareNet, a global sales and marketing knowledge management system that for many years was a poster child for applications of its kind (see here for the HBR case study). The secret behind Siemens' success with ShareNet was the flexibility with which it could adapt what was captured, and how, to make it easy for people to contribute and consume. Today, fortunately, the tools and costs for building capabilities like this are far more accessible. And now as attribution analysis moves closer to the center of the marketing analytics agenda, we have the opportunity to put the resulting data to work in a way that moves the dominant motivation for this kind of behavior beyond altruism to proper credit.
So if you'd like to improve your organization's gol-orientation, perhaps it's time to compile and publish your own assist chart?
In the June 23 edition of The New Yorker, in an article titled "The Disruption Machine", Harvard history professor Jill Lepore critiques the theories of Harvard Business School professor Clayton Christensen. Her article is wonderfully written of course, but curiously sharp in tone (or is it me?). Re-reading it, I have the impression she's sick of the Manifest Destiny - like over-simplification and over-application of Christensen's theories:
Most big ideas have loud critics. Not disruption. Disruptive innovation as the explanation for how change happens has been subject to little serious criticism, partly because it’s headlong, while critical inquiry is unhurried; partly because disrupters ridicule doubters by charging them with fogyism, as if to criticize a theory of change were identical to decrying change; and partly because, in its modern usage, innovation is the idea of progress jammed into a criticism-proof jack-in-the-box.
But unfortunately the baby appears to go out with the bathwater. Lepore suggests -- accuses -- Christensen of cherry-picking his data, and describes a dynamic in business scholarship (as distinguished from practices in other fields) that over-promotes findings. In taking aim, fairly or not, at Christensen (who beyond his reputation as a scholar is even more respected for his integrity), Lepore goes for the jugular of the academic-industrial complex.
Professor Christensen replied at length to Lepore's piece in an interview he gave last Friday to Bloomberg Businessweek reporter Drake Bennett, taking significant issues with her points -- maybe too personally?
Well, in the first two or three pages, it seems that her motivation is to try to rein in this almost random use of the word “disruption.” The word is used to justify whatever anybody—an entrepreneur or a college student—wants to do. And as I read that, I was delighted that somebody with her standing would join me in trying to bring discipline and understanding around a very useful theory. I’ve been trying to do it for 20 years.
And then in a stunning reversal, she starts instead to try to discredit Clay Christensen, in a really mean way. And mean is fine, but in order to discredit me, Jill had to break all of the rules of scholarship that she accused me of breaking—in just egregious ways, truly egregious ways. In fact, every one—every one—of those points that she attempted to make [about The Innovator’s Dilemma] has been addressed in a subsequent book or article. Every one! And if she was truly a scholar as she pretends, she would have read [those]. I hope you can understand why I am mad that a woman of her stature could perform such a criminal act of dishonesty—at Harvard, of all places.
Why am I interested, and why should you care?
Christensen's research into innovation, and his books like The Innovator's Dilemma and its successors have pretty much dominated the landscape on the subject for nearly the last twenty years. If you're a senior executive at a big established firm, he's made you paranoid. If you're an entrepreneur at a small, ambitious firm, you're telling your investors, customers, and employees you're one of the "disruptive innovations" making those executives paranoid. There's even a slim chance you might be right! Lepore is a Pulitzer Prize and National Book Award winner (as well as a department chair at Harvard). Scholars of her prominence usually don't take on other scholars, particularly outside their field, without good reason or care. Christensen's reply suggests she had neither, and prior critiques of her work suggest this might be the latest in a pattern of occasional trip-ups. In any event, the critique's attracted some attention. If you follow either of these folks for professional reasons (or simply are drawn to MMA-style clashes between academic titans) you might be interested to track this story too.
I'm interested because the controversy highlights a pattern I often see, and as I describe in my new book Marketing and Sales Analytics, I use a variety of techniques to compensate for.
The pattern is to treat ideas and supporting evidence like Christensen's as the last word on a subject. After all, he teaches at Harvard Business School, right? But even Christensen says (in his Bloomberg Businessweek interview) that models and supporting stories like his are at best just partly predictive points of departure for any specific "truth" I need to be tracking, and hedging against.
The psychologist Daniel Kahneman, who wrote the best-seller Thinking, Fast and Slow about his research into cognitive bias, talks about the power of stories, and in particular about our susceptibility to them and to other biases when we're tired, frazzled, juggling a thousand things. I think Christensen's theory of disruptive innovations invites criticisms like Lepore's because it mirrors, purposefully or not, an especially powerful archetypal story, in this case of the hero who battles long odds but ultimately triumphs grandly.
Recent history and scholarship has reinforced the power of this particular theme. The Internet revolution has created a number of poster children for it (I'm reminded of Paul Simon's line "Every generation thows a hero up the pop charts", and in our generation we've had several), and created conditions for accelerating the pace of disruption. Nassim Taleb's scholarship, in particular his own best-selling book The Black Swan, has sensitized us to the existence of outlier possibilities our models often miss (and even explicitly discard to improve fit).
So the combination of a story model we're especially tuned for, and recent examples that magnify its power, position it uniquely in our consciousness. The implication for executives is to be mindful of the bias this creates, and to have appropriate mechanisms in place for managing it.
In my book I describe a portfolio-driven approach to managing analytic efforts that can help with this. In brief, the idea is to manage analytic projects not just as a list of questions to be checked off as you answer them, but as a venture capitalist's portfolio of investments that need to generate target returns appropriate to their size and riskiness. In the governance of such portfolios we set up in our work with clients, we manage to a rule we call "3-2-1". In any given quarter, we aim to have the collection of analytic initiatives we're pursuing yield (notionally) three "news you can use" insights, two experiments based on those insights, and one outcome we're putting into production at scale to help pay the freight for the overall investment in the portfolio (including us). The portfolios are constructed to reflect priorities across a grid that itself is a collection of different business opportunities ("Need X for Customer Y") we're targeting and different purchase funnel or customer journey stages for these opportunities. This grid helps you judge how concentrated you are on existing versus new opportunities, and on whether your investments are appropriately focused on bottlenecks in the relevant funnels or journeys.
The main point of using a grid like this, as opposed to just a list of individual projects, is that it forces you to think backwards in a data-driven way from the customer-defined strategies, goals, and objective performance of the business, and about whether you've got sufficient attention on the hot spots thereof. In the parallel world of innovation, I believe the opportunity for subscribers to Christensen's ideas is to neither accept nor reject "disruption dogma", but to ask themselves whether they've got sufficient "3-2-1" style attention and results on the hot spots in their business. "What customers are important to us?" "What needs are important to them?" "How well are we serving those needs?" "What's / who's out there trying to serve these same folks and their needs?" "What kind of progress are they making, and what's contributing or impeding that progress?" "What research, analysis, or testing should we be doing to stay close to potentially meaningful threats?" "Are the hedges these efforts represent probabilistically - in likely magnitude and yield - a good match for the risk the potential competition out there represents to our business?"
A "Clash Of The Titans" around business theories, while good sport for some, can cause a lot of anxiety for many others trying to build and run their businesses. Framing your approach specifically and objectively using the techniques for managing analytic efforts can help you get past these concerns in a practical, tailored - and maybe even heroic - way.
Mary Meeker's annual Internet Trends report is out. It's a very helpful survey and synthesis of what's going on, as ever, all 164 pages of it. But for the past few years it's contained a bit of analysis that's bugged me.
Page 15 of the report (embedded below) is titled "Remain Optimistic About Mobile Ad Spend Growth... Print Remains Way Over-Indexed." The main chart on the page compares the percentage of time people spend in different media with the percentage of advertising budgets that are spent in those media. The assumption is that percentage of time and percentage of budget should roughly be equal for each medium. Thus Meeker concludes that if -- as is the case for mobile -- the percentage of user time spent is greater than budget going there, then more ad dollars (as a percent of total) will flow to that medium, and vice versa (hence her point about print).
I can think of demand-side, supply-side, and market maturity reasons that this equivalency thesis would break down, which also suggest directions for improving the analysis.
On the demand side, different media may have different mixes of people, with different demographic characteristics. For financial services advertisers, print users skew older -- and thus have more money, on average, making the potential value to advertisers of each minute of time spent by the average user there more valuable. Different media may also have different advertising engagement power. For example, in mobile, in either highly task-focused use cases or in distracted, skimming/ snacking ones, ads may be either invisible or intrusive, diminishing their relative impact (either in terms of direct interaction or view-through stimulation). By contrast, deeper lean-back-style engagement with TV, with more room for an ad to maneuver, might if the ad is good make a bigger impression. I wonder if there's also a reach premium at work. Advertisers like to find the most efficient medium, but they also need to reach a large enough number of folks to execute campaigns effectively. TV and print are more reach-oriented media, in general.
On the supply side, different media have different power distributions of the content they can offer, and different barriers to entry that can affect pricing. On TV and in print, prime ad spots are more limited, so simple supply and demand dynamics drive up prices for the best spots beyond what the equivalency idea might suggest.
In favor of Meeker's thesis, though representing another short term brake on it, is yet another factor she doesn't speak to directly. This is the relative maturity of the markets and buying processes for different media, and the experience of the participants in those markets. A more mature, well-trafficked market, with well-understood dynamics, and lots of liquidity (think the ability for agencies and media brokers to resell time in TV's spot markets, for example), will, at the margin, attract and retain dollars, in particular while the true value of different media remain elusive. (This of course is one reason why attribution analysis is so hot, as evidenced by Google's and AOL Platform's recent acquisitions in this space.) I say in favor, because as mobile ad markets mature over time, this disadvantage will erode.
So for advertisers, agency and media execs, entrepreneurs, and investors looking to play the arbitrage game at the edges of Meeker's observation, the question is, what adjustment factors for demand, supply, and market maturity would you apply this year and next? It's not an idle question: tons of advertisers' media plans and publishers' business plans ride on these assumptions about how much money is going to come to or go away from them, and Meeker's report is an influential input into these plans in many cases.
A tactical limitation of Meeker's analysis is that while she suggests the overall potential shift in relative allocation of ad dollars (her slide suggests a "~$30B+" digital advertising growth opportunity in the USA alone - up from $20B last year*), she doesn't suggest a timescale and trendline for the pace with which we'll get there. One way to come at this is to look at the last 3-4 annual presentations she's made, and see how the relationships she's observed have changed over time. Interestingly, in her 2013 report using 2012 data, on page 5, 12% of time is spent on mobile devices, and 3% of ad dollars are going there, for a 4x difference in percentages. In the 2014 report using 2013 data, 20% of time is spent on mobile, and 5% of media dollars are going there -- again, a 4x relationship.
So, if the equivalency zeitgeist is at work, for the moment it may be stuck in a phone booth. But in the end I'm reminded of the futurist Roy Amara's saying: "We tend to overestimate the effect of a technology in the short term and underestimate its effect in the long term." Plus let's not forget new technologies (Glass, Occulus Rift, both portable and large/immersive) that will further jumble relevant media categories in years to come.
The book marks a watershed moment in the Big Data zeitgeist. Much of the literature on the topic to this point has been more evangelical, telling us how analytics will make us all taller, smarter, and more handsome. But the general sense for me has been of stories that are "way out there" for most organizations. This latest book is much more about how to realize these visions with tactical, practical prescriptions across a range of issues.
Perhaps the most important of these dimensions is having a clear idea of the challenges or opportunities for which Big Data might be a part of the solution. In Chapter Two, Davenport presents a very helpful series of use cases for using Big Data in several industry applications, including business travel, energy management, retail, and home education. He pushes further to examine the relative readiness of a number of different industries and business functions, including marketing and sales (which are the particular focus of my own upcoming book, Marketing and Sales Analytics). In Chapter Three he builds on these examples and sector assessments to offer a framework for shaping business strategies that leverage Big Data. He suggests cost reduction, time reduction, new offerings, and decision support as broad objectives for focusing Big Data initiatives, and then further suggests a useful distinction between discovery-oriented application of Big Data (say, for sorting out emergent patterns of behavior to address) and production-oriented usage (applying Big Data to personalize experiences based on which emergent patterns might be worth the effort).
This "ends" focused approach to applying Big Data, in contrast to an "If I build it (my giant Hadoop Cluster) they will come" is an extremely valuable perspective to have introduced at this point in the evolution of this trend, and Davenport has wrapped it in a clean, well-organized package of specific advice executives interested in this space can profit from.
I've written a second book. It's called Marketing and Sales Analytics: Proven Techniques and Powerful Applications From Industry Leaders (so named for SEO purposes). Pearson is publishing it (special thanks to Judah Phillips, author of Building A Digital Analytics Organization, for introducing me to Jeanne Glasser at Pearson). The ebook version will be available on May 23, and the print version will come out June 23.
The book examines how to focus, build, and manage analytics capabilities related to sales and marketing. It's aimed at C-level executives who are trying to take advantage of these capabilities, as well as other senior executives directly responsible for building and running these groups. It synthesizes interviews with 15 senior executives at a variety of firms across a number of industries, including Abbott, La-Z-Boy, HSN, Condé Nast, Harrah's, Aetna, The Hartford, Bed Bath & Beyond, Paramount Pictures, Wayfair, Harvard University, TIAA-CREF, Talbots, and Lenovo. My friend and former boss Bob Lord, author of Converge was kind enough to write the foreword.
I'm in the final editing stages. More to follow soon, including content, excerpts, nice things people have said about it, slideshows, articles, lunch talk...
I'm working on a book. It will be titled Marketing and Sales Analytics: Powerful Lessons from Leading Practitioners. My first book, Pragmalytics, described some lessons I'd learned; this book extends those lessons with interviews with more than a dozen senior executives grappling with building and applying analytics capabilities in their companies. Pearson's agreed to publish it, and it will be out this spring. Right now I'm in the middle of the agony of writing it. Thank you Stephen Pressfield (and thanks to my wife Nan for introducing us).
A common denominator in the conversations I've been having is the importance of culture. Culture makes building an analytics capability possible. In some cases, pressure for culture change comes outside-in: external conditions become so dire that a firm must embrace data-driven objectivity. In others, the pressure comes top-down: senior leadership embodies it, leads by example, and is willing to re-staff the firm in its image. But what do you do when the wolf's not quite at the door, or when it makes more sense (hopefully, your situation) to try to build the capability largely within the team you have than to make wholesale changes?
There are a lot of models for understanding culture and how to change it. Here's a caveman version (informed by behavioral psychology principles, and small enough to remember). Culture is a collection of values -- beliefs -- about what works, and doesn't: what behaviors lead to good outcomes for customers, shareholders, and employees; and, what behaviors are either ignored or punished.
Values, in turn, are developed through chances individuals have to try target behaviors, the consequences of those experiences, and how effectively those chances and their consequences are communicated to other people working in the organization.
Chances are to culture change as reps (repetitions) are to sports. If you want to drive change, to get better, you need more of them. Remember that not all reps come in games. Test programs can support culture change the same way practices work for teams. Also, courage is a muscle: to bench press 500 pounds once, start with one pushup, then ten, and so on. If you want your marketing team to get comfortable conceiving and executing bigger and bolder bets, start by carving out, frequently, many small test cells in your programs. Then, add weight: define and bound dimensions and ranges for experimentation within those cells that don't just have limits, but also minimums for departure from the norm. If you can't agree on exactly what part of your marketing mix needs the most attention, don't study it forever. A few pushups won't hurt, even if it's your belly that needs the attention. A habit is easier to re-focus than it is to start.
Consequences need to be both visible and meaningful. Visible means good feedback loops to understand the outcome of the chance taken. Meaningful can run to more pay and promotion of course, but also to opportunity and recognition. And don't forget: a sense of impact and accomplishment -- of making a difference -- can be the most powerful reinforcer of all. For this reason, a high density of chances with short, visible feedback loops becomes really important to your change strategy.
Communication magnifies and sustains the impact of chances taken and their consequences. If you speak up at a sales meeting, the client says Good Point, and I later praise you for that, the culture change impact is X. If I then relate that story to everyone at the next sales team meeting, the impact is X * 10 others there. If we write down that behavior in the firm's sales training program as a good model to follow, the impact is X * 100 others who will go through that program.
Summing up, here's a simple set of questions to ask for managing culture change:
Ask these questions daily, tote up the score -- chances taken, consequences realized, communications executed -- weekly or monthly. Track the trend, slice the numbers by the behaviors and people you're trying to influence, and the consequences and communications that apply. Don't forget to keep culture change in context: frame it with the business results culture is supposed to serve. Re-focus, then wash, rinse, repeat. Very soon you'll have a clear view of and strong grip on culture change in your organization.
So healthcare.gov launched, with problems. I'm trying to understand why, so I can apply some lessons in my professional life. Here are some ideas.
First, I think it helps to define some levels of the problem. I can think of four:
1. Strategic / policy level -- what challenges do the goals we set create? In this case, the objective, basically, is two-fold: first; reduce the costs of late-stage, high-cost uncompensated care by enrolling the people who ultimately use that (middle-aged poor folks and other unfortunates) in health insurance that will get them care earlier and reduce stress / improve outcomes (for them and for society) later; second; reduce the cost of this insurance through exchanges that drive competition. So, basically, bring a bunch of folks from, in many cases, the wrong side of the Digital Divide, and expose them to a bunch of eligibility- and choice-driven complexity (proof: need for "Navigators"). Hmm. (Cue the folks who say that's why we need a simple single-payor model, but the obvious response would be that it simply wasn't politically feasible. We need to play the cards we're dealt.)
2. Experience level -- In light of that need, let's examine what the government did do for each of the "Attract / Engage / Convert / Retain" phases of a Caveman User Experience. It did promote ACA -- arguably insufficiently or not creatively enough to distinguish itself from opposing signal levels it should have anticipated (one take here). But more problematically, from what I can tell, the program skips "Engage" and emphasizes "Convert": Healthcare.gov immediately asks you to "Apply Now" (see screenshot below, where "Apply Now" is prominently featured over "Learn More", even on the "Learn" tab of the site). This is technically problematic (see #3 below), but also experientially lots to ask for when you don't yet know what's behind the curtain.
3. Technical level -- Excellent piece in Washington Post by Timothy B. Lee. Basically, the system tries to do an eligibility check (for participation and subsidies) before sending you on to enrollment. Doing this requires checking a bunch of other government systems. The flowchart explains very clearly why this could be problematic. There are some front end problems as well, described in rawest form by some of the chatter on Reddit, but from what I've seen these are more superficial, a function of poor process / time management, and fixable.
Second, here are some things HHS might do differently:
1. Strategic level: Sounds like some segmentation of the potential user base would have suggested a much greater investment in explanation / education, in advance of registration. Since any responsible design effort starts with users and use cases, I'm sure they did this. But what came out the other end doesn't seem to reflect that. What bureaucratic or political considerations got in the way, and what can be revisited, to improve the result? Or, instead of allowing political hacks to infiltrate and dominate the ranks of engineers trying to design a service that works, why not embed competent technologists, perhaps drawn from the ranks of Chief Digital Officers, into the senior political ranks, to advise them on how to get things right online?
2. Experience level: Perhaps the first couple of levels of experience on healthcare.gov should have been explanatory? "Here's what to expect, here's how this works..." Maybe video (could have used YouTube!)? Maybe also ask a couple of quick anonymous questions to determine whether the eligibility / subsidy check would be relevant, to spare the load on that engine, before seeing what plans might be available, at what price? You could always re-ask / confirm that data later once the user's past the shopping /evaluation stage, before formally enrolling them into a plan. In ecommerce, we don't ask untargeted shoppers to enter discount codes until they're about to check out, right?
3. I see from viewing the page source they have Google Tag Manager running, so perhaps they also have Google Analytics running too, alongside whatever other things... Since they've open-sourced the front end code and their content on Github, maybe they could also share what they're learning via GA, so we could evaluate ideas for improving the site in the context of that data?
Postscript, November 1:
Food for thought (scroll to bottom). How does this happen? Software engineer friends, please weigh in!
I originally got to know Judah Phillips through Web Analytics Wednesdays events he organized, and in recent years he's kindly participated on panels I've moderated and has been helpful to my own writing and publishing efforts. I've even partnered with some of the excellent professionals who have worked for him. So while I'm biased as the beneficiary of his wisdom and support, I can also vouch first-hand for the depth and credibility of his advice. In short, in an increasingly hype-filled category, Judah is the real deal, and this makes "Building The Digital Analytics Organization" a book to take seriously.
For me the book was useful on three levels. One, it's a foundational text for framing how to come at business analysis and reporting. Specifically, he presents an Analytics Value Chain that reminds us to bookend our analytic efforts per se with a clear set of objectives and actions, an orientation that's sadly missing in many balkanized corporate environments. Two, it's a blueprint for your own organization-building efforts. He really covers the waterfront, from how to approach analysis, to different kinds of analysis you can pursue, to how to organize the function and manage its relationships with other groups that play important supporting roles. For me, Chapter 6, "Defining, Planning, Collecting, and Governing Data in Digital Analytics" is an especially useful section. In it, he presents a very clear, straightforward structure for how you should set up and run these crucial functions. Finally, three, Judah offers a strong point of view on certain decisions. For example, I read him to advocate for a strongly centralized digital analytics function, rooted in the "business" side of the house, to make sure that you have both critical mass for these crucial skills, as well as proximity to the decisions they need to support.
These three uses had me scribbling in the margins and dog-earing extensively. But if you still need one more reason to pull the trigger, it helps that the book is very up-to-date and has a final chapter that looks forward very thoughtfully into how Judah expects what he describes as the "Analytical Economy" to evolve. This section is both a helpful survey of the different capabilities that will shape this future as well as an exploration of the issues these capabilities and associated trends will raise, in particular as they relate to privacy. It's a valuable checklist, to make sure you're not just building for today, but for the next few years to come.
I moderated this panel at the Massachusetts Innovation and Technology Exchange's (mitx.org)"The Science of Marketing: Using Data & Analytics for Winning" summit on August 1, 2013. Thanks to T. Rowe Price's Paul Musante, Visual IQ's Manu Mathew, iKnowtion's Don Ryan, and Google's Sonia Chung for participating!
On July 27, I'll be riding once again in the annual Nashoba Learning Group bike-a-thon, and I'd really appreciate your support:
(Note: please also Like / Retweet / forward to friends, etc. using links at bottom!)
This is a great cause, and an incredibly effective and well-run school. Your contribution will make a big difference. (And thank you to everyone who'd been so generous so far!)
For kicks, here's my recap of my 2007 ride:
Thank you all for being so generous on such short notice!
Fresh off a flight from London that arrived in Boston at midnight on Friday, I wheeled myself onto the starting line Saturday morning a few minutes after eight . Herewith, a few journal entries from the ride:
Mile 2: The peloton drops me like a stone. Dopeurs! Never mind; this breakaway is but le petit setback. Where are my domestiques to bring me back to the pack?
Mile 3: Reality intrudes. No domestiques. Facing 47 miles' worth of solo quality time, I plot my comeback...
Mile 10: 1st major climb, L'Alpe de Bolton (MA), a steep, nasty little "beyond classification" grade. I curse at the crowds pressing in. 'Allez! Allez!' they call, like wolves. A farmer in a Superman cape runs alongside.
Mile 10.25: Mirages disappear in the 95-degree heat. (First time I've seen the Superman dude, though. Moral of this story: lay off the British Airways dessert wines the night before a big ride.)
Mile 10.5: Descending L'Alpe de Bolton, feeling airborne at 35 MPH
Mile 10.50125: Realizing after hitting bump that I am, in fact, airborne. AAAAARRH!!!
Mile 14: I smell sweet victory in the morning air!
Mile 15: Realize the smell is actually the Bolton dump
Mile 27: Col d'Harvard (MA). Mis-shift on steep climb, drop chain off granny ring. Barely click out of pedal to avoid keeling over, disappointing two buzzards circling overhead.
Mile 33: Whip out Blackberry, Googling 'Michael Rasmussen soigneur' to see if can score some surplus EPO
Mile 40: I see dead people
Mile 50: I am, ahem... outsprinted at the finish. Ride organizers generously grant me 'same time' when they realize no one noticed exactly when I got back."
We're now in the blood-sugar-crash phase of the Analytics / Big Data hype cycle, where the gap between promise and reality is greatest. Presenting symptoms of the gap include complaints about alignment, access to data, capacity to act on data-driven insights, and talent. This September 2012 HBR blog post by Paul Barth and Randy Bean of NewVantage Partners underscores this with some interesting data.
Executives' anxiety about this gap is also at its peak. Many of them turn to organization as their prime lever for solving things. A question I get a lot is "How should we organize our analytic capabilities?" Related ones include "How centralized should they be?", and "What should be on the business side, and what belongs in IT?"
This post suggests a few criteria for helping to answer these questions. But first, I'd like to offer a principle for tackling this generally:
Think organization last, not first.
A corollary to this might be, "Role is as role does." Too much attention today is paid to developing and organizing for analytic capability. Not enough attention is paid to defining and managing a portfolio of important business opportunities that leverage this capability. In our work with clients, we focus on building capability through practice and results. Our litmus test for whether we're making progress is a rule we call "3-2-1": In each quarter, the portfolio of business opportunities we're supporting with analytic efforts has to yield at least three "news you can use" insights, two experiments based on these insights, and one "scaling" of prior experiments to "production", with commensurate results. (The specific goals we set for each of these varies of course from situation to situation, but the approach is the same.)
Approaching things this way has several benefits:
Now, two critiques that can be made of this approach are, first, that it's too ad hoc and therefore misses opportunities to leverage experience beyond each individual opportunity addressed, and second, that it ignores that most people are "tribal" and that their behaviors are shaped accordingly. So once you've got a decent portfolio assembled and you're managing it along, here are some organizational considerations you can apply to help decide where folks should "live":
In our work we'll typically apply these criteria using scoresheets to evaluate either or both the specific business challenges we're solving for or the organizational models we're evaluating as possible options. Sometimes we just use "high-medium-low" assessments, and other times we'll do the math to help us stay objective about different ways to go. The main things are to keep attention to organization in balance with attention to progress, and to keep discussions about organization focused on the needs of the business, rather than allowing them to devolve into proxy battles for executive power and influence.
This week, one of the big news items is the disclosure of the NSA's Prism program that collects all sorts of our electronic communications, to help identify terrorists and prevent attacks.
I was struck by three things. One is the recency bias in the outrage expressed by many people. Not sixty days ago we were all horrified at the news of the Boston Marathon bombings. Another is the polarization of the debate. Consider the contrast the Hullabaloo blog draws between "insurrectionists" and "institutionalists". The third was the superficial treatment of the tradeoffs folks would be willing to make. Yesterday the New York Times Caucus blog published the results of a survey that suggested most folks are fence-sitters on the tradeoff between privacy and security, but left it more or less at that. (The Onion wasn't far behind with a perfect send-up of the ambivalence we feel.)
In sum, biased decision-making based on excessively simplified choices using limited data. Not helpful. Better would be a more nuanced examination of the tradeoff between the privacy you would be willing to give up for the potential lives saved. I see this opportunity to improve decision making alot, and I thought this would be an interesting example to illustrate how framing and informing an issue differently can help. So I posted this survey: https://t.co/et0Bs0OrKF
Here are some early results from twelve folks who kindly took it (please feel free to add your answers, if I get enough more I'll update the results):
(Each axis is a seven point scale, 1 at lowest and 7 at highest. Bubble size = # of respondents who provided that tradeoff as their answer. No bubble / just label = 1 respondent, biggest bubble at lower right = 3 respondents.)
Interesting distribution, tending slightly toward folks valuing (their own) privacy over (other people's) security.
Now my friend and business school classmate Sam Kinney suggested this tradeoff was a false choice. I disagreed with him. But the exchange did get me to think a bit further. More data isn't necessarily linear in its benefits. It could have diminishing returns of course (as I argued in Pragmalytics) but it could also have increasing value as the incremental data might fill in a puzzle or help to make a connection. While that relationship between data and safety is hard for me to process, the government might help its case by being less deceptive and more transparent about what it's collecting, and its relative benefits. It might do this, if not for principle, then for the practical value of controlling the terms of the debate when, as David Brooks wrote so brilliantly this week, an increasingly anomic society cultivates Edward Snowdens at an accelerating clip.
I'm skeptical about the value of this data for identifying terrorists and preventing their attacks. Any competent terrorist network will use burner phones, run its own email servers, and communicate in code. But maybe the data surveillance program has value because it raises the bar to this level of infrastructure and process, and thus makes it harder for such networks to operate.
I'm not concerned about the use of my data for security purposes, especially not if it can save innocent boys and girls from losing limbs at the hands of sick whackos. I am really concerned it might get reused for other purposes in ways I don't approve, or by folks whose motives I don't approve, so I'm sure we could improve oversight, not only for what data gets used how, but of the vast, outsourced, increasingly unaccountable government we have in place. But right now, against the broader backdrop of gridlock on essentially any important public issue, I just think the debate needs to get more utilitarian, and less political and ideological. And, I think analytically-inclined folks can play a productive role in making this happen.
(Thanks to @zimbalist and @perryhewitt for steering me to some great links, and to Sam for pushing my thinking.)
The May 20th 2013 edition of The New Yorker has an article by Vogue writer Nathan Heller on Massive Online Open Courses (MOOCs) titled "Laptop U: Has the future of college moved online?" The author explores, or at least raises, a number of related questions. How (well) does the traditional offline learning experience transfer online? Is the online learning experience more or less effective than the traditional one? (By what standard? For what material? What is gained and lost?) What will MOOCs mean for different colleges and universities, and their faculties? How will the MOOC revolution be funded? (In particular, what revenue model will emerge?)
Having worked a lot in the sector, for both public and private university clients, developing everything from technology, to online-enabled programs themselves, to analytic approaches, and even on marketing and promotion, the article was a good prompt for me to try to boil out some ways to think about answering these questions.
The article focuses almost exclusively on Harvard and EdX, the 12-school joint venture through which it's pursuing MOOCs. Obviously this skews the evaluation. Heller writes:
Education is a curiously alchemical process. Its vicissitudes are hard to isolate. Why do some students retain what they learned in a course for years, while others lose it through the other ear over their summer breaks? Is the fact that Bill Gates and Mark Zuckerberg dropped out of Harvard to revolutionize the tech industry a sign that their Harvard educations worked, or that they failed? The answer matters, because the mechanism by which conveyed knowledge blooms into an education is the standard by which MOOCs will either enrich teaching in this country or deplete it.
For me, the first step to boiling things out is to define what we mean by -- and want from -- an "education". So, let's try to unpack why people go to college. In most cases, Reason One is that you need a degree to get any sort of decent job. Reason Two is to plug into a network of people -- fellow students, alumni, faculty -- that provide you a life-long community. Of course you need a professional community for that Job thing, but also because in an otherwise anomic society you need an archipelago to seed friendships, companionships, and self-definition (or at least, as scaffolding for your personal brand: as one junior I heard on a recent college visit put it memorably, "Being here is part of the personal narrative I'm building.") Reason Three -- firmly third -- is to get an "education" in the sense that Heller describes. (Apropos: check this recording of David Foster Wallace's 2005 commencement address at Kenyon College.)
Next, this hierarchy of needs then gives us a way to evaluate the prospects for MOOCs.
If organization X can produce graduates demonstrably better qualified (through objective testing, portfolios of work, and experience) to do job Y, at a lower cost, then it will thrive. If organization X can do this better and cheaper by offering and/or curating/ aggregating MOOCs, then MOOCs will thrive. If a MOOC can demonstrate an adequately superior result / contribution to the end outcome, and do it inexpensively enough to hold its place in the curriculum, and do it often enough that its edge becomes a self-fulfilling prophecy -- a brand, in other words -- then it will crowd out its competitors, as surely as one plant shuts out the sunlight to another. Anyone care to bet against Georgia Tech's new $7K Master's in Computer Science?
If a MOOC-mediated social experience can connect you to a Club You Want To Be A Member Of, you will pay for that. And if a Club That Would Have You As A Member can attract you to its clubhouse with MOOCs, then MOOCs will line the shelves of its bar. The winning MOOC cocktails will be the ones that best produce the desired social outcomes, with the greatest number of satisfying connections.
Finally, learning is as much about the frame of mind of the student as it is about the quality of the teacher. If through the MOOC the student is able to choose a better time to engage, and can manage better the pace of the delivery of the subject matter, then the MOOC wins.
Beyond general prospects, as you consider these principles, it becomes clear that it's less about whether MOOCs win, but which ones, for what and for whom, and how.
The more objective and standardized -- and thus measurable and comparable -- the learning outcome and the standard of achievement, the greater the potential for a MOOC to dominate. My program either works, or it doesn't.
If a MOOC facilitates the kinds of content exchanges that seed and stimulate offline social gatherings -- pitches to VCs, or mock interviewing, or poetry, or dance routines, or photography, or music, or historical tours, or bird-watching trips, or snowblower-maintenance workshops -- then it has a better chance of fulfilling the longings of its students for connection and belonging.
And, the more well-developed the surrounding Internet ecosystem (Wikipedia, discussion groups, Quora forums, and beyond) is around a topic, the less I need a Harvard professor, or even a Harvard grad student, to help me, however nuanced and alchemical the experience I miss might otherwise have been. The prospect of schlepping to class or office hours on a cold, rainy November night has a way of diluting the urge to be there live in case something serendipitous happens.
Understanding how MOOCs win then also becomes a clue to understanding potential revenue models.
If you can get accredited to offer a degree based in part or whole on MOOCs, you can charge for that degree, and gets students or the government to pay for it (Exhibit A: University of Phoenix). That's hard, but as a variant of this, you can get hired by an organization, or a syndicate of organizations you organize, to produce tailored degree programs -- think corporate training programs on steroids -- that use MOOCs to filter and train students. (Think "You, Student, pay for the 101-level stuff; if you pass you get a certificate and an invitation to attend the 201-level stuff that we fund; if you pass that we give you a job.")
Funding can come directly, or be subsidized by sponsors and advertisers, or both.
You can try to charge for content: if you produce a MOOC that someone else wants to include in a degree-based program, you can try to license it, in part or in whole.
You can make money via the service angle, the way self-publishing firms support authors, with a variety of best-practice based production services. Delivery might be offered via a freemium model -- the content might be free, but access to premium groups, with teaching assistant support, might come at a price. You can also promote MOOCs -- build awareness, drive distribution, even simply brand -- for a cut of the action, the way publishers and event promoters do.
Perhaps in the not-too-distant future we'll get the Academic Upfront, in which Universities front a semester's worth of classes in a MOOC, then pitch the class to sponsors, the way TV networks do today. Or, maybe the retail industry also offers a window into how MOOCs will be monetized. Today's retail environment is dominated by global brands (think professors as fashion designers) and big-box (plus Amazon) firms that dominate supply chains and distrubution networks. Together, Brands and Retailers effectively act as filters: we make assumptions that the products on their shelves are safe, effective, reasonably priced, acceptably stylish, well-supported. In exchange, we'll pay their markup. This logic sounds a cautionary note for many schools: boutiques can survive as part of or at the edges of the mega-retailers' ecosystems, but small-to-mid-size firms reselling commodities get crushed.
Of course, these are all generic, unoriginal (see Ecclesiastes 1:9) speculations. Successful revenue models will blend careful attention to segmenting target markets and working back from their needs, resources, and processes (certain models might be friendlier to budgets and purchasing mechanisms than others) with thoughtful in-the-wild testing of the ideas. Monolithic executions with Neolithic measurement plans ("Gee, the focus group loved it, I can't understand why no one's signing up for the paid version!") are unlikely to get very far. Instead, be sure to design with testability in mind (make content modular enough to package or offer a la carte, for example). Maybe even use Kickstarter as a lab for different models!