I lead Force Five Partners, a marketing analytics consulting firm (bio). I've been writing here about marketing, technology, e-business, and analytics since 2003 (blog name explained).
Email or follow me:
So healthcare.gov launched, with problems. I'm trying to understand why, so I can apply some lessons in my professional life. Here are some ideas.
First, I think it helps to define some levels of the problem. I can think of four:
1. Strategic / policy level -- what challenges do the goals we set create? In this case, the objective, basically, is two-fold: first; reduce the costs of late-stage, high-cost uncompensated care by enrolling the people who ultimately use that (middle-aged poor folks and other unfortunates) in health insurance that will get them care earlier and reduce stress / improve outcomes (for them and for society) later; second; reduce the cost of this insurance through exchanges that drive competition. So, basically, bring a bunch of folks from, in many cases, the wrong side of the Digital Divide, and expose them to a bunch of eligibility- and choice-driven complexity (proof: need for "Navigators"). Hmm. (Cue the folks who say that's why we need a simple single-payor model, but the obvious response would be that it simply wasn't politically feasible. We need to play the cards we're dealt.)
2. Experience level -- In light of that need, let's examine what the government did do for each of the "Attract / Engage / Convert / Retain" phases of a Caveman User Experience. It did promote ACA -- arguably insufficiently or not creatively enough to distinguish itself from opposing signal levels it should have anticipated (one take here). But more problematically, from what I can tell, the program skips "Engage" and emphasizes "Convert": Healthcare.gov immediately asks you to "Apply Now" (see screenshot below, where "Apply Now" is prominently featured over "Learn More", even on the "Learn" tab of the site). This is technically problematic (see #3 below), but also experientially lots to ask for when you don't yet know what's behind the curtain.
3. Technical level -- Excellent piece in Washington Post by Timothy B. Lee. Basically, the system tries to do an eligibility check (for participation and subsidies) before sending you on to enrollment. Doing this requires checking a bunch of other government systems. The flowchart explains very clearly why this could be problematic. There are some front end problems as well, described in rawest form by some of the chatter on Reddit, but from what I've seen these are more superficial, a function of poor process / time management, and fixable.
Second, here are some things HHS might do differently:
1. Strategic level: Sounds like some segmentation of the potential user base would have suggested a much greater investment in explanation / education, in advance of registration. Since any responsible design effort starts with users and use cases, I'm sure they did this. But what came out the other end doesn't seem to reflect that. What bureaucratic or political considerations got in the way, and what can be revisited, to improve the result? Or, instead of allowing political hacks to infiltrate and dominate the ranks of engineers trying to design a service that works, why not embed competent technologists, perhaps drawn from the ranks of Chief Digital Officers, into the senior political ranks, to advise them on how to get things right online?
2. Experience level: Perhaps the first couple of levels of experience on healthcare.gov should have been explanatory? "Here's what to expect, here's how this works..." Maybe video (could have used YouTube!)? Maybe also ask a couple of quick anonymous questions to determine whether the eligibility / subsidy check would be relevant, to spare the load on that engine, before seeing what plans might be available, at what price? You could always re-ask / confirm that data later once the user's past the shopping /evaluation stage, before formally enrolling them into a plan. In ecommerce, we don't ask untargeted shoppers to enter discount codes until they're about to check out, right?
3. I see from viewing the page source they have Google Tag Manager running, so perhaps they also have Google Analytics running too, alongside whatever other things... Since they've open-sourced the front end code and their content on Github, maybe they could also share what they're learning via GA, so we could evaluate ideas for improving the site in the context of that data?
Postscript, November 1:
Food for thought (scroll to bottom). How does this happen? Software engineer friends, please weigh in!
I originally got to know Judah Phillips through Web Analytics Wednesdays events he organized, and in recent years he's kindly participated on panels I've moderated and has been helpful to my own writing and publishing efforts. I've even partnered with some of the excellent professionals who have worked for him. So while I'm biased as the beneficiary of his wisdom and support, I can also vouch first-hand for the depth and credibility of his advice. In short, in an increasingly hype-filled category, Judah is the real deal, and this makes "Building The Digital Analytics Organization" a book to take seriously.
For me the book was useful on three levels. One, it's a foundational text for framing how to come at business analysis and reporting. Specifically, he presents an Analytics Value Chain that reminds us to bookend our analytic efforts per se with a clear set of objectives and actions, an orientation that's sadly missing in many balkanized corporate environments. Two, it's a blueprint for your own organization-building efforts. He really covers the waterfront, from how to approach analysis, to different kinds of analysis you can pursue, to how to organize the function and manage its relationships with other groups that play important supporting roles. For me, Chapter 6, "Defining, Planning, Collecting, and Governing Data in Digital Analytics" is an especially useful section. In it, he presents a very clear, straightforward structure for how you should set up and run these crucial functions. Finally, three, Judah offers a strong point of view on certain decisions. For example, I read him to advocate for a strongly centralized digital analytics function, rooted in the "business" side of the house, to make sure that you have both critical mass for these crucial skills, as well as proximity to the decisions they need to support.
These three uses had me scribbling in the margins and dog-earing extensively. But if you still need one more reason to pull the trigger, it helps that the book is very up-to-date and has a final chapter that looks forward very thoughtfully into how Judah expects what he describes as the "Analytical Economy" to evolve. This section is both a helpful survey of the different capabilities that will shape this future as well as an exploration of the issues these capabilities and associated trends will raise, in particular as they relate to privacy. It's a valuable checklist, to make sure you're not just building for today, but for the next few years to come.
I moderated this panel at the Massachusetts Innovation and Technology Exchange's (mitx.org)"The Science of Marketing: Using Data & Analytics for Winning" summit on August 1, 2013. Thanks to T. Rowe Price's Paul Musante, Visual IQ's Manu Mathew, iKnowtion's Don Ryan, and Google's Sonia Chung for participating!
On July 27, I'll be riding once again in the annual Nashoba Learning Group bike-a-thon, and I'd really appreciate your support:
(Note: please also Like / Retweet / forward to friends, etc. using links at bottom!)
This is a great cause, and an incredibly effective and well-run school. Your contribution will make a big difference. (And thank you to everyone who'd been so generous so far!)
For kicks, here's my recap of my 2007 ride:
Thank you all for being so generous on such short notice!
Fresh off a flight from London that arrived in Boston at midnight on Friday, I wheeled myself onto the starting line Saturday morning a few minutes after eight . Herewith, a few journal entries from the ride:
Mile 2: The peloton drops me like a stone. Dopeurs! Never mind; this breakaway is but le petit setback. Where are my domestiques to bring me back to the pack?
Mile 3: Reality intrudes. No domestiques. Facing 47 miles' worth of solo quality time, I plot my comeback...
Mile 10: 1st major climb, L'Alpe de Bolton (MA), a steep, nasty little "beyond classification" grade. I curse at the crowds pressing in. 'Allez! Allez!' they call, like wolves. A farmer in a Superman cape runs alongside.
Mile 10.25: Mirages disappear in the 95-degree heat. (First time I've seen the Superman dude, though. Moral of this story: lay off the British Airways dessert wines the night before a big ride.)
Mile 10.5: Descending L'Alpe de Bolton, feeling airborne at 35 MPH
Mile 10.50125: Realizing after hitting bump that I am, in fact, airborne. AAAAARRH!!!
Mile 14: I smell sweet victory in the morning air!
Mile 15: Realize the smell is actually the Bolton dump
Mile 27: Col d'Harvard (MA). Mis-shift on steep climb, drop chain off granny ring. Barely click out of pedal to avoid keeling over, disappointing two buzzards circling overhead.
Mile 33: Whip out Blackberry, Googling 'Michael Rasmussen soigneur' to see if can score some surplus EPO
Mile 40: I see dead people
Mile 50: I am, ahem... outsprinted at the finish. Ride organizers generously grant me 'same time' when they realize no one noticed exactly when I got back."
We're now in the blood-sugar-crash phase of the Analytics / Big Data hype cycle, where the gap between promise and reality is greatest. Presenting symptoms of the gap include complaints about alignment, access to data, capacity to act on data-driven insights, and talent. This September 2012 HBR blog post by Paul Barth and Randy Bean of NewVantage Partners underscores this with some interesting data.
Executives' anxiety about this gap is also at its peak. Many of them turn to organization as their prime lever for solving things. A question I get a lot is "How should we organize our analytic capabilities?" Related ones include "How centralized should they be?", and "What should be on the business side, and what belongs in IT?"
This post suggests a few criteria for helping to answer these questions. But first, I'd like to offer a principle for tackling this generally:
Think organization last, not first.
A corollary to this might be, "Role is as role does." Too much attention today is paid to developing and organizing for analytic capability. Not enough attention is paid to defining and managing a portfolio of important business opportunities that leverage this capability. In our work with clients, we focus on building capability through practice and results. Our litmus test for whether we're making progress is a rule we call "3-2-1": In each quarter, the portfolio of business opportunities we're supporting with analytic efforts has to yield at least three "news you can use" insights, two experiments based on these insights, and one "scaling" of prior experiments to "production", with commensurate results. (The specific goals we set for each of these varies of course from situation to situation, but the approach is the same.)
Approaching things this way has several benefits:
Now, two critiques that can be made of this approach are, first, that it's too ad hoc and therefore misses opportunities to leverage experience beyond each individual opportunity addressed, and second, that it ignores that most people are "tribal" and that their behaviors are shaped accordingly. So once you've got a decent portfolio assembled and you're managing it along, here are some organizational considerations you can apply to help decide where folks should "live":
In our work we'll typically apply these criteria using scoresheets to evaluate either or both the specific business challenges we're solving for or the organizational models we're evaluating as possible options. Sometimes we just use "high-medium-low" assessments, and other times we'll do the math to help us stay objective about different ways to go. The main things are to keep attention to organization in balance with attention to progress, and to keep discussions about organization focused on the needs of the business, rather than allowing them to devolve into proxy battles for executive power and influence.
This week, one of the big news items is the disclosure of the NSA's Prism program that collects all sorts of our electronic communications, to help identify terrorists and prevent attacks.
I was struck by three things. One is the recency bias in the outrage expressed by many people. Not sixty days ago we were all horrified at the news of the Boston Marathon bombings. Another is the polarization of the debate. Consider the contrast the Hullabaloo blog draws between "insurrectionists" and "institutionalists". The third was the superficial treatment of the tradeoffs folks would be willing to make. Yesterday the New York Times Caucus blog published the results of a survey that suggested most folks are fence-sitters on the tradeoff between privacy and security, but left it more or less at that. (The Onion wasn't far behind with a perfect send-up of the ambivalence we feel.)
In sum, biased decision-making based on excessively simplified choices using limited data. Not helpful. Better would be a more nuanced examination of the tradeoff between the privacy you would be willing to give up for the potential lives saved. I see this opportunity to improve decision making alot, and I thought this would be an interesting example to illustrate how framing and informing an issue differently can help. So I posted this survey: https://t.co/et0Bs0OrKF
Here are some early results from twelve folks who kindly took it (please feel free to add your answers, if I get enough more I'll update the results):
(Each axis is a seven point scale, 1 at lowest and 7 at highest. Bubble size = # of respondents who provided that tradeoff as their answer. No bubble / just label = 1 respondent, biggest bubble at lower right = 3 respondents.)
Interesting distribution, tending slightly toward folks valuing (their own) privacy over (other people's) security.
Now my friend and business school classmate Sam Kinney suggested this tradeoff was a false choice. I disagreed with him. But the exchange did get me to think a bit further. More data isn't necessarily linear in its benefits. It could have diminishing returns of course (as I argued in Pragmalytics) but it could also have increasing value as the incremental data might fill in a puzzle or help to make a connection. While that relationship between data and safety is hard for me to process, the government might help its case by being less deceptive and more transparent about what it's collecting, and its relative benefits. It might do this, if not for principle, then for the practical value of controlling the terms of the debate when, as David Brooks wrote so brilliantly this week, an increasingly anomic society cultivates Edward Snowdens at an accelerating clip.
I'm skeptical about the value of this data for identifying terrorists and preventing their attacks. Any competent terrorist network will use burner phones, run its own email servers, and communicate in code. But maybe the data surveillance program has value because it raises the bar to this level of infrastructure and process, and thus makes it harder for such networks to operate.
I'm not concerned about the use of my data for security purposes, especially not if it can save innocent boys and girls from losing limbs at the hands of sick whackos. I am really concerned it might get reused for other purposes in ways I don't approve, or by folks whose motives I don't approve, so I'm sure we could improve oversight, not only for what data gets used how, but of the vast, outsourced, increasingly unaccountable government we have in place. But right now, against the broader backdrop of gridlock on essentially any important public issue, I just think the debate needs to get more utilitarian, and less political and ideological. And, I think analytically-inclined folks can play a productive role in making this happen.
(Thanks to @zimbalist and @perryhewitt for steering me to some great links, and to Sam for pushing my thinking.)
The May 20th 2013 edition of The New Yorker has an article by Vogue writer Nathan Heller on Massive Online Open Courses (MOOCs) titled "Laptop U: Has the future of college moved online?" The author explores, or at least raises, a number of related questions. How (well) does the traditional offline learning experience transfer online? Is the online learning experience more or less effective than the traditional one? (By what standard? For what material? What is gained and lost?) What will MOOCs mean for different colleges and universities, and their faculties? How will the MOOC revolution be funded? (In particular, what revenue model will emerge?)
Having worked a lot in the sector, for both public and private university clients, developing everything from technology, to online-enabled programs themselves, to analytic approaches, and even on marketing and promotion, the article was a good prompt for me to try to boil out some ways to think about answering these questions.
The article focuses almost exclusively on Harvard and EdX, the 12-school joint venture through which it's pursuing MOOCs. Obviously this skews the evaluation. Heller writes:
Education is a curiously alchemical process. Its vicissitudes are hard to isolate. Why do some students retain what they learned in a course for years, while others lose it through the other ear over their summer breaks? Is the fact that Bill Gates and Mark Zuckerberg dropped out of Harvard to revolutionize the tech industry a sign that their Harvard educations worked, or that they failed? The answer matters, because the mechanism by which conveyed knowledge blooms into an education is the standard by which MOOCs will either enrich teaching in this country or deplete it.
For me, the first step to boiling things out is to define what we mean by -- and want from -- an "education". So, let's try to unpack why people go to college. In most cases, Reason One is that you need a degree to get any sort of decent job. Reason Two is to plug into a network of people -- fellow students, alumni, faculty -- that provide you a life-long community. Of course you need a professional community for that Job thing, but also because in an otherwise anomic society you need an archipelago to seed friendships, companionships, and self-definition (or at least, as scaffolding for your personal brand: as one junior I heard on a recent college visit put it memorably, "Being here is part of the personal narrative I'm building.") Reason Three -- firmly third -- is to get an "education" in the sense that Heller describes. (Apropos: check this recording of David Foster Wallace's 2005 commencement address at Kenyon College.)
Next, this hierarchy of needs then gives us a way to evaluate the prospects for MOOCs.
If organization X can produce graduates demonstrably better qualified (through objective testing, portfolios of work, and experience) to do job Y, at a lower cost, then it will thrive. If organization X can do this better and cheaper by offering and/or curating/ aggregating MOOCs, then MOOCs will thrive. If a MOOC can demonstrate an adequately superior result / contribution to the end outcome, and do it inexpensively enough to hold its place in the curriculum, and do it often enough that its edge becomes a self-fulfilling prophecy -- a brand, in other words -- then it will crowd out its competitors, as surely as one plant shuts out the sunlight to another. Anyone care to bet against Georgia Tech's new $7K Master's in Computer Science?
If a MOOC-mediated social experience can connect you to a Club You Want To Be A Member Of, you will pay for that. And if a Club That Would Have You As A Member can attract you to its clubhouse with MOOCs, then MOOCs will line the shelves of its bar. The winning MOOC cocktails will be the ones that best produce the desired social outcomes, with the greatest number of satisfying connections.
Finally, learning is as much about the frame of mind of the student as it is about the quality of the teacher. If through the MOOC the student is able to choose a better time to engage, and can manage better the pace of the delivery of the subject matter, then the MOOC wins.
Beyond general prospects, as you consider these principles, it becomes clear that it's less about whether MOOCs win, but which ones, for what and for whom, and how.
The more objective and standardized -- and thus measurable and comparable -- the learning outcome and the standard of achievement, the greater the potential for a MOOC to dominate. My program either works, or it doesn't.
If a MOOC facilitates the kinds of content exchanges that seed and stimulate offline social gatherings -- pitches to VCs, or mock interviewing, or poetry, or dance routines, or photography, or music, or historical tours, or bird-watching trips, or snowblower-maintenance workshops -- then it has a better chance of fulfilling the longings of its students for connection and belonging.
And, the more well-developed the surrounding Internet ecosystem (Wikipedia, discussion groups, Quora forums, and beyond) is around a topic, the less I need a Harvard professor, or even a Harvard grad student, to help me, however nuanced and alchemical the experience I miss might otherwise have been. The prospect of schlepping to class or office hours on a cold, rainy November night has a way of diluting the urge to be there live in case something serendipitous happens.
Understanding how MOOCs win then also becomes a clue to understanding potential revenue models.
If you can get accredited to offer a degree based in part or whole on MOOCs, you can charge for that degree, and gets students or the government to pay for it (Exhibit A: University of Phoenix). That's hard, but as a variant of this, you can get hired by an organization, or a syndicate of organizations you organize, to produce tailored degree programs -- think corporate training programs on steroids -- that use MOOCs to filter and train students. (Think "You, Student, pay for the 101-level stuff; if you pass you get a certificate and an invitation to attend the 201-level stuff that we fund; if you pass that we give you a job.")
Funding can come directly, or be subsidized by sponsors and advertisers, or both.
You can try to charge for content: if you produce a MOOC that someone else wants to include in a degree-based program, you can try to license it, in part or in whole.
You can make money via the service angle, the way self-publishing firms support authors, with a variety of best-practice based production services. Delivery might be offered via a freemium model -- the content might be free, but access to premium groups, with teaching assistant support, might come at a price. You can also promote MOOCs -- build awareness, drive distribution, even simply brand -- for a cut of the action, the way publishers and event promoters do.
Perhaps in the not-too-distant future we'll get the Academic Upfront, in which Universities front a semester's worth of classes in a MOOC, then pitch the class to sponsors, the way TV networks do today. Or, maybe the retail industry also offers a window into how MOOCs will be monetized. Today's retail environment is dominated by global brands (think professors as fashion designers) and big-box (plus Amazon) firms that dominate supply chains and distrubution networks. Together, Brands and Retailers effectively act as filters: we make assumptions that the products on their shelves are safe, effective, reasonably priced, acceptably stylish, well-supported. In exchange, we'll pay their markup. This logic sounds a cautionary note for many schools: boutiques can survive as part of or at the edges of the mega-retailers' ecosystems, but small-to-mid-size firms reselling commodities get crushed.
Of course, these are all generic, unoriginal (see Ecclesiastes 1:9) speculations. Successful revenue models will blend careful attention to segmenting target markets and working back from their needs, resources, and processes (certain models might be friendlier to budgets and purchasing mechanisms than others) with thoughtful in-the-wild testing of the ideas. Monolithic executions with Neolithic measurement plans ("Gee, the focus group loved it, I can't understand why no one's signing up for the paid version!") are unlikely to get very far. Instead, be sure to design with testability in mind (make content modular enough to package or offer a la carte, for example). Maybe even use Kickstarter as a lab for different models!