This is part two of a series of short columns on problems in data analysis. I will use the lens of COVID-19-related data and interpretation (and misinterpretation) to shed some light on how nonprofit fundraising operations can avoid similar data-driven issues.
In my last column I mentioned the fad of referring to one’s nonprofit, philosophy of fundraising or grantmaking, or personal leadership style as “data-driven.” It’s one among many clichés ripped from the standard for-profit playbook.
On the one hand, it’s hard to argue with: who among us would want to be known for the implied alternative, completely ignoring the data or doing the opposite of what the data suggests?
But orienting leadership around the false notion that the data always points in a clear direction, or refusing to take action until one’s preferred type of data can be collected and analyzed, can actually be a recipe for paralysis and poor decision making.
Many events and changes in the course of humanity—both globally and at the more local scale that most nonprofits are dealing with—are not the sort of thing where we have the luxury of abundantly clear data. And many events that demand our attention are not the sort of thing where we can put off decision-making about until that data arrives—even if it happens to be the sort of phenomenon which is conducive to data analysis.
Take our COVID case, for example. While many widespread social problems stubbornly resist being reduced to a few sets of charts and graphs, a pandemic like COVID-19 is nearly a textbook case where good data could make all the difference when it comes to policy.
With COVID-19 (at least on the medical questions specifically), we were dealing with “unknown knowns.” The most relevant questions as the situation heated up in March could be boiled down to just a handful: how many people have COVID? How quickly does it spread and grow? Of those that have it, how many will require hospitalization, and of those how many will pass away?
And yet, at the time that swift decisions needed to be taken, public-policy makers could not answer any of these questions. Some of these questions we had no way to even make an estimate, and on other questions we made estimates that turned out to be wildly wrong (remember the incredible 180-degree turnaround on the question of masks?).
If you were a governor or legislator on, say, March 15th of this year, you were faced with an imperative to take dramatic actions—for better or worse—affecting nearly every area of the economy and public life, knowing that you couldn’t be sure, and maybe wouldn’t ever be sure, whether you did the right thing according to the data. Why? Because that basic data didn’t yet exist—and much of the existing data we did have at that time already looks incomplete or misleading even just two months later.
So in this sense, to insist on being “data driven” is actually the abnegation of leadership. The “data-driven” philosophy says that decisions are only justified if appropriately backed by data that indicates a clear course of action. But if there exists such data, leadership is no longer necessary: anyone with the data in hand should be able to make the correct decision. Leadership qualities, entrepreneurial spirit, and the rest are unnecessary.
But in the real world, true leadership very often involves making tough decisions with little information, misleading information, or even contradictory information. Cheap demands for rigorous empiricism can therefore be quite counterproductive and even backfire spectacularly. The good leaders are those who are able to act, well and responsibly, even with that limited data.
Imagine a world where all public leaders insisted on seeing the results of rigorous randomized control trials before limiting public gatherings of 50 people or did nothing until they could confirm the exact rate of exponential growth of the virus. It’s an absurd notion, and yet I see analogous situations in nonprofits all the time (albeit usually with much lower stakes!).
Before doubling down on empiricism and data obsession in your organization, first ask:
- Is the question we are trying to answer one that is conducive to empirical confirmation, or is it too complex for simple metrical answers?
- What is the potential cost of delaying action until this data can be generated or confirmed?
- What will it cost us to acquire this data in both resources and time?
- What will this data actually tell us, and will it be worth the resources required?
- Would some plausible range of findings actually change any of our decisions or strategies, or are we already committed to a course of action which numbers and charts would merely make us feel better about?
Coming up with honest answers to these questions will help you become a better nonprofit leader or grantmaker and escape the pitfalls of naïve data obsession. Leadership is the driver, not data. And data at its best is a helpful partner in the backseat.