I have written extensively about how I believe practitioners and researchers can work most effectively together in reducing poverty, often using some of the missed opportunities for this kind of collaboration in microfinance as a point of departure. In the process, quite a few people have come to believe that I am opposed to randomized controlled trials (RCTs). Perhaps part of the issue is the tendency of many people, especially those in the media, to reduce a point of view to whether it is “for” or “against” something else.
In fact, my position has always been that RCTs have an important place in improving development policy and practice. I have, however, been uneasy with the way they have been promoted – dare I say “over-hyped”? – as a “gold standard” and also with the misleading assumptions and comparisons some of its proponents routinely make. The limitations of RCTs are frequently minimized, as are the potential contributions of other research methodologies.
In the case of microcredit, the way RCT studies have been digested, interpreted, and acted upon has done considerable and unnecessary damage to policy and practice, even as it has increased our knowledge of some important issues. No one group bears responsibility for this damage, but too many continue to ignore or minimize it.
Enter Angus Deaton
Like many others, I have struggled at times to articulate the source of my unease. Now comes powerful commentary on RCTs and related topics by 2015 Nobel laureate economist Professor Angus Deaton in his interview with Tim Ogden of NYU’s Financial Access Initiative.
Deaton, a Professor of Economics and International Affairs at the Woodrow Wilson School, is currently focusing on the broad topics of health, well being, and economic development. In announcing the Nobel Prize, the Royal Swedish Academy of Sciences noted his “analysis of poverty, consumption and welfare.”
In the interview, Deaton comes across as a blunt, old school economist who resists hyperbolic claims by proponents of the latest fads, especially if those fads are rebranded versions of earlier ones that were discovered to have notable strengths but also significant limitations.
I have a lot of sympathy for this type of realistic, long-term thinking. In fact, my blog post about what impact investors could learn from microfinance began as a reflection on the ahistorical thinking of many impact investors who failed to appreciate their forerunners and what they accomplished and learned.
In the interview with Ogden, the overarching theme is that development and development research, besides being prone to fads, are simply hard to do well. There are no short-cuts, no silver bullets, and above all (to use a term he favors), no “magic.” Each generation seems to believe they have found the answer, and are ultimately proved wrong, in part because they missed opportunities to learn from prior generations and to replace hubris with humility and curiosity.
Certainly Grameen Foundation, for example, has learned how difficult it is to make tangible progress on leveraging the power of information technology for poverty reduction. But with sustained effort, and at times aided by RCTs, we have been able to make real headway, especially in the areas of financial services, subsistence agriculture, and using field data to improve decision-making impacting poor clients. Other members of the Microfinance CEO Working Group are advancing the state of practice through similarly rigorous and painstaking approaches.
As we go about our work, Grameen Foundation staff and volunteers continuously familiarize themselves with and promote broad industry awareness of the widest possible range of research findings and methodologies, most notably in our “Measuring the Impact of Microfinance” series, which is coming out with its third installment next month. (This forthcoming publication is written by Professor Kathleen Odell of Dominican University, who also authored the 2010 report.)
To get a flavor of Deaton’s commentary, I have extracted some of the most trenchant quotes from the interview and grouped them thematically. I urge people to read the entire interview and also the forthcoming book Experimental Conversations by Ogden that it will be part of. (I am also interviewed in the book.)
There Is No Magic
“[The randomistas believe that they] have a magic tool but they don’t yet have much of an idea of the problems with that magic tool. And there are a lot of them…. I think RCTs rarely meet the hype.”
“[The randomistas] like to argue that RCTs don’t need assumptions but they’re loaded with assumptions at least if you are going to use them for anything.”
“There is a sort of belief in magic, that RCTs are attributed with properties that they do not possess.”
Failing to Learn from History
“If you go back to the 70s and 80s and you read what was written then, people thought quite hard about how you take the result from one experiment and how it would apply somewhere else. I see too little of that in the development literature today… [My] reading of the J-PAL webpage makes me think that when they list estimates they seem to suggest you can use them pretty much anywhere. Which is pretty weird when you think about it.”
“I think [prior generations of economists] thought very hard about the strengths and limitations of RCTs, and that much of that seems to have been lost today, which is a pity.”
“I think some of it is a youth over old age sort of idea. Here’s a new tool, we can rethink the world with it.”
Beyond Its Capabilities
“…[A] reporter wrote that an RCT allows you to establish causality for sure. But that statement is absurd.”
“I think [RCT advocates] thought they were going to solve a problem which is not solvable.”
“I think that right now the literature is claiming way too much for RCTs.”
Part of A Toolbox
“RCTs are fine, but they are just one of the techniques in the armory that one would use to try to discover things. Gold standard thinking is magical thinking, … a belief that this method can do something which it can’t, the replacement of statistics with magical thinking.”
“I’m not blinkered by the craziness that if it’s not a randomized control trial I shouldn’t pay any attention to it. Which I’ve heard more times than I can count.”
“[E]xtended case studies are enormously important. Most of the great ideas in the social sciences over the last 100 years came out of case studies like [the financial diaries]. Because people are open to hearing things they didn’t necessarily plan…”
“I think [the RCT movement] will likely fade in the same way it faded 30 years ago and for much of the same reasons…. RCTs will have more of a place in the economist’s toolbox than was true twenty years ago, and as with other methods, we will have a well thought out view of when and where they are useful. More tools are always welcome, as long as we don’t think one of them is a magic tool, or that it is the only tool we need.”
To which I simply say, “Amen.”
Have you read?
Reflections on the Latest Studies on Microcredit
Considerations for the Pursuit of Financial Inclusion – Beyond Account Access