Reasons to Disagree

We believe we have developed an approach to measuring and optimizing impacts that will cause the poor to realize greater benefits from efforts aimed at helping them.  We are committed to gathering evidence of the marginal impact our approach has for programs that adopt them.  Despite this, SMI wants to surface our understandings of the most powerful criticisms of our approach which have not been raised and addressed elsewhere on our site – and, in some cases, riposte.

We approach our entire venture with considerable and deserved humility.  This is a new field. While we are experienced and knowledgeable in our areas of strength, there is a lot to be learned in applications to this field.  We invite dissent and welcome such viewpoints any readers of this web site may wish to offer.  We commit to read them carefully and, if at all appropriate, include a summary of them in future versions of this web site in this space.  We seek to promote discussion and debate.  We are committed to changing our viewpoints as evidence and learning require.  We welcome your feedback anytime here.

Positions different from ours could include:

  • SROIs are an inadequate means of measuring most important elements of an inherently subjective process of bettering peoples’ lives.  Susan Stout, retired Manager of the World Bank’s Results Secretariat, captured this sentiment in the following cautionary remarks:

“There is incredible ‘silver bulletism’ around in the donor (and perhaps foundation) worlds—seeking that one special number that will tell us if we are succeeding or failing.  This is driven by bureaucratic fantasy, not reality.  The chances that we could come up with a metric that avoids an inevitably subjective process of judgment and choice are infinitely small (else politics would be a much simpler and boring topic).  It’s usually driven by a desire to define ‘a bottom line’ that will do for philanthropy and public sector management what profit/loss statements do for the private sector.  It’s just not going to happen that way.”

In a similar vein, some feel that there is not yet enough consensus in the policy or philanthropic communities for it to use SROIs heavily at present.

  • An inappropriate emphasis on SROIs could lead to ‘skimming the cream,’ taking on only easy charity cases and leaving hard cases behind, even if SROI studies were careful to account for and price such behaviors appropriately.  Also, many high SROI programs are successful because people who decided to improve their lives self-select into the programs.  This concern is a problem especially if SROI estimates do not account properly for the needs of recipients and what would happen to them absent intervention.
  • Even if, according to ‘revealed preference’ theory, donors implicitly quantify and value the charitable benefits they fund when they choose to fund one project and not another, analyzing benefits using these approaches could be misleading.  For example, the quality of analysis or types of benefits reported may not be comparable across projects donors are likely to consider for funding.
  • Some needs are so urgent and life-critical that taking time to do even crude SROIs before giving is inappropriate.  Can anyone say what saving one life in a disaster is worth?
  • There are many non-quantifiable results from giving that cannot be included in SROI analyses.  The value of being kind-hearted, of being obedient to a sense that it is right to give, and to friendships that result from giving are examples.
  • Valuations of benefits by recipients or larger society, even if reasonably approximated in each case, are not sufficient for project comparisons in different cultures, at different income levels, and in different fields of endeavor as diverse as higher education and AIDs prevention.  Even purchasing power parity devices used to compare values of money in different countries are not adequate.  One could make primary SROI comparisons only in each field of philanthropy for similar income levels in each culture and neglect SROIs in other areas.
  • Credibility for SROIs is so important and randomized controlled trials (RCTs) are sufficiently inexpensive that, if others were to adopt the approaches SMI champions, there is a risk that most donors could be misled significantly.  This is especially a concern if inexperienced or wrongly intentioned people are involved. 
  • In estimating expected SROIs, SMI transparently multiplies probabilities of program success times the values of results in the event programs are successful and invites donors and others to cross examine or change any of these probabilities.  Even with the significant discounting of expected values SMI does to account for risks, some donors may want to minimize risks still further and avoid this ‘expected value’ element of SROIs SMI uses (this criticism would not apply to SMI’s ex post measurements of benefits).
  • It is hard to estimate accurately the risks of project non-success and the share of result benefits properly credited to an evaluated charity SMI uses in computing SROIs.

We respect these positions and believe each has merit.  We intend to investigate many of them further and address them again in the future.  We recommend that donors consider them.  This is why we feel SROIs should inform donations along with other information.  We suggest that charities that do wish to report SROIs address issues such as limitations on the credibility of SROIs, risk mitigation efforts, the particular benefits and costs measured, and, where appropriate, the methods used to audit results directly in written statements.  However, if charities do so, we recommend that they also show that such disclosures are not a cover for management ineffectiveness that could show up in low SROI numbers.

SMI believes that its approach to developing both standard SROIs and customized variations on them may address the concerns expressed in the ‘silver bullet’ single criterion and other positions reflected above.  At a minimum, we believe our approach may help modify these positions or show where our position fits with them in a wise, balanced view.

We believe that issues and risks like the ones cited above ought to be disclosed to donors. We are committed to helping donors give knowledgeably.  We think it is possible to find satisfying, high-impact places to help others, even with risks, and to have reasonably credible, cost-effective estimates of actual impact.  We also hope to discover best ways to minimize or eliminate risks to donors, for example by providing results’ guarantees in certain cases.