Methodological and Policy Limitations of Quantifying the Saving of Lives: A Case Study of the Global Fund’s Approach.

Summary Points

·         A recent trend in global health has been a growing emphasis on assessing the effectiveness and impact of specific health interventions.

·         For example, it has been estimated that 8.7 million lives were saved between 2002 and mid-2012 by “Global Fund–supported programmes” (as distinct from The Global Fund alone) through antiretroviral therapy (ART); directly observed tuberculosis treatment, short course (DOTS); and distribution of insecticide-treated mosquito nets (ITNs).

·         This paper assesses the methods used by The Global Fund to quantify “lives saved,” highlights the uncertainty associated with the figures calculated, and suggests that the methods are likely to overestimate the number of “lives saved.”

·         The paper also discusses how the attribution of “lives saved” to specific programmes or actors might negatively affect the overall governance and management of health systems, and how a narrow focus on just ART, DOTS, and ITNs could neglect other interventions and reinforce vertical programmes.

·         Furthermore, the attribution of “lives saved” to Global Fund–supported programmes is potentially misleading, because such programmes include an unstated degree of financial support from recipient governments and other donors.


This paper argues that the number of “lives saved” that are attributed to Global Fund–supported programmes is not as certain as has been suggested by The Global Fund, and is likely to be an overestimate. Furthermore, estimating the “lives saved” by Global Fund–supported programmes is confusing and potentially misleading, because such programmes include a considerable but unstated amount of financial support from other sources. Finally, a number of potentially negative policy effects are associated with the selective impact estimation of downstream clinical interventions.

While this paper focuses on The Global Fund, the issues raised here apply to other global health partnerships and international donor agencies that are increasingly under pressure to quantify the health impact of their investments. The methods for estimating and attributing “lives saved,” and the consequences of doing so, should be questioned and subjected to critical debate.

In the case of The Global Fund, for a start, greater clarity and explanation about the assumptions and generalisations of the methods are required; this should include publication of uncertainty ranges and of disaggregated estimates of “lives saved” for each of the three interventions and for each year. The Global Fund should also conduct and publish sensitivity analyses, particularly in relation to treatment effectiveness, and publish estimates of “lives saved” through DOTS based on alternative counterfactual scenarios.

If the health impact of ART, DOTS, and ITNs is to be estimated in the form of “lives saved,” we argue that this should not be done as an exercise focused on individual external agencies, but rather on the collective contributions of governments and development partners within countries. This would confer a number of benefits. First, the monitoring of service delivery outputs and the estimation of their health impact would be linked to an assessment of the performance of national health systems (a more appropriate unit for assessment) and the degree to which development partners are working in harmonisation with each other and in alignment with ministries of health and their national plans and priorities. This would help shift more attention towards the strengthening of integrated national plans and information systems.

Second, holistic assessments of service delivery results and health improvement at the country level would allow for a context-based analysis of performance, including assessments of efficiency and equity. This would be aided by cross-country comparisons that would reveal variations in effectiveness (and efficiency) of ART, DOTS, and ITNs that arise from differences in, amongst other things, access to health care, quality of care and treatment adherence, and population coverage of nonclinical determinants of health such as access to clean water and nutrition. By describing this variation, policy attention can be directed not just at the delivery of selected clinical interventions, but also at the social, economic, and environmental conditions that influence the degree to which those interventions are effective. This stands in marked contrast to a modelling approach that assumes standardised levels of effectiveness across countries or regions.

Third, estimates of “lives saved” at the country level might be more valid and less uncertain because they would be derived from more appropriate and country-specific modelling assumptions, and because it would motivate countries to improve the quality of their data. In addition, it could stimulate other actors within countries, such as parliamentary health committees, universities, and local nongovernmental organizations, to develop the capacity to scrutinise the performance of the health system. While many countries produce annual health reports, health needs assessments, and national health plans, which provide some description of progress in the health sector, they are often incomplete or weak. Subnational analyses are frequently absent or superficial; and the fragmented and piecemeal nature of reporting systems, encouraged by vertical and donor-driven DAH, still undermines the development of coherent planning, budgeting, management, and information systems.

While an estimate of “lives saved” by ART, DOTS, and ITNs at country level would still be limited by its narrow focus on three interventions, it would provide a platform for monitoring and evaluating other aspects of HIV, TB, and malaria programmes and be more easily incorporated into a national system of data collection and evaluation that takes into account a wider package of health systems inputs, processes, and outputs, enabling policy makers and planners to consider the importance of investments that do not have a measurable or immediate mortality impact.

If individual external agencies need to estimate their specific contribution to “lives saved,” this could be done more simply by apportioning a share of a country’s estimated number of lives saved on the basis of their proportional financial contribution to THE or total HIV/AIDS, TB, and malaria programme financing. This would provide a more meaningful assessment of the contribution of individual agencies, avoid double-counting in reported estimates of “lives saved” by external agencies, and incentivise external agencies to promote coherent national health planning and reporting.

Many of these recommendations  are applicable to external agencies in general. However, since 2012, The Global Fund has been providing more active support for detailed national evaluations of programme performance and impact, and more accurate measures of disease incidence, prevalence, mortality, and morbidity in 20 to 25 “high-impact” countries. This provides it with an opportunity to shift emphasis away from estimating “lives saved” by individual interventions and donor-supported programmes, towards an assessment of health systems performance and impact that incorporates all major actors, programmes, and interventions, and a fuller assessment of the contribution of social, economic, and other upstream determinants of health.


Saving Lives in Health: Global Estimates and Country Measurement.

One of the most compelling reasons for development aid to health is that it saves lives, often for a few hundred dollars per year of life saved. Relatively uniquely in development, health has a set of high-impact interventions that can save lives directly. Insecticide-treated bednets (ITNs) protect families from malaria, antiretrovirals (ARVs) reduce mortality from HIV, and tuberculosis detection and treatment reduce TB mortality. Prevention activities, particularly for HIV, can save millions more lives. Yet, health programs have not always communicated with simple methods the lives they save.

In this week’s PLOS Medicine David McCoy and colleagues discuss the “lives saved” model of The Global Fund to Fight AIDS, Tuberculosis and Malaria (The Global Fund). The Global Fund, together with WHO, UNAIDS, and scientists from the article by McCoy and colleagues [1],[2], have published simple peer-reviewed methods to calculate the lives saved from a restricted set of HIV, TB, and malaria interventions that have known mortality outcomes [3][7]. Our method includes only those health interventions with known, documented mortality effects: ARV treatment; directly observed treatment, short-course (DOTS); and ITNs. Our methodology uses documented data reported to the Global Fund on the individuals receiving these services. These results are first verified by national disease programs (we invest 5%–10% of our funds to build the capacity of country monitoring and evaluation systems), then by the Global Fund (which uses independent local fund agents to check the national data systems measuring these services every six months), and finally by on-site checks in a sample of health facilities to verify that people receive these services (as part of performance-based funding) [8].

In addition, the Global Fund’s method applies the agreed, partner mortality estimates and models from WHO and UNAIDS [4] to these service results—for example, the latest scientific data on how HIV treatment or TB treatment will reduce the chance that a person will die of HIV or TB.

Extensive criteria are used to exclude countries where The Global Fund is not a significant contributor; that is, where The Global Fund does not contribute at least US$50 million; is a significant percentage of HIV, TB, and malaria spending; and does not support a key national-level activity, such as drug procurement. Where this does not occur, as has been the case in Uganda, Kenya, or South Africa in recent years, the results are not included.

The method to assess lives saved provides a conservative estimate. The estimate [3],[4] does not include the impact of HIV prevention (which in certain countries—e.g., Thailand, Uganda, Kenya, and Zimbabwe—has saved several million lives per country); the impact of malaria outside Africa and among adults; and the significant, secondary impact of DOTS treatment on reducing TB (as shown by the declines in TB prevalence in China, and in TB prevalence by 45% in Cambodia). Furthermore, reporting of services by programs in country are subject to substantial delays before they are reported globally. The most recent scale up in ITNs and ARV treatment are not fully included; for example, the lives saved are only half the number of people reported on ARVs. We do acknowledge the method [3],[4] has major limitations. Most importantly, it does not directly measure mortality, because in many countries in which we work vital registration systems are too weak, so the method is based on the latest partner estimates of mortality from WHO and UNAIDS.

The article in this week’s PLOS Medicine by David McCoy and colleagues has great value in discussing the assumptions in the methods the Global Fund uses to assess lives saved and the partner estimates—of ARV adherence, use of ITNs, and the limitations of focusing only on a limited set of services. We agree that assumptions require additional sensitivity analysis, and we will update our estimates in 2014 as modeling is refined with new and improved data from country impact evaluations and updated WHO and UNAIDS estimates. We have published more detailed analysis of the ARV, ITN, and DOTS estimates as used by the McCoy and colleagues[4]. Yet, the uncertainty ranges, with the lives saved from ITNs as low as 27,000, were based on very limited data and provided little additional value. We fully agree with the need for increased country data on estimates and mortality assumptions of lives saved. Most importantly, global modeling needs strengthening with wider and deeper country measurement of epidemic trends and lives saved.

To significantly strengthen our assessment of lives saved, The Global Fund approved a new evaluation plan in 2012, which includes country health, HIV, TB, and malaria reviews to more directly measure mortality and impact on HIV, TB, and malaria trends in 25 countries, where 65% of the global burden of these diseases occurs [9]. To strengthen these direct country data and to put global commitments made with WHO and other partners into practice, we are also investing in five components of data systems: surveys, health information systems, vital registration, financial tracking, and country analytical capacity [10]. These investments in country data and analysis will form a basis to implement some of the recommendations to global modeling provided by McCoy and colleagues [1]. Our initial country reviews provide direct evidence of impact. For example, in Cambodia, malaria deaths have been reduced by over 80%, child mortality has declined in Tanzania, TB has declined in China, HIV prevalence has declined in Zimbabwe (though not in other countries, such as Uganda in recent years) [7],[11]. Some of these declines are not captured by global estimates of lives saved, and suggest these estimates may be conservative and require updating as country-level analysis of mortality is available.

The need for improved country data does highlight some of the weaknesses we see in the paper by David McCoy and colleagues [1]. Their analysis provides very little additional country data on uncertainties, or on the significant changes in child mortality and mortality among adults of working ages, associated with the impact of HIV, TB, and malaria on Millennium Development Goals (MDGs) 4 and 6. It draws upon one meta-analysis of ARVs suggesting 62% retention at 24 months, but our recent country reviews suggest most have moved between districts rather than have died, and mortality rates were often less than 10% [11][13]. Removing a “counterfactual” of TB deaths is unclear and difficult to explain why TB deaths in 1995 might be reduced from deaths reported now, unless we are evaluating a global TB strategy. The aim of The Global Fund’s lives saved figure is not to evaluate a particular strategy, as McCoy and colleagues suggest, nor to “attribute” lives saved to the agency, but to highlight the lives saved each year from services delivered by the programs we support. The adjustment of ITNs based on data on use is important—WHO reports this as over 90% from surveys [14], and we will further adjust our estimates of lives saved as partner estimates from WHO and UNAIDS on the key parameters are updated. However, we believe McCoy and colleagues overstate the figures on ARV retention, ITN use, and issues of removing TB deaths from the 1990s.

The Global Fund is investing in improved country financial data, including funding national health accounts to improve data on spending on health, HIV, TB, and malaria by different partners in 46 countries. At present, it would not be accurate to use the reported share of total health or disease expenditure to attribute lives saved to an agency. Our communications on lives saved [3],[4] are clear that the estimates aim to show the lives saved of the programs we support together with other partners, civil society, and country HIV, TB, and malaria programs themselves. As described above and in our publications [1],[3], we use extensive criteria to exclude countries where The Global Fund does not provide significant financial and programmatic support. We stress that we play an important financing role, but the results are first and foremost those of country HIV, TB, and malaria programs. We will communicate this more clearly going forward and as we refine our methods with improved country data.

Finally, we understand the argument on vertical programs by McCoy and colleagues that reporting on individual HIV, TB, and malaria services can distort health priorities. The Global Fund is clear that it encourages countries to align funding with their health and disease strategies, and uses indicators of individual HIV, TB, and malaria services to measure progress and performance. However, we do think clear targets on HIV, TB, and malaria are important, as shown by the MDGs.

Lives saved is an important measure for health programs. We have based our estimates on real, individual verified data on a limited set of services, which have clear, documented mortality outcomes. We welcome the paper by McCoy and colleagues in this week’s PLOS Medicine, as it discusses more fully the assumptions, explores the potential pitfalls in communication, and stresses the importance of investments in country financial and impact data. Our new evaluation plan fully supports this country investment in country data and analysis. We will update our global estimates with country data from our impact studies in 25 countries where 65% of the global burden of HIV, TB, and malaria occurs. Through these investments in global estimates and country measurement, we are confident the programs The Global Fund supports will save more than the 8.7 million lives estimated so far.


Malaria programme gets kiss of death from Global Fund.

It appears that the Affordable Medicines Facility — Malaria (AMFm) is being scuttled by the Global Fund to Fight AIDS, Tuberculosis and Malaria. The AMFm is a multimillion-dollar programme to get effective drugs — artemisinin-based combination therapies (ACTs) — to remote rural villages, where local stores are often the main providers of medicines.

That’s my first take on the Global Fund’s announcement this afternoon, delivered towards the end of a two-day meeting of its board, which began yesterday in Geneva, Switzerland. The fund intends to end AMFm’s stand-alone status, and instead “integrate” the AMFm into its existing core system of providing grants to countries to purchase drugs, bed nets and other malaria-control measures.

Meanwhile, just in: the Fund announced a few minutes ago that it has nominated as its new executive director Mark Dybul, a former head of the President’s Emergency Program for AIDS Relief (PEPFAR), who now co-directs the Global Health Law Program at the O’Neill Institute for National and Global Health Law at Georgetown University in Washington DC. It follows the resignation of Michel Kazatchkine in February (see ‘Global health hits crisis point‘).  More on that later.

The idea behind the AMFm was to make ACTs the frontline treatment, something which has been badly hindered by their much higher price than older antimalarials. Although these older drugs, such as chloroquine, are cheap, they are often ineffective because the parasite has developed resistance to them.

AMFm’s strategy was to first guarantee a market of large bulk orders, allowing it to negotiate large cuts in the price of the drugs from pharmaceutical companies. It then gave importers in affected countries — both in the public and private sector — a subsidy on their purchases, to bring the final price to the consumer down to a level competitive with older drugs. It was also intended to compete with artemisinin monotherapies, which are a recipe for generating drug resistance against the only drug class that is truly effective against malaria.

A full-blown country-level trial of the AMFm concept in eight countries — Cambodia, Ghana, Kenya, Madagascar, Niger, Nigeria, Tanzania (including Zanzibar) and Uganda — began in July 2010 and runs until December 2012. It has had its critics — see ‘Malaria plan under scrutiny’ — but in anyone’s reasonable terms it has already been remarkably successful in many important ways within a very short period ( see the 31 October Lancet paper ‘Effect of the Affordable Medicines Facility—malaria (AMFm) on the availability, price, and market share of quality-assured artemisinin-based combination therapies in seven countries: a before-and-after analysis of outlet survey data’).

The Global Fund’s press release detailing its plans is entitled ‘Board approves integration of AMFm into core Global Fund grant processes‘, and much of its soothingly reassuring content would perhaps have many thinking that the AMFm’s integration into the Global Fund’s core grants system is good news. But it is in effect being killed. There’s will be no new money ringfenced for the AMFm once it runs through its current funding up to the end of 2013, which means that any countries wanting to set aside cash for the private sector will be required to take this from their country grants from the Global Fund. In reality, that will probably translate into AMFm activities simply being terminated in most countries, leading to local price rises in ACTs, and the drugs disappearing off the shelves of local pharmacies. AMFm’s clout in negotiating bulk pricing deals internationally will also probably be weakened.

Here’s what a group of AMFm’s supporters wrote in the Lancet on 31 October about what ‘integration’ of the AMFm into the Global Fund would mean (from’The Affordable Medicines Facility—malaria: killing it slowly‘):

“In November, 2012, the Board of the Global Fund will vote to either continue AMFm in a modified form after December, 2013, or terminate the programme. There is a strong push from donors (though not from countries) to integrate AMFm into the regular Global Fund model, whereby countries would choose how much of their country budget envelopes, which are already committed to other priorities supporting the public sector, to reallocate to AMFm. We believe that this approach will create instability in artemisinin demand, lower the number of ACT manufacturers, increase ACT prices, and abandon the millions who depend on AMFm-subsidised ACTs. Most importantly, it will kill a programme that, when fully implemented, rapidly met its benchmarks despite the many constraints, expectations, and unrealistic timelines imposed on it. We must acknowledge that an efficient approach to subsidising antimalarial drugs has worked, making them available in the private sector where people go to buy them.”

The Global Fund’s board has decided AMFm’s fate, and there is probably no going back on that. But the world now urgently needs a strategy that works with the private sector to keep down ACT prices, particularly in Africa.

Souce: Nature blog