6 Interpreting and comparing cost-effectiveness estimates
6.1 Limitations
When using the modeled cost, effectiveness, or cost-effectiveness results generated using the MAPS tool cost-effectiveness module, it is important to keep the limitations of (and caveats associated with using) the underlying data sources in mind.
6.1.1 HCES data
Using household-level food consumption data to estimate the adequacy of the household diet without and with LSFF has several limitations (Adams, Vosti, Mbuya et al., 2022). Because the food consumption data are collected at the household level, it is not possible to generate estimates of individual-level food consumption and micronutrient intake without imposing assumptions about the intrahousehold distribution of food. In the MAPS tool, we assess apparent dietary micronutrient adequacy using the adult female equivalent method and assume that if the diet is adequate to meet the requirements of an adult female, it is likely adequate to meet the requirements of all household members. However, this does not mean that all household members are receiving an equitable share of the household’s total food consumption, so the micronutrient requirements of some household members may not be met even if the household diet is adequate to meet the requirements of a non-pregnant, non-lactating adult female.
Another limitation is that household food consumption data are typically collected using a closed food list that may include aggregate food items (e.g., fresh fish), which can error in the estimated micronutrient content of the food, and/or may be missing key foods needed to accurately characterize diets, capture all important sources of nutrients, and accurately estimate the potential for LSFF, biofortification, or agronomic biofortification to improve the micronutrient adequacy of diets. Related, foods consumed away from home are often inadequately captured in HCESs or are not captured at all. Estimates of household food consumption are typically based on the recall of one (or several) household members, so underreporting of foods consumed by individuals, particularly outside the home, is possible. Because foods consumed away from home are, in many low- and middle-income countries, an increasingly important source of nutrients, the inadequate accounting of foods consumed outside the home could lead to an underestimation of total nutrient intake and overestimation of the prevalence of inadequate apparent intake. If foods consumed away from home are fortifiable or biofortifiable foods, this will also lead to an underestimate of the impact of these interventions on the micronutrient adequacy of diets. Finally, because HCESs do no collect data on consumption of micronutrient supplements, our estimates of micronutrient adequacy do not account for supplement use.
6.1.2 Cost data
The default cost parameters used in the MAPS cost models are, to the extent possible, based on the best available data and information sources. However, there is undoubtedly some level of error in these parameters, as, for example, it was not possible to interview all wheat flour refineries to collect data on the costs associated with fortification at their facilities, so many of our estimates are based on interviews with one or two refineries and extrapolated to the entire industry. Likewise, information provided from a few government personnel about the activities and costs associated with, e.g., regulatory monitoring of LSFF programs were assumed to reflect the situation nationally. It is also possible that some of the activities included in our cost models may not be relevant, or some relevant activities may not be included (this may be especially true for ex-ante cost models that estimate the cost of hypothetical micronutrient intervention programs, making locally-specific data collection particularly challenging). As such, it is critical for users to carefully scrutinize each cost parameter for accuracy in their local context and to make changes where needed. Conducting sensitivity analyses around particularly uncertain parameters (described below) is also very important.
6.2 Sensitivity analysis
Assessing the influence of uncertainty is an important part of conducting a comprehensive cost-effectiveness analysis. When estimating the cost of a micronutrient interventions, there is typically some degree of uncertainty around most parameter values. However, some sources of uncertainty will typically be much more important than other sources. Key sources may include parameter values that have a large impact on the total cost estimate, for example uncertainty about the cost of micronutrient premix for LSFF or the incremental cost of a biofortified seed variety compared to a traditional variety. Or, if the cost estimates will be used to help allocate government resources, paying special attention to uncertainty in M&E costs would be important. There can also be uncertainly in estimated program impacts, and it is important to conduct sensitivity analyses around assumptions or uncertain parameter values that influence the impact, or effectiveness, of the intervention. In the context of estimating effective coverage in MAPS, this could include uncertainty around adherence with LSFF standards or framer adoption rates of biofortified crop production.
Ultimately, we hope to add functionality to the cost and effectiveness module of the tool to allow users to define and automatically conduct sensitivity analyses around the cost, effectiveness, and/or cost-effectiveness of each intervention they define. In the meantime, we urge all users to manually conduct sensitivity analyses after estimating the cost, effectiveness, or cost-effectiveness of an intervention by copying the primary intervention and adjusting key parameters up and/or down to reflect uncertainty, and re-estimating cost, effectiveness, and/or cost-effectiveness.
For cost estimates, this could begin by first identify key program cost drivers. Once these parameters are identified, data can be collected (e.g., time series data, if available) or assumptions made to establish ranges of uncertainty (e.g., plus or minus 25%) regarding the amounts of specific inputs (e.g., person-days), input prices/values (e.g., micronutrient fortificant prices, seed prices, wage rates, etc.), or other key sources of uncertainty. Then, in the copy of the primary cost model, these parameters can be ‘shocked’ by the estimated or assumed uncertainty, and the change in estimated costs noted and reported alongside the primary cost estimate.
For effectiveness estimates, it is important to identify key sources of uncertainty for a specific intervention scenario. For LSFF this could include the percent of a food vehicle that is fortifiable, the expected micronutrient loss from point of fortification to households, or expected adherence with fortification standards. For biofortification via crop breeding, key sources of uncertainty could include the expected micronutrient contents of the biofortified variety compared to the traditional variety. For agronomic biofortification, an important parameter to include in sensitivity analysis might be expected increase in the mineral contents of the edible portion of the crop as a result of mineral-enhanced fertilizer application. And for both types of biofortification, modeling uncertainty in farmer adoption rates may be important.
When there is more than one source of uncertainty or you are aiming to simultaneously assess the impact of uncertainty in both costs and impacts of an intervention, uncertainly can be group into best-case and worst-case scenarios to identify ranges of possible costs, effectiveness, and cost-effectiveness for a specific intervention.
Interpreting and using the results of sensitivity analyses can be challenging, but doing so can lend credibility to the results. For example, if the results of best-case/worst-case sensitivity analyses suggest that Program A will always be more cost-effective than Program B, this can lend confidence to choosing Program A. On the other hand, if a given program under- or out-performs another depending on sensitivity analyses, this may leave decision-makers with doubts regarding which program to choose. Ultimately, it is up to the users of the evidence generated in the MAPS tool cost and effectiveness module to decide how much uncertainty they are comfortable with, but it is important to be transparent about uncertainty and integrate it into your economic evaluations using sensitivity analysis.