Methods to assess farming sustainability in developing countries. A review

Sustainable agricultural development is linked to the improvement of food security and poverty alleviation, especially in developing countries where 98 % of the chronically hungry population lives (WSSD 2002; FAO 2013). Sustainable agriculture is socially just, ecologically sound, economically viable, and a paradigm that aims to produce the food needed to achieve food security (IAASTD 2009; Cavatassi 2010; FAO 2013). In developing countries, mainly smallholder farmers supply local and national markets by providing 80 % of the food (IAASTD 2009; Dethier and Effenberger 2012; IFAD and UNEP 2013). In Africa, 90 % of agricultural production is derived from smallholder farmers, where the average farm size is about one hectare (IAASTD 2009; IFAD and UNEP 2013). It is a well-known contradictory fact that farmers who supply the main part of agricultural produce are the poorest and most hungry population group within developing countries (IAASTD 2009; Dethier and Effenberger 2012; IFAD and UNEP 2013). Several development initiatives focus on enhancing agricultural production and productivity of smallholder farmers in developing countries. Because their livelihoods are fragile, the impact of development initiatives needs to be assessed before implementation, with the primary goal of increasing the probability that these initiatives will improve the livelihoods of impoverished people in the project regions, and to avoid negative impacts (EIARD 2003). This can be done by applying impact assessment in order to adapt development interventions to the local context and to steer towards sustainable development (Pope et al. 2013). There is a great emphasis on the suitability and sustainability of project interventions, and assessing this requires appropriate methods. Theory and practice of impact assessment are expanding rapidly. There is a large number of impact assessment methods related to different types: environmental, social, economic […]; levels: local, national, regional […]; targets: policies, programs, projects […]; and timing: ex ante, during and ex post. The new approach to impact assessment is sustainability impact assessment, which places emphasis on delivering positive net sustainability gains now and in the future (Bond et al. 2012).

Today, we see a wide diversity of practices and methodology in the field of sustainability impact assessment (Sadok et al. 2008; Pope et al. 2013). There has been a lot more focus on ex post than on ex ante impact assessment at the project level (Silvestrini 2011). Presently, there are no internationally established standards regarding the methodological approaches for carrying out ex ante evaluations at the project level (Silvestrini 2011; GIZ 2013). Pope et al. (2013) highlight that “[…] the lack of consistent and agreed methodology is perceived as a strength allowing for flexibility and context-specific approaches.” The number of methods and tools available is continually increasing (Sadok et al. 2008). We find several approaches even when narrowing the field of interest down to ex ante impact assessment for the sustainability of agricultural development interventions.

The objective of this paper is to analyze and compare existing ex ante sustainability impact assessment methods. We verify whether the requirements of sustainability impact assessment, as given in the recent theoretical discourse, are represented in sustainability impact assessment approaches applied. We analyzed ten methods currently applied within the scope of sustainable agricultural development, with a focus on crop farming in developing countries. Although we are aware that analyzing only ten methods does not address all existing approaches, it will nevertheless demonstrate the variety of methods that are currently applied in the context of sustainability impact assessment. We first present the current state of the theory of sustainability impact assessment, followed by an analysis of the methodological approaches. Then we critically discuss the methods with respect to the theoretical requirements of sustainability impact assessment. Finally, we outline the utility and the informative value of the different approaches in regards to their potential of achieving sustainable development.

To identify and select sustainability impact assessment methods applied in the context of developing countries, we searched in scientific databases, such as the ISI Web of Science and Scopus, using the following keywords: (ex ante) impact assessment, evaluation, sustainability, agriculture, (small-scale) farming, strategic project planning, food security, decision support, and project assessment. In addition, we searched on the websites of national and international institutions for research and implementation to find methodological approaches for the ex ante impact assessment of development projects. We limited our focus to the methodological approaches that have been applied in the context of agricultural development initiatives in developing countries. To present the current state of practice, the case studies in developing countries date from 2000 until 2013. Methods that have been widely used for project assessment but which only focus on one particular dimension of the sustainability paradigm (for instance only on economic aspects) were excluded from this review.

We analyzed the ten methods with regard to the following characteristics:

• General application characteristics including objectives of the method, thematic agricultural sector of application, moment of application (ex ante, during, ex post), time needed for application, spatial analysis scale, data input and analysis (qualitative, quantitative), applying user, end user of results, and timely horizon of assessed impacts.

• Representation of sustainability dimensions including number of sustainability pillars, number of indicators per dimension, indicators predefined or open to be defined, interrelation of dimensions, context-relatedness of indicators.

• Level of stakeholder involvement including representation of involved stakeholder groups, moment of stakeholder involvement during the assessment, level of interactive stakeholder involvement, integration of collective learning process, stakeholder’s influence on indicators, assessment, and decision-making.

2.1 Definition

Sustainability impact assessment is a process that supports decision-making towards sustainable development in a given context. The International Association for Impact Assessment (IAIA) defines impact assessment in general as a process that identifies future consequences of an intended action. The terms “impact” and “effect” are frequently considered as synonyms (IAIA 2009). In the context of project planning, “ex ante” impact assessment is used in order to predict environmental, social, and economic consequences of an intervention in order to approve, adapt, or reject a development project proposal. The use of impact assessment of projects dates back to the 1950s (Maredia 2009). According to the IAIA (2009), impact assessment procedures generally aim at providing information for decision-making, the participation of the public, and the contribution to sustainable development as primary goals.

Sustainability impact assessment is the only type of impact assessment that considers and integrates the three pillars of sustainable development equally. It is simply defined as the process that steers decision-making towards sustainability, as well as ensuring net sustainability gains in the present and the future (Ness et al. 2007; Hacking and Guthrie 2008; Bond and Morrison-Saunders 2011; Bond et al. 2012; Singh et al. 2012). Sustainability impact assessment does not only consider environmental, social, and economic implications but also analyzes the interrelations between the three pillars (Pope et al. 2004). It is becoming common as a decision-making procedure whose goal is to anticipate the sustainability of policies, plans, projects, programs, or a current practice or activity (Pope et al. 2004; Hacking and Guthrie 2008; Bond and Morrison-Saunders 2011; Bond et al. 2012). Gibson (2013) lists six imperative criteria of sustainability impact assessment:

• The development initiative must have a positive contribution to a sustainable future.

• All key factors that affect a sustainable future as well as their interlinkages are considered.

• The interdependence of ecology, economy, and society is respected.

• Trade-offs are minimized.

• Context is respected, and effective criteria in regard to the people and the context concerned are used.

• Participation at all levels from government to society is emphasized, and simplistic technical procedures are avoided.

In relevant literature, the potential and limitations of quantitative versus qualitative research have been discussed, as have reductionist versus holistic approaches for sustainability impact assessment (Ness et al. 2007; Gasparatos et al. 2008; Bond et al. 2012; Bond and Morrison-Saunders 2013b).

2.2 Reductionism vs. complexity

Reductionism in sustainability impact assessment means that only a few indicators are used to assess the sustainability of a whole system. They simplify, quantify, analyze, and therefore facilitate the communication of complex and complicated information (Singh et al. 2012). Bebbington et al. (2007) and Singh et al. (2012) argue that there is a widely recognized need for individuals, organizations, and societies to find models, metrics, and tools in order to assess sustainability. The risk of quantitative and reductionist assessment is that it can lose sight of the complex and often characteristic picture of reality, as well as what is important at the local level (Cosyns et al. 2013). Analytical tools and techniques that only use quantitative approaches, often reductionist, are less important in sustainability impact assessment, and decision-making has to follow more qualitative analysis (Cashmore 2004; Bond and Morrison-Saunders 2011; Bond et al. 2012). Bell and Morse (2008) as well as Bond and Morrison-Saunders (2013b) emphasize that pluralism is central to sustainability impact assessment in order to get a broader understanding of the given context. Sustainability impact assessment gives direct, effective, and efficient attention to social, economic, and ecological factors and their interaction, and it therefore supports the development of site-adapted solutions. But pluralism may also be considered as a weakness due to its reliance on (often costly and time demanding) stakeholder involvement strategies in order to integrate the diversity of a context. A reductionist approach often simplifies a system in an effort to have clearly defined components for analysis, while the holistic approach often comprises more qualitative aspects to reflect the complexity of a system. The broad scope of the assessment allows a more holistic picture, which should reduce the need for trade-off decisions (Bond et al. 2012). Therefore, sustainability impact assessment should provide sufficient space and time with stakeholder groups to understand the local context and to elaborate indicators, which fully represent the analyzed system. The number and nature of indicators are not restricted or predefined, but need to be elaborated in an interactive process. Sustainability impact assessment allows complexity and enables the collection of primary data useful for a subsequent qualitative data analysis.

2.3 Stakeholder involvement

Spohn (2004 cited Singh et al. (2012)) distinguishes between a “top-down” and a “bottom-up” approach in the assessment procedure. The top-down approach “[…] enables experts and researchers to define the overall structure for achieving the sustainability and subsequently it is broken down into set of indicators” while a “bottom-up approach requires systematic participation of various stakeholders to understand the framework as well as the key sustainable development indicators.” Different authors emphasize the importance of “embedded learning” due to stakeholder involvement in the practice of sustainability impact assessment (Gibson 2006; Bond et al. 2012; Bond and Morrison-Saunders 2013a). Sustainability cannot be simply measured by categories and indicators that are invented on the drawing board. The impact assessment process is therefore highly context-related and needs active stakeholder involvement (Bond et al. 2012). As context changes with time, so too does sustainability. Adaptation and flexibility, willingness to learn, and changing perspectives are therefore essential requirements within the sustainability impact assessment process. The view of all affected parties needs to be integrated (Gibson et al. 2005; Gibson 2006; Bond et al. 2012).

Thus, it is recommended that stakeholder involvement be considered from the planning phase through to the final evaluation, which is so critical to sustainable development, and thereby move away from simple technocratic decision-making towards a more dialogic approach (EIARD 2003; Bond and Morrison-Saunders 2011; Kiara 2011; Morgan 2012). Bebbington et al. (2007), Maredia (2009) and Singh et al. (2012) mention that indicators of sustainable development should be selected, revisited, and refined upon based on the appropriate communities of interest. Stakeholder involvement is not only important in order to identify different perspectives, objectives, and values but also to align those different views and to reduce the chance that a conflict will arise. Participation adds to the understanding of the project itself, and thereby helps with the acceptance of decisions and ownership. The process ensures that results are locally adapted and relevant and, therefore, contribute to the overall sustainability (Cashmore et al. 2009; Cosyns et al. 2013; Stoeglehner and Neugebauer 2013). This also requires a learning attitude to be shown by all the stakeholders involved. Stoeglehner and Neugebauer (2013) argue that stakeholder implication should even be added as “the fourth pillar” of sustainable development. Stakeholder involvement may have different levels of intensity. Considering stakeholder involvement in the context of sustainability impact assessment means to actively involve stakeholders at all stages of a development initiative. Simply informing and consulting them is insufficient (Stoeglehner and Neugebauer 2013).

In the following section, we briefly describe the objective and the methodological procedure of ten approaches used in sustainability impact assessment. We present them regarding their objectives, context of application (thematic agricultural sector of application, moment of application [ex ante, during, ex post], time needed for application, data input, spatial analysis scale, analysis [qualitative, quantitative], applying user, end user of results), indicators, stakeholder involvement, and case study examples. The results are presented in Table 1.

Ex ante poverty impact assessment (PIA)

The PIA guideline developed by the OECD (2007) intends to provide a framework that integrates the already existing approaches of different donors, their procedures, and terminologies. PIA is a process that examines the intended and unintended consequences of projects, programs, sector interventions, and policy reforms and focuses on impoverished and vulnerable people. The OECD considers five poverty dimensions: economic, human, political, socio-cultural, and protective security, whereby gender equity and environmental sustainability cut across all dimensions of poverty.

PIA is based on balancing existing quantitative and qualitative information to achieve a sound and reliable assessment. It is an iterative process involving decision-makers and stakeholders from both donor and partner countries. PIA consists of five steps:

1. 1.

   Outline the poverty situation and the relevance of the intervention.

2. 2.

   Identify stakeholders and institutions.

3. 3.

   Summarize the process by which the interventions are expected to influence the target group (transmission channels: prices, employment, transfers, access, assets, authority, and productivity).

4. 4.

   Outline the likely results on the stakeholder groups and their ability to escape from or to avoid poverty: economic-, human-, political participation-, security capabilities (to lessen vulnerability).

5. 5.

   Present the impacts of the intervention on the Millennium Development Goals regarding extreme poverty and hunger, primary education, gender equality/empowerment of women, child mortality, maternal health, HIV/AIDS, malaria/other diseases, environmental sustainability, and global partnership.

Impacts of the intervention are assessed in terms of their “transmission channels,” the enhancement of “stakeholders capabilities,” as well as the improvements related to the Millennium Development Goals. The PIA results are summarized in simple matrices.

Framework for participatory impact assessment (FoPIA)

Originally, FoPIA was developed for land-use policy impact assessment in Europe. In this context, the framework was described by Morris et al. (2011). At the same time, the FoPIA framework was adapted by König et al. (2010) and further developed to be applicable in the developing context (König et al. 2012, 2013, 2015; Purushothaman et al. 2012). FoPIA is structured around the DPSIR framework (driver-pressure-state-impact-response) (OECD 1993). It considers the relationships between environmental, economic, and social issues, as well as national and regional sustainability priorities. By exploring alternative scenarios, the FoPIA aims to inform stakeholders about possible sustainability trade-offs and compromises and possible win-win situations. During the participatory and iterative process, alternative land-use scenarios are elaborated, and assessment criteria and indicators are developed and evaluated. Finally, the plausibility and the acceptability of the impacts are analyzed, and recommendations for improved decision-making then formulated. The procedural steps of FoPIA are the following:

1. 1.

   Nationally and regionally relevant scenarios are identified and analyzed together with stakeholders.

2. 2.

   Criteria based on land-use functions that present the key social, economic, and environmental functions of land are elaborated. Stakeholders rank the perceived importance of each criterion for the sustainability of the region (Fig. 1).

   

   Ex ante sustainability impact assessment workshop: example of a workshop applying the framework for participatory impact assessment (FoPIA). The participants elaborate in a participatory process the sustainability criteria relevant in their locality

   Full size image

3. 3.

   Assessment indicators are assigned to the land-use function criteria (expert-based).

4. 4.

   Impacts on each scenario (ex ante) are assessed using indicators ranked by stakeholders, while the definition of the time horizon remains flexible.

5. 5.

   Land-use function criteria are re-evaluated and scored with the knowledge of trade-offs.

6. 6.

   Reflection on the final results with stakeholders.

Participatory impact pathways analysis (PIPA)

PIPA (Douthwaite et al. 2007a; Alvarez et al. 2010) is a methodological approach that can be applied at different stages of a project cycle: planning (ex ante), monitoring, and ex post evaluation. With PIPA, an outcome and impact logic model is developed that describes how the project’s outcomes will be scaled out (horizontal spread of project outputs, i.e., farmer to farmer) and scaled up (vertical institutional expansion) to achieve environmental, social, and economic impacts. Stakeholders are involved throughout the whole process developing all the results. According to Alvarez et al. (2010), stakeholders in PIPA are defined as next users, end users, politically important actors, and project implementers. The procedural steps of PIPA are the following:

1. 1.

   Stakeholders elaborate a problem or objective tree that links the problem to be addressed by the project to the socio-economic and environmental situation.

2. 2.

   Elaboration of network maps showing how actors, relationships, and interactions influence the general environment for the new knowledge or technology.

3. 3.

   The two aforementioned perspectives are integrated via an outcomes logic model that describes the project’s strategies, knowledge, and practice changes, as well as outputs and outcomes associated with realizing the project’s vision. A monitoring and evaluation scheme is developed on the basis of the outcomes logic model.

DESIRE-Decision Support Systems (DESIRE-DSS)

DESIRE-DSS is a three part participatory methodology for selecting sustainable land management options. The approach was elaborated in the context of the EU-DESIRE project (www.desire-project.eu) and applied in 16 case studies in 14 countries within the World Overview of Conservation Approaches and Technologies network (WOCAT: www.wocat.org). The methodology was presented by Schwilch et al. (2009). It is a practical, structured, and flexible methodology that can be applied in diverse contexts. The methodological procedure emphasizes a multi-stakeholder-learning process (land users, technicians, researchers, governmental, and non-governmental officials) and combines in its solutions local knowledge with global expertise. DESIRE-DSS provides a pragmatic approach that considers time and financial restrictions, as well as the limited availability of facilitators and experts that need to guide the process. The procedure consists of the following constituent parts (Schwilch et al. 2012a):

1. 1.

   Identify land degradation problems and existing and potential solutions: 3-day workshop with a series of exercises. Linking scientific and local knowledge makes it possible to derive a range of alternative measures and mitigation strategies.

2. 2.

   Evaluate and document the identified existing and potential prevention and mitigation strategies in the 2 to 3 months following the workshop by using questionnaires and a database system developed by WOCAT. Then appraise the ecological, economic, and socio-cultural advantages and disadvantages of the strategies identified.

3. 3.

   Participatory selection of potential options to test the implementation by weighing relevant criteria (e.g., technical requirements, costs and benefits of implementation, social acceptability, etc.) and ranking the presented strategies while taking into account the technical, bio-physical, socio-cultural, economic, and institutional dimensions.

MESMIS framework

MESMIS (Spanish acronym for Indicator-Based Sustainability Assessment Framework) was developed by the Interdisciplinary Group for Appropriate Rural Technology (GIRA) and presented by López-Ridaura et al. (2002). It is an iterative, holistic, and interdisciplinary framework for evaluating sustainability to improve the design and the implementation of development projects (Astier et al. 2012). The determination of sustainability criteria and indicators varies according to the approach followed by the evaluation team and is specific for each case study. The framework allows for the derivation, measurement, and monitoring of sustainability indicators and is often applied in peasant natural resource management systems. Sustainability is not measured per se, but assessed through the comparison of two or more systems. Sustainability is here defined by productivity, stability, reliability, resilience, adaptability, equity, and self-reliance (self-empowerment) (López-Ridaura et al. 2002). López-Ridaura et al. (2002) and Astier et al. (2012) present the MESMIS operational structure consisting of six steps forming a cyclical process:

1. 1.

   Define the evaluation objective, the context, and the system under analysis: Identify existing and alternative management systems (components, inputs, outputs, socio-economic characteristics) and their socio-environmental context; identify spatial and time span of evaluation.

2. 2.

   Determine the system’s critical features in relation to sustainability attributes: productivity, stability, reliability, equity, adaptability, and self-reliance.

3. 3.

   Select diagnostic criteria and, based on these strategic indicators, address the seven sustainability attributes and the social, economic, and environmental dimension.

4. 4.

   Measure indicators and monitor: Measurement techniques are flexible and vary from literature review, direct measurement, or simulation model use to participatory group techniques in the field. The dynamic consequences of management, thresholds, and trade-offs are elaborated.

5. 5.

   Integrate the results by means of multi-criteria graphic tools: Quantitative, qualitative, and graphical as well as mixed procedures can be used to integrate results. Indicators are presented in a AMOEBA diagram (Ten Brink et al. 1991) to demonstrate the present and the alternative system at the same time and compare the indicator features.

6. 6.

   Offer conclusions and recommendations that reflect on how the different systems compare in terms of sustainability, main limitations, and possibilities. A selection of scenarios and a translation into adaptive or corrective actions to improve the natural resource management system is then carried out.

ScalA

ScalA is a tool that aims to systematically evaluate, communicate, and disseminate successful agricultural practices at a community level. ScalA was developed within the framework of two research projects (www.sustainet.org). The tool was then adapted and applied in further research projects (www.reacctanzania.com, www.better-is.com). The present state of the tool described here is the version of the ReACCT project and deals with the scaling up of good agricultural practices (Crewett et al. 2011). It is specifically designed for the evaluation of an enhanced crop production system prior to its implementation. It compares specific requirements of a crop-production system and the specific conditions that are relevant for the production in a certain locality. ScalA argues that an intervention is sustainable if it enhances at least one of the three sustainability dimensions (environmental, economic, and social) without the deterioration of another (Crewett et al. 2011). The potential of scaling up is defined by 61 success indicators. Those indicators are linked to preconditions for the project’s successful implementation, namely the financial, human, institutional, and infrastructural preconditions. ScalA consists of eight steps:

1. 1.

   Step 1: Sustainability assessment regarding 5–7 indicators for each sustainability dimension (environmental, economic, social).

2. 2.

   Steps 2–5: Climate change responsiveness assessment (project contribution to adaptive capacity, resilience to climate change, employment of climate change adaptation strategies, and adoption of greenhouse gas mitigation measures). For the steps 2 to 5, factors, indicators, and key questions are given.

3. 3.

   Steps 6–8: Assessment of the scaling up potential (fulfillment of the basic requirements for project implementation; assessment of how the scaling up factors relate to financial aspects, human resources as well as institutional and infrastructural considerations). In step 8, the actual situation is compared with the optimal situation for scaling up.

The final outcome of the ScalA tool is a rating figure that enables a comparison between the failure or success of the analyzed project.

Nutrient use in animal and cropping systems–efficiencies and scales framework (NUANCES)

NUANCES was presented by Giller et al. (2011) on the basis of works by Giller et al. (2006) and continues further developments based on results from field experiences and experiments (Rufino et al. 2007; Tittonell et al. 2007, 2010; van Wijk et al. 2009). NUANCES assesses the impact and trade-offs of agricultural management and explores the potentials of best-fit technologies and promising management alternatives at farm level before they are promoted to farmers. Different analytical methods are combined, such as participatory research, farm typologies, data-mining, experiments, and modeling. NUANCES outlines a four step approach:

1. 1.

   Describe and group farming systems and constraints (socioeconomic, institutional, agro-ecological data, and farming system descriptions).

2. 2.

   Describe the consequences of farmers’ decisions regarding resource allocation: detailed farm descriptions are entered into different models (FARMSIM, FIELD, LIVSIM, HEAPSIM (Giller et al. 2011)) in combination with secondary data, expert knowledge, and experiments. The key processes of the farms are described.

3. 3.

   Draw future scenarios. Here, a series of agro-technologies to improve productivity and trade-offs between resource allocation are discussed.

4. 4.

   Elaborate with farmers and agents of new management systems that are contributing to sustainable smallholder agriculture. The farmers weigh their farming strategy priorities and plan the interventions.

Response-inducing sustainability evaluation (RISE)

RISE was developed by the School of Agricultural, Forest and Food Sciences (HAFL, www.hafl.bfh.ch) (Grenz et al. 2012). A previous version of the tool for holistic sustainability assessment at farm level was presented by Häni et al. (2003). RISE is an indicator and interview-based method for assessing the sustainability of farm management that considers the economic, social, and environmental dimensions. Grenz et al. (2012) define sustainable agriculture as follows: “The farm produces food, feed and further agricultural products and services in amounts and qualities that meet the demands of population and trade and that reflect the local production potential, as defined by climate, soils, and socio-economic framework conditions.”

All sustainability dimensions (ecologic, economic, and social) are covered by ten indicators (soil use, animal husbandry, nutrient flow, water use, energy and climate, biodiversity and plant production, working conditions, quality of life, economic viability, and farm management). Each indicator is calculated from four to seven parameters. According to Häni et al. (2003), for each indicator the current situation “state” (S) and the pressure “driver” (D) that the farming system puts on the indicator are identified. The “Degree of Sustainability” is calculated by the equation “DS = S − D” to identify the strong and weak aspects of the farm. RISE is used for the comparative evaluation of the sustainability degree of different farms, and for the enhancement of the sustainability level of a certain farm. Since the year 2000, RISE has been used on more than 1000 farms worldwide. It has been applied in 36 countries on various farm types from large commercial farms in Europe to smallholder farms in developing countries. The steps of application are the following:

1. 1.

   A trained agronomist performs a field visit to the farm and collects data by filling in the RISE questionnaire.

2. 2.

   Data are entered into the RISE 2.0 software (can be used online and offline, available at: http://www.farmrise.ch), and the degree of sustainability is calculated and visualized in a sustainability polygon.

3. 3.

   Four dimensions are assessed on three scales, namely whether they are strong, acceptable, or not favorable for sustainable development. The four dimensions are (a) stability of the social, economic, and ecological framework; (b) farmer’s risk awareness, attitudes, and management; (c) gray energy (machines, buildings, external inputs); and (d) animal health and welfare (Häni et al. 2003).

4. 4.

   The final results are discussed with the farmers or technical advisors to develop management practices that will add to the sustainability of the farm.

Interactive multi-goal agro-ecological generation and evaluation of systems (Farm-Images)

The methodological procedure of Farm-Images was presented by Dogliotti et al. (2005). The approach is generic. It integrates complex crop rotations and spatial heterogeneity on farms in one method to support the re-design of farming systems. The method explores sustainable development options and trade-offs at the farm level. It can design a diversity of land-use alternatives describing entire crop rotations based on given criteria. The model creates alternative farm systems by allocating production activities to different land units within a farm while taking into account the socio-economic and environmental objectives as well as the specific production conditions (i.e., land, labor, capital, machinery, and irrigation) and the farmer’s preferences (i.e., type of crops, rotation length, and number of land-use types). The approach follows the steps:

1. 1.

   Field scale design: A list of crops suitable to be grown at field scale in the case study site is elaborated. These crops are combined in crop rotations with the help of the computer model ROTAT (Dogliotti et al. 2003). The crop rotations are then combined with production techniques. This results in a number of production activities and land-use options at field scale. Coefficients are quantified for each production activity related to their economic performance, resource requirement, and impact on the environment.

2. 2.

   Farm scale design: The optimal production activities identified at field scale are used to produce optimal farm systems that take into account the farmer’s resource endowment limitations in the region. Farming systems are designed by optimally allocating production activities to different fields on the farm using the model Farm-Images (Dogliotti et al. 2005). The “[…] model has seven alternative objective functions: farm gross margin, family income, capital requirement, soil erosion, soil organic matter rate, N surplus and environmental exposure to pesticides” (Dogliotti et al. 2006). These functions can be defined as constraints or objectives. Functions that are not required in the study can be left out. Farm-Images gives optimal combinations for each farm, satisfying the farmer’s interest, and minimizing the negative side effects.

3. 3.

   The farm types are categorized and are required to represent the diversity of farm types existing in the case-study site. From those different farm types, endowment scenarios are constructed to study the influence of resource availability on options of sustainable farm development. The scenarios are based on the objective function and the sustainability thresholds set.

Trade-off analysis model for multi-dimensional impact assessment (TOA-MD)

TOA-MD 5.0 is a computerized model publicly available at http://tradeoffs.oregonstate.edu. It was developed by Antle and Valdivia (2006) and further developed and applied in several case studies (Claessens et al. 2008; Antle 2011; Tran et al. 2013). The approach addresses the economic, environmental, and social impacts of agricultural technologies, and also assesses adoption rates. The model allows for a quantitative analysis of agricultural systems. The data that is introduced in the model is mainly based on existing secondary, quantitative data derived from existing studies. Additional or missing data is collected during a farm survey. The model can be flexibly set up to calculate a variety of indicators. Any quantifiable outcome variable can be used, for instance, environmental variables like water quality or the amount of protein consumed per household member. The model can simultaneously calculate four outcomes in addition to income and poverty indicators, which are built into the model. It is possible to model whole farm systems, simulate economic indicators and farmer’s participation, trade-offs of technology adoption, as well as mean and threshold indicators for any other quantifiable economic, environmental, or social outcome of the agricultural system. Assumptions are set for the case-study side during the model’s calculations, i.e., the interest of farmers to obtain the highest economic returns.

The ten methods described above present a variety of methodological approaches that are applied for the assessment of sustainability in crop-farming development projects. They range from quantitative modeling approaches (NUANCES, Farm-Images, TOA-MD) over indicator/interview-based approaches (PIA, ScalA, RISE) to more participative frameworks (FoPIA, PIPA, MESMIS, DESIRE-DSS).

4.1 General application characteristics

The methodological approaches vary regarding their initial point of observation. FoPIA, DESIRE-DSS, MESMIS, RISE, NUANCE, and Farm-Images start by analyzing the local context and challenges. Based on the current state of farm management, development initiatives are planned to achieve improved sustainability (Table 1). PIA, PIPA, ScalA, and TOA-MD take as the initial point of observation an intended development initiative and assess its future impacts from there. The ex ante impact assessment methods are applied in various thematic agricultural sectors (Table 1), for instance, PIA focuses on poverty reduction, FoPIA assesses the sustainability of land-use policies, and ScalA analyzes the scaling up potential of sustainable crop production. Six of the ten approaches are exclusively used for ex ante assessments, while PIPA, PIA, MESMIS, and TOA-MD can also be applied for monitoring and ex post analysis. The majority of approaches make use of a method mix, i.e., of qualitative as well as of quantitative analysis practices (PIA, FoPIA, DESIRE-DSS, MESMIS, ScalA, and NUANCES). PIPA uses only qualitative data while the modeling approaches RISE, Farm-Images, and TOA-MD work only quantitatively. The level of application and spatial scale of impact interpretation greatly varies between the approaches. Particularly, the models Farm-Images and RISE give information at a farm level, while other approaches may also interpret impact outreach at a wider local (village) to regional level (for instance DESIRE-DSS, ScalA, FoPIA). The time needed for the application of the methodological procedures varies because it depends on the scale of the analysis (farm level, village level, or regional level). While RISE is assumed to only take 2 days for an analysis at the farm level, DESIRE-DSS needs about 3 months for an assessment at the regional level. The majority of approaches presented are applied by research institutes. PIA, which was developed by the OECD, is widely applied by donor and technical agencies and advisors. Only a few of the methods, such as RISE, are frequently used by implementing institutions.

The end user of the results also varies. Some methods foresee that the results of the analysis are discussed with the farmers and decision-makers (for instance FoPIA, DESIRE-DSS, MESMIS, PIPA, RISE), who form the actual target group for finding solutions in a participative way. Others instead use the results for internal decision as to whether projects will be implemented, or how they will have to be modified in order to reach the set sustainability objectives (for instance PIA and ScalA). The time horizons used to project impacts into the future vary from method to method. While NUANCES interprets short-term impacts of one season and longer-term impacts of between 5–10 years, for DESIRE-DSS the short-term impacts are interpreted as lying within 1–3 years and long-term impacts up to 10 years. Some methods, such as FoPIA, PIA, and MESMIS do not specify, but note only that the impact assessment focuses on short- and long-term impacts. Time horizons of impacts can vary for different interventions and sustainability dimensions, but in the assessment process, they have to be more accurately defined in order to gain improved estimations of impacts, which vary considerably along the time scale (Table 1).

4.2 Stakeholder involvement and learning

The involvement of stakeholders is a central aspect of sustainability impact assessment, and this involvement is not limited to consultation and information, but includes providing local stakeholders with the capacity to shape decisions (O’Faircheallaigh and Howitt 2013; Stoeglehner and Neugebauer 2013). “In terms of sustainability outcomes, it is critical that voices representing affected economic, environmental, cultural, and social values and interests are heard, accurately and fully […]” (O’Faircheallaigh and Howitt 2013). The level of stakeholder involvement varies considerably in the methodological procedures presented (Table 1). DESIRE-DSS, MESMIS, PIPA, and FoPIA foresee active participation of multiple-level stakeholder representatives at several stages of the assessment procedure. They are involved to describe the context, define the challenges, elaborate, and weigh sustainability indicators, evaluate the intended activities, and in the final decision process on what to implement. These methods base their analysis mainly on primary data collected in the field.

Other methods involve stakeholders during context analysis and in discussions and decision-making after the assessment process (NUANCES, RISE). In PIA, ScalA, NUANCES, RISE, Farm-Images, and TOA-MD the sustainability indicators are predetermined and based on theoretical assumptions of sustainable development. Farm-Images at its current stage and TOA-MD foresee little to no involvement of stakeholders. TOA-MD is mainly based on expert estimations and secondary data. In restrictive terms, those approaches (PIA, ScalA, NUANCES, RISE, Farm-Images, and TOA-MD) cannot be considered true forms of sustainability impact assessment as defined by Gibson (2013). To contribute to the improvement of sustainable livelihoods, it is necessary to address the needs of the target group. Reductionist approaches with predefined sustainability criteria (for instance RISE, Farm-Images, ScalA) simplify but also dictate what “sustainability” means and neglect the local perspective and development priority, while a more holistic approach (DESIRE-DSS, PIPA, MESMIS, FoPIA) allows the stakeholders to define their own priorities and understanding of sustainability within the local context. Stoeglehner and Neugebauer (2013) emphasize the need of participation and collective learning as central features of sustainability impact assessment, allowing for learning about facts and values within groups of relevant decision-makers, stakeholders, and planners. Sustainability is not a fixed state, but a moving target. Learning and exchange is an essential element of sustainability assessment (Gibson 2006; Bond et al. 2012; Bond and Morrison-Saunders 2013a, b). Therefore, the learning process is critical for all stakeholders involved. It requires horizontal as well as vertical interaction of multiple level stakeholders (Bond and Morrison-Saunders 2013a). It is therefore not sufficient to involve only one stakeholder group in the impact assessment process, but to integrate stakeholders of different levels, to bring them together for exchange and to involve them from the planning through to the final evaluation stage of an initiative (EIARD 2003; Bond and Morrison-Saunders 2011; Morgan 2012). Only the methodological approaches DESIRE-DSS, FoPIA, and PIPA particularly focus on the learning process during the assessment process.

4.3 Sustainability dimensions

Sustainability impact assessment is the only type of assessment that considers and integrates the three pillars of sustainable development (economy, environment, and social dimensions) equally while also analyzing the interrelations between the three pillars (Pope et al. 2004; Ness et al. 2007; Hacking and Guthrie 2008; Bond and Morrison-Saunders 2011; Bond et al. 2012; Singh et al. 2012). All the above-mentioned methodological approaches present an assessment for sustainable development and consider criteria related to all three sustainability dimensions. But not all methodological approaches consider these dimensions in equal terms as required above. NUANCES and Farm-Images mainly focus on economic and environmental criteria. PIA, on the other hand, mainly analyzes the economic and social dimensions. In RISE, the social dimension is under-represented. FoPIA makes use of the land-use functions framework to link regional sustainability issues to land use, which helps to select and assign indicators for each sustainability dimension but recommends restricting them to a feasible number of approximately nine. DESIRE-DSS and MESMIS leave it to the stakeholders to define the nature and number of indicators for each dimension. RISE, PIA, ScalA, NUANCES, and Farm-Images predetermine and restrict the number of indicators that are included in the assessment or calculation process. The interrelation between the three dimensions of sustainability are approached in FoPIA, PIPA, DESIRE-DSS, and also in MESMIS in a participative way, discussing trade-offs as well as positive implications. The models NUANCES, Farm-Images, and TOA-MD calculate trade-offs due to given thresholds, but those are not weighed, strategically discussed, or evaluated with the stakeholders. A structured discussion and reflection process with the stakeholders is indispensable for the effective and locally adapted analysis of interrelations.

In all the methodological approaches presented, understanding sustainability involves the consideration of all three dimensions. However, not one of the methodological approaches also factors in the institutional dimension. Sustainability, we argue, has not only three, but four dimensions: social, economic, environmental, and institutional (UNDESA DSD 2001; Spangenberg et al. 2002). Institutional capacity is a significant means for facilitating movement towards sustainable development (UNDESA DSD 2001). The Norwegian Agency for Development Cooperation (Norad 2000) defines an institution as sustainable “[…] if it has the strength to survive and develop to fulfil its functions on a permanent basis with decreasing levels of external support.” For sustainable development, the involved individuals, organizations, and social systems need to increase their capacities and performance in relation to sustainability goals, resources, and environment. Core institutional objectives are accountability, civil society empowerment, gender equity, and knowledge formation (Spangenberg 2004). Spangenberg (2002) describes the institutional dimension of the sustainability paradigm “[…] as the result of interpersonal processes, such as communication and co-operation, resulting in information and systems of rules governing the interaction of members of a society.” Participation and governance are critical elements of the institutional dimension and are indispensable for sustainable development—particularly in the development context. Institutional criteria and indicators therefore need to be integrated in the sustainability impact assessment framework of development initiatives.

4.4 Intervention planning and sustainable development

Sustainability assessment has the potential to considerably increase the sustainability performance of planning outcomes and decision-making (Bond et al. 2013). Therefore, it should be an integrated step of development initiative planning (Sadok et al. 2008; Stoeglehner and Neugebauer 2013). To date, there has been a lot more focus on the ex post rather than the ex ante impact assessment of development initiatives, and currently there is no internationally established standard of methodological approaches to carry out ex ante sustainability impact assessment on planning development initiatives (GIZ 2013). Both implementing institutions and scientific discourse still has an extensive backlog demand regarding ex ante impact assessment (Silvestrini 2011).

A challenge also lies in formulating a common understanding of what impact assessment means. Several institutions and also research reports use the term impact assessment for ex post evaluation only. But impact assessment is a process that identifies future consequences of an intended action (IAIA 2009) and consequently is part of the planning process of interventions. Several approaches analyzed in this framework did not use the term impact assessment, but decision support. A common understanding of impact assessment in the context of sustainable development therefore is crucial.

Ex ante sustainability impact assessment as part of the planning process is essential for the sustainability of development initiatives. The assessment process respects the three dimensions of the sustainability paradigm. The active involvement of multiple-level stakeholders in the assessment process is crucial to adapting development initiatives to the locally specific conditions. It was the objective of this paper to review, analyze, and compare ex ante sustainability impact assessment methods that are applied within the framework of sustainable agricultural development, with a focus on crop farming in developing countries (Fig. 2). We have presented the variety of currently applied methods of ex ante impact assessment by analyzing a sample of ten methodological approaches. The minority of methodological approaches analyzed, follow this holistic understanding of sustainability impact assessment. Only those approaches which (a) integrate equally all three sustainability dimensions, (b) respect their interrelations, (c) involve stakeholders actively at every step of the assessment process, and (d) also focus on exchange and learning can be considered as a complete or holistic method of sustainability impact assessment. Methodological approaches of sustainability impact assessment have to be adapted to different local contexts and need to respect the requirements mentioned above. Only this kind of ex ante sustainability impact assessment has a large potential to avoid negative outcomes and to improve stakeholder understanding, acceptance, and ownership of the intervention. Therefore, it should be mandatory within the planning process of development initiatives for sustainable development.

Challenges for sustainable agricultural development: a Pearl Millet field in an arid area of Tanzania during the main rainy season. Improved yields may enhance the livelihood and the food security of the rural poor population. Ex ante impact assessment reduces the risks of negative impacts of agricultural development interventions

Full size image