Chapter 8: Impact-oriented Monitoring and Evaluation

Annex A: Glossary contains definitions of some of the technical terms used in SEADS.

This chapter describes the minimum standards for the monitoring and evaluation of crop-related crisis responses. There is a need for more impact evaluation of crop-related crisis responses, while also recognizing the operational challenges and the diversity of actors in humanitarian contexts. At present, many organizations focus on monitoring project implementation and expenditure, rather than impact.

Impact evaluation in crop-related crisis responses is challenging because:

Livelihood impacts are often not expected until many months after a crop-related crisis response ends, particularly for short-term projects that respond to a rapid-onset crisis.
Supporting crop production does not automatically lead to livelihood impacts.

The SEADS minimum standards recognize that most humanitarian organizations already have monitoring and evaluation systems in place. However, these systems are seldom specific to crop-related responses. The minimum standards in this chapter provide guidance on how to refine existing systems to improve understanding of crop-related responses.

Impact-oriented monitoring and evaluation minimum standards

Minimum Standard 8.1: Participatory approaches

Active community participation is part of monitoring and impact evaluation of crop-related crisis responses.

Key actions

Work jointly with communities to agree to expected impacts of crop-related responses and identify relevant impact indicators (see Guidance note 1).
Involve communities actively in the monitoring of response implementation and in end-of-project reviews and impact evaluations of crop-related crisis responses (see Guidance note 2).
Use locally appropriate participatory methods during monitoring and impact evaluation (see Guidance note 3 ).
Ensure that vulnerable and disadvantaged groups are involved in monitoring and impact evaluation (see Guidance note 4).

Guidance notes

Impacts and impact indicators. Participatory initial assessment should lead to a rapid understanding of the role of crop production in livelihoods, the crop production cycle, and specific crop production practices. As responses are identified and discussed, the likely impacts of these responses can also be discussed. An initial assessment is therefore an opportunity to gather information for the design of monitoring and evaluation activities and the selection of impact indicators. Example questions in Appendix 4.1 will support data collection to monitor impacts. Appendix 8.1 provides examples of impact indicators.
Participatory monitoring, review, and impact evaluation. In line with SEADS Principle 2: Use a participatory approach in all aspects of crisis response. Participatory approaches should be used for monitoring, impact evaluation, end-of-project reviews, and learning activities. participatory approaches, such as focus group discussions, key informant interviews, and proportional piling, require working relationships with communities that involve joint identification of relevant impacts (see Guidance note 1). They also require regular meetings and dialogue during response implementation to jointly track planned and actual implementation and discuss any changes to the response. An end-of-project review (see Minimum Standard 8.5) or impact evaluation (see Minimum Standard 8.6) should also use participatory approaches to compare actual results to the project objectives. Appendix 8.2 illustrates various participatory methods that can be used to monitor, review, or evaluate crop-related crisis responses.
Qualitative and quantitative data. Participatory methods are commonly associated with qualitative information. However, they can also produce quantitative information, such as market prices, crop production data, or proportion of income derived from crop sales. Some of this information can be cross-checked with secondary data, such as earlier livelihood or food security assessments and research reports or statistics from local agriculture offices or markets. Larger development projects often use quantitative baseline surveys in monitoring and evaluation. However, quantitative surveys contradict SEADS Principle 2 of participation. Even in development projects, quantitative surveys are relatively uncommon and tend to be used by larger organizations with relatively more financial and human resources. In crises, the added value of quantitative baselines over rapid participatory assessment is open to question, especially if quantitative information can be obtained from participatory methods and secondary data.
Vulnerable and disadvantaged groups. An important aspect of community participation is understanding the needs of marginalized or vulnerable groups within communities and tailoring responses accordingly. Inclusive monitoring can mean holding separate meetings with poorer households or people who are marginalized due to ethnicity, faith, gender, disability, or other reasons. Design and implement project reviews and impact evaluations in ways that enable marginalized and vulnerable groups to participate, such as by using participatory methods specifically with these groups. Information collected during monitoring, reviews, and impact evaluations should be disaggregated according to local contexts and the presence of marginalized sub-groups within communities.

Minimum Standard 8.2: Project objectives

The design of crop-related crisis responses shows clear technical plausibility of livelihood impacts as the basis for monitoring and impact evaluation.

Key actions

Use specific, measurable, achievable, relevant, and time-bound (SMART) project objectives to express and quantify expected livelihood impacts (see Guidance note 1).
Incorporate SMART objectives into project design tools, such as a theory of change or logical framework (see Guidance note 2).

Guidance notes

SMART objectives. A monitoring and evaluation system that supports learning about livelihood impacts should include objectives that state and quantify the intended livelihood impacts on specific households within a specific timeframe. SMART objectives are an important outcome of good-quality initial assessment and response identification. As explained in Chapter 2: The Scope and Approach of SEADS, crop-related crisis response should aim to achieve one or more of the SEADS livelihood objectives. Therefore, SMART objectives at the project level should directly relate to one or more of the SEADS livelihood objectives. They should also align with the SEADS principles in Chapter 3: SEADS Principles. See Tables 5.5, 5.6, 6.5, 6.6, 7.3, and 7.4 for considerations related to timing for different response areas.
Appendix 8.3 provides an example of applying SMART objectives to a crop-related crisis response. In this example, key information for developing the SMART objective is drawn from participatory initial assessment and response identification activities at community level (see Minimum Standard 4.2: Initial assessment approach). This approach is efficient in humanitarian situations because it does not require return visits to the project areas for the purpose of designing a monitoring and evaluation system or collecting baseline data; the key livelihood impacts have already been identified jointly with local stakeholders during the initial assessment and response identification.
Technical plausibility and theory of change. SMART objectives indicate whether a project objective is technically plausible, which means that a stated type of response, timing, and amount of inputs (including services, repairs, and reconstruction) will likely lead to an expected impact at household level. During project design, technical plausibility can be demonstrated using simple project design tools, such as a theory of change or logical framework. Different organizations have different preferences for the use of these tools, and there are various versions of each tool. When used properly a theory of change (or logical framework) will show:
- the technical plausibility of project inputs leading to project livelihood impacts
- how project managers can identify and prioritize items that need to be monitored and the frequency of monitoring
- the key predicted livelihood impacts in a quantified form and in terms of when these impacts should occur
- the key assumptions and risks that affect the progression of inputs to impacts; these assumptions and risks need to be tracked during the project.
In crisis responses, attention to technical plausibility and use of a theory of change are especially useful because livelihood impacts are often predicted to occur months after a project ends. In these cases, participatory end-of-project reviews can ensure some understanding of likely impacts, even if an impact evaluation is not possible.

Appendix 8.4 is an example of a simple theory of change for a crop-related crisis response that delivered seeds and fertilizer. This example shows that a theory of change does not need to be complex in order to show technical plausibility and that various assumptions and risks affect how crop production leads to livelihood impacts. The theory of change also indicates areas of additional support that a project might provide to try to reduce the risk of low impact. For example, if there is a high risk of households selling the inputs to meet their basic food needs, the project could consider adding food or cash transfers to prevent seed sale.

A detailed example of technical plausibility is available in Case Study 8.1 (see SEADS website).

Minimum Standard 8.3: Process monitoring and indicators

Regular and systematic process monitoring allows tracking of and adjustments to response implementation, and ensures alignment with SEADS minimum standards.

Key actions

Monitor process indicators to check the progress of response implementation, check alignment with SEADS minimum standards, and make real-time adjustments as needed (see Guidance note 1).
Integrate the tracking of context and response risks and assumptions into routine process monitoring (see Guidance note 2).
Update the response area identification tool (RAIT), the decision trees, and the theory of change as implementation evolves (see Guidance note 3).

Guidance notes

Process indicators. This type of indicator measures the progress of response implementation, or what is being done. In most crop-related crisis responses, monitoring focuses on measuring inputs using process indicators. This often involves simple counts of people targeted and/or reached or items provided. Examples include the amounts of inputs provided to target households, the value of vouchers distributed, or the length of irrigation channels rehabilitated. The timing and duration of activities is also commonly included in process indicators. Process indicators track the quality and accountability of implementation. They:
- assist project managers to track response implementation and, when needed, make timely adjustments
- often relate to project expenditure and so support financial accountability of a project
- contribute to impact evaluation by describing activities relative to the project design and implementation plan; this enables predictions of outcomes and impact to be revised if needed.
Process indicators are used to check how implementation aligns with SEADS minimum standards. Examples of key process indicators for relevant minimum standards are provided in Appendix 8.5.

Use process indicators to monitor response implementation regularly, according to organizational and donor requirements. When combined with community focus groups and tracking of risks and assumptions, process monitoring enables timely adjustments to response implementation.
Context, assumptions, and risks tracking. During response implementation, the wider context can change, requiring adjustments in project design. Therefore, it is important to track the operational context over time as well as the assumptions and risks in the project theory of change or logframe, and make adjustments to project implementation as needed. Checks on project implementation can involve activities such as community focus groups. This dialogue can include the use of a checklist of assumptions and risks, drawn from the theory of change. It can also potentially expose the exploitation of children, which is covered by the Child Protection Minimum Standard on child labor (The Alliance 2020). As illustrated in Figure A8.1, some assumptions and risks are constant throughout the project, whereas others are only relevant at certain phases of the project. Therefore, the checklist can vary over time to cover topics that are relevant at each phase. In insecure environments, very frequent tracking of the security situation might be needed such as daily reviews of security issues. Tracking is particularly important when a crisis is complex, long term, or slow onset, and whenever the context changes.
Updating decision support tools during implementation. As implementation progresses, the crop, crisis, or operational context may change significantly. Participatory approaches to process monitoring will provide early signals of these changes. Return to your completed RAIT (see Minimum Standard 4.4: Selecting response areas) when changes are evident to ensure that a crop-related response remains necessary, appropriate, and feasible. Review your responses to questions in the decision trees to ensure that your selected technical option remains likely to have livelihood impacts. It is particularly important to update the assumptions and risks that are included in your theory of change as they can also change over time. A survey done by the Global Food Security Cluster’s Agriculture Working Group (2022) of partners engaged in crop-responses in 2022 indicated that approximately 40% of partners engage with farmers at some level to monitor outcomes and see if anything needs to change in the project. However, 78% of partners say that more and better engagement is needed.

Minimum Standard 8.4: Impact indicators

Response reviews and impact evaluations use meaningful impact indicators.

Key actions

Select livelihood impact indicators for crop-related crisis response that directly show impacts on households (see Guidance note 1).
Select indicators that show impacts on pre-existing systems, services, or markets where relevant (see Guidance note 2).
Ensure that household-level livelihood impacts are meaningful (see Guidance note 3).

Guidance notes

Household-level impact indicators. In general, household-level impact indicators are not well defined or used in crop-related crisis responses. SEADS aims to ensure that responses have livelihood impacts, and these impacts tend to coincide with one or more of the five main assets in typical livelihood frameworks: financial, natural, physical, human, and social. Definitions of these assets, and specific examples of livelihood impact indicators are provided in Appendix 8.1. Further reading at the end of this chapter provides additional examples of process indicators (Appendix 8.5).
Some organizations use a composite index, such as a coping strategy index or a dietary diversity score, to measure impact. In terms of participatory approaches, a challenge with these measures is that populations affected by a crisis are unlikely to propose them as indicators of livelihood impact. Instead, local people tend to express intended crop-related impacts in terms of basic needs, such as amounts of money and food or shares of income and self-sufficiency. Where possible, use impact indicators that are based on local expressions and needs and which all stakeholders will understand (see SEADS Principle 2). A composite index may be useful to measure vulnerability or for comparability across time and space as a basis for learning.

In contrast to process monitoring, it is usually not efficient to measure impact indicators during the implementation of crop-related crisis responses. This is because livelihood impacts are most likely to occur toward the end of a response or after it ends. Impact indicators can be measured during impact evaluation, and the analysis of the findings can be supported by collated process-monitoring data and records of any changes during implementation.

Case Study 8.2 (see SEADS website) provides examples of crop-related responses that led to impacts on household incomes.
Other impact indicators. SEADS Principle 1: Use livelihoods-based programming highlights the need for crop-related crisis responses to support pre-existing systems, services, and markets where possible. Therefore, livelihood impact indicators can also measure impacts on systems, services, and markets. See examples in Appendix 8.1.
Meaningful impact indicators. Impact indicators for crop-related crisis responses are usually quantitative, expressing impact in terms of an absolute number, such as “average household income from crop sales.” However, this indicator has limited meaning unless it is related to a reference point, which in this case could be “total required household income” or “income required from crop sales.” Similarly, a project might report that a household consumed 30 kg of maize that was produced because of project support. But this measure has limited meaning unless the household size and composition is known and the period of consumption is stated. A stronger indicator would relate the maize consumption to the household’s nutritional needs.

Further examples of meaningful impact indicators with a reference point are provided in Appendix 8.1.

Minimum Standard 8.5: Participatory end-of-project review

A participatory end-of-project review predicts future livelihood impacts.

Key actions

Conduct a participatory end-of-project review if livelihood impacts are expected months after a project ends (see Guidance note 1).
Consider using benefit-cost analysis to complement a participatory end-of-project review (see Guidance note 2).

Guidance notes

Participatory end-of-project review. A participatory end-of-project review is needed for projects for which the main livelihood impacts are expected months after a project ends. In these situations, the implementing organization might no longer be present, or might not have the resources to evaluate future impacts. The end-of-project review is an opportunity for community members and project staff to review the implementation process, including the theory of change (or logical framework) and risks and assumptions that were applied. participants can also consider if the completed project is likely to achieve the desired livelihood impacts, even though those impacts may not yet be evident. The end-of-project review need not be elaborate or time-consuming; it can simply involve community meetings supported by collated process-monitoring data and records of risks and assumptions. Simple ranking or scoring tools can assess the likelihood that project impacts will be achieved, assuming that certain conditions remain in place.
This type of predictive review process is imperfect because future events are uncertain. In general, the longer the time between the end of the project and the expected impacts, the greater the likelihood of impact predictions being inaccurate. However, a well-documented end-of-project review can contribute to learning and future project design.

Some organizations have a long-term presence in the areas where crop-related crisis responses are used. Two common situations are long-term development projects and repeated short-term humanitarian projects in a complex crisis. In both situations, this continued presence can enable follow-up of previous crop-related crisis responses many months after the responses took place. This follow-up can involve repeating the end-of-project review process, or a participatory impact evaluation (see Minimum Standard 8.6).
Benefit-cost analysis. In some situations, benefit-cost analysis can complement a participatory review. In livelihoods-based programming there is often a trade-off between the number of people or households who receive support and the type or amount of support they receive. If a donor or organization prioritizes the number of beneficiaries over the quality or relevance of the support, there is a risk that large numbers of households will receive support that is not sufficient to support livelihoods. Benefit-cost analysis can provide useful information on the cost efficiency of a response for a given level of livelihood impact. It can also identify program costs that can be reduced without substantially reducing the impacts. This can lead to future programs being designed to provide similar livelihood benefits to a larger number of people. When applied to different responses in similar socioeconomic and operational contexts, benefit-cost analysis enables a comparison of the cost efficiencies of different responses.

For projects with SMART objectives, a theory of change, and either an end-of-project review or an impact evaluation (see Minimum Standard 8.6), benefit-cost analysis is straightforward. You will need to know the project costs, including the cost of inputs, such as seeds or tools, and the cost of overheads and technical, logistical, monitoring and evaluation, and support staff. You will also need to estimate a monetary value of the livelihood impacts predicted in the end-of-project review. Because it cannot be based on actual livelihood impacts, a benefit-cost analysis done as part of an end-of-project review will only be indicative.

Benefit-cost analysis also requires a good understanding of non-project costs that contributed to the impacts. For example, the value of any labor, seed, fertilizer, or other inputs provided by crop producers themselves should be calculated and included in the analysis.

Case Study 8.3 (see SEADS website) shows how costs and benefits can be estimated for crop-related projects.

Minimum Standard 8.6: Participatory impact evaluation

Participatory impact evaluation measures livelihood impacts during or at the end of a project.

Key actions

Compare the project objectives with the actual livelihood impacts that are evident at the end of the project (see Guidance notes 1, 3, and 4).
Ensure that participatory impact evaluation assesses project contribution (see Guidance note 2).
Consider the use of benefit-cost analysis to complement participatory impact evaluation (see Guidance note 2 of Minimum Standard 8.5).

Guidance notes

Participatory impact evaluation. Participatory impact evaluation of projects is needed where livelihood impacts are expected during a project (or soon after) or to complement an end-of-project review where additional impacts are expected after a project ends. An important advantage of using SMART objectives and a theory of change that both specify livelihood impacts is that the key questions for the evaluation can relate directly to the project objectives, theory of change, and the corresponding indicators. For example, were the project objectives/livelihood impacts achieved and if not, why not? Other questions might examine impacts on specific gender or wealth groups or aim to identify changes to project design that might improve future impacts. A further question might cover the cost efficiency of the project and include a benefit-cost analysis.
Assessing project contribution. An impact evaluation should aim to not only measure livelihood impacts, but also to understand how these impacts happened. What were the specific causes of the impacts and the relative importance of these causes? These questions are often captured using the concept of contribution, which assesses a project’s contribution to impact relative to non-project factors (ALNAP 2016). For example, a household might need 75 kg of seed but only be able to produce 25 kg of seed (that is, 33% of their needs) themselves. If the balance of 50 kg of seed (that is, 67% of the household’s needs) is provided as part of a crop-related crisis response, then the response contributes to 67% of the harvest. Household contribution is often overlooked in impact evaluations. When this or other non-project factors are omitted from evaluations, contribution cannot be fully assessed and project impacts can be inflated.
Using the theory of change in Appendix 8.4 as an example, the project provided seeds and fertilizer but it did not provide other inputs, such as tools, labor, and pest control, that were needed to ensure production. Production also needed the right climatic conditions and secure land access. Market access and reasonable prices were needed to ensure livelihood impacts. While crop producers themselves might have provided tools and pesticide, these inputs might also have been provided by other humanitarian organizations. Therefore, although the theory of change in this case is relatively simple, the assessment of contribution is complex. Understanding the project contribution of crop-related crisis response requires you to:
- Examine the project objectives and theory of change (or logframe), taking account of any changes to implementation that occurred; revise the expected impacts as needed.
- Use simple participatory scoring or proportional piling methods with targeted households to list and score the main factors that are viewed as contributing to the impacts.
Use probing questions to check the technical plausibility of the responses.
Crop production. Many crop-related crisis responses will have a theory of change that includes crop production as an output or outcome. Therefore, a measure of crop yields is useful for understanding contribution, while also recognizing that crop production alone is not a livelihood impact. Obtaining the target yield does not automatically mean the expected livelihood impact will be achieved. Commodity prices can fluctuate hugely, losses in crop storage may be significant, and/or there may be limited access to markets. If you make assumptions about the livelihood impact of a particular yield, test these through discussions with crop producers.

Methods for estimating yields vary in terms of their cost-effectiveness, scale, and accuracy. Select one that fits the context you are operating in and makes the best use of available resources (see Appendix 8.6).
Quantitative impact evaluation. Quantitative impact evaluation uses evaluation designs and methods that are similar to those that are used in quantitative research. It often uses techniques such as random sampling and sample size estimates using mathematical formulas; this requires specialized technical support. The effectiveness of an intervention is usually based on statistical tests. Quantitative impact evaluation might produce statements such as, “Post-harvest household income was significantly higher for crop producers who received project inputs relative to those who received no inputs, and therefore the project was effective in supporting livelihoods.” Quantitative impact evaluation can complement participatory review or impact evaluation, especially if policy makers or senior technical experts prefer to use quantitative research to guide decision making or for advocacy. In humanitarian contexts, it is important that quantitative research approaches be as flexible and participatory as possible, and that assessment and evaluation teams have the required technical expertise to carry them out. Although tools such as mobile apps can speed up data collection in quantitative surveys, to align with SEADS Principle 2 individual surveys should be complemented with other participatory impact and monitoring methods (see Appendix 8.2).

A further challenge with some quantitative impact evaluations of crop-related crisis response relates to the use of randomized control trials. This approach can exclude some people from assistance by using them as a control group. This contradicts the basic principles of humanitarian action as defined by the Core Humanitarian Standard. Randomized control trials that take place when assistance is being rolled out in stages or scaled up over seasons can avoid this exclusion.

Recommended reading

Details of references cited in this chapter are in Annex C. Further recommended reading includes:

Byrne, K. (2022). Applying adaptive management to livelihoods in emergency settings: challenges and opportunities. Mercy Corps (as part of the Strengthening Capacity in Agriculture, Livelihoods, and Environment (SCALE) Associate Award). https://www.fsnnetwork.org/resource/applying-adaptive-management-livelihoods-emergency-settings-challenges-and-opportunities

Foresti, M. (2003). A practical guide to assessment, monitoring, review and evaluation. Save the Children. https://resourcecentre.savethechildren.net/document/practical-guide-assessment-monitoring-review-and-evaluation-2nd-edition/

Global Food Security Cluster (2020). Food security and livelihoods indicator handbook. Programme Quality Working Group. https://fscluster.org/page/indicators

Guenet, D. & Uyen, V.N. (2011). Cost-benefit analysis for interventions supported by the Swiss Agency for Development and Cooperation (SDC) in Vietnam through the PSARD project. Swiss Agency for Development and Cooperation (SDC).

Inter-Agency Standing Committee (IASC) (2012). Accountability to affected populations: Tools to assist in implementing the IASC AAP commitments. https://interagencystandingcommittee.org/system/files/legacy_files/TOOLS%20to%20assist%20in%20implementing%20the%20IASC%20AAP%20Commitments.pdf

Save the Children, International Rescue Committee (IRC) & Mercy Corps (2022). Multi-purpose cash assistance M&E toolkit. https://www.fsnnetwork.org/resource/multi-purpose-cash-assistance-mpca-me-toolkit

Sud, U., Ahmad, T., Gupta, V. & Chandra, H. (2017). Methodology for estimation of crop area and crop yield under mixed and continuous cropping. Technical report series: GO-21-2017. FAO. https://www.researchgate.net/publication/349075583

Willenbockel, D. (2011). A cost-benefit analysis of practical action’s livelihood-centred disaster risk reduction project in Nepal. Institute of Development Studies at the University of Sussex, Brighton, UK. http://hdl.handle.net/11283/366214

Appendix 8.1: Examples of livelihood impact indicators for crop-related crisis response

Table A8.1: Examples of livelihood impact indicators for crop-related crisis response

Livelihood asset	Illustrative impact indicators for crop-related crisis responses
Financial	Income derived from the sale of crops per household, relative to total household income. Secondary impact income indicators could focus on the specific uses of income, such as to buy food, crop production supplies, or medicines, or to pay school fees
Natural	Area of land accessed per household relative to baseline (for a response that aims to ensure access or expand crop production land)
Physical	Use of new or rehabilitated infrastructure relative to baseline (for a project aiming to construct or rehabilitate crop-related infrastructure)
Human	Proportion of household dietary energy requirements from own-crop consumption
Social	Strengthening of social networks and collaboration between network members

Livelihood impact indicators are most meaningful if they include a reference point, such as the total amount of a certain crops consumed in a normal year or the total household income required.

For example, for a project that provides rice seed and fertilizer, you might measure:

average proportion of household rice food needs met relative to a normal year
average proportion of total household income derived from rice sales
average uses of income derived from rice sales.

This type of project might produce impacts such as, “On average, project households were able to meet 62% of their rice food (own consumption) needs, and an average of 27% of household income was derived from rice sales. Of this income, 40% of income was spent on agricultural inputs, 40% on food, 10% on school fees, and the remaining 10% on other items.”

For a project that aims to rehabilitate a damaged community irrigation scheme, you might measure:

average area of irrigated land planted relative to the pre-crisis situation
average proportion of crop-based food needs met per household
average income from crop sales relative to average household income needs.

This type of project might produce impacts such as, “On average, project households were able to plant 0.3 hectares (ha) relative to pre-crisis cultivation of 0.4 ha. The average income from irrigated crop sales was US$400 relative to annual household income needs of US$5,200.”

For projects that aim to support pre-existing systems or services, impact indicators could include:

number of private sector input sellers at seed fair and average value of vouchers redeemed per seller and type of inputs sold
partnership with local government extension agents for detecting and controlling crop pests
use of loans or subsidies to private sector transporters
use of local tool producers and average value of tools purchased per producer.

ICRC’s Economic Security Indicators Cookbook provides additional examples (2021).

Appendix 8.2: Participatory methods for monitoring and impact evaluation of crop-related crisis responses

Table A8.2: Examples of participatory methods and illustrative uses for monitoring and impact evaluation of crop-related crisis responses

Participatory method	Illustrative uses	Notes
Interviewing methods
Individual semi-structured interviews	Can be used to collect information on any topic Information can be quantitative (for example, income from crop sales) or qualitative (for example, security situation during project implementation) Important method for cross-checking and probing information from visualization and scoring methods	Semi-structured approach combines structure with flexibility Cross-check with other methods and process-monitoring data Repetition of interviews that produce quantitative data enables data to be summarized using statistics as needed
Group semi-structured interviews (focus group discussions)	As above	As above
Visualization methods
Timeline	Timing of responses against crop production cycle	Useful for assessing the timeliness and relevance of project support
Participatory mapping	Community and land boundaries Areas of land rehabilitated relative to all farming land Areas of land cultivated relative to all farming land	Areas of land cultivated or rehabilitated can be quantified if a scale is applied to maps
Service maps	Before-and-after project access to markets and input sellers	Useful for projects that aim to improve market access or links between producers and sellers
Transect walks	Observations of crop production Observations of use of project-supplied tools Observations of crops managed for pest control	Useful especially when cross-checked with informal interviews
Causal diagrams	Project contribution—visualizing linkages from project and non-project inputs and issues to livelihood impacts	Review findings against project theory of change Linkages can be weighted or scored; repetition of method produces data that can be summarized using statistics as needed
Participatory video and photographs	Recording issues and impacts during a project as selected by community members	Can use photovoice approaches Can be used to compare priorities and perspectives of different social or wealth groups
Ranking and scoring methods
Proportional piling	Before-and-after proportions of types of household income and changes in total income Before-and-after proportions of food types consumed and changes in total food consumption	Most useful when the method is repeated with different individual informants or groups of informants; results can then be summarized using statistics as needed Cross-check results against theory of change and a review of technical plausibility; support with causal diagrams
Matrix scoring	Comparison of issues and impacts of different responses: agricultural and non-agricultural different crop-related responses	Useful for understanding the relative impacts of different responses Findings might be compared with benefit-cost analyses of the responses Repetition enables statistical summaries and analyses as needed

Adapted from Gujit (1998) and Catley et al. (2014)

Appendix 8.3: An example of applying SMART objectives to a crop-related crisis response

An NGO project proposal states the following objective: “Improve the agricultural production of crisis-affected households through provision of seed and other inputs.” This objective outlines what the project is trying to achieve but is far too vague to support an assessment of technical plausibility or impact. Although impact indicators could be identified for the objective, it is more efficient to use a SMART objective.

A SMART objective for the same project could be: “In Hamsi district of Amuristan, provide wheat seed and diammonium phosphate and urea fertilizer to 700 vulnerable households so that production covers own consumption and seed needs, and sales contribute to at least 15% of household income in the 2005 post-harvest season.” This objective is:

Specific because it states the project area, the type of household to be targeted, and the specific type of seed and fertilizer to provide.

Measurable because it states a quantified target for the income impact (15% of income) and targets for wheat consumption (100%) and wheat seed needs (100%), as identified with focus groups during the response identification.

Achievable because it is based on a participatory initial assessment in which focus groups described sources of crop and non-crop-related income, uses of wheat, and average incomes from wheat sales in a typical year.

Relevant because in the participatory initial assessment, focus groups described the characteristics of more-vulnerable households, the relative importance of different sources of crop- and non-crop-related income, and the relative importance of different crops as sources of income.

Time-bound because it specifies when the livelihood impact will occur and draws on the crop production cycle described by focus groups during the initial assessment.

Appendix 8.4: Example of a simple theory of change for a crop-related crisis response that delivered seeds and fertilizer

This example of a theory of change is from a project in south-central Asia that responded to a complex crisis. A kit of improved certified wheat seeds was provided for each household for irrigated cultivation, along with diammonium phosphate (DAP) fertilizer and urea fertilizer. Different organizations define outputs, outcomes, and impacts differently when using a theory of change, but in this example, agriculture production has been positioned as an outcome.

Figure A8.1: A simple theory of change for a crop-related crisis response that delivered seeds and fertilizer

Appendix 8.5: Tracking alignment with SEADS—key process indicators

Table A8.3 provides examples of process indicators that help monitor alignment with SEADS principles and minimum standards. Note that disaggregation of indicators by vulnerability status can help distinguish how key processes may or may not have been appropriate for certain groups.

Table A8.3: Key process indicators to monitor alignment with SEADS principles and minimum standards

No.	Key process indicator	Principles and standards monitored
1	Key response team members trained in use of SEADS	All
2	Key response team members with essential knowledge and skills as in Annex B	All
3	Targeted participants are actively and equitably engaged in assessment, response identification, targeting, implementation, and evaluation of crop-related responses	SEADS Principle 2, Minimum Standards 4.1, 8.1, 8.6
Initial assessment
4	Appendixes 4.1, 5.1, 6.1, 7.1 were used to select assessment questions	Minimum Standards 5.1, 6.1, 7.1
5	Technical data were collected at the same time as initial assessment data where possible	Minimum Standards 4.2, 4.3, 5.1, 6.1, 7.1
6	Initial assessment is completed in a participatory manner and written report(s), in appropriate local language(s), is/are available to team members	Minimum Standards 4.1, 5.1, 6.1, 7.1
7	Acute constraints to crop production have been identified and links have been made with development programs to address chronic constraints	Minimum Standards 5.1, 6.1, 7.1
8	Approach to data collection draws on existing secondary data and information	Minimum Standards 4.2, 4.3
9	Initial assessment and technical data from different sources cross-checked for consistency	Minimum Standards 4.1, 4.2
10	Initial assessment determined that a crop-related response is or is not appropriate, necessary, and feasible	SEADS Principle 1, Minimum Standard 4.2
11	Criteria of selection for targeting the most in need are defined and relevant to the context	Minimum Standard 4.3
12	Different targeting methods are considered and the most suitable identified	Minimum Standard 4.3
13	Where livestock play a critical role in livelihoods alongside crop production, Livestock Emergency Guidelines and Standards Chapter 3: Initial Assessment and Identifying Responses was used to inform initial assessment	SEADS Principles 1 and 5, Minimum Standard 4.3
Response identification
14	SEADS livelihood objectives and their relevance to the context are understood	SEADS Principle 1, Minimum Standards 4.4, 5.2, 6.2, 7.2, 8.2, 8.4
15	A participatory RAIT is complete and potential response areas identified	Minimum Standard 4.4
16	Scoring of the RAIT was based on initial assessment and technical data and consensus of participants	SEADS Principles 1 and 2, Minimum Standard 4.4
17	Decision tree results identifying appropriate technical options and related minimum standards are available and show links between assessment results and technical options selected	Minimum Standards 4.4, 5.2, 6.2, 7.2
18	Market-based responses have been considered and favored where possible	SEADS Principle 1, Minimum Standards 5.2, 6.2, 7.1
19	Timeline(s) with potential delivery bottlenecks relative to critical seasonal tasks for all technical options identified	Minimum Standards 5.2, 6.2, 7.3
20	Quality criteria for seed, tools, equipment, and other non-seed inputs were established that were at least as good as crop producers routinely used, and were acceptable to farming communities, donors, authorities, and practitioners	Minimum Standards 5.5, 6.5
21	Quality and technical specifications verification throughout the response timed to reject bad-quality seed, tools, equipment, other non-seed inputs or infrastructure, if needed	Minimum Standards 5.5, 6.5, 7.3
22	Crop-related responses are coordinated with other actors delivering responses in the target area	SEADS Principle 5
23	Crop-related responses include activities to ensure environmental protection	SEADS Principle 4, Minimum Standard 7.2
24	Calculations of the scale of assistance are based on appropriate targeting, real-time costs, sowing rates, access to assets, and appropriate and safe delivery mechanisms	Minimum Standards 5.2, 6.3
25	Written timeline showing consideration of sowing season and bottlenecks	SEADS Principle 1, Minimum Standards 4.2, 5.2, 6.2, 7.3
26	A participatory assessment identifies the key infrastructure rehabilitation needs, the technical requirements, and the location, lifetime, and timing for implementing the rehabilitation, including: understanding inequalities in ownership, access, and who would benefit most (security of tenure) local legislation, preferred construction techniques, building codes positive and negative environmental impacts risk mapping how to build back better	SEADS Principles 1, 2, Minimum Standard 7.1
Implementation of SEADS response areas
27	Data on market prices and crop producers’ buying power is available	SEADS Principle 1, Minimum Standards 5.3, 6.3
28	Local and existing seed, tools, equipment, and other non-seed input sources and systems leveraged	SEADS Principle 1, Minimum Standards 5.3, 6.3
29	All seed, tools, equipment, other non-seed inputs, and infrastructure used in a crop-related response: were known, tried, and tested in the area (varieties, types) were acceptable to all participants (choice, quality, amounts) met different requirements and regulations of donors and governments (certifications, planning and building codes) perform well under crop producers’ realistic management conditions were tailored to the needs and capacities of different crop producers	SEADS Principle 1, Minimum Standards 5.4, 5.5, 6.4, 6.5
30	Consideration of crop producer choice, limits, and final recommendations documented	Minimum Standards 5.4, 6.4, 7.2
31	Seed treatment and labeling associated with all inputs were assessed and completed as needed	Minimum Standards 5.4, 5.5, 6.4, 6.5
32	Safety training delivered as required to protect participants	Minimum Standard 6.4
33	Crop-related infrastructure is rehabilitated according to agreed safe practices for the specific context and hazards	Minimum Standards 7.2, 7.3
35	Target participants report the seed, tools, equipment, other non-seed inputs, or infrastructure was functional and accessible and safe when it was needed (disaggregated by in kind or cash and delivery mechanism)	SEADS Principle 1, Minimum Standards 5.2, 6.2, 7.2, 7.3
Monitoring and impact evaluation
36	Project objectives are specific, measurable, achievable, relevant, and time-bound (SMART)	Minimum Standard 8.2
37	A theory of change or logical framework exists for the project and includes SMART objectives, and risks and assumptions	Minimum Standard 8.2
38	Process monitoring was done and real-time adjustments made as needed. The RAIT, the decision trees, and the theory of change were updated as needed	Minimum Standard 8.3
39	Impact indicators directly show meaningful impacts on households	Minimum Standard 8.4
40	If livelihood impacts are expected months after a project ends, a participatory end-of-project review was completed and benefit-cost analysis considered	Minimum Standard 8.5
41	If livelihood impacts are expected at the end of the project, a participatory impact evaluation was completed	Minimum Standard 8.6
42	Rationale for the choice of evaluation is available	Minimum Standards 8.5, 8.6

Appendix 8.6: Methods for estimating crop yields

Table A8.4: Cost-effectiveness, scale, and accuracy in estimation, errors, and biases of methods for estimating crop yields

Method	Cost-effectiveness	Scale	Accuracy in estimation, errors, and biases
Crop cut	Time and labor intensive	Field, farm, and sometimes landscape level	Tendency to overestimate
Farmer’s estimate	Cheap and quick method that saves time and money	Farm to landscape	Fairly accurate estimation, but needs adequate supervision Subjective
Sampling harvest unit	Cost-effective	Farm to landscape	Error prone in the condition where crop producers harvest from multiple areas at a time and not possible with staggered harvesting
Whole plot harvest	Cost and labor intensive	Plot level, farm level, case study	Almost bias/error-free
Expert assessment	Moderately cost-effective	From farm to landscape level	Chance of error increases if different teams of experts are used or extension people are used to estimate yield in their own area Subjective
Crop cards	Cost and labor intensive	Field to farm level	Bias due to illiteracy, use of local units, etc.
Crop modeling	Cost-effective	Landscape	More accurate if adequately parameterized and calibrated. Does not include induced improvements in agricultural technology
Purchaser’s/insurance record	Cost-effective	Field scale	Suitable for cash crops only with no household consumption
Allometric models	Cost-effective	Field scale	Suitable for few crops
Remote sensing	Cost-effective	Landscape	Chance of error in cases where different crops have the same signature

Adapted from Sapkota et al. (2016)