The notion of evidence-based information, decision-making and policies is increasingly used in the development vocabulary. Providing evidence, however, is a tedious and tricky undertaking, due to the complex nature of development processes. Merely using external evaluation processes assessing achievement of log frame indicators is often insufficient and, in fact, inadequate to understand how and why change has, or has not, happened following a development intervention. Increasingly, projects are required to have monitoring and evaluation systems incorporated in project design.
Sustainable development can be defined as a change process to improve people's livelihood in the short and long term, taking into consideration sociocultural, economic, environmental and political aspects. In the context of agriculture, this implies a need for farmers to become knowledgeable and skilled managers of the agroecosystem and agribusiness (van de Fliert, 2014). In many Asian countries, the majority of farm landholdings are highly diverse but generally small, the financial capacity of farming families is often limited and labour is relatively cheap. Farmers, therefore, need to be able to adapt information and innovations conveyed to them to the agroecological and socioeconomic conditions prevailing on their farms.
The processes through which farmers get access to information and innovations from a variety of sources, such as rural advisory services and development agents, mass media or input retailers, strongly influence their ability to adopt and adapt innovations and give direction to sustainable improvement of their livelihood. This paper argues that to inform policymakers, development organizations and communities in a meaningful way, evaluation of rural advisory services should seek evidence of impacts that address process indicators as much as outcome indicators. Understanding how change happened, and particularly how it sustained, or not, rather than just recording that it happened at a particular moment in time, is vital to planning effective initiatives for further development.
Advisory services to support sustainable rural development
Before touching on ways to evaluate rural advisory services in a more comprehensive way, it is important to define what the functions are of advisory services that support sustainable rural development. Where the overall goals of sustainable development imply reduction of poverty, attainment of food and nutrition security and preservation of the environment, at the level of individuals and families this means people should have a fair income, sufficient quality food, good health (services), equitable chances and a healthy environment (see Figure 1). People achieve positive mpacts in these areas by making the right decisions towards supportive behaviours. In this regard, 'right' is what provides the best possible balance in attaining economic, social and environmental outcomes for a specific family context to support and improve their livelihood. There is no 'one size fits all' formula. The consequence of this is that to make better informed decisions farmers need knowledge, skills and access, as further detailed in Figure 1.
Of particular importance is the development of critical skills, which implies the capability to sift available information, decide what is valuable and worth trying out, observe, experiment, conduct an economic analysis, and develop a farm management system that suits the prevailing socioeconomic and biophysical conditions. With such skills, farmers will be able to adapt to an ever-changing environment and give direction to their own development as they see fit.
To support such an approach to development, the role of rural advisory service providers, whether it be government extension officers, NGO development workers or private sector advisors, is more one of facilitating access to innovations, including support mechanisms such as credit facilities, and to opportunities to develop knowledge and skills, rather than merely transferring technologies. The content and process of facilitation need to be carefully designed and aligned to one another. Learning about and acting upon complex issues, such as climate change adaptation, for instance, cannot be done through simple messages and usually requires provision of multiple options, dialogue and collective action. Impacts achieved from such change processes should be emergent and long lasting.
Evaluation approach, outcomes and utilization
Methodologies to evaluate sustainable development, in general, and advisory services, in particular, should apply the same principles that sustainable development is defined with. The sustainable livelihood framework initially developed by DFID (1999) provides a good basis for unravelling some of the complexities involved in understanding change. In particular, identifying major relevant indicators for change within the five livelihood capitals, i.e. human, social, natural, physical and financial capital, helps to recognize the interrelatedness between the various aspects of people's lives, and, hence, to understand how they make major decisions towards changing behaviours.
A second aid to understanding and assessing results of rural advisory services is by reviewing change at progressive levels and recognizing favouring and hampering factors at each level. As described above, advisory services involve processes that provide access to learning and innovations, through which farmers are capacitated to gain knowledge and develop skills to make better informed decisions. Whether this happens depends on the quality of the process, as well as on the ability of the farmers to learn, which in turn is determined by their existing levels of awareness and knowledge and skills, their values, beliefs and prior experiences. The next change level involves how changed awareness, knowledge and skills is utilized to change practices and behaviours, which in addition to ability will be influenced by the socioeconomic conditions farmers face, other interventions they are exposed to and power relations that may exist. Next is how changed behaviours result in livelihood effects, including increased yields and income, improved health, a better environment, and more equitable gender roles, and so on. This, again, will also depend on external factors such as climatic, market, environmental and infrastructure influences. Finally, collective livelihood effects can lead to impacts at the community level, that are usually anticipated in the overall goals of development initiatives, but achievement will also be determined by sociopolitical and legal contexts, resolution of conflicting interests and available resource base, to name a few aspects.
By defining how each of the change levels affects each of the livelihood capitals, we can build a framework that helps to understand the complexity and causalities of events and effects in an advisory services initiative. This is illustrated in Figure 2. Each of the cells in the grid represents an indicator area for which specific indicators can be defined that then form the basis for an evaluation methodology. This will result in the identification of both outcome and process indicators and should preferably be based on the expected outcome set by the targeted stakeholders at the start of the intervention. The active participation of all stakeholder groups in designing, implementing and utilizing evaluation is key to obtaining the evidence of processes and outcomes that will work at all levels in the long run.
Effectively assessing outputs, outcomes and impacts at the various change levels, as a whole labelled as 'evaluation', involves ongoing monitoring of activities, systematic evaluation upon finalization of each activity and overall evaluation at the end of the initiative, and impact assessment after a long enough period upon completion of the initiative for sustainable change to have become internalized. Particularly where sustained behavioural change is expected based on improved abilities for better informed decision- making, a substantial amount of time and a broad scope of assessment is required to collect the evidence that will make an interesting and convincing case. Unfortunately, not many organizations have the capacity to provide the resources for such long-lasting investigations.
The case for more comprehensive evaluation of advisory services is, of course, only valid if we intend to utilize the results effectively. This implies not just for accountability purposes to serve donor requirements, but primarily for learning, adaptive project management and planning of follow-up activities by all stakeholder groups. Consequently, both data collection methods and reporting formats are required to be stakeholder specific and to recognize the different abilities and needs of stakeholders to generate, absorb and utilize information. Participatory and utilization oriented evaluation and impact assessment processes will need to be planned and budgeted for at the inception of interventions.
Principles and methods
A people-oriented evaluation system of rural advisory services that would address smallholders', communities' and policy makers' sustainability goals should be based on the following principles (after Lennie and Tacchi, 2013). First, it should be participatory by developing mutual trust, partnerships, interactive communication, and mutual learning. Second, it should be holistic as a basis for understanding systems, contexts, inter-relationships and networks. Third, it should recognize complexity, involving the analysis of dynamic contexts and social norms. Fourth, it should be critical and allow for a focus on difference, including gender related aspect, power inequity and contradictions in social change processes. Fifth, it should be emergent, in that intermediate outcomes should inform the direction of further change, hence requiring adjustment of expected outcomes and evaluation indicators and methods. It should contain continuous feedback loops and capture unexpected outcomes and ripple effects. And sixth, it should be realistic, focused on how systems actually behave and what kind of impact can be reasonably expected.
Evaluation and impact assessment that is based on the above principles is not served by a blueprint methodological design as it needs to reckon with the situation-specific nature of the contexts, goals and stakeholder characteristics of an initiative. There should, first, be consistency between the objectives, expected outputs and intended utilization of evaluation and the selected approach and methods. A mixed methods approach suits best to evidencing both the suitability and impact of innovations and the appropriateness and long-lasting effects of advisory services processes towards enhancing farmers' decision-making capacity. Methods should be simple and practical while rigorous and grounded in realities. Results should provide a good balance between numeric values and narratives that not only provide proof of achievements but also illustrate how achievements came about and can be sustained. As indicated above, monitoring, evaluation and impact assessment methods should be closely interlinked and complementary.
Ideally, evaluation methodologies are built in project design but regularly adjusted to match emerging needs and conditions. Integrating such evaluation systems in projects and organizations requires expertise to design, implement and facilitate the methodologies within project teams, on the one hand, and understanding and commitment by donors and policymakers, on the other.
The above sections argued that efforts to support and promote sustainable rural development are best served by holistic people-centred approaches to advisory services, and consequently to evaluation. Wicked problems have wicked solutions, and one-size-fits-all approaches are unlikely to result in long-lasting impact. Evaluation should therefore function as an ongoing mechanism to inform adaptive management and generate evidence of achievements, both in terms of outcomes and process. In addition to being conducted in a participatory and comprehensive manner, outcomes should be evidence-based and well communicated in appropriate formats to relevant stakeholder groups. Proper allocation of time and resources is needed if we are serious about making a case for assessing, documenting and utilizing sustained impacts.
(References will be made available upon request)
(References will be made available upon request)