Toolkit+-+gathering+evidence

Previous section: Iterative design Back to Framework & Toolkit home Next section: What to do with the evidence CCSL Framework and Toolkit Documentation and data collection

This section can help you get started with the monitoring and evaluation (M&E) of your social learning approach, or to improve your existing monitoring strategy (from a social learning perspective). As an important element of an iterative, adaptive management approach, M&E should be carried out continually and inform actors and decisions in short feedback loops, to facilitate looped learning. In the context of social learning, monitoring should be as social as other aspects of a project’s approach, involving all actors in a project.

This section acts as a portal through which you can explore different monitoring approaches and tools, as well as examples of their successful implementation for social learning initiatives (work in progress). toc

1. M&E Framework
The CCSL team has recently developed a common M&E framework for new projects and programs using a social learning approach, with the aim of being able to systematically collect evidence on social learning's added value in the context of climate change adaptation and food security. The framework has been published as [|CCAFS Working Paper #98, 'Monitoring and Evaluating Social Learning: A Framework for Cross Initiative Application']. It provides a theory of change for social learning and 30 key indicators spread across four monitoring areas: iterative learning, engagement, capacity development and challenging institutions. These four monitoring areas, which form the basis of the framework, build on those selected by participants of the June 2014 Social Learning Evidence Gathering workshop in London. In 2015 a minimum of 5 projects will receive peer assistance from the CCSL Initiative in order to pilot the framework; if you wish to join the pilot phase please get in touch.

If you are developing your own project- or programme-specific M&E framework, the information on the rest of the page may be useful:

Theories of change may be a familiar tool. Establishing a theory of change is an important first step in building an M&E framework in many contexts, and can be useful in a social learning approach as well. A theory of change maps all the changes that a given intervention hopes or expects to have. It also highlights 'impact pathways' (how project/program activities or "inputs" lead to the outputs, outcomes, and eventually impacts desired, given the context and the people/groups involved). In this sense, it is essential to visualize the effects that a social learning approach is meant to bring about. Crucially, a theory of change also helps you map out the assumptions and risks involved in a (social learning) approach. It is often helpful to have a visual representation of your theory of change and to consider revisiting that theory of change, impact pathway(s) and set of assumptions and risks that guide the initiative. The conversation that leads to developing the theory of change and impact pathway is arguably more important than the actual map that results from it. Read some recent experiences of CCAFS in developing impact pathways and theory of change.
 * Establishing impact pathways and a theory of change **

The processes around your impact pathways and/or theory of change are as important in a social learning approach as the tools themselves. Often, these tools are created in a one-sided process that does not involve stakeholders and then abandoned once the project begins. In a social learning approach, these tools should reflect the social, flexible, and iterative nature of social learning (as is the case with //participatory impact pathways// - see below). Some topics and questions to reflect on, in this respect:
 * Guidance on impact pathways and theories of change**
 * 1) **Process**: Are your M&E tools created through a collaborative process? Which stakeholders are involved or excluded?
 * 2) **Iteration**: Are your M&E tools dynamic, living documents - are you revisiting them periodically to reflect, learn and revise them?

Resources:
 * [|Strategic overview of CGIAR Research Programs: Part 1. Theories of change and impact pathways]: for general information on theories of change and logframes
 * [|CCSL Learning Brief #6: "Lessons in theory of change: experiences from CCAFS"]: on theories of change at the institutional level, building iteration and flexibility into the process of developing a theory of change, and the use of log frames
 * [|Participatory impact pathway analysis: a practical method for project planning and evaluation]: a guide to designing and monitoring impact pathways using participatory workshop

2. M&E Toolkit
Traditional M&E approaches look for evidence of whether a project/program's approach is resulting in the changes set out in the theory of change. In the context of a social learning approach, social learning goals can be built into the theory of change, and subsequently monitored along with other desired outputs, outcomes, and impacts. For instance, you may want to collect evidence in any of the following areas:
 * Key areas of evidence to collect **
 * **Intent**: the discourse and its manifestations (in documents, meetings, official communications etc.) about purposefully using social learning and the results of having this intent
 * **Looped learning**: mechanisms in place to ensure reflection, learning and iteration and success or failure of these mechanisms; examples of looped learning, adaptation and flexibility
 * **Social differentiation:** makeup of groups impacted by social learning (homogenous/heterogenous) and how this is being addressed successfully or not
 * **Power**: power relations between stakeholder groups and how this is being addressed successfully or not; impact of social learning on these relationships (social transformation)
 * **Capacity**: ways in which stakeholder capacity (particularly to contribute to successful social learning through e.g. process facilitation, interpersonal communication, empathy, listening, teamplay etc.) is being identified, assessed, integrated, and built (or not), and the effect of this on the wider process
 * **Institutional opportunities and barriers**: elements of institutional systems, governance and culture that are helping or hindering social learning
 * **Endogenous processes**: processes and policies in local government that can be linked to or integrated with
 * **Scaling up and out**: changes over time in the size of the area and number of people impacted

Some other areas of more traditional M&E can be useful in monitoring social learning:
 * **Relationships:** the quality of partnership development, and the way specific actors and 'boundary partners' may be influenced by the process. Associated methodologies include Social Network Analysis.
 * **Knowledge and learning**: intangible assets - broader than the 'looped learning' mentioned above.
 * **Outcomes, change and impact at scale**: outcomes and changes in behaviour, discourse, practice, policy and the livelihoods of people. Associated methodologies include Outcome Mapping, stories of change such as Most Significant Change, etc.

A number of different tools have been used to collect and analyze evidence for the M&E of social learning. To make sense of the wide range, we have grouped tools by type of evaluation (process/outcome) and scale (individual/network/system):
 * Evidence gathering tools **


 * Scale (right)

Inquiry type (below) || **Individual** || **Network** || **System** || // This table has been adapted from [|CCAFS Working Paper #38], “Social learning in practice: a review of lessons, impacts and tools for climate change adaptation”. //
 * **Process evaluation** || * Participant observation / diary
 * (pre/post questionnaires) Knowledge / Attitude / Skills surveys and e.g. analysis of the transformation of participants’ narratives
 * Stories of change
 * Appreciative Inquiry || * After action reviews, community self-assessments etc.
 * Formal and i nformal observations, focus group discussions and e.g. post-workshop discussions
 * Social network analysis
 * Stories of change
 * Outcome Mapping
 * Appreciative Inquiry || * Participatory Mapping
 * Social network analysis
 * Stories of change
 * Outcome Mapping ||
 * **Outcome evaluation** || * (pre/post questionnaires) Knowledge / Attitude / Skills surveys and e.g. analysis of the transformation of participants’ narratives
 * Stakeholder portraits and follow up interviews
 * Psychologists’ evaluation of the actors’ experiences
 * Stories of change
 * AppreciativeInquiry || * Knowledge gained by e.g. farmers and their plan to apply it
 * Post-workshop survey to assess attitudes
 * M&E by the community of their actions
 * Mixed method/triangulation approach used to collect data
 * Amount of change in behaviour, attitude, skills, knowledge or condition (situation) of programme participants
 * Environmental stress reduction indicators and environmental status indicators that measure actual success in environmental outcomes
 * Social network analysis
 * Social return on investment
 * Stories of change
 * Outcome mapping
 * Appreciative Inquiry || * Policy change assessment (based on e.g. citation analysis)
 * Social network analysis
 * Stories of change and e.g. //SenseMaker//TM narrative filtering
 * Developmental evaluation ||

Throughout, methodologies such as 'Process documentation' and 'Reflexive monitoring' are applicable to all cells of this table.

While evidence collection does not necessarily need to be a participatory process, the analysis of that evidence needs to lead to social learning. Even if your project's monitoring efforts are participatory, they may not facilitate genuine social learning--it is important to make the distinction between participation and social learning. For more ideas on how to proceed, please see the next section of the Framework and Toolkit, What to do with the evidence.
 * Guidance on gathering evidence**

3. Resources for gathering evidence

 * General resources about assessing social learning:**
 * [|CCAFS Working Paper 38], "Social learning in practice: A review of lessons, impacts and tools for climate change adaptation": Appendix 2 contains information on the evaluation approach taken in 5 social learning case studies
 * Creating robust evidence through knowledge integration (by Wenny Ho): this paper explains how the integration of different knowledge perspectives (using double-loop learning and dealing with power) to provide more robust evidence of complex change at play
 * [|How long does social learning take? Insights from a longitudinal case study paper] by Measham, T.: based on the findings presented here, comparable social learning programs should consider a minimum of 3 years to allow enough time to develop new knowledge for tackling complex problems
 * [[file:ccsl/Learning to learn Mahanty.pdf|Learning to learn: designing monitoring plans in the Pacific Islands International Waters Project paper ]]by Mahanty S., et al.: outlines the project monitoring approach developed in the Pacific Islands International Waters Project (IWP) and finds that a monitoring approach based on indicators to assess supportive processes, behavioural change and human–environmental conditions is useful for monitoring the long- and short-term impacts associated with integrated coastal management programs.
 * [[file:ccsl/measuring social learning.pdf|Measuring Social Learning in Participatory Approaches to Natural Resource Management (academic paper]]) by van der Wal, M. et al.: this paper proposes a method for quantitatively and qualitatively measuring social learning, using a perspective scoring table based on cultural theory perspective categories. Though this method is not presented specifically in the context of climate change adaptation and food security, it may prove useful or could be adapted.
 * Monitoring and evaluating learning networks by Rick James (2010) is another INTRAC Praxis paper which considers ways to assess the value of learning networks through e.g. learning activities, engagement, learning outputs, outcomes and impact, relationships between members etc.
 * [|The acoustics of social learning: designing learning processes that contribute to a more sustainable world] by Wals, A., van der Hoeven, N., and Blanken, H.: see section "Evaluating: looking back and looking ahead", p. 23.


 * General resources about M&E:**
 * Better Evaluation is a general portal on Monitoring and Evaluation that highlights a number of these methods and approaches, and probably the best place to find information related to monitoring and evaluation in general, hence with a lot of relevant information for social learning too
 * Impact, innovation and learning: Towards a research and practice agenda for the future (event report) reports key observations from a workshop that took place in March 2013 and is very useful to provide important advice regarding systems approaches and complex environments where emergence matters very much, as is the case in social learning initiatives
 * Discussion paper: Innovations in monitoring and evaluating results by UNDP (2013) focuses on M&E innovations that encourage shorter feedback loops and are therefore more likely to help with complex processes where results are less obvious (e.g. social learning initiatives)
 * Monitoring and evaluating capacity building: is it really that difficult? by Nigel Simister and Rachel Smith (2010) is a Praxis paper by INTRAC looking at ways to assess capacity development at various levels (as means, process ends, inside out and outside in)
 * [|Duncan Green's blog: From Poverty to Power] : contains some interesting and relevant posts on learning at all levels, e.g. [|"When we (rigorously) measure effectiveness, do we want accountability or learning?"].

**Examples in practice**:
 * [|CGIAR Challenge Program on Water and Food Outcome Stories]: case studies on the impact of CPWF projects, with information about their M&E strategies (projects do not necessarily involve social learning).
 * [|Monitoring social learning processes in adaptive comanagement: three case studies from South Africa paper] by Cundill, G.: presents the results of an experimental research process that identified a small set of key variables that influence effective collaboration and learning, and tested a methodology for monitoring these in a collaborative way in three case studies in South Africa
 * [|CGIAR Research Program on Aquatic Agricultural Systems Monitoring and Evaluation Strategy Brief]: an M&E strategy with a co mponent focused specifically on M&E for learning.
 * Monitoring the composition and evolution of the research networks of the CGIAR Research Program on Roots, Tubers and Bananas (RTB) by Ekboir, J., Canto, G.B. and Sette, C. (2013).
 * Full paper: Series on Monitoring Research Networks No. 01. Rome, Institutional Learning and Change (ILAC) Initiative
 * ILAC Brief: ILAC Brief No. 27. Rome, Institutional Learning and Change (ILAC) Initiative
 * Developmental evaluation in practice : Developmental evaluation is one of the most relevant M&E approaches to social learning. These examples give a better idea of how such a methodology could be applied. The landing page of 'Developmental Evaluation' on the Better Evaluation portal offers other resources with examples.


 * Specific M&E methodologies and tools ** :
 * [|Developmental Evaluation book] by Patton, M.: describes the process of conducting developmental evaluations for ongoing program development. Developmental evaluation is a methodology developed by Michael Quinn Patton, which "is designed for adaptive and emergent interventions, such as social change initiatives or projects operating in complex and uncertain environments" and is therefore particularly well suited for social learning initiatives
 * Documenting change: an introduction to process documentation (2011) by Carmen DaSilva Wells et al.: a publication introducing 'process documentation' as a whole methodology (coming with concrete tools and methods) to document important areas (and assumptions) of a theory of change or of a complex initiative
 * Horizontal evaluation: stimulating social learning among peers by Graham Thiele et al. (2006) is a flexible evaluation method that combines self-assessment and external review by peers
 * Outcome Mapping is a methodology developed by IDRC and ODI, aiming at mapping specific outcomes, i.e. behaviour changes of 'boundary partners' that a given initiative intends to influence
 * Most Significant Change is the most prominent of change story techniques and was developed in the 2000s by Rick Davies and Jessica Dart. It helps collect a series of the most important changes that protagonists have observed
 * [|Participatory Impact Assessment: A Design Guide] by Tuft University's Feinstein International Center: an 8-stage approach to designing better impact assessments
 * Participatory Impact Pathways Analysis by Boru Douthwaite, et.al as used extensively in the design of the M&E system of the CGIAR Challenge Program on Water and Food
 * Reflexive monitoring in action by Barbara van Mierlo et al. (2010): this methodology also offers a set of tools that help monitor changes towards sustainable development of a sector or a region, using innovation systems
 * Social return on investment offers a method to calculate the return on investment of network interactions and as such can be useful to understand the value of social learning interventions in terms of the relationships developed among actors

4. Questions from the Social Learning Checklist
These questions/pointers from the Social Learning Checklist can help those already using social learning approaches to assess whether initiatives are addressing social learning as meaningfully and completely as they could/should.

This is about who decides what evidence is gathered and how: Q. What measures are being used to engage beneficiaries in monitoring? Q. What indicators are being used to determine whether the initiative is achieving transformational learning? Q. How will change be monitored and accounted for in an iterative manner? Q. What is the theory of change behind the outcome that is hoped for? Q. What indicators are being used to determine whether the initiative is socially and ecologically beneficial?
 * Monitoring Change**
 * How are beneficiaries defining their own indicators for successful change
 * To what extent are those with responsibility held accountable for initiating real world change?

This is to understand what parts of a given initiative were affected by social learning - to get a sense of scope and possible impact, geographically, thematically etc. Q. Who did the social learning approach initially affect Q. Did the social learning approach eventually expand
 * Evaluating Extent of Change**
 * One or more distinct geographic area (with no easy possibility to interact face-to-face)?
 * One or more team (as part of the people originally involved in this approach)?
 * A homogeneous or heterogeneous group of people (e.g. single discipline/type, multi-disciplinary, transdisciplinary?)
 * Scale up in time (affecting more people in the same area[s])?
 * Scale out in space (affecting more areas)?

Previous section: Iterative design Back to Framework & Toolkit home Next section: What to do with the evidence