This section can help you get started with the monitoring and evaluation (M&E) of your social learning approach, or to improve your existing monitoring strategy (from a social learning perspective). As an important element of an iterative, adaptive management approach, M&E should be carried out continually and inform actors and decisions in short feedback loops, to facilitate looped learning. In the context of social learning, monitoring should be as social as other aspects of a project’s approach, involving all actors in a project.
This section acts as a portal through which you can explore different monitoring approaches and tools, as well as examples of their successful implementation for social learning initiatives (work in progress).
The CCSL team has recently developed a common M&E framework for new projects and programs using a social learning approach, with the aim of being able to systematically collect evidence on social learning's added value in the context of climate change adaptation and food security. The framework has been published as CCAFS Working Paper #98, 'Monitoring and Evaluating Social Learning: A Framework for Cross Initiative Application'. It provides a theory of change for social learning and 30 key indicators spread across four monitoring areas: iterative learning, engagement, capacity development and challenging institutions. These four monitoring areas, which form the basis of the framework, build on those selected by participants of the June 2014 Social Learning Evidence Gathering workshop in London. In 2015 a minimum of 5 projects will receive peer assistance from the CCSL Initiative in order to pilot the framework; if you wish to join the pilot phase please get in touch.
If you are developing your own project- or programme-specific M&E framework, the information on the rest of the page may be useful:
Establishing impact pathways and a theory of change Theories of change may be a familiar tool. Establishing a theory of change is an important first step in building an M&E framework in many contexts, and can be useful in a social learning approach as well. A theory of change maps all the changes that a given intervention hopes or expects to have. It also highlights 'impact pathways' (how project/program activities or "inputs" lead to the outputs, outcomes, and eventually impacts desired, given the context and the people/groups involved). In this sense, it is essential to visualize the effects that a social learning approach is meant to bring about. Crucially, a theory of change also helps you map out the assumptions and risks involved in a (social learning) approach. It is often helpful to have a visual representation of your theory of change and to consider revisiting that theory of change, impact pathway(s) and set of assumptions and risks that guide the initiative. The conversation that leads to developing the theory of change and impact pathway is arguably more important than the actual map that results from it. Read some recent experiences of CCAFS in developing impact pathways and theory of change.
Guidance on impact pathways and theories of change The processes around your impact pathways and/or theory of change are as important in a social learning approach as the tools themselves. Often, these tools are created in a one-sided process that does not involve stakeholders and then abandoned once the project begins. In a social learning approach, these tools should reflect the social, flexible, and iterative nature of social learning (as is the case with participatory impact pathways - see below). Some topics and questions to reflect on, in this respect:
Process: Are your M&E tools created through a collaborative process? Which stakeholders are involved or excluded?
Iteration: Are your M&E tools dynamic, living documents - are you revisiting them periodically to reflect, learn and revise them?
Key areas of evidence to collect
Traditional M&E approaches look for evidence of whether a project/program's approach is resulting in the changes set out in the theory of change. In the context of a social learning approach, social learning goals can be built into the theory of change, and subsequently monitored along with other desired outputs, outcomes, and impacts. For instance, you may want to collect evidence in any of the following areas:
Intent: the discourse and its manifestations (in documents, meetings, official communications etc.) about purposefully using social learning and the results of having this intent
Looped learning: mechanisms in place to ensure reflection, learning and iteration and success or failure of these mechanisms; examples of looped learning, adaptation and flexibility
Social differentiation: makeup of groups impacted by social learning (homogenous/heterogenous) and how this is being addressed successfully or not
Power: power relations between stakeholder groups and how this is being addressed successfully or not; impact of social learning on these relationships (social transformation)
Capacity: ways in which stakeholder capacity (particularly to contribute to successful social learning through e.g. process facilitation, interpersonal communication, empathy, listening, teamplay etc.) is being identified, assessed, integrated, and built (or not), and the effect of this on the wider process
Institutional opportunities and barriers: elements of institutional systems, governance and culture that are helping or hindering social learning
Endogenous processes: processes and policies in local government that can be linked to or integrated with
Scaling up and out: changes over time in the size of the area and number of people impacted
Some other areas of more traditional M&E can be useful in monitoring social learning:
Relationships: the quality of partnership development, and the way specific actors and 'boundary partners' may be influenced by the process. Associated methodologies include Social Network Analysis.
Knowledge and learning: intangible assets - broader than the 'looped learning' mentioned above.
Outcomes, change and impact at scale: outcomes and changes in behaviour, discourse, practice, policy and the livelihoods of people. Associated methodologies include Outcome Mapping, stories of change such as Most Significant Change, etc.
Evidence gathering tools A number of different tools have been used to collect and analyze evidence for the M&E of social learning. To make sense of the wide range, we have grouped tools by type of evaluation (process/outcome) and scale (individual/network/system):
Scale (right)
Inquiry type (below)
Individual
Network
System
Process evaluation
Participant observation / diary
(pre/post questionnaires) Knowledge / Attitude / Skills surveys and e.g. analysis of the transformation of participants’ narratives
Stories of change
Appreciative Inquiry
After action reviews, community self-assessments etc.
Formal and informal observations, focus group discussions and e.g. post-workshop discussions
Social network analysis
Stories of change
Outcome Mapping
Appreciative Inquiry
Participatory Mapping
Social network analysis
Stories of change
Outcome Mapping
Outcome evaluation
(pre/post questionnaires) Knowledge / Attitude / Skills surveys and e.g. analysis of the transformation of participants’ narratives
Stakeholder portraits and follow up interviews
Psychologists’ evaluation of the actors’ experiences
Stories of change
AppreciativeInquiry
Knowledge gained by e.g. farmers and their plan to apply it
Post-workshop survey to assess attitudes
M&E by the community of their actions
Mixed method/triangulation approach used to collect data
Amount of change in behaviour, attitude, skills, knowledge or condition (situation) of programme participants
Environmental stress reduction indicators and environmental status indicators that measure actual success in environmental outcomes
Social network analysis
Social return on investment
Stories of change
Outcome mapping
Appreciative Inquiry
Policy change assessment (based on e.g. citation analysis)
Social network analysis
Stories of change and e.g. SenseMakerTM narrative filtering
Developmental evaluation
This table has been adapted from CCAFS Working Paper #38, “Social learning in practice: a review of lessons, impacts and tools for climate change adaptation”.
Throughout, methodologies such as 'Process documentation' and 'Reflexive monitoring' are applicable to all cells of this table.
Guidance on gathering evidence
While evidence collection does not necessarily need to be a participatory process, the analysis of that evidence needs to lead to social learning. Even if your project's monitoring efforts are participatory, they may not facilitate genuine social learning--it is important to make the distinction between participation and social learning. For more ideas on how to proceed, please see the next section of the Framework and Toolkit, What to do with the evidence.
3. Resources for gathering evidence
General resources about assessing social learning:
CCAFS Working Paper 38, "Social learning in practice: A review of lessons, impacts and tools for climate change adaptation": Appendix 2 contains information on the evaluation approach taken in 5 social learning case studies
Creating robust evidence through knowledge integration (by Wenny Ho): this paper explains how the integration of different knowledge perspectives (using double-loop learning and dealing with power) to provide more robust evidence of complex change at play
Learning to learn: designing monitoring plans in the Pacific Islands International Waters Project paper by Mahanty S., et al.: outlines the project monitoring approach developed in the Pacific Islands International Waters Project (IWP) and finds that a monitoring approach based on indicators to assess supportive processes, behavioural change and human–environmental conditions is useful for monitoring the long- and short-term impacts associated with integrated coastal management programs.
Measuring Social Learning in Participatory Approaches to Natural Resource Management (academic paper) by van der Wal, M. et al.: this paper proposes a method for quantitatively and qualitatively measuring social learning, using a perspective scoring table based on cultural theory perspective categories. Though this method is not presented specifically in the context of climate change adaptation and food security, it may prove useful or could be adapted.
Monitoring and evaluating learning networks by Rick James (2010) is another INTRAC Praxis paper which considers ways to assess the value of learning networks through e.g. learning activities, engagement, learning outputs, outcomes and impact, relationships between members etc.
Better Evaluation is a general portal on Monitoring and Evaluation that highlights a number of these methods and approaches, and probably the best place to find information related to monitoring and evaluation in general, hence with a lot of relevant information for social learning too
Impact, innovation and learning: Towards a research and practice agenda for the future (event report) reports key observations from a workshop that took place in March 2013 and is very useful to provide important advice regarding systems approaches and complex environments where emergence matters very much, as is the case in social learning initiatives
Discussion paper: Innovations in monitoring and evaluating results by UNDP (2013) focuses on M&E innovations that encourage shorter feedback loops and are therefore more likely to help with complex processes where results are less obvious (e.g. social learning initiatives)
Monitoring the composition and evolution of the research networks of the CGIAR Research Program on Roots, Tubers and Bananas (RTB) by Ekboir, J., Canto, G.B. and Sette, C. (2013).
Full paper: Series on Monitoring Research Networks No. 01. Rome, Institutional Learning and Change (ILAC) Initiative
Developmental Evaluation book by Patton, M.: describes the process of conducting developmental evaluations for ongoing program development. Developmental evaluation is a methodology developed by Michael Quinn Patton, which "is designed for adaptive and emergent interventions, such as social change initiatives or projects operating in complex and uncertain environments" and is therefore particularly well suited for social learning initiatives
Documenting change: an introduction to process documentation (2011) by Carmen DaSilva Wells et al.: a publication introducing 'process documentation' as a whole methodology (coming with concrete tools and methods) to document important areas (and assumptions) of a theory of change or of a complex initiative
Outcome Mapping is a methodology developed by IDRC and ODI, aiming at mapping specific outcomes, i.e. behaviour changes of 'boundary partners' that a given initiative intends to influence
Most Significant Change is the most prominent of change story techniques and was developed in the 2000s by Rick Davies and Jessica Dart. It helps collect a series of the most important changes that protagonists have observed
Participatory Impact Pathways Analysis by Boru Douthwaite, et.al as used extensively in the design of the M&E system of the CGIAR Challenge Program on Water and Food
Reflexive monitoring in action by Barbara van Mierlo et al. (2010): this methodology also offers a set of tools that help monitor changes towards sustainable development of a sector or a region, using innovation systems
Social return on investment offers a method to calculate the return on investment of network interactions and as such can be useful to understand the value of social learning interventions in terms of the relationships developed among actors
4. Questions from the Social Learning Checklist
These questions/pointers from the Social Learning Checklist can help those already using social learning approaches to assess whether initiatives are addressing social learning as meaningfully and completely as they could/should.
Monitoring Change
This is about who decides what evidence is gathered and how:
Q. What measures are being used to engage beneficiaries in monitoring?
How are beneficiaries defining their own indicators for successful change
Q. What indicators are being used to determine whether the initiative is achieving transformational learning?
To what extent are those with responsibility held accountable for initiating real world change?
Q. How will change be monitored and accounted for in an iterative manner?
Q. What is the theory of change behind the outcome that is hoped for?
Q. What indicators are being used to determine whether the initiative is socially and ecologically beneficial?
Evaluating Extent of Change
This is to understand what parts of a given initiative were affected by social learning - to get a sense of scope and possible impact, geographically, thematically etc.
Q. Who did the social learning approach initially affect
One or more distinct geographic area (with no easy possibility to interact face-to-face)?
One or more team (as part of the people originally involved in this approach)?
A homogeneous or heterogeneous group of people (e.g. single discipline/type, multi-disciplinary, transdisciplinary?)
Q. Did the social learning approach eventually expand
Scale up in time (affecting more people in the same area[s])?
CCSL Framework and Toolkit
Documentation and data collection
This section can help you get started with the monitoring and evaluation (M&E) of your social learning approach, or to improve your existing monitoring strategy (from a social learning perspective). As an important element of an iterative, adaptive management approach, M&E should be carried out continually and inform actors and decisions in short feedback loops, to facilitate looped learning. In the context of social learning, monitoring should be as social as other aspects of a project’s approach, involving all actors in a project.
This section acts as a portal through which you can explore different monitoring approaches and tools, as well as examples of their successful implementation for social learning initiatives (work in progress).
Table of Contents
1. M&E Framework
The CCSL team has recently developed a common M&E framework for new projects and programs using a social learning approach, with the aim of being able to systematically collect evidence on social learning's added value in the context of climate change adaptation and food security. The framework has been published as CCAFS Working Paper #98, 'Monitoring and Evaluating Social Learning: A Framework for Cross Initiative Application'. It provides a theory of change for social learning and 30 key indicators spread across four monitoring areas: iterative learning, engagement, capacity development and challenging institutions. These four monitoring areas, which form the basis of the framework, build on those selected by participants of the June 2014 Social Learning Evidence Gathering workshop in London. In 2015 a minimum of 5 projects will receive peer assistance from the CCSL Initiative in order to pilot the framework; if you wish to join the pilot phase please get in touch.
If you are developing your own project- or programme-specific M&E framework, the information on the rest of the page may be useful:
Establishing impact pathways and a theory of change
Theories of change may be a familiar tool. Establishing a theory of change is an important first step in building an M&E framework in many contexts, and can be useful in a social learning approach as well. A theory of change maps all the changes that a given intervention hopes or expects to have. It also highlights 'impact pathways' (how project/program activities or "inputs" lead to the outputs, outcomes, and eventually impacts desired, given the context and the people/groups involved). In this sense, it is essential to visualize the effects that a social learning approach is meant to bring about. Crucially, a theory of change also helps you map out the assumptions and risks involved in a (social learning) approach. It is often helpful to have a visual representation of your theory of change and to consider revisiting that theory of change, impact pathway(s) and set of assumptions and risks that guide the initiative. The conversation that leads to developing the theory of change and impact pathway is arguably more important than the actual map that results from it.
Read some recent experiences of CCAFS in developing impact pathways and theory of change.
Guidance on impact pathways and theories of change
The processes around your impact pathways and/or theory of change are as important in a social learning approach as the tools themselves. Often, these tools are created in a one-sided process that does not involve stakeholders and then abandoned once the project begins. In a social learning approach, these tools should reflect the social, flexible, and iterative nature of social learning (as is the case with participatory impact pathways - see below). Some topics and questions to reflect on, in this respect:
Resources:
2. M&E Toolkit
Key areas of evidence to collect
Traditional M&E approaches look for evidence of whether a project/program's approach is resulting in the changes set out in the theory of change. In the context of a social learning approach, social learning goals can be built into the theory of change, and subsequently monitored along with other desired outputs, outcomes, and impacts. For instance, you may want to collect evidence in any of the following areas:
Some other areas of more traditional M&E can be useful in monitoring social learning:
Evidence gathering tools
A number of different tools have been used to collect and analyze evidence for the M&E of social learning. To make sense of the wide range, we have grouped tools by type of evaluation (process/outcome) and scale (individual/network/system):
Inquiry type (below)
Individual
Network
System
Throughout, methodologies such as 'Process documentation' and 'Reflexive monitoring' are applicable to all cells of this table.
Guidance on gathering evidence
While evidence collection does not necessarily need to be a participatory process, the analysis of that evidence needs to lead to social learning. Even if your project's monitoring efforts are participatory, they may not facilitate genuine social learning--it is important to make the distinction between participation and social learning. For more ideas on how to proceed, please see the next section of the Framework and Toolkit, What to do with the evidence.
3. Resources for gathering evidence
General resources about assessing social learning:
General resources about M&E:
Examples in practice:
Specific M&E methodologies and tools:
4. Questions from the Social Learning Checklist
These questions/pointers from the Social Learning Checklist can help those already using social learning approaches to assess whether initiatives are addressing social learning as meaningfully and completely as they could/should.Monitoring Change
This is about who decides what evidence is gathered and how:
Q. What measures are being used to engage beneficiaries in monitoring?
- How are beneficiaries defining their own indicators for successful change
Q. What indicators are being used to determine whether the initiative is achieving transformational learning?- To what extent are those with responsibility held accountable for initiating real world change?
Q. How will change be monitored and accounted for in an iterative manner?Q. What is the theory of change behind the outcome that is hoped for?
Q. What indicators are being used to determine whether the initiative is socially and ecologically beneficial?
Evaluating Extent of Change
This is to understand what parts of a given initiative were affected by social learning - to get a sense of scope and possible impact, geographically, thematically etc.
Q. Who did the social learning approach initially affect
- One or more distinct geographic area (with no easy possibility to interact face-to-face)?
- One or more team (as part of the people originally involved in this approach)?
- A homogeneous or heterogeneous group of people (e.g. single discipline/type, multi-disciplinary, transdisciplinary?)
Q. Did the social learning approach eventually expandPrevious section: Iterative design Back to Framework & Toolkit home Next section: What to do with the evidence