Monitoring, Evaluation, Holidays and Poetry

I recently received a holiday greeting from a colleague working in monitoring and evaluation (Kylie Hutchinson of Community Solutions). The poem she wrote is brilliant and I want to share it with you!

It Came From Using a Systems Lens
It came from using a systems lens,
Such insights so glorious – behold!
This program in its entirety,
Is really a part of a whole.
So often when we evaluate,
We think only outputs, outcomes,
And fail to see the complexity,
The system from which it comes.

Consider the interactions,
Within and between system parts,
They help or hinder the outcomes,
Ignoring them isn’t smart.
Look at the interrelationships,
Of policies, patterns, and norms,
These interact with the program,
And impact how it performs.

Three interconnected elements,
Can help show you where to begin,
They’re Boundaries and Relationships,
And Perspectives within.
A deeper look at these elements,
Will open up more leverage points,
If you are looking for system change,
These actions won’t disappoint.

But how to choose what to include?
What’s evaluated or not?
Your framing is quite critical,
Requiring some thought.
Be conscious of what is left out,
Think of the three factors above,
And how they impact your outcomes,
The influences thereof.

Why do we bother with systems?
When logic models are so nice?
Well life is not really linear,
They clearly can’t suffice.
We’re living in a complex world,
Simple answers rarely exist,
It’s only through complexity,
That we can learn what we missed.

And this is a great opportunity to wish friends, family and colleagues a very happy holiday and New Year.  I have many blog posts turning in my mind and will work on getting them posted in the near future.  Namaste from Kathmandu!

My Harvest from a Think Tank on Assessing Collaboration

The International Institute for Sustainable Development (IISD) has worked for over a decade on research and capacity building for performance assessment and improvement of collaboration in order to help partnerships partner, alliances ally and networks work.

Our most recent paper was presented at the Canadian Evaluation Society Conference in Toronto (June 2013) via a Think Tank session and provided preliminary reflections on performance improvement and assessment of groups of individuals working together in networks and communities of practice.

The session started with a summary of our initial thinking and was followed by a World Café brainstorm to help further our collective understanding using the knowledge in the room.

My introduction covered four main points:

1. Collaboration and knowledge sharing are increasingly viewed as essential for achieving the sustainable development agenda.
2. Don’t get caught in the tangle of terminology between learning networks, knowledge networks, CoPs.
3. Do distinguish between:
   1. Multistakeholder, multi-sectoral, networked governance process
   2. Relationship among group of organizations
   3. Collaboration between individuals
4. Once the distinction made, determine:
   1. Focus and extensiveness
   2. Emergent structure
   3. Social capital
   4. Desired outcomes and value creation of the collaboration

The full presentation via powerpoint is here.

The paper can be downloaded here.

World Café is a participatory technique I use to host meaningful dialogues with groups.  You can learn about the technique here if you are interested.

The theme of our café was: Assessing Individual Purposeful Collaboration. People were seated at tables of 4-5 people and we had three quick rounds of discussion, each with a different question.  After each round, participants got up and moved tables so we cross-pollinated ideas across diverse groups of individuals.  One person remained at the table as the host and was responsible for documenting key messages from the conversation.

Visual Summary of our Think Tank Session: Performance Improvement & Assessment of Collaboration:
Starting Points for networks and CoPs

Key messages were then placed on a visual template and were clustered by emerging themes.  At the end of the café, we gathered around the landscape created and reviewed what we created.  The messages are highlighted below as a reminder for participants and as a way to share our learning with those who were not able to attend but are interested in this topic.

Question 1: What constitutes success in collaboration?  What are indicators of that success?

(These are the green trees on the landscape)

Themes that emerged:

°    Relationships (sense of belonging, trust)

°    Participation (engagement, culture of knowledge sharing individuals, motivation)

°    Diversity of knowledge exchange

°    Agreement on objectives (commitment to shared purpose)

°    Sustainability

 

Question2: What are the potential challenges evaluators should be aware of?

 (These are the grey boulders in the river)

Themes that emerged:

°    Measurement:

• Inadequate assessment tools
• How to measure contribution, attribution and social capital
• Agreement on what to measure
• Nonresponse and selection bias

°    Frameworks:

• Finding a framework inclusive of diverse groups and diverse knowledge bases
• Changing goals and visions, shifing focus

°    Group and organizational dynamics

• Competing agendas, conflicting interests
• Culture that doesn’t support collaboration
• Lack of trust
• Changing membership, turnover

°    Capacity:

• Lack of skills, time, resources

 

Question 3: What ideas do you have to strengthen this area of practice?

(These are the yellow and orange clouds in the sky)

Themes that emerged

°    Establish shared vision and measures at outset:

• Governance structure
• Common understanding of goals
• Clear outputs and outcomes
• Develop measurable indicators

°    More reflective practice throughout network life cycle:

• Evaluator actively engagement with the network/CoP
• Developmental evaluation techniques
• Resources to undertake reflection (time and money)
• Generate interest among participants

°    Measurement:

• Use a proxy such as common elements with other CoPs, identify value-added
• Assess the dynamics of the group (the individuals)
• Track online data (usage/number of hits)
• Assess for alignment of purpose vs. structure
• Use the spectrum of engagement (observer-participant-leader)
• Test via small group experiment/ pilots

°    Improved information sharing and networking tools:

• Data repository
• Open sharing

Overall, there is a need for more data collection and research on this topic.

Thank you to everyone who participated in the session and generating ideas for further study.  It was a great experience to dive deeper into the topic of  improving the performance and ability to assess collaboration with a group of such passionate people.  Please keep in touch!