Is it working?

Kate Graham

The Nuffield Foundation Commonwealth Programme’s mission is to support initiatives that bring about long-term improvements in health, education and civic justice in Eastern and Southern Africa. Change is at the root of each project – change at individual, community, organizational and societal level. But Nuffield wanted to know whether the projects were actually making progress towards effecting the desired changes.

How did projects know that their activities were changing things? For example, was training orthopaedic clinical officers for district hospitals in Malawi resulting in increased survival rates from accidents and more secure referrals?

In 2000, Nuffield changed its grantmaking strategy from offering a number of small grants each year to a biannual grant round, making four five-year grants of £250,000 each. Each was to a UK NGO or university, working with a local partner in Malawi, Uganda, Tanzania and South Africa.While accepting the risk involved in this new approach, the Nuffield Foundation sought at the same time to minimize it. Each grantee was asked to identify target outputs (ie activities delivered) at the halfway point so that progress could be reviewed. These mid-term assessments were both a process of due diligence and an opportunity for the implementers to get some objective input and review their original plans.

My role, two years later, was to carry out this review – visiting each to find out how they were getting on – and finding out how they knew that these activities were bringing them closer to their desired outcomes. I was interested in the assumptions people were making about how their activities contributed to the desired changes. People’s beliefs about change are often not explicit, despite their importance, and thus are difficult to test. It was surprising how many people assumed that if the planned activities were completed, the anticipated results would follow. They may do so, but isn’t it time to find out? And to budget for doing so?

A different kind of North-South partnership

Nuffield is relatively unusual in actively seeking partnerships between northern and southern NGOs where the northern NGO is providing more than money and monitoring. Unlike many funders, Nuffield values expatriate involvement – where it appears helpful.

This approach is typified by the grant to the Cambridge-based International Extension College (IEC). The project aim is to improve the quality of distance education provided by Kyambogo University for Primary School teachers in Uganda, and, in turn, the quality of classroom teaching. IEC’s expertise in distance learning was invaluable to Kyambogo, especially as many of the people involved have worked together in projects in Uganda over the last ten years. My impression was that this worked because of this history of cooperation and the trust and respect that had developed as a result. Kyambogo Distance Education team knew what they wanted and how to make the most of IEC’s expertise – and the IEC team in turn knew how to listen and propose accordingly.

IEC were engaged and committed, and offered very specific skills, in addition to the more frequent role of the northern ‘partner’ ie a ‘safe’ conduit for cash, a reporting procedure, expertise in monitoring and evaluation, and links to the outside world for dissemination. Some of the other partnerships offered the latter role only, where the added value of the northern partner was less easy to see.

Seeing the other side

The IEC/KU partnership above is both achieving and measuring change at multiple levels. The training they devised took 70 people (lecturers, administrators, local centre principals, etc) through an experience that placed them in the shoes of their students, and enabled them to see the other side of the distance learning coin. They struggled, suffered and supported each other. Virtually all completed the course, and were delighted to have done so, even the most cynical.

Participants said they had changed, in attitude and behaviour, as well as knowing much more about distance learning. Organizationally, they have made some significant operational changes, all indicating movement towards the project’s desired outcomes. In appraisals after the course, the students graded the trained lecturers considerably better than those who hadn’t been on the IEC course. The next stages will be to monitor student exam results and then classroom performance. So far, it seems to have worked beyond anyone’s expectations – and IEC/KU have the information to know this.

The South African project aims to change the treatment of women and children who have been raped and abused. The project ran a series of workshops bringing police, doctors, nurses, social workers, community leaders and prosecutors together for the first time and forced them to address deeply painful and often taboo issues. Individuals were clearly transformed by the experience. The challenge is to gain a critical mass in order to change institutional practice, but small steps count – like the policeman now calling a nurse, health workers now gathering the right evidence, doctors now recording the injury as domestic violence rather than just a broken arm. If they invest in tracking these changes, they will start to see the results of their work.

Evaluation: resource and capacity constraints

Asking people about measuring outcomes did meet with some resistance; in the case above, the NGO felt it only had the capacity to carry out the activities, not to evaluate the effect. Nuffield has invited them to apply for funding for a consultant to help them do this. The main problem seemed to be that people weren’t sure how to go about it – and consequently, had not budgeted for the time and resources needed to gather information systematically over time. Nor had Nuffield asked them to, a defect which it will remedy in future, by finding out earlier whether organizations have the capacity to develop monitoring systems themselves, and whether they have budgeted for this.

Nuffield had, however, believed in their desired outcomes – that was the ‘dream’ they bought into when supporting each grantee. One effect of the review was to make considerably more information about the project available, and a greater understanding of the values and vision of the grantees. This enabled the Nuffield staff to make decisions about the second parts of the grant with a ‘warts and all’ picture of each grant – something which can be quite uncomfortable, as of course none of them was perfect, nor quite as expected.

Round two

Did this change the actual decisions? Not in this case: the second stage of the grant was awarded in all four cases. However, the real learning will be applied in the next grant round, in two main areas. First, discovering where the ‘dream’ is held in the partnership – where is the vision and the aspirations? The review showed that in some cases, this was firmly with the overseas partner. In another, it was equally clearly held by the UK partner. In the IEC/KU partnership described above, the vision seemed to be jointly held – another reason for its success to date. So the extent to which there really is a shared vision beneath the rhetoric is something the Foundation will explore.

Secondly, we will ask from the beginning ‘how are you going to know throughout the project if this activity is contributing to the stated aim?’ We hope that this will make everyone think about change in a realistic way – which can still be excited and curious!

Looking at outcomes and at how each activity contributed towards the proposal aim made me wonder how far people really believe the aims they set out in their proposals. Are we all too used to big problems which we don’t really believe we can solve and therefore don’t feel we have the responsibility to do so? What would it be like if people put in achievable changes and we all got used to success? This isn’t saying we should lower our standards, but we should consider what we can do. For instance, we want to change society’s attitudes to rape and the South Africa project is a first step on the way, by increasing the skills and attitudes of service providers. Not quite so exciting maybe, but it’s what we can do right now. What can funders like Nuffield do to encourage this?

A learning process for funders

Despite the current trend towards advocacy, the Foundation remains interested in service delivery. It aims to increase the skills of the policymakers and people who provide effective public services in a rapidly changing world. What will make this succeed is an intriguing question. Old assumptions about the role of government in providing services and how things should be done are being increasingly challenged. How relevant are public service values to a rapidly privatizing country like Tanzania? How well equipped are UK NGOs to assist partners struggling to engage with the market? Are we working with the right people – might social entrepreneurs and local business people have more relevant skills to share?

The Nuffield Foundation wants to learn what UK NGOs offer as models of change for partners trying to improve services in Eastern and Southern Africa, and how other foundations have addressed these issues. It is especially interested in learning from other initiatives in what areas particular skills and expertise from the UK are required.

Kate Graham is a partner in Triangle Consulting and is a consultant to the Nuffield Foundation. She can be contacted at Kate@triangleconsulting.co.uk


Comments (0)

Leave a Reply

Your email address will not be published.



 
Next Analysis to read

Johns Hopkins first East Africa findings published

Andrew Kingman