Alliance Online - November 2007

Interview Fay Twersky
An internal critical friend for the Gates Foundation?

The December issue of Alliance will have a special feature on measuring impact, which will look critically at the range of approaches that are being taken. So how do they do it at the Bill & Melinda Gates Foundation, the world’s largest private foundation, which will soon be spending as much as $3 billion a year? In the run-up to publication of the Alliance special feature, Alliance talked to Fay Twersky, recently appointed to head up the Foundation’s new Impact Planning and Improvement Unit.

You have an interesting job title – Director of Impact Planning and Improvement. What does the Gates Foundation want to improve?

Bill and Melinda want us to learn from everything we do and that’s really the ethic here. They are willing to take risks and it’s our responsibility to make sure that we learn. So we are putting in place systems of measuring results and feedback loops that will help us to learn in a continuous fashion and then use that learning to improve our grantmaking practices, using our resources most effectively, driving towards impact.

What is the system of measuring results going to look like?

Well there is no one system. A good measurement system begins with good questions, and good questions emanate from good strategy. Our approach is to blend planning with measurement, learning and improvement, so Impact Planning and Improvement will be helping develop strategy, bringing evidence base to strategy, and crafting learning questions. We’ll then work in partnership with programmes and grantees to set up systems of measurement that provide us with answers to both our short-term and our long-term questions. It’s very important for us to have systems of measurement that we have evolved collaboratively with our partners and also to use that data to inform our decision-making and course direction.

It sounds as if your department will be involved in the planning of all the programmes at the foundation?

We will. We see ourselves as an internal consultancy within the foundation and in a sense as an internal critical friend. We will work with all the programme teams, asking them questions as they’re developing their theories of change, articulating the problems we are trying to solve, the strategies for doing so, and the short- and long-term outcomes and impacts.

Will programme beneficiaries be involved in designing these measurement systems at all?

Yes, and we know that we can do even better in this regard. One of the things that we’re trying to do is really listen to outside voices.

An example is our Avahan initiative in India, which is devoted to HIV prevention. We’re targeting people who are highest risk for HIV, many of whom are sex workers. The Avahan team involved our target beneficiaries in the programme development, and, when it came to designing the evaluation and measurement tools, we took survey instruments to them too, and they helped to craft better questions. So we’ve involved them in the programme design and in the measurement design, and the next step is to involve them in the analysis, to say, OK, what are the implications of these results and how might we be doing our work better?

When you’re implementing the results of the findings, do they then get asked again?

Yes they do. To stay with that example in India, there are a lot of data that are gathered across all the districts that the programme is working in and these performance data are reviewed on a regular basis. If there are things that jump out, then the team asks the clinics and the women who are outreach workers – because a lot of the programme is based on a pure outreach model – to talk about what’s going on in the particular clinic or community and to identify what we might do differently.

Is this more about informing foundation programmes as they’re running or about informing decision-making about future programmes or both?

Both. With a learning process, you have to be open to what you’re going to learn. One of the things we’ve done recently at the Foundation is adopt a formal annual review process, where we review the results of all our grantmaking programmes with our executive teams and with our co-chairs Bill and Melinda. We look at whether we achieved the results that we expected to see and whether we should continue on course as planned or make different decisions.

In our Pacific North-West programme, for example, we have an initiative called the Sound Families Initiative, focused on homeless families in Washington State. Seven years ago, we set a goal of building 1,500 transitional units, which we accomplished, and we helped more than 2,000 families move out of the streets and shelters into transitional housing. But we realized that, in spite of that, we haven’t solved the problem – overall homelessness was still increasing as was family homelessness. Around 40 per cent of the homeless people in Washington State are individuals living in families. But we are using what we’ve learned to continue to tackle the problem and we’re developing a new strategy for trying to end family homelessness in Washington State.

And where things aren’t going well, will you be able to make sure that you learn from failures?

Warren Buffett said a few months ago when he was here that if we don’t fail, if we hit it out of the park every time, then we’re not going after the right problems. We’ve set a tone at the Foundation that we want to address some of the world’s greatest inequities, some of the most intractable problems, that’s why we’re here. We’re going to make some big bets, we’re going to take some risks, and some of the time it’s going to be a great success. But other times, a fair amount of the time, we’re not going to succeed at first go, but we’re going to stick with it because we’ve chosen a limited set of problems and we’re going to continue to fail and learn and fail and learn until we succeed.

The other thing to add is that we’ve put in place these formal systems, including our annual reviews, as a way of forcing ourselves to ask some critical questions. Where did we get to this year in our malaria work, for example, and where did we not? And what do we need to do differently or more of next year?

You’ve described how you share results of evaluations and what you find with your immediate stakeholders. Do you have plans to share these results with others funders?

Yes! We recognize that we are in a place of incredible privilege in being able to make big bets to try to solve these important social problems, and we’ve also recognized that with that privilege comes responsibility to share what we’ve learned honestly and directly. We’re revamping our website and, as part of that process, we’re going to be very focused on publishing our evaluations and reports, making our work more public and accessible. Of course, it is not just about what we share but also what we learn from others. We recognize that there’s a lot of good work that has come before us that informs our thinking.

It seems to me that there isn’t really an infrastructure – such as a web-based system – for easy sharing between different foundations. Do you think the Gates Foundation might in the long term be willing to help build that sort of infrastructure?

I hope that we will contribute to the field of evaluation and learning. It’s not a direct part of our mission, but we recognize its importance, and sometimes we have put ourselves in the role of building data infrastructure. For instance, we’re now one of a few funders who are funding the International Initiative for Impact Evaluation (the 3IE work), which will be doing independent, very rigorous impact evaluation to help identify what are the most effective development projects. We see data infrastructure and building an evidence base for what works as a kind of public utility and the 3IE is a good example of that.

With the Avahan example, the Avahan team found we needed to create some of the data infrastructure before we could collect and analyse the data we were interested in. Even in the US, in our work in public education, there are so many school districts and they all collect data in slightly different ways, so really understanding the effects of different educational reforms is challenging.

Going back to your question about an online system for sharing information, there have been a few efforts in the sector that I know of – some groups have put out a call to ‘send us your reports’, ‘send us your lessons’ – but there’s been no robust effort to build a real infrastructure as far as I know. I would be interested in something like that happening and I hope that we could contribute to it in some way.

Given the scale of your grantmaking and your commitment to working in depth, isn’t the Gates Foundation in a unique position to take a lead on that?

It is an area to which I hope we can make a contribution. I don’t know that we would take a lead in building such an information repository. We are a young foundation, our Impact Planning and Improvement unit is one year old and I’ve been in this position for six months, so I think it’s something for us to consider down the road.

Do you think there is any danger that implementing measurement systems and collecting data could be a burden to your grantees, or maybe even to the programme beneficiaries?

I think sometimes funders don’t recognize the tremendous burden that measurement can place on organizations. I think the highest leverage way to avoid undue burden is to make the measurement meaningful to grantees and meaningful to the people that we are ultimately trying to reach. What that means is that we have to involve our grantees in the design to make sure that the measurement is also answering questions that are fundamentally important to them. It means that we need to provide adequate resources because measurement is never cost-neutral and we need to engage our grantees in learning together about the results and identifying implications for practice.

So the two key aspects are making sure measurement is meaningful and making sure the costs are covered?

Meaningful and costs covered, but also that we’re in a partnership and that we’re learning together. We’ve just launched a design process with our partners in the education division in the US Program to conduct a rigorous evaluation of Charter Management Organizations. We’ve given a grant to one of our key partners and they’re convening a group of Charter Management Organizations, researchers and experts, school developers and practitioners and other funders to articulate the questions that we have about Charter Management Organizations, to select an evaluation vendor and to collectively guide the data collection, analysis and interpretation processes. We want to make sure that everybody’s questions and interests are in the pot and that the design is made, and lessons learned, together. We are then in a much better position to move the field forward efficiently, drawing upon a common evidence base.

Are external evaluations part of your approach, or are they all initiated and carried out within the Foundation and your partners?

Many of our evaluations are external. We have external evaluation partners and often use an external advisory board for evaluations. In the Charter Management Organizations project, a third party evaluator is going to be selected so that we ensure a true independent assessment of that body of work.

Do you see any danger that the need to measure might limit the innovativeness of what funders are prepared to undertake?

I think the key is to strike the right balance; we want to be both innovative and effective. We want to explore new ideas and also be accountable stewards. What that means in practical terms is that at the beginning of a new initiative, often there’s a lighter touch approach to measurement, where the evaluation work really needs to facilitate the programme’s development. If an idea takes root and the model matures, it’s time for a more rigorous approach and to go deeper and look at whether our initiatives are having a meaningful and sustainable impact. Again, it really goes back to making sure that measurement is the right size, that it aligns with the strategy. If it is, if strategy is well integrated with measurement and learning, then it is less likely to limit the innovativeness because it’s right there in step with the innovation.

What are the promising new ideas in the field of impact planning and improvement?

It’s interesting that you call it a field! There is a field of evaluation and there is a field of planning, I’m not sure there is a field of improvement. There probably should be! What we’re trying to do here is really blend a few different concepts and disciplines to support an integrated effort of planning, measurement and improvement. That said, I think there are some new ideas in the field, and we’re trying to learn from what many others are doing and incorporate them into our work. We’re looking at the work that groups like the Foundation Strategy Group and Mission Measurement have done around more timely performance metrics. I think it’s important to gather more timely data and use it in ongoing regular management to understand where are we most productive and where are we meeting our benchmarks.

I think that there are interesting ideas percolating around social return on investment and blended value measurement. Some of the work the Center for Effective Philanthropy (CEP) is doing on grantee perception surveys, and other surveys that allow foundations to benchmark performance against the performance of their peers, has been a helpful addition to the field. We foundations don’t always hear what our grantees are saying, and CEP has given the sector a window into those voices. Some of David Bonbright’s work on constituency feedback loops is good and important and we’re drawing on that too in some of our global work in agriculture. Here in the US in our education work, we are doing more and more of thinking about how we hear from students directly, how we hear from parents, and David’s work has been a catalyst for us in that regard.

We’re also taking some of the emergent learning work that originally came out of the military and applying it to the sector, and it’s been helpful for harvesting knowledge that resides in people’s own experiences and generates more real-time feedback and actionable ideas. There are other movements towards more robust impact studies, like the 3IE, that will design robust evaluations for development projects and provide evidence to guide funding and policy decisions. Yet, we also know that some of the best work in the social sector is developmental in nature and not necessarily meant to be fixed – what does that mean in terms of making sure that evaluation and measurement is integrated into development so as to be constantly feeding back what’s working and what’s not working?

All these different ideas are moving the field in a way that I think is productive and we’re trying to learn from and draw on all of them.

So keeping up with all this new thinking is a big part of your job?

Yes, but not the whole of it! I think that some of the best ideas still are old ones that typically aren’t very well or consistently done, and which we’re trying to do better – thinking about evaluation and measurement early; integrating evaluation, planning and strategy development – I think that the theory of change is the principal tool for that. Also, making sure that evaluation findings are available in accessible language and that they’re not so technical that lay people can’t read and understand them. And, as we said earlier, involving stakeholders in the processes of planning for impact, measuring results, and learning from the results. The process is often as important as the product in making sure that we’re involving our partners in constructive and generative ways.

We’re not called the evaluation unit, but the Impact Planning and Improvement unit, because we’re about planning, measuring and improving, and integrating those is key, and not necessarily easy to do. Often, in our fields you find folks who may be very good at measurement but not necessarily good at planning, or people who know something about learning but don’t know anything about data, so trying to blend those three in an intentional way is a struggle, which is why it hasn’t always been done consistently. But I think it has such tremendous benefits when you can do that, and that’s our aim.

If you were called the evaluation department, would that suggest more a process after the programme is over? I know the expression is often ‘monitoring and evaluation’ but nevertheless there might be more of an emphasis on evaluating after the event rather than being in there from the beginning.

If you polled people in the sector on their feeling about monitoring and evaluation, my guess is that you wouldn’t find too many cheerleaders! People tend to have pretty negative experiences with monitoring and evaluation, and often with good reason – because of the burden it places on folks, because they haven’t been involved in the process, because they aren’t supported properly to do it, because it can take resources away from the mission.

And it’s not always about improvement. People have sometimes experienced evaluation as being about judgement and punishment. So the idea of approaching the planning and measurement work for the purposes of learning and improvement is a variation on the traditional theme of evaluation, but an important variation – one that is about adding value and aiding decision-making rather than retrospective or detached from implementation realities.

Even if it’s not for punishment, there’s often a feeling that evaluation reports are just shelved and not used for anything.

That’s right. I used to talk about people putting evaluation reports in the evaluation library in the sky, where things get checked in but hardly anything ever gets checked out! That’s so much of what happens in our sector. So we have to make sure that those reports are read, that the data are reliable, that they are written in a language that is accessible, and that people will engage with the results and ask ‘what does this mean to our work?’ People are out in the field doing the work because they want to make a difference, so the evaluation has to help them make the best difference they can.

Fay Twersky is Director of Impact Planning and Improvement at the Bill & Melinda Gates Foundation. Email Fay.Twersky@gatesfoundation.org

Click here to send this article to a friend

Why should I subscribe to Alliance?

Subscribe now to Alliance magazine and receive:

4 issues of Alliance to your door
8 issues of Alliance Bulletin to your inbox
Access to all back issues (since 1998) in PDF format
Click here for more information or call +44 20 7239 1405.