Hi 'M&E is cool' - thanks for your contributions here. Some interesting new ideas and perspectives.  

What do others think of these ideas? What about...:

• Could a simplified version of the 'Performance Story Report work? (see http://www.nrm.gov.au/publications/books/pubs/performance-story.pdf)
• What about ways of getting good M&E info like mixing text and online reporting, with talking to the people involved?
• How do you capture the context and honour the work being done on your programs? Do you find the ongoing and evolving story helps?
• Do you also start with an exit strategy in mind... to identify who will continue the work and manage dependence on repeat funding?

Please post up any ideas or comments you like in this final day before voting starts...

Also, further to starting with baseline and outcomes in mind, it is so important to also start with an exit strategy from the outset.  To identify who will continue the work into the future (where relevant) and to manage dependence on repeat funding.

While I understand the need for measurement of some key (clear, simple) outputs, in my experience (environment programs – govt and ngo) monitoring and evaluation is required to evaluate what the funding/project has achieved and to learn from the projects experience to improve in the future.  In the programs I have worked on, what I need to see to achieve this is the story, the ongoing and evolving story, that gives context and tells me where we are, what we have achieved and where we need to go next.  To capture the context and honour the work being done. If such an evaluation could be combined with latest advances in effective aid/program delivery to plot the path forward I think it would be ideal.  

 

I think text and online reporting is fantastic (who uses paper, honestly). Another really great way to get good M&E information is to just talk to the people involved, by phone or in person, and find out what’s going on.  I have always found people are incredibly honest, helpful and so happy to hear that someone cares about what they are doing and how it is going.  In my view this is worth every second. I’ve managed projects from a distance and I think it is possible, but this kind of contact definitely helps.

 

I wouldn’t recommend an approach this complex, but you might be interested in this approach to M&E which mixes quantitative and qualitative info: http://www.nrm.gov.au/publications/books/pubs/performance-story.pdf. Maybe a simplified version could be workable?

Thanks, John, for your more recent comments here.  

Would people like to share any further comments on monitoring aid projects?

Please feel free to drop in a line, a referral, a link, a story, or any ideas you might like to share here over the next few days.... We welcome your ideas.

Some interesting discussion here, points of agreement

• Aid is a frustrating, risky and worthwhile activitiy, AusAid should communicate this to the electorate, who are a key ultimate supporter (Diplomats, traders and the military are others)
• M&E is similarly difficult, expensive and often misleading as nobody likes to be put in the position where they may be criticised so good baselines are often not established at the start and the data collection poor, almost nobody gets it right based on 30 years looking at these things in most IFIs Development agencies and many bi lateral agencies 
• Participatory M&E does not work in many cultures, quote from one former command economy  official "the stakeholder is the person who hangs onto the stake while you shoot him"  
• M&E does tend to focus on outputs and process, this is unfortunate but understandable as accountability is fundamental to it continuing
• It should devote much more attention to outcomes and the wider context and this is very difficult , periodic qualitative evaluations by experience people is one solution, there is a GEF system that focusses on reassessing goals, effectiveness effciency and cost effectiveness (the Mid term evaluations system, this can be found on UN web sites, I think this source may be useful, but I have not read this new publication  http://www.undp.org/evaluation/handbook/
• there should also be more emphasis on the use of M&E for adaptive management by participants
• evaluation in areas of poor security is very difficult, needs the most experienced people and more open TOR (something AuAid is not good at!) 

Hi Tony Voutas, Thanks for sharing your insights.  

What do others think about placing M&E in the wider context? Do you agree or disagree this should be part of designing an M&E framework, and also a means to ensure we don't ignore the impact of other factors (that could lead us to falsely attribute "success" to the project)?

Do you have examples of M&E mechanisms where the poor have anonymously and freely expressed their views on sub-projects?

It would be interesting to hear your views.

After years of M&E involvement, my feeling is that we have to relegate standard M&E model processes to guidance only status.  The wider context in which a project is operating should be a major factor in designing an M&E framework for a specific project.

There is a case for placebo areas being included within a project area to more accurately ascertain with and without project  results

While community participatory M&E is laudatory in terms of Western democratic tradition, in certain contexts it may return false information proclaiming successes which have not occurred.  Here is one anedotal example here in China.  It was a project aimed at demonstrating that NGO involvement at village level would have greater poverty reduction impacts than the predominant Government only poverty reduction drive.  Several highly competent NGOs took on different villages.  Most of the NGOs set up community participatory M&E committees ensuring that the poor were proportionately represented or over-represented on these committees.  The attendance of the poor at committee meetings dropped significantly over time. The wealthy and the semi-wealthy were dominated the committees.   They focussed their M&E attention on those sub-projects that benefitted their own interest groups.  When the external M&E team asked some of these committee members why the poor were no longer participating the most common answer was that the poor were not articulate enough, did not understand about the execution of subprojects, had nothing to say, and anyway, we more articulate committee members are able to speak on their behalf. (The NGOs could not regularly attend these meetings and facilitate greater participation by the poor).

The M&E mechanisms in this case needed a venue where the poor could anonymously and freely express their observations of progress on sub-projects and have their  comments on performance deficiencies acted on in a power structure where they were the lowest occupants

Focussing on the broader context is so important, as, in the case of poverty reduction, a range of factors external to the project may also be vectors for poverty reduction (e.g. reduction in agricultural taxes, improved access to markets through a recently completed road project).  Ignoring the interaction and impact of these external factors will most probably result in false attribution of "success" to the project.

Hi Dianne, Donna and Zach.  This is an interesting line of discussion – thanks for raising these ideas, suggestions and questions.

Here’s an extension on a couple of these… what do others think?

1. How can we best mix quantitative and qualitative assessment measures of aid programs? Could less (e.g. fewer participants in training (outputs side)) actually/sometimes mean more (e.g. if better training results (on outcomes side))?
   
2. Zach’s question: Given Australia's commitments there [in Uruzgan Province, Afghanistan], does anyone have feelings, suggestions on how best to monitor & assess ODA in insecure environments?

 

Australia plays a significant role in Uruzgan Province, Afghanistan,- estimated 123 million AUD this year to Afghanistan, about 20% of that to Uruzgan. 

Uruzgan is a volatile place where direct monitoring and implementation of projects is quite difficult.  Given Australia's commitments there, does anyone have feelings, suggestions on how best to monitor & assess ODA in insecure environments?

There does need to be more qualitative assessment of aid projects. Quantitative will capture outputs, but not necessarily outcomes and long term effects. For example an education program to improve English through enhanced teaching methods could be measured according to number of teachers trained and number of students receiving improved instruction. Alternatively, qualitative assessment methods applied consistently over several years would measure techer effectiveness (and therefore training effectiveness), methods effectivesness, and actual learning outcomes. This might mean fewer participants because of the intensive nature of the project but it would ultimately be a better investment.  

I don't believe you can effectively manage and monitor development projects from afar nor by using a partner to do the management. It is essential to have someone at the point of delivery. I have seen what can happen too many times to think it can be done any other way.

I also agree with Mr Grumpy! Development is not a science and sometimes hard data (actual numbers) on success is difficult to come by.

Nevertheless all projects should have approaches to monitoring and evaluation with clear baselines included in their proposals

The comments made below made me realize how complex development is, the complexity of which is reinforced if an organisation like our International Trade Centre (UN/WTO) has no field office. You have to rely on the performance of a project coordinator and an Impact Measurement Tool with a baseline and two evaluations conducted half way and towards the end of projects. Now, no doubt that monitoring in between those evaluations is still highly needed to guarantee the best results possible. Electronically I agree is only possible in limited cases, and rarely in LDCs. Unless we partner with an NGO who can follow it up with the beneficiaries or train them properly on how to use the monitoring system and double checks every given period that everything is reported adequately.

Thanks, Mr Grumpy. You have some interesting points here.

What do others think? Do you agree or disagree that we need to accept that there are inherent risks with aid?  How would you approach the proposition that aid can prevent something (negative) from happening?... and what would you say to the Review Panel - how do you think they could craft their views of the aid program's approach to efficiency and effectiveness?

Really, all wonderfully warm and cosy. Success of success. But thin about it! Aid is necessarily difficult, complicated and risky. Whilst feedback loops etc are a good idea, you guys need to prepare the Australian public for the reality of aid. It is a high risk venture. The public need to understand that they will lose money and that this is necessary. Absolutely necessary because of geo-political and economic reasons - but it is NOT A SCIENCE. As Soros recently said - economists suffer from physics envy. If we only give aid to things we can "prove" work we will end up only giving aid in emergency. We wont be able to give aid for preventative reasons - how can you prove you prevented something from happening?

Matt - thanks for your original post, and Ann - for your comment on this discussion thread. These raise a range of interesting points, well worth more debate. The Review team is looking at the best approach to 'efficiency and effectiveness', so let's explore this. What do others think about tools or approaches that could make the aid program most 'effective' (achieving result(s)), and 'efficient' (well organised, without waste)? Matt suggests practical tools can help show if aid is effective, like SMS-messaging on: reporting crises; if water-points are working; if books are received in schools. Ann adds in the need for efficient processes: making sure beneficiaries/communities can measure what is there before and after. What do you think? Do you have some specific ideas or suggestions to offer on these topics?

No argument that monitoring for efficiency and effectiveness is fundamental to understanding what has been achieved, and how to do better. The process itself needs to be lean and efficient. Essential to this is that every project and programme needs to establish a clear baseline and monitoring parameters. Too often evaluation is frustrated by poor data about the status quo antea. Involving beneficiaries is great, so long as they have the capability; it helps them towards awareness of where they came from, and how they themselves have managed to change.

Hi Tony. There are some new perspectives here, thank you.  Here's a few things (para-phrased) that seemed to stand out. What did others think?

'... Effectiveness could be seen as narrowing the gap between our planning and the total context in which a project will unfold.'

'... Ideas: give a project manager the flexibility of a CEO in a small to medium sized enterprise; study the uptake of project evaluation recommendations; don't limit effective projects to a set ministy/province/country for the 1, 3 or 5 year project.'

Did you agree or disagree with some of these ideas on effectiveness?

There are so many ways "aid effectiveness" can be defined and that is clear from comments made already.  I had a wild, cynical thought that it could also be defined as minimizing the "evaporation" (what a nice nuetral word) that often occurs during project implementation.

Logically, from the meaning of "effectiveness", we appear to be talking out outcomes and impacts.  These are posited in project design, and as other commentators have said, outcomes and impacts are often not visible until several years after project completion.  Also, by the time several years have transpired after project completion, the desired outcome may be evident but what caused it?  Your project?  Other external factors? A combination of your project and other factors?  If so. how much causality can you attribute to your project?

In the folowing comments I am somewhat influenced by the work of psychologist Dr Richard E Nisbett in his book "The Geography of Thought.  How Asians and Westerners Think Differently...and Why" Free Press, New York 2004.

One could start with the view that it is quite a gamble to posit what inputs (nature, size and timing) will, when completed (outputs) will produce the presupposed outcomes.

We might be blinded by our Western thought process of focussing on single objects or systems that are themselves interdependent with a wider and very dynamic context.  One could argue that the log frame, risk analysis, stakeholder analysis, participation analysis, etc contributes to this inadequate vision of reality.

Maybe increased effectiveness is about narrowing the gap between our specific guesses (planning) and the total context (relationship analysis) in which the project will unfold.

To start with, why not acknowledge that we are guessing when we predict the future through our modalities of planning and the project implementation discipline that follows.  All of us would have seen project sub-components persisted with even though it becomes obvious they should be dropped.

Of course country strategies can be a guide to donors provided the recipient countries, though both their political and administrative hierarchies have "bought in" to those strategies.

What if the project manager had the same flexibility as a CEO in a small to medium sized enterprise?  This includes constant attention to the external enviroment, seeing opportunities generated by changes in the external environment and responding promptly to them, building up "person based" relationships with the external environment, quickly cancelling activities where little progress is being made, reallocating financial and staff resources promptly, and progressively making the outcomes more specific from their original higher level of generality.  Of course there would need to be regular consultation with an easily accessible corporate board.

Another wild thought on measuring aid effectiveness is to study how many recommendations in project evaluation reports have been noted and then included in future aid projects in that sector in that country.

 

Of course, how effective an aid project is also related to where (institutionally and in terms of "mover and shaker" personnel) it is embedded.  When reviewing overall  aid projects in a recipient country , it is fairly obvious which institutions with their lead and middle level personnel have 100% bought into the projects and have driven them with gusto and little or no evaporation.  When these conducive "hot spots" are found, why need feed success and put more projects in their direction?  Why limit assistance to that ministry or province or county to one three to five year project?

Enough for now!

In order to increase the efficiency and effectiveness of the aid program in relation to the inclusion of people with disabilities, people with disabilities in partner countries should be actively involved and consulted as far as practicable. Furthermore, disaggregated data should identify if and how people with disabilities are being included in development initiatives.

Hi All, Thanks for your perspectives and contributions here.

What do others think of some of these? What about…:

• The mismatch between the donor’s and recipient’s concept of aid effectiveness? (interesting points from Donna Vaughan)
• Broadening our thinking of effectiveness to include other factors from the recipient perspective, like trade policy and investment? (thanks, Joel Negin)
• The unexpected outcomes that could denote a program’s effectiveness – as assessed by a participant, rather than a project manager or donor’s view? (thx for alternative viewpoint, Gillian Fletcher)

It seems to me that a key question here is: what do we mean by 'effectiveness'? Further, who gets to set the measures of effectiveness, and who gets to assess it? (To put it crudely, assuming we are essentially talking about 'bang for buck', which type of bang, and who gets to say if it was a good one or not?)

The current M&E focus on collation and analysis of quantitative data can produce highly skewed pictures of effectiveness; number of people reached per program, for instance, is often presented as a measure of effectiveness. What about consideration of what was achieved by a program, both in terms of outcomes identified from the start of the program and also in terms of unexpected outcomes? A sex worker's view of the effectiveness of a project in which she is involved is likely to be greatly different to a project manager's view, or a donor's view. In HIV, for instance, a sex worker who has sat through literally hundreds of condom demonstrations is unlikely to consider the demonstration itself effective, but will value the opportunity to chat with friends, take a shower, learn about new brothels from other women... in my experience, little of this appears to be either captured or considered.

Following on from Donna's point below, aid effectiveness is very much a term that emanates from the donor perspective.  Developing country governments are more likely to speak of development effectiveness which broadens the scope to include trade policy, investment, and other areas of engagement beyond aid that contribute to growth and poverty reduction.

The aid review is focused on effectiveness for Australia but we shouldn't be too narrow in our thinking about other factors that impact on developing countries.

It is important to distinguish the donor's concept of aid effectiveness from the recipient's concept. These could be quite distinct. A sole focus on return on investment (economic or social) will tend to limit funding horizons and place unsustainable financial burdens on recipients once initial funding expires. Aid recipients at a grassroots level are often focused on long term benefits, and may have quite different needs or expectations of support from donors. This mismatch results in project failure. The wto must be aligned from the start through a consultative, participatory process.

Hi Susan, Thanks for contributing this link.

What do others think? It would be interesting to hear some views from charities or NGOs on this.

The article seems to point to the use of a tool (called Social Return On Investment (SROI)) that measures charities' impact. SROI attributes a financial value to inputs and outcomes and calculates these as a ratio.  A report by the UK-based Third Sector Research Centre (TSRC) finds some limitations to SROI though - including that it is difficult to use as a comparative tool (between charities).

Or - did you find something quite different on your reading of this article? And, are there other ROI tools used - whether by charities or others involved with delivery of Australian aid? It would be interesting to hear from you.

This article may be of interest http://www.ethicalinvestor.com.au/index.php?option=com_content&task=view&id=4055&Itemid=402

Thanks, John for your extra comments and insights here.

What do people think? Do you agree or disagree with this post topic i.e. the need to "invest directly in partner government capacity?" 

Maybe you have some brief examples or links you could share with us to support your thinking...? Interested to hear from you.

Here is another aspect to partner participation that may help aid agency staff see how difficult it is to acheive good partner relations.

The paris declaration says hopefully that donors will collaborate with each other in signatory countries

Don't hold your breath! my experience says they will hardly ever really collaborate with each other, they are huge unwieldy administrative instraments that compete visciously and thoughtlessly at the institutional level.

The whole is unfortunately much less than the sum of the usually quite capable staff! 

When dealing with (an unequal) recipient, the agencies they say they 'devise and assist', they actually 'devise and insist'  

It takes extraordinary far sightedness and political will at a high level in both donor and recipient countries for the partnership to be otherwise and I use extra ordinary in it original sense. 

First to comment on the recurrent expenditure item

• the only solution is to put the time into getting it into the national recipient country budget, very hard, it takes years and it is necessary to demonstrate worth, this is unlikely in the short term aid budget cycle, needs to be in loan or grant conditionality that this be reviewed after the post project evaluation - and before consideration of new projects, also hard, IFI's don't like things to stand in the way of a 'full program pipe line'
• There can be reasons to fund recurrant expenditure for some planned learning experience to have time to mature and so prove its worth to the recipient Government, donors do not like this, part of the short term focus of most programs (even though most programs saynthey have a long term focus in practical terms they do not, even though long term evaluation by the WB shows that projects with pillots and successeeding phases are much more successful than short term ones that tend to repeat lessons (Einstien is reported to have said that insanity is doing the same thing time and time again and expecting a different outcome) 

Secondly, the question of partner government participation.

• there is no alternative to building partner capacity, most effectively by making them take responsibility (except in emergencies and projects that are risky because they are technically demanding such as bridges and large infrastructure where lives are at stake)
• In my experience the instinct of beauracrats to protect them selves means the donors wont let go because of the risk something will go wrong and while the recipients will say they want them to let go they often don't want them to for the same reason (unless it makes it easier to skim money off) It takes a very determined effort and acceptance of risk on the part of the donor for this to occur. This too needs to be explained to the public. This is not easy to solev as Australia has hound out in PNG   

Thanks, David for the post - and Peter for the comment and different viewpoints here.

There are a variety of views on capacity development efforts with partner governments. What are some of people's own experiences?

Do you agree or disagree that funding recurrent expenditure could assist to address bottlenecks in a country’s recurrent (maintenance) budget?

David

Agree. The ADB and World Bank tended to bypass recurrent expenditure in the belief that the partner govt would have the incentive to look after it themselves. End result was replacement became the means to maintain. I remember a pumping installation which had 1 duty and 5 standbys. As each pump broke, another standby would be used.

Thank you, Martin and Colin for your Vanuatu, Viet Nam and Cambodian stories and examples here. Thanks, too, John, for your views.

Do others have country-specific case studies you would like to share, even if brief? Or constructive ideas on what you have seen work to good effect in countering aid dependency?

Please consider posting your thoughts here. We'd welcome your contributions, and have a few more days before public voting starts.

As stated in other parts of this site.

• Aid is frustrating, difficult, risky but worthwhile, there is no other source of money for development
• the answer does not lie in soley bottom up or top down or middle out. different objectives require different tactics
• a lot of these points relate to politcal learning, education, including experience of things that work, this takes a very long time although it is getting much quicker, magna carta to universal sufferage took the brits 800 years, the Japanese 70, Guy fawkes was only a few hundred years ago, Libyans are only now contemplating whether it is better to blow up the houses of parliament or try and get elected to it to get their own way 
• I think there is room for much more sensitively applied conditionality at a high level between donor and recipient, 'ou do something and we will continue and increase', AusAid has been doing this in the Pacific, it is not easy and takes political will, and this is impossible from a middle level in either recipient or donor agencies   

Your experience in Vanuatu of corruption and wasted aid funding is not unique to that region I would suggest. The problem I believe, stems from the Australian Governments and other world powers desire to not offend the ruling clique of other countries. A classic example of this can be seen with what’s happening with the Middle East currently where international governments have supported these corrupt and despotic regimes ad infinitum.

 

International aid appears to be intensely focused on working from the TOP DOWN and this feature, although successfully avoiding friction with the authorities, makes aid donations particularly vulnerable to corruption and inefficiency.

 

My AVI experience in Viet Nam last year was a classic example, where my experience and advice went ultimately toward empowering the ruling gentry to exploit a river, newly flooded for irrigation purposes, for their own benefit. I will admit that there will likely be some flow on effect in possibly providing employment for the local population and in educating visitors in organic and sustainable building techniques, but by and large, the wealthy and powerful will profit from land that the local land holders have now lost.

 

I recently read The Road of Lost Innocence (2007, Virago press Great Britain) written by Somaly Mam, who escaped from sex slavery in Cambodia. She has become internationally famous for her work in rescuing and rehabilitating children from similar situations throughout SE Asia and elsewhere.

 

I read in her book (page 180), that their foundation is having trouble with finding adequate funding to continue rescuing, feeding and retraining their rescued charges and she is proposing that they set up some kind of a farming venture, where they could educate and feed the girls as well as possibly self funding their operation in order to lessen their dependence upon the often fickle supply of international aid money. She proposes that this would allow them to assist not only former prostitutes, but also orphans, ethnic minorities and the elderly. She has called it the Somaly Mam Foundation and I have located the website, see below:

http://www.somaly.org/whoweare

 

I also saw on SBS evening news on Saturday 26^th Feb that she is currently in Sydney doing talks at schools, trying to raise awareness and funds in order to continue her work, which certainly falls within the UN and Australian millennium development goals.

 

However, I would be most surprised if AusAid or any of the large foundations such as the Clinton foundation will in any way offer support this foundation. Why you might ask? Because it functions from a BOTTOM UP perspective. On reading this book it becomes clear that members of the Cambodian Government, police and military all have vested interests in the sex slavery and human trafficking business and the trouble with diplomats is that we wouldn’t want to offend the ruling gentry in any way would we? This I believe is the key to understanding why international aid will always be largely inefficient at assisting those most in need. The evidence seems crystal clear to me and I have no idea what the solution might be other than to leave the real on the ground effective work to the privately funded volunteer who is prepared to work in an unofficial capacity.

 

 

Hi Martin, Thanks for your ideas and your detailed, considered contribution here.

What do others think? Do you agree or disagree with some of these comments?

Do you have constructive suggestions on any of the topics raised... whether on aid dependency, small island states like Vanuatu, or practically, things you may have seen work in these sorts of situations?

Please feel free to share some of your views, too.

 

This subject has been covered a bit under other sections, here is some one liners about how effectiveness and efficiency is achieved in the long term 

• building capacity is essential but difficult, and with governance issues takes political will at a high level in both donor and receipient countries
• real partner participation over the long term is essential but difficult as it is risky for the donor agencies and their staff
• M&E should be more about outcomes and adaptive management, than outputs and process, accountability is of course necessary but difficult and the best is not mechanical, it takes judgement by experienced people to see the context, which is usually complex and often culturally specific, 
• good evaluation should consider efficiency and effectiveness but in a wide sense, a poor log frame is a dam nuscience! 
• insulating bureacrats from the results of their decisions through committees is not a recipe for learning