Home Blog

Latest blogs

Written by Richard Mallett on 03 December 2012 15:58

“What do we do if we need results to justify a development or humanitarian programme, but don’t have the evidence to demonstrate what works?”

SLRC Research Officer, Richard Mallett

As Rachel Slater and Samuel Carpenter recently argued, getting aid programming right in fragile and conflict-affected situations is challenging but hugely important. From a purely monetary perspective, there’s $46.7 billion at stake – which is a lot of money to demonstrate good value for.

A sizable chunk of this is spent on programmes aimed at supporting livelihoods and stimulating economic recovery, and many donors, DFID included, are increasingly looking to justify the specifics of this spend on the basis of results. Many aspects of this approach are hard to disagree with: the argument that we should know what works before throwing around funds is a no-brainer, and programme effectiveness is to many a more sensible mechanism for allocating aid than, say, the logics of self-interest and soft power.

But do we actually know what works?

Partly as a result of the sharpened focus on results and value-for-money, DFID country offices are now required to assemble business cases for new spending that cite evidence to justify their decisions. However, a new review of the evidence on growth and livelihoods in conflict-affected situations suggests that there is surprisingly little out there for them to draw on. Despite the range of programmes on offer to aid agencies and governments wanting to protect livelihoods and promote economic recovery – from public works programmes to the distribution of seeds and tools – in many cases the impact data just aren't there. Much of the time, it seems, we simply don’t know whether programmes are working for beneficiaries, having no effect at all or, worst case scenario, making things worse. (It should be noted that although we are talking primarily about the micro-level impacts of programmes here rather than the meso- and macro-level impacts of reforms, it is also understood that many developing countries similarly suffer from a lack of data on macroeconomic performance – see page 8 of this newsletter from the Centre for the Study of African Economies).

This may come as some surprise to those who have spent any time with the burgeoning literature on livelihood and economic programming in conflict-affected situations. There is no shortage of claims and recommendations to be found within the abundance of donor reports and policy briefs, suggesting that the impact evidence base is pretty strong and that our level of knowledge is pretty good. But as soon as we start asking serious questions about the sources for claims and the basis for recommendations, their mask of certainty and assuredness starts to slip. Most of the time, study methodologies are rarely discussed in any detail; sometimes, they are barely mentioned at all. For something so straightforward – and so fundamental – this is baffling.

Studies that are clear on methodology and that examine impact are massively in the minority. One illustration of this emerges from our review. As part of our review methodology – and in an attempt to inject some additional rigour into the process – we undertook two systematic reviews in addition to more orthodox review practices. We wanted to know about the impacts of two separate interventions – seeds-and-tools programmes and ‘markets for the poor’ (M4P) interventions – in countries defined as fragile and / or conflict-affected. Even without specifying which outcomes we were interested in, our two systematic reviews yielded a depressingly low number of relevant studies – nine on seeds-and-tools and just three on M4P – and, of these, the quality was generally low.

What might explain this sizeable gap in the evidence base?

It’s difficult to be sure, but there may be a number of reasons why there is so little evidence of impact. In no particular order:

  • Doing impact evaluation well is not easy or cheap. Studies that take impact, causality and attribution seriously take a long time to do and attract substantial costs – even more so in difficult contexts.
  • Fund programmes, not studies. In conflict-affected situations, donors are faced with a huge number of urgent humanitarian and recovery needs. Funding research may not be at the top of their list of priorities when there are other, more pressing things to invest in.
  • Assumptions of effectiveness can prove remarkably resilient. To many, it may seem obvious that giving people jobs in war zones is a good thing to do – why spend money on research that will simply tell us what we already know? Deductive logic such as this is certainly compelling, and often convincing, but research can turn conventional wisdom on its head.
  • The truth might hurt. If a donor has been funding programme x for several years, it may not be in their interest to then fund research that tells them they’ve been doing it wrong.
  • We are measuring impact! Many studies we came across in our review used ‘impact’ to refer to how well a programme functions in terms of its own design – i.e. was it completed on time? Was the right amount of, say, seed distributed? This may be one way of measuring success, but it doesn’t tell us anything about what the programme did for beneficiaries.



Managing the gap

So, what do we do if we need results to justify a programme, but don’t have the evidence to demonstrate what works? In such circumstances, it may be tempting to argue for donors to reduce the burden of proof required to justify decisions, to ‘lower the bar’ for evidence-based policy making in conflict-affected countries.

But we’d argue that this would be the wrong approach to take. It is possible to do high quality, methodologically rigorous research in difficult places, and there are plenty of cases of where this has been done. Take, for example, the Households in Conflict Network and MICROCON research programme who, together, have generated a valuable body of robust, fascinating and methodologically clear evidence on the micro-level causes and consequences of war. Or how about the multi-year study of rural change in eight Afghan villages conducted by researchers at the Afghanistan Research and Evaluation Unit? Such examples provide clear demonstrations that doing high quality research in conflict-affected environments is not beyond the limits of possibility.

Thus, rather than throw out the results-based agenda altogether, we would instead suggest a number of recommendations that might help move us forward.

First, there should be an obligation for people doing both research and monitoring and evaluation in conflicts to be much more systematic and rigorous in presenting their methodologies.

Second, given the current lack of impact evidence, donors and aid agencies need to be more cautious in their policy recommendations.

Third, there’s also a need to guard against conflict exceptionalism – not everything about conflict-affected places is qualitatively different. Aid actors working in conflicts could do better at drawing on evidence from other development contexts and considering how approaches could be adapted to deal with the challenges of conflict (as has been argued in relation to delivering social protection).

Finally, we simply need more and better research…which is what researchers always think but this time there really is a clear cut case for it!

Read the full Working Paper: Growth and Livelihoods in Fragile and Conflict-Affected Situations and the accompanying 4-page Briefing Paper: Growth and livelihoods in conflict-affected situations: what do we know?.

22 November 2012 16:20

"For those living in the villages we visited, it is often impossible to carry out daily activities without being taxed... But can an 'all-pervasive system of taxation' have an upside?"

Katherine Haver, Humanitarian Outcomes. 

In 2009, I carried out some research with Oxfam GB and four local organisations in eastern DRC. Our question was: how do people cope with extreme violence, and is there anything aid agencies can do to support ‘self-protection’ mechanisms? We spoke with over 700 people in 24 communities across North Kivu and South Kivu. Our report concluded, perhaps unsurprisingly, that there was no ‘magic cape’ that could easily be supported or replicated. Rather, the most successful strategies – fleeing, submitting – also entail the worst negative side-effects.

Along the way, however, we learned a lot about the links between protection and livelihoods. An informal, predatory, almost omnipresent state is continually eroding people’s ability to earn a living. It is not so much that the state does not function, but that it functions too much — violently, without rules and without limits. As summarised in a recent SLRC report, much of this was legitimised under the Mobutu regime, which encouraged public servants, including the army and the police, to self-finance salaries and operational costs through informal taxation and extortion of the general public. 

Taxing livelihoods

For those living in the villages we visited, it is often impossible to carry out daily activities without being taxed. Need to travel to the next village to visit a sick relative? Pedestrians pay $0.24, those on a bicycle $0.35. Want to produce some palm oil? For every 20-litre bottle (which can be sold for around $10), the state takes 7 litres as well as $0.12 and the military takes 7 litres as well as $0.25, and there’s another $10 per year to access the trees plus a tax on the machine to extract the oil. Want to buy some food? In one town, the number of tax inspectors (and their agents) had become so high that people were afraid to go to the market. 

It is difficult to distinguish illegal taxes from simple theft. Those who wield more power often justify their theft as an official requirement or payment. Lacking information about what taxes are legal, civilians find it difficult to challenge those making demands, who in any case are often armed. Seemingly everyone collects taxes: agents of the state (including the military (FARDC), the police, the national intelligence agency (ANR) and even ‘anti-fraud’ agents in one place); customary authorities, such as local chiefs; and armed groups such as the Mai Mai and Democratic Forces for the Liberation of Rwanda (FDLR). 

Armed groups will often justify taxes as payment for their ‘protection’ of the population, whereas local chiefs will refer to the need to respect tradition. All of these taxes leave people wondering what the government does with the money. As one participant said, ‘We do everything for the state, but it does nothing for us.’ 

Can an ‘all-pervasive system of taxation’ have an upside?

Given all this, what was surprising was that people did not want the state to disappear. Rather, one of their most frequent (and spontaneous) requests was for information and training – for both themselves and the authorities – about their rights and obligations, and the relevant laws, especially tax laws. When asked how this would help keep them safe, people said they felt that increased clarity would itself prevent abuses by making it more difficult for state agents to justify their actions. People seemed to feel that this knowledge would give them power to stand up for their rights. 

I approached this request with a healthy dose of scepticism. Could it be that people perceived that Oxfam or its partners were providing this type of information, and hence that it was what they thought they should ask for? Had they heard about someone who had received a per diem for attending a training session? In the end, the number of times it was repeated, in different ways by different people, convinced me that there existed a genuine desire for a credible outside actor to help facilitate some kind of dialogue between residents and the relevant duty-bearers. 

I also came away impressed by people’s understanding of the purpose and potential utility of taxes. People want to see something for their money. They want schools with teachers in them, markets with shelters from the rain, drugs in the clinics and roads that are passable. There is a sense that the contract between the people and the leaders who tax them could thrive, even in an informal, local, perhaps even technically illegal way. After all, people are used to paying school and health fees, even if they receive only a rudimentary service in return. Could this ‘all-pervasive system of taxation’, which has become so much a part of the culture, have an upside? I came away thinking that we shouldn't assume that taxes are bad just because they’re informal. 

There are heaps of questions to ask before venturing to say what changes in taxation policy and practice would be of benefit to people’s livelihoods. A good first step would be a better understanding of who is being taxed, how and how much. What kinds of contracts – good or bad, peaceful or violent – exist between the people taxing and the people being taxed? Among the many aid organisations operating in eastern DRC, there is surprisingly little knowledge of how taxation affects people’s livelihoods — and very little understanding of how aid interventions enrich or impoverish the taxers and the taxed. But it is relatively easy to find out this information, if you ask.


This is a guest post by Katherine Haver. Katherine is a partner with Humanitarian Outcomes where she works on operational security management, cash based responses, humanitarian coordination, and disaster preparedness. Before joining Humanitarian Outcomes, she worked as a policy advisor for Oxfam based in DRC, where she authored reports on developing responses to displaced people in host families; improving the targeting of humanitarian assistance; and designing programmes to better support community self-protection mechanisms. 

16 November 2012 16:20
 
"Are 'fragile states' failing to meet the MDGs? The answer might not be as black and white as one might expect."


SLRC Director, Paul Harvey

Almost whenever you read anything about fragile states, the introduction notes that, ‘no low-income fragile or conflict-affected country has yet to achieve a single United Nations Millennium Development Goal'. This is a quote from the overview from the 2011 World Development Report on conflict, security and development. It seems to be an elaboration of a quote from the main body of the report that is subtly but importantly different.  ‘No low-income, fragile state has achieved a single MDG, and few are expected to meet targets by 2015.’

The WDR report was focussed on a clear definition of fragile and conflict affected countries or countries affected by very high levels of violence which were countries with:

  1. homicide rates greater than 10 per 100,000 per population per year; 
  2. major civil conflict with battle deaths greater than 1,000 per year between 2006 and 2009;
  3. UN or regionally mandated peacebuilding or peacekeeping missions; and 
  4. low income countries with institutional levels in 2006-09 (World Bank’s CPIA less than 3.2) correlated with high risks of conflict and violence

This definition does produce a set of countries that as of 2009 (the most up to date data when the WDR was being written) hadn’t met any MDGs. The more up to date MDG data now available does suggest that some of the countries in this definition have or will achieve some MDGs.

The quote from the overview that ‘no low income or conflict affected country has yet to achieve a single MDG’ however doesn't match this definition.  It implies that any conflict affected country hasn’t met any MDGs not just low income and conflict affected countries.  There are four countries that the Uppsala Conflict Data Program (UCPD) categorises as conflict affected in 2011 that have achieved MDGs. They are Colombia and Russia, which have achieved universal primary education, Thailand which has achieved seven of the eight MDGs, and Turkey which has achieved two (see MDG Monitor). All four governments claim to be affected not by conflict but by terrorism but that in itself highlights the important point that being labelled (or accepting the label) as conflict affected is an intensely political as well as technical issue. Countries with very homicide rates such as Mexico and Brazil have also achieved MDGs.  

But the quote is also misleading because there are plenty of conflict affected countries that are on track to meet several MDGs by 2015. Seven countries listed as conflict affected in the background paper to the WDR on the MDGs (Algeria, Afghanistan, Bosnia, Ethiopia, Philippines, Sri Lanka and Syria) are on track / very likely to meet at least five of the eight MDGs (See Gates et al 2010 and MDG Monitor).

These aren't all countries that would fall within the list of countries that the Bank used for the WDR, as the conflicts haven’t reached the threshold of 1,000 battle deaths per year and there are plenty of reasons to doubt the quality of the data on MDG progress in many fragile and conflict affected countries. But they are countries that fit within broader definitions of conflict and as the quote ‘no low-income fragile or conflict-affected country has yet to achieve a single MDG’ is reproduced elsewhere, the narrowness of the footnoted Bank definition gets lost. 

Is this any more than quibbling? Arguably, it’s just an editing issue – using ‘fragile or conflict affected’ rather than ‘fragile and conflict affected countries’. Or a definitional one – resting on whether or not you see Colombia, Russia, Turkey and Thailand as conflict affected and what threshold of battle deaths you use for the definition of ‘conflict affected’. Conflict certainly has dire economic consequences and the effects of conflict have made it more difficult for the countries affected to meet MDGs, shown in detail in Gates et al (2012 and 2010). Given the Bank’s poverty mandate, their main objective in the opening WDR chapter was explaining the poverty reduction challenges associated with violence and fragility.  

But I would argue that the quote is problematic both because it is so widely cited and because it reflects a wider tendency to conflate conflict and fragility in problematic ways. History shows that rich, middle income and poor countries can be affected by conflict but the current conflation of fragile and conflict affected situations risks equating conflict with poor and fragile countries. The conflation of conflict and fragility is often politically convenient because it allows people to ignore conflicts in richer, more powerful or strategically important countries. So debates around conflict and fragility focus more on countries such as DRC, Somalia and Liberia and less on places like the Philippines (Mindanao), India (Naxalites) and Russia (Chechnya) or indeed the USA and the United Kingdom (costs of wars in Afghanistan, Iraq and Libya).  That in turn risk setting up conflict as something ‘other’ – something that affects poor and difficult places rather than something that affects a much bigger swathe of the globe and that rich as well as poor countries are involved in.

Source: MDG Monitor. For the full table click here.
Notes: In order to work out the ‘overall’ level of progress for each country, we took the mode status for each country (e.g. Angola is ‘off track’ against one MDG, ‘likely’ against two, ‘possible’ against four, and has ‘insufficient data’ for one – therefore, for our purposes here we classify Angola as ‘possible’).  
The following countries have not been included in the overall map due to them having two modes: DRC, Djibouti, Sierra Leone, Timor Leste.
In SLRC’s work we prefer to talk about conflict rather than fragility and stress that strong as well as weak and rich as well as poor states are affected by conflict. The challenges governments face in delivering services and supporting livelihoods in South Sudan and DRC are clearly very different from those in Pakistan and Sri Lanka where state capacities are very different. Policy recommendations relating to recovery from conflict need to be tailored to recognise this diversity. 
Written by Rachel Slater on 12 November 2012 16:24
  "Since 2007, OECD DAC members have been signed up to state-building as the ‘central objective’ of their engagement in fragile and conflict-affected situations. It is a compelling narrative: deliver basic services (health, education water and sanitation) and – voilà! – better prospects for long-term peace and stability."

SLRC Research Director, Rachel Slater and Samuel Carpenter

Read more: http://www.odi.org.uk/opinion/6884-service-delivery-state-building-conflicted-affected-states

Written by Richard Mallett on 12 November 2012 12:55
"For me, the volume is more than anything about the blurring of boundaries – analytical 

and spatial – and the (increased) hybrid nature of politics and power during and after 

conflict."

SLRC Research Officer, Richard Mallett
Read more: http://africanarguments.org/2012/08/24/big-men-african-conflicts-and-informal-power-a-review-by-richard-mallett-odi/
Written by Richard Mallett on 12 November 2012 12:53
  "How useful is the concept of political settlement? Not very, according to a recent post by Mick Moore. Taking particular issue with the lack of consensus regarding definition, Mick questions the legitimacy of the concept, closing with a somewhat pessimistic evaluation of its added value."

SLRC Research Officer, Richard Mallett

Read more: http://www.odi.org.uk/opinion/6816-political-settlements-matter-response-mick-moore

Written by Richard Mallett on 12 November 2012 12:51
  "Although well established in the natural sciences, systematic reviews are relatively new to the world of international development research. But they are being increasingly promoted as an important step in strengthening evidence-informed policy-making amongst aid agencies."

SLRC Research Officer, Richard Mallett

Read more: http://www.odi.org.uk/opinion/6283-systematic-reviews-international-development-slrc

Welcome to SLRC's blog.

This blog will feature reflections from our team of researchers on the practicalities of actually conducting research in conflict-affected situations. We will also be posting guest blogs written by key researchers and practioners working on livelihoods, basic services and social protection in conflict-affected situations.