“Are people right to be so scathing or sceptical about us trying to use quantitative survey methods in the Secure Livelihoods Research Consortium, or about us attempting a panel?”
SLRC is doing a quantitative survey in its seven focus countries to explore how livelihoods recover following conflict and how access to basic services affects perceptions of government. Not only are we trying to do a representative, statistically significant survey, we aim to make it into a longitudinal panel by returning to the same households in three years time. We have just completed the first round of the survey in DRC, Nepal, Pakistan and Sri Lanka, are currently in the field in Uganda, and will go to the field soon in Afghanistan.
When I tell people that we are doing a quantitative panel survey, I get three reactions from people. The first is a frown. It may be accompanied by the second reaction – (often fierce) shaking of the head. The third (definitely the worst!) is a reaction which feels like I’m being patted on the head and told ‘never mind, my dear, you’ll soon learn’. So, are people right to be so scathing or sceptical about us trying to use quantitative survey methods in the SLRC, or about us attempting a panel?
We certainly recognise that our approach brings risks – especially the security of our respondents and enumerators – but after spending the time last month with the SLRC Nepal survey team, in Ilam district in the far west of the country, I’m more convinced that our approach is justified. This is why:
It starts with the journey into Ilam. We fly to Biratnagar and then on by road. At one stage we stop for tea (a famous Ilam product) and across the valley see a huge landslip scarring the mountainside. On closer inspection, we realise that it is the landslip of 30th September. I can feel the heart of Bishnu (SLRC’s leader in Nepal) grow heavy as we contemplate the 22 bodies forever entombed in the mud, soil and rocks. A few days later, I find myself gingerly picking my way across smaller landslip, urging the team of ten enumerators in front of me not to dawdle and to stop chattering so they can hear any rock fall coming from above. All this reminds me that, as a team, we frequently catch ourselves sliding into what we’ve called ‘conflict exceptionalism’, where we automatically assume that conflict is the cause, or outcome of all that we find. Here in Nepal, I’m reminded at every turn of the path winding from one household to the next that there are numerous threats to people’s livelihoods and well-being and that our survey will help to capture that multiplicity.
We also face other challenges in the survey that are not directly related to conflict: enumerators walking up to five hours between households; difficulties preventing a small (or large!) crowd gathering to listen to (and sometimes attempt to join in with) the interview; very elderly respondents who, although we’d love to know more about how they perceive the government, cannot understand our questions well. In Bardiya district, where young women and girls were being kidnapped and taken across the border into service in India, we had to reorganise our enumerator teams to ensure that all our female enumerators worked far from the border.
So, should we avoid doing quantitative work in conflict-affected places because of these challenges? I’m left feeling that conflict should not be an automatic excuse for not doing quantitative survey work and that we need to look at situations on a case by case basis. Nepal may currently be safer than Pakistan or DRC and these will certainly present other challenges, but nothing we have confronted so far is a reason not to try and deliver a high quality survey.
I find myself coming face to face with the second question – should we try a longitudinal panel? – in the first Ward where I accompany enumerators. Beside the logistical challenges (for example attrition in our sample), we’ve been told by many people that that in three years we’ll never see enough shifts, either in people’s access to health, water, education and social protection, or in their attitudes towards government, for us to say anything useful about how far delivering services can contribute towards state legitimacy and state building. And the truth is that we won’t and can’t know what change we’ll get until 2015. Until now, we’ve been arguing that if we find very little change in access to services or to attitudes, this will at least contribute to a recognition of the sorts of timelines that are required to get changes in people’s perceptions of their governments. Then, en route to our first interview, I’m taken by Suresh Prasain to look at the local health post. It’s a small, dilapidated building but right next to it the foundations are being laid for a new 20 bed inpatient facility. So here, at least, we are likely to see either some change in access to health service provision. And if construction should grind to a halt, or the facility is not staffed, we can expect to hear our respondents voice their frustrations in three years’ time.
So, my visit to Ilam has left me feeling that in Nepal we can deliver a high quality survey and there is huge value in tracking change over the next 3 – 4 years. We face significant logistical constraints, especially tracking down our respondents in the future, but we’ve a strong plan to tackle them. Finally, the survey has thrown up some new analytical challenges for us to deal with – not least how we analyse ‘cross-border perceptions’ – but Bishnu [^] and I will tell you more about that next week!