As the discussions on post-2015 targets continue, the way we currently monitor global WASH progress is coming under increasing scrutiny. This can only be a good thing – debate leads to fresh ideas and new consensus – but only if viable alternatives to the status quo are suggested.
When the new JMP report came out a week or two ago, there was a swift set of complaints about the data (1). Many of these are well-founded. After all, when measuring the same indicator across all 193 UN member states, depth is sacrificed for breadth to allow measurement of all countries by the same yardstick. The JMP does not tell us everything we want to know about a national WASH sector – it does not adequately address water quality, distance travelled to get it, or water point functionality. Some of these arguments are rehearsed in a recent Waterlines crossfire debate between Clarissa Brocklehurst and Ned Breslin.
However, to use these criticisms as a stick to bash the JMP with, without proposing workable solutions (readily applicable for all developing countries), is to miss the point about its objectives and mandate. In this blog I set out what the JMP is for and what it is not for and why, in my view, some of the criticisms are misplaced.
What is the JMP for?
The primary objective of the JMP is to monitor the WASH MDGs at the global level. In order to achieve this, WHO and UNICEF collate national-level data for every country in the world (the JMP does not collect primary data itself). Underlying this are further objectives: (i) compare countries based on a single method, (ii) statistically estimate progress over time. One of the overarching goals is for the international community to be able to make inter-country resource allocation decisions, i.e. which countries are most in need of support towards the MDGs. In addition, it is used to see which countries are making progress compared to their peers, and learning from those successes and failures.
All this is only possible when one uses the same method and indicator for all countries, or the comparison is not valid. At the national level, different countries collect data on different indicators using different mechanisms, so a key challenge in this work is manipulating national-level data to arrive at the MDG indicators. Currently, the only workable method for doing this for all developing countries is to collate household survey data, mainly existing surveys such as DHS and MICS. To use very patchy data from utilities, governments and sporadic water point mapping (WPM) efforts would leave huge gaps and result in unacceptable levels of inaccuracy (see this report p.5-6 for some discussion).
The key point about the household survey data is that it is already collected in all developing countries using pretty much the same questions and response categories (though not always, see the same report p.7-9)
What is the JMP not for?
The JMP is not designed to assist complex policy decisions at the national level. It does not aim to tell you about water point functionality, water quality or district-level inequalities. Other instruments such as WPM, routine data collection, and WASH-specific surveys are useful for this.
The nationally-representative household survey approach allows us to see a snapshot of WASH infrastructure usage from the perspective of the household. It can tell us nothing about water points themselves, as the household is the unit of analysis (2).
So, valid critiques of the JMP include that it discounts water quality (because the water may not be “safe”), time taken (because if this is too long it might not constitute “access” by national definitions), or the fact that people use multiple sources. Similarly, with sanitation, there are critiques around interpretation of the definitions of improved and unimproved, and the status of shared sanitation. Furthermore, only the ‘collection’ stage of the sanitation chain and not disposal, treatment etc.,
While these critiques are valid, they are only useful when workable solutions are proposed which would still allow data to be collected using a single method for all developing countries. That is why the current post-2015 process is so important, as it allows people to make suggestions.
Measuring functionality is not suited to household surveys
There is a school of thought arguing that JMP data should be cast aside because it is not perfect and it doesn’t cover functionality. Such arguments are unrealistic because they make the (unachieveable) best the enemy of the good. For example, in the crossfire piece mentioned above, Ned Breslin suggests that an ideal indicator would cover
“the percentage of people who have access to an improved water supply where water flows regularly and reliably, where breakdowns are limited to less than 1 day per month (for repairs), and where the conditions are evident to eventually replace the water system when its operational life has expired.” (Waterlines Oct. 2012, p.259)
Data on that indicator would be nice to have, but it mixes up household experience with infrastructure, so would be extremely difficult (if not impossible) to measure with a household survey. Nationally-representative data could not be collected through WPM without unacceptable blanket assumptions of ‘people per water point’ (see report links above for why this introduces huge errors). Even then, it would require such data to exist in every country and be nationally representative if a future JMP method were to be able to use it, which is not likely in the next 10-15 years. So, to expect JMP to cover functionality in the short to medium-term is to expect too much (notwithstanding footnote (3) below), as health and demography-focused household surveys and censuses are not suited to complex questions around infrastructure management. For the long term, the onus is on those criticising to make workable, costed suggestions of how things could be done better.
Global estimates for MDG indicators should not be confused with the important task of so-called ‘post-implementation monitoring’ at the national level. Such monitoring data is self-evidently important and useful for policy and planning. It would, however, be prohibitively expensive and undesirable to attempt to provide regular global estimates of this kind.
It is also important to note the inherent differences in what is measured by household surveys as opposed to WPM. It is wrong to make direct comparisons between household survey data regarding what infrastructure people say they use (such as JMP) and proportions of functional of water points. For example, the JMP estimates that 60% of the rural population of Liberia used an improved infrastructure as their main source of drinking water in 2011, based on 6 surveys since 2000. It is a coincidence that a recent WPM exercise in Liberia found that 60% of rural WPs are fully-functional, a figure which reduces to 38% when problems of seasonality and quantity are taken into account (3). The household survey is measuring use of infrastructure, whereas WPM is measuring the infrastructure itself. The two are of course related, but not comparable, because vastly different numbers of people use each water point (see e.g. WASHcost 2013 data for 3 African countries).
Proposing solutions
So, it is not valid to say “data from these 4 or 5 countries show that many water points are not working, so JMP is useless”. Full WPM inventories only exist for a handful of countries, and the extent to which they will be updated regularly is unclear. Until nationally-representative WPM data exists for the vast majority of countries, there is no way for WHO and UNICEF to use the data consistently (4).
So, we should recognise that, yes, the JMP data is not perfect and, yes, it does not provide the level of detail we need for policy-making at the national level. However, for the global level and inter-country comparison, survey-based methods are the best thing we have, and the best we are likely to have for the next 10-20 years.
At the national level, no single source of data is useful for policy-making – a combination must be used. WPM can tell you about infrastructure, but not about who uses it. Household surveys can tell you about the household experience and equity, but not about infrastructure management and sustainability. Utility databases can tell you about water usage and pricing, but not about those not on their network. Ultimately, the answer at the national level has to be to use a combination of these sources. For the global level and inter-country comparison, however, the only thing we have in the medium-term is JMP (with minor tweaks for post-2015). While acknowledging JMP’s weaknesses and looking to improve it, we should therefore support it.
===========
(1) Monitoring the current MDG targets and indicators is the responsibility of the WHO/UNICEF Joint Monitoring Programme (JMP).
(2) As an aside, WP functionality is theoretically built into the question, as DHS asks “what water source do you use most of the time” – if the village handpump is broken, people would presumably not respond “borehole” if they are not in fact using one.
(3) See p.5 of this report http://www.lr.undp.org/documents/pdf/preliminary_analysis_report_liberia11.pdf
(4) Even then, this misses the point that WPM can only tell you about infrastructure, whereas household surveys tell you about people’s actual reported experience, which is more relevant for human development
SqWASH - searching questions for WASH policy 
Well articulated Ian. Each monitoring approach and the related indicator/s has specific relevance and one should not aim to get an overly complex monitoring system that ticks all the boxes. That would be nearly impossible to implement in most countries, quality of information would be questionable and hence, would not support timely decision making or policy formulation !
Thanks Girish – completely agree. It’s a question of balance between breadth , depth and quality, whether you’re talking about the national level or comparing between countries. The point I’m making here is mainly about the latter. Maybe something on the national level another time, but I’d always press your above point about timely decision-making. No point in having a bells and whistles system with mobile phones etc. when data on half the indicators are missing, there’s no local incentive or capacity to update, and nobody pays any attention to the outputs. Regular updating of a few simple indicators and strong links to decision-making processes are key
Indeed – the question of balance is important. And it is also about the back-to-basics question – monitoring for what ? That purpose level clarity is important to get a fit-for-purpose system / process / mechanism. Thanks on your work – very useful insights !
Nicely put, and excellent blog name! The pending question for me is: How far does ‘the international community’ actually make ‘inter-country resource allocation decisions’ on the basis of JMP data? It’s presumably improving with greater accountability between donor and recipient in initiatives like Sanitation and Water for All, but disentangling allocation decisions, actual spending and country progress on WASH over decades is extremely difficult. So far, I’ve only really seen anecdotal evidence that ‘donor x’s’ decision to allocate to ‘country y’ is based on their WASH coverage and/ or progress (as opposed to the broader politics, and geopolitics, of aid). I’d love to be convinced otherwise so if you’ve got any pointers, let me know! This is pretty critical but somewhat overlooked as each sector piles in for a piece of the post-2015 pie. Getting an issue included within the post-2015 agenda at target and indicator level may mean more resources and political attention, but will it necessarily mean better allocation decisions?
Good question Nat – something I was going to write about, but the post was already too long! I essentially agree with your points around the political economy of aid.allocations. However, there are a couple of reasons to suggest JMP data is at least part of the inter-country decision. First, most of the key donors fund it, which suggests that they value the role it plays (http://www.wssinfo.org/about-the-jmp/structure-governance/)
Second, some donors use JMP data explicitly. For example, an early version of my ‘MDG support needs tool’ was used in DFID’s Bilateral Aid Review back in 2010, to justify selection of priority countries (see here – http://bit.ly/11FZzhm). The tool uses JMP data alongside disease burden and WASH aid per capita to inform resource allocation. This is perhaps an isolated case, but I’m sure most funders at least look at the data (and perhaps use it to justify what they want to do anyway for political reasons…).
However, your point essentially stands, and colonial history and current geopolitics clearly decide a great deal – the data supports this. The big picture hasn’t changed that much since I did the number-crunching for the WaterAid/Tearfund report calling for SWA back in 2008, so the data supports your argument (http://bit.ly/15nYRUh).
One of the challenges for SWA is to try and change this, and ensure allocations are made on a more rational basis. That means not only a focus on low-income countries with the lowest coverage but also those making slow progress or slipping backwards
Other comments from Twitter users, and my responses:
Certainly we can improve on JMP in its current incarnation! Survey questions can be tweaked and added to gain more rounded perspectives. My point about the “10-20 years” is that household survey-based methods are likely to remain the backbone of our approach to inter-country comparable statistics on WASH outcomes. Maybe “big data” has the potential to help in the long-term, but there’s nothing out there right now that can deliver on the scale of household surveys.
Global targets and numbers do help, in my view. This is not because of the absolute number (it’s not that policy-relevant whether it’s 2.5bn or 2.7bn), but whether the number is going up or down in different regions, and how fast. However, while the current method was designed for monitoring the MDGs, the national-level JMP data arguably has more value, in terms of the inter-country comparison it allows. It is indeed a very low-cost way of getting these numbers – I don’t know their budget but it’s perhaps 3-4 staff in WHO and UNICEF working on other projects as well, using existing survey data. That’s pretty good value-for-money, when a recent WPM survey in Tanzania cost $4m. By saying that. I’m not setting the two methods against each other (they have different and important uses as outlined in the blog) but pointing out that Tim is correct – “need low-cost counting, and JMP is a steal”
Couldn’t agree more. This blog focused on JMP as it’s been taking quite a lot of flak recently and I think it’s worth defending.
[…] space!). And this has led to much discussion about the merits of these estimates. Ian Ross has a very thorough blog-post arguing the merits of the JMP estimation […]
[…] space!). And this has led to much discussion about the merits of these estimates. Ian Ross has a very thorough blog-post arguing the merits of the JMP estimation […]