As the discussions on post-2015 targets continue, the way we currently monitor global WASH progress is coming under increasing scrutiny. This can only be a good thing – debate leads to fresh ideas and new consensus – but only if viable alternatives to the status quo are suggested.
When the new JMP report came out a week or two ago, there was a swift set of complaints about the data (1). Many of these are well-founded. After all, when measuring the same indicator across all 193 UN member states, depth is sacrificed for breadth to allow measurement of all countries by the same yardstick. The JMP does not tell us everything we want to know about a national WASH sector – it does not adequately address water quality, distance travelled to get it, or water point functionality. Some of these arguments are rehearsed in a recent Waterlines crossfire debate between Clarissa Brocklehurst and Ned Breslin.
However, to use these criticisms as a stick to bash the JMP with, without proposing workable solutions (readily applicable for all developing countries), is to miss the point about its objectives and mandate. In this blog I set out what the JMP is for and what it is not for and why, in my view, some of the criticisms are misplaced.
What is the JMP for?
The primary objective of the JMP is to monitor the WASH MDGs at the global level. In order to achieve this, WHO and UNICEF collate national-level data for every country in the world (the JMP does not collect primary data itself). Underlying this are further objectives: (i) compare countries based on a single method, (ii) statistically estimate progress over time. One of the overarching goals is for the international community to be able to make inter-country resource allocation decisions, i.e. which countries are most in need of support towards the MDGs. In addition, it is used to see which countries are making progress compared to their peers, and learning from those successes and failures.
All this is only possible when one uses the same method and indicator for all countries, or the comparison is not valid. At the national level, different countries collect data on different indicators using different mechanisms, so a key challenge in this work is manipulating national-level data to arrive at the MDG indicators. Currently, the only workable method for doing this for all developing countries is to collate household survey data, mainly existing surveys such as DHS and MICS. To use very patchy data from utilities, governments and sporadic water point mapping (WPM) efforts would leave huge gaps and result in unacceptable levels of inaccuracy (see this report p.5-6 for some discussion).
The key point about the household survey data is that it is already collected in all developing countries using pretty much the same questions and response categories (though not always, see the same report p.7-9)
What is the JMP not for?
The JMP is not designed to assist complex policy decisions at the national level. It does not aim to tell you about water point functionality, water quality or district-level inequalities. Other instruments such as WPM, routine data collection, and WASH-specific surveys are useful for this.
The nationally-representative household survey approach allows us to see a snapshot of WASH infrastructure usage from the perspective of the household. It can tell us nothing about water points themselves, as the household is the unit of analysis (2).
So, valid critiques of the JMP include that it discounts water quality (because the water may not be “safe”), time taken (because if this is too long it might not constitute “access” by national definitions), or the fact that people use multiple sources. Similarly, with sanitation, there are critiques around interpretation of the definitions of improved and unimproved, and the status of shared sanitation. Furthermore, only the ‘collection’ stage of the sanitation chain and not disposal, treatment etc.,
While these critiques are valid, they are only useful when workable solutions are proposed which would still allow data to be collected using a single method for all developing countries. That is why the current post-2015 process is so important, as it allows people to make suggestions.
Measuring functionality is not suited to household surveys
There is a school of thought arguing that JMP data should be cast aside because it is not perfect and it doesn’t cover functionality. Such arguments are unrealistic because they make the (unachieveable) best the enemy of the good. For example, in the crossfire piece mentioned above, Ned Breslin suggests that an ideal indicator would cover
“the percentage of people who have access to an improved water supply where water flows regularly and reliably, where breakdowns are limited to less than 1 day per month (for repairs), and where the conditions are evident to eventually replace the water system when its operational life has expired.” (Waterlines Oct. 2012, p.259)
Data on that indicator would be nice to have, but it mixes up household experience with infrastructure, so would be extremely difficult (if not impossible) to measure with a household survey. Nationally-representative data could not be collected through WPM without unacceptable blanket assumptions of ‘people per water point’ (see report links above for why this introduces huge errors). Even then, it would require such data to exist in every country and be nationally representative if a future JMP method were to be able to use it, which is not likely in the next 10-15 years. So, to expect JMP to cover functionality in the short to medium-term is to expect too much (notwithstanding footnote (3) below), as health and demography-focused household surveys and censuses are not suited to complex questions around infrastructure management. For the long term, the onus is on those criticising to make workable, costed suggestions of how things could be done better.
Global estimates for MDG indicators should not be confused with the important task of so-called ‘post-implementation monitoring’ at the national level. Such monitoring data is self-evidently important and useful for policy and planning. It would, however, be prohibitively expensive and undesirable to attempt to provide regular global estimates of this kind.
It is also important to note the inherent differences in what is measured by household surveys as opposed to WPM. It is wrong to make direct comparisons between household survey data regarding what infrastructure people say they use (such as JMP) and proportions of functional of water points. For example, the JMP estimates that 60% of the rural population of Liberia used an improved infrastructure as their main source of drinking water in 2011, based on 6 surveys since 2000. It is a coincidence that a recent WPM exercise in Liberia found that 60% of rural WPs are fully-functional, a figure which reduces to 38% when problems of seasonality and quantity are taken into account (3). The household survey is measuring use of infrastructure, whereas WPM is measuring the infrastructure itself. The two are of course related, but not comparable, because vastly different numbers of people use each water point (see e.g. WASHcost 2013 data for 3 African countries).
So, it is not valid to say “data from these 4 or 5 countries show that many water points are not working, so JMP is useless”. Full WPM inventories only exist for a handful of countries, and the extent to which they will be updated regularly is unclear. Until nationally-representative WPM data exists for the vast majority of countries, there is no way for WHO and UNICEF to use the data consistently (4).
So, we should recognise that, yes, the JMP data is not perfect and, yes, it does not provide the level of detail we need for policy-making at the national level. However, for the global level and inter-country comparison, survey-based methods are the best thing we have, and the best we are likely to have for the next 10-20 years.
At the national level, no single source of data is useful for policy-making – a combination must be used. WPM can tell you about infrastructure, but not about who uses it. Household surveys can tell you about the household experience and equity, but not about infrastructure management and sustainability. Utility databases can tell you about water usage and pricing, but not about those not on their network. Ultimately, the answer at the national level has to be to use a combination of these sources. For the global level and inter-country comparison, however, the only thing we have in the medium-term is JMP (with minor tweaks for post-2015). While acknowledging JMP’s weaknesses and looking to improve it, we should therefore support it.
(1) Monitoring the current MDG targets and indicators is the responsibility of the WHO/UNICEF Joint Monitoring Programme (JMP).
(2) As an aside, WP functionality is theoretically built into the question, as DHS asks “what water source do you use most of the time” – if the village handpump is broken, people would presumably not respond “borehole” if they are not in fact using one.
(3) See p.5 of this report http://www.lr.undp.org/documents/pdf/preliminary_analysis_report_liberia11.pdf
(4) Even then, this misses the point that WPM can only tell you about infrastructure, whereas household surveys tell you about people’s actual reported experience, which is more relevant for human development