The 2015 target date for achieving the Millennium Development Goals (MDGs) is fast approaching, but there is very little discussion of the validity of the indicators used to measure progress.

In particular, there has been little attention given to the problems that arise when assessments of progress are based on household surveys. These are inappropriate for obtaining information about the poorest of the poor. Typically, they omit by design those not in households because they are homeless; those who are in institutions; and mobile, nomadic or pastoralist populations; and, in practice, household surveys will typically under-represent those in fragile, disjointed or multiple occupancy households; and those in urban slums and insecure areas of a country.

Those six subgroups constitute a pretty comprehensive ostensive definition of the "poorest of the poor." Between 300 and 500 million people – mainly in developing countries – will be missed worldwide from the sampling frames of household surveys.

Introduction

Millennium Development Goals

The 2015 target date for achieving the Millennium Development Goals (MDGs) is fast approaching. Moreover, the most recent report on progress (United Nations Development [UNDP] 2012) claims that three important targets have already been reached, including reducing poverty by half from the 1990 level, and significantly improving the lives of 200 million slum dwellers, well ahead of the 2020 deadline. Already civil society, academics, governments and the UN system are thinking about what the next era of development should look like, but there is very little discussion of the validity of the indicators used to measure progress. In particular, there has been little attention to the problems that arise when, as is increasingly common in developing countries, population estimates and assessments of progress toward more than half (29 of the 54) of the indicators used for monitoring progress toward the Millennium Development Goals are based on household surveys (see Appendix 1).

Population Counts

For several decades and in some countries for centuries, populations have been counted through national, usually decennial, censuses in which enumerators go to households. Inter-censal population estimates have usually depended on reliable birth and death registration systems. In most middle- and low-income countries, however, vital registration systems have never been fully functioning (Powell 1981), and there has been a similar decline in donor interest in censuses and vital registration systems (Setel et al. 2007). Instead, there is an increasing reliance on large-scale standardized household surveys for the basic data.

Many countries run national economic and social surveys to provide detailed information on consumer prices, income and employment, and other relevant data for planning. This move away from censuses to relying on surveys raises the obvious problem that drawing a sample for a survey depends on having a sampling frame in the first place which is frequently based on the census. Clearly, any problem with the census, if used as the sampling frame for a national survey, will lead to that sampling frame being biased. But there is –strangely – little recognition of these problems.

Censuses

Population censuses have always faced problems of complete enumeration. Groups of adults have been excluded from censuses in some countries for political and/or practical reasons. Non-citizens, cultural minorities or marginalized groups, and specific categories of prisoners or rebels who object to government oversight have often been excluded for political reasons (Buettner and Garland 2008), and although less frequent and certainly more transparent, this still continues (Indigenous Portal 2011; Abbasi-Shavazi and Sadeghi 2011).

Therefore, the general problem that censuses are not necessarily complete is well understood (Carr-Hill 2009). At the same time, there is an emerging consensus as to what constitutes good census practice (United Nations 2008); and censuses that follow those UN guidelines will usually overcome many of the deficiencies in earlier censuses. The guidelines are clear but there can still be problems in practice, as described below.

Housekeeping concept: While Cinderella is a fairy tale, the exclusion of poor servants from the census count in rich households (even though they usually share some of the household food), especially in Asia, is a reality, and their personal poverty is therefore missed.

Mobile populations: In developed countries, the young, highly mobile – usually male – population is also difficult to count, especially when they live in collective households, but they are relatively well-off; in developing countries, they may well be among the poorest.

Homelessness and counting de facto rather than de jure populations: These are difficult to count, especially where there are disputes over nationality (Refugees International 2007); equally, there are several millions internally displaced in many countries, either as a result of civil war or because environmental change (e.g., floods, nuclear accidents) makes their homes uninhabitable; and, although there are periodic counts, there is no regular database anywhere.

Institutional populations: There are a number of different types of institutions (care homes, [some] factory barracks, hospitals, the military, prisons, refugee camps, religious orders and school dormitories), and there is considerable variation in opinion on whether or how they should be included in the population count. They are often simply counted as special census blocks.

Careful census reporting documents how well these groups have been enumerated, and most categories are included in estimated census population counts of developed countries but not in those of many developing countries. Moreover, in many developing countries the census enumerators are often police or other government officials, who tend to use security-based national identity cards or family registration cards to validate the citizenship status of those they are enumerating (Di 2010; Dwinosumono 2006).

Assessing Poverty

In assessing the absolute level of poverty or the absolute levels of illness, household surveys are an inappropriate instrument for obtaining information about the poorest of the poor, especially in developing countries. This is because household surveys, with rare exceptions, typically omit, by design,

1. those not in households because they are homeless,
2. those in institutions, including refugee camps, and
3. mobile, nomadic or pastoralist populations.

In addition, in practice, because these individuals are difficult to reach, household surveys will typically under-represent

1. those in fragile, disjointed or multiple occupancy households (because of the difficulty of identifying them),
2. those in urban slums (because of the difficulty of interviewing), and
3. certain areas of a country deemed to pose a security risk.

If one wanted a practical – as distinct from theoretical – definition of the "poorest of the poor," the above collection of six population subgroups could hardly be bettered.

Census officials, because of the difficulty of enumeration, even in developed countries, often only estimate those groups' size and location, so their members are not included in the available sampling frames for household surveys. In developing countries, these marginalized groups may not be included at all, even in the estimated population counts. The lack of recognition of these problems with the design and implementation of household sample surveys, particularly in developing countries, has meant that there has been no systematic attempt to estimate the size and distribution of the population groups "missing" from the sampling frames of national household surveys.

How Many Are Potentially "Missing" from Population Counts and from Sampling Frames of Household Surveys?

The focus here is on groups for which there are credible sources, and that are normally among the poorest. Other groups, such as economic and environmental migrants (Myers 1997), are not considered below.

The Homeless

Rather obviously, household surveys omit the homeless and street children. Estimating numbers is very difficult. Over 20 years ago, the United Nations Children's Fund (UNICEF) estimated that there were about 100 million street children (UNICEF 1989) worldwide. The figure is still commonly cited but has no evidence base (Green 1998; Hecht 1998). But, however many there are, they will not be covered by household surveys.

Institutionalized Populations

Household surveys, by definition, omit from their sampling frame those in institutions: care homes, (some) factory barracks, hospitals, the military, prisons, refugee camps, religious orders and school dormitories.

Care Homes and Hospitals: Those in hospitals and care homes will on average be poorer because morbidity is associated with poverty (Lopez 2002), although that is less true for older people. There are an estimated 20 million hospital beds worldwide (World Health Organization [WHO] 2011).

Military: The Central Intelligence Agency's (CIA) World Factbook (CIA 2011) documents 92 million military personnel worldwide (including reservists).

Prison: Those in prisons will usually be poorer, and estimates of the world's total prison population are around 9.8 million (Walmsley 2003). None of these 9.8 million will be included in the sampling frame of household surveys.

Refugees: Refugees are not considered part of any nation's population so they cannot be included in survey sampling frames nor make any contribution to survey-based estimates. However, the United Nations High Commissioner for Refugees (UNHCR 2010) publishes figures annually on numbers of officially registered refugees, internally displaced persons and stateless persons. The worldwide total is 36.5 million, but these figures do not include the large number of illegal immigrants, most of whom will not be counted in a national population and, of course, not in the sampling frames of household surveys.

Nomadic and Pastoralist Groups by World Region

The permanently mobile are usually excluded from household surveys. In particular, censuses and surveys in developing countries have difficulty enumerating nomadic/pastoralist populations, who have much less access to services (UNICEF 2007); and, while it is difficult to assess their income and wealth, and there clearly are some who are rich-in-kind (or asset rich), the majority are usually poor in all senses.

There is no reliable information available on the number of nomadic pastoralists, including sea-faring mobile communities, worldwide. The only internationally comparable source is that compiled by the International Livestock Research Institute (Thornton et al. 2003), based partly on livestock numbers, although they will include agro-pastoralists who are semi-sedentarized. These estimates suggest there are 217.5 million in pastoralist households.

The Difficult to Reach

A) Fragile and Disjointed Households

The task of the census enumerator or survey interviewer is made much more difficult when the household structure is ambiguous or undefined so that either identifying the household head and/or counting the numbers in the household is almost impossible.

B) Urban Slums

The most recent estimates from UN-Habitat (2011) are that more than a billion people are living in urban slums in developing countries, but information is rarely disaggregated according to intra-urban location, and the poorest urban populations are often simply not included in data gathering.

The few surveys conducted in those slums show sharp gradients according to income quintiles within urban populations (UN-Habitat 2003). But, given the very high levels of mobility, it would seem reasonable to assume that a substantial minority of those households in the slum areas of cities in developing country are uncounted in any census. Moreover, even where they are counted in censuses, many would (because of interviewer reluctance) in practice, be excluded from sampling frames.

C) Insecure or Isolated Areas

This will obviously vary according to context and so will be a much larger problem in specific countries. Given the security situation – or simply difficulty of transport – in many countries, it can often be difficult for the implementing institutions to carry out a fully representative survey or census.

Overall Estimates, Discussion and Conclusions

Overall Estimates

The numbers vulnerable to undercounting are summarized in columns 3 and 4 of Table 1. Not all those missing populations are poor. Estimates of the poor are assumed to include all those in prison and all refugees (and there are assumed to be at least as many unofficial refugees as official refugees); all those hospitalized in low- or low-middle-income countries (about half), but only those in the military in low-income countries (about 20%); and between 20% and 40% of nomadic populations. In addition, based on informal conversations with census officials in several countries and on the difficulties of using satellite imagery in slum areas (Schurman 2009) (because of the need to rely on key informants and the visual obscurity of some structures), the number of slum dwellers missing is estimated at between 10% and 20%.

Worldwide, the totals in the subsections above add up to between 194 and 370 million (Table 1). Moreover, the estimates do not include the homeless, those in fragile or disjointed households, or those in areas where there are security risks. Most of the homeless are probably from urban slums, so there would be double counting, but the other two categories (large, but of unknown size) are definitely additional. Estimates of between 300 and 500 million worldwide would be reasonable.

Censuses and Sampling Frames

Counting Displaced and Illegal Groups

Census organizations in developed countries use several procedures for estimating the numbers of illegal immigrants. But those procedures would not work for South–South illegal migration. Moreover, there are other omitted subgroups, often quite large: for example, scheduled castes and tribes in India (Government of India, Ministry of Social Justice and Empowerment 2012) and illegal servants in rich households.

Counting and Sampling Nomads and Pastoralists

These are difficult to count simply because they are moving. Reasonable samples have been obtained through combining local-level surveys with remote sensing of livestock (Galvin et al. 2001), but documenting change in their human population remains, on the whole, elusive.

Counting Urban Slum Populations

Any face-to-face interviewing approach will be unreliable both because of the lack of a sampling frame and because respondents will be suspicious of the reason for the questions because they are illegally resident. But even if the census organization were to make available a listing of "houseless" people, given the high levels of intra-slum mobility, this would be an unreliable sampling frame for surveys.

Solutions and Conclusions

Accurate Censuses?

International organizations should support national census organizations in developing and testing procedures for counting pastoralists and other nomads (gypsies, highly mobile workers, long-distance truck drivers, travellers, etc.) and those in urban slums (Angeles et al. 2009; Schurman 2009); and in adopting best practice from UN guidelines. But the same procedures do not solve the sampling frame problem of household surveys, because of the delays between census and survey. In the absence of any simple solution, this author (Carr-Hill 2012) has shown that it may be possible to make estimates of the missing populations. It is crucial to develop similar methods more systematically, with an agreed theoretical basis.

Measuring Progress toward the Millennium Development Goals

These problems are urgent because these data are input for more than half the indicators used for assessing progress toward the Millennium Development Goals (Appendix 1) in developing countries. In particular, the UNDP (2012) claim that the goal of halving the poverty rate has been reached cannot be taken at face value. Even if there has been no growth in the number of uncounted – which seems unlikely given the collapse of vital and registration systems – the missing population means that the reported reduction of 50% from 2 billion to 1 billion in fact reflects a reduction of 43% from 2.3 billion to 1.3 billion (and the latter figure is probably a substantial underestimate).

Population undercounting means that any social program risks ignoring the poorest of the poor. This blindness is a public scandal affecting between 300 and 500 million of the poorest people in developing countries (between 4.5% and 7.0% of the total world population). It should be addressed immediately by international and national organizations, in terms of developing and testing appropriate procedures for counting.

Appendix I. Indicators for Monitoring Millennium Development Goals that are dependent on population data

Comments

Be the first to comment on this!