Surveying Income

For a long time now, I’ve been sceptical of the practice of finding out the average income in a country or state or city or locality by doing a random survey. The argument I’ve made is “whether you keep Mukesh Ambani in the sample or not makes a huge difference in your estimate”. So far, though, I hadn’t been able to make a proper mathematical argument.

In the course of writing a piece for Bloomberg Quint (my first for that publication), I figured out a precise mathematical argument. Basically, incomes are distributed according to a power law distribution, and the exponent of the power law means that variance is not defined. And hence the Central Limit Theorem isn’t applicable.

OK let me explain that in English. The reason sample surveys work is due to a result known as the Central Limit Theorem. This states that for a distribution with finite mean and variance, the average of a random sample of data points is not very far from the average of the population, and the difference follows a normal distribution with zero mean and variance that is inversely proportional to the number of points surveyed.

So if you want to find out the average height of the population of adults in an area, you can simply take a random sample, find out their heights and you can estimate the distribution of the average height of people in that area. It is similar with voting intention – as long as the sample of people you survey is random (and without bias), the average of their voting intention can tell you with high confidence the voting intention of the population.

This, however, doesn’t work for income. Based on data from the Indian Income Tax department, I could confirm (what theory states) that income in India follows a power law distribution. As I wrote in my piece:

The basic feature of a power law distribution is that it is self-similar – where a part of the distribution looks like the entire distribution.

Based on the income tax returns data, the number of taxpayers earning more than Rs 50 lakh is 40 times the number of taxpayers earning over Rs 5 crore.
The ratio of the number of people earning more than Rs 1 crore to the number of people earning over Rs 10 crore is 38.
About 36 times as many people earn more than Rs 5 crore as do people earning more than Rs 50 crore.

In other words, if you increase the income limit by a factor of 10, the number of people who earn over that limit falls by a factor between 35 and 40. This translates to a power law exponent between 1.55 and 1.6 (log 35 to base 10 and log 40 to base 10 respectively).

Now power laws have a quirk – their mean and variance are not always defined. If the exponent of the power law is less than 1, the mean is not defined. If the exponent is less than 2, then the distribution doesn’t have a defined variance. So in this case, with an exponent around 1.6, the distribution of income in India has a well-defined mean but no well-defined variance.

To recall, the central limit theorem states that the population mean follows a normal distribution with the mean centred at the sample mean, and a variance of \frac{\sigma^2}{n} where \sigma is the standard deviation of the underlying distribution. And when the underlying distribution itself is a power law distribution with an exponent less than 2 (as the case is in India), \sigma itself is not defined.

Which means the distribution of population mean around sample mean has infinite variance. Which means the sample mean tells you absolutely nothing!

And hence, surveying is not a good way to find the average income of a population.

Surveying Priorities

Earlier today the Lowy Institute put out the results of a survey it conducted on “India’s views of the world ahead”. While the report contains some excellent insights (including Indians’ perception of various countries), the problem is that it doesn’t establish what people’s priorities are.

For example, there is a question that asks people how important it is that “India has the largest navy in the Indian Ocean”. Some 94% of respondents think it is important, but neither the question nor the answer acknowledges the cost of being the largest navy in the Indian Ocean. Of course, having the largest navy in the Indian Ocean is a great thing to have, but what about the cost?

This is the problem with “uni-directional surveys” – where questions are independent of each other and no relation between factors is established. For example, everyone wants low taxes, high level of government-sponsored welfare, full employment, good wages and a strong military. The reason differences between political parties occur is because it is impossible to have all of it at the same time, and different parties have different positions on the trade-offs.

Table 24 of the Lowy survey illustrates this. The question is about domestic policy goals, and respondents are asked about the importance of each. Is it of any surprise that over 90% of respondents think each and every one of these goals is important?

Extracted from the Lowy Institute report on Indian Views of the World Ahead (http://www.lowyinstitute.org/files/india_poll_2013_0.pdf)
Extracted from the Lowy Institute report on Indian Views of the World Ahead (http://www.lowyinstitute.org/files/india_poll_2013_0.pdf)

In order to capture trade-offs, I propose a different kind of survey. One where the respondent is told “The government suddenly gets an extra Rs. 100 which it has to spend on either strengthening our military or providing food security. What do you choose?”. The survey I propose will have a series of such “binary” questions, where respondents have to allocate the government budget between various programs. That way, the true preferences of the respondents can be captured.

One last point on the presentation of the above table. The survey uses a “4 point Likert scale” (“not at all important”, “not very important”, “fairly important”,”very important”) to record responses. First off, marketing research theory recommends that such scales have an odd number of choices (3 and 5 are the recommended numbers). Secondly, the report has chosen to group the first two choices under “total not important” and the latter two under “Total important”. As you can see from the table, these “total” columns are presented in boldface, thus drawing attention. Consequently, given the amount of information in each table, no one really looks at the columns not in bold face. In other words, the Likert scale could have had only two points (important – not important)!