SK – Page 132 – Pertinent Observations

Rare observations and observed distributions

Over the last four years, one of my most frequent commutes in Bangalore has been between Jayanagar and Rajajinagar – I travel between these two places once a week on an average. There are several routes one can take to get to Rajajinagar from Jayanagar, and one of them happens to be from the inside of Chamrajpet. However, I can count the number of times I’ve taken that route in the last four years on the fingers of one hand. This is because the first time I took that route I got stuck in a massive traffic jam.

Welcome to the world of real distributions and observed distributions. The basic concept is that if you observe a particular event rarely, the distribution you observe can be very different from the actual distribution. Take for example, the above example of driving through inner Chamrajpet. Let us say that the average time to drive through that particular road on a Saturday evening is 10 minutes. Let us say that 99% of the time on a Saturday evening, you take less than 15 minutes to drive through that road. In the remaining 1% of the time, you can take as much as an hour to drive through the road.

Now, if you are a regular commuter who drives through this road every Saturday evening, you will be aware of the distribution. You will be aware that 99% of the time you will take at most 15 minutes to get past, and base your routing decision based on that. When it takes an hour to drive past, you know that it is a rare event and discount it from your future calculations. If, however, you are an irregular commuter like me and happened to drive through that road on that one day when it took an hour you get past, you will assume that that is the average time it takes to get past! You are likely to mistake the rare event as the usual, and that can lead to suboptimal decisions in the future.

In his book The Black Swan, Nassim Nicholas Taleb talks about the inability of people to model for rare events. He says that the problem is that people underestimate the probability of rare events and fail to account for it in their models, leading to blow ups when they do occur. While I agree that is a problem, I contend that the opposite problem can also be not ignored. Sometimes you fail to recognize that what has happened to you is a rare event and thus end up with a wrong model.

Let me illustrate both problems with the same example. Think of a game where 99 times out of 100 you win a rupee. The rest of the time (i.e. 1%) you lose fifty rupees. Regular players of the game, who have “sampled” this enough will know the full distribution, and will take that into account when deciding on whether to play the game. Non-regular players, however, don’t have complete information.

Let us say there are a hundred cards. 99 of them have a +1 written on it, and the 100th has a -50. Let us suppose you pick ten cards. Ninety percent of the time, all ten cards you pick will be a “+1”, and you will conclude that all cards are “+1”. You will model for the game to give you a rupee each time you play. The other 10% of the time, however, you will draw nine +1s and one -50. You will then assume that the expected value of playing the game is Rs. -4 .1( (9 * 1 + 1 * (-50))/10 ). Notice that both times you are wrong in your inference!

So while it is important that you recognize black swans, it is also important that you don’t overestimate their probability. Always remember that if you are a rare observer, the distribution you observe may not reflect the real distribution.

India: Disinvestment Receipts

Common wisdom is that disinvestment in India was on a high back in the days when Atal Behari Vajpayee was Prime Minister, when there was a dedicated Ministry of Disinvestment under Arun Shourie. The UPA, upon coming to power in 2004, disbanded this ministry and common wisdom is that disinvestment stopped as a result of that.

Here, we take a look at disinvestment in India over the years. Here is the total disinvestment amount by year:

You can see that there was a spike in disinvestment in 2003-04, which was Vajpayee’s last year as Prime Minister. You can also see that disinvestment ground to a halt in the first term of the UPA government – possibly as a result of the presence of the Left Front as part of the government. However, you may not have realized that in its second avatar the UPA government has taken up disinvestment with a vengeance, with the receipts in the last four years far exceeding the receipts during Vajpayee’s tenure as Prime Minister.

However, the picture becomes clear if we look at the method of disinvestment. Most disinvestment receipts in the 1990s and in the last five years have come through a sale of minority stakes in PSUs. The disinvestment receipts in the Vajpayee years, however, came mostly through majority stake sales and strategic sales. In other words, there has been no big bang disinvestment in the last ten years – the money the government has made is through quiet sales of minority stakes in PSUs. So one can say that big bang disinvestment has ground to a halt after Vajpayee’s tenure.

Gold: Currency or Commodity?

In today’s Hindu Business Line, S Gurumurthy of the Swadeshi Jagran Manch has an insightful article on the Indian affinity for gold. In this, he talks about gold being the preferred form of savings among the poor and mentions that the preferred form of financing for poor and/or rural households is the “gold loan” (loan issued keeping gold as collateral), often arranged by an informal moneylender. He argues that attempts to regulate gold imports are futile and what instead needs to be done is formalization and regulation of the gold loan industry.

The question one needs to answer when trying to regulate gold is whether it is a currency or a commodity. Or, to “segment along another axis”, whether it is a “conventional asset” or “financial asset”. The thing with “conventional assets” (as opposed to financial assets) is that demand decreases as price increases (most goods and services fall under this category). “Financial assets” on the other hand see the reverse relation – price increases are usually followed by an increase in demand.

Conventional wisdom which governs gold regulation in India (and elsewhere) is that it is a commodity, and a conventional asset. Gurumurthy’s argument is that it should rather be treated as a currency or a financial asset.

The concept of gold being a currency is not new. In fact, if you look at the way currencies were traditionally traded (by the “gold standard”) gold was a de facto currency. The gold standard can be described as gold being the only convertible currency, which could be converted to any national currency at a fixed rate. In the era of the gold standard, it can be argued that all international transactions were effectively priced in gold, and only notionally paid for by means of a national currency.

Despite this background of gold being a currency, however, in India it is regulated as a commodity. Take for example, the customs duty on gold. Drawing an analogy, think of what would happen if a “15% customs duty” were imposed on US Dollars. In other words, every time I converted my US dollars into Indian Rupees, I would have to pay 15% of the value of the transaction to the government as “customs duty”. You might say that is absurd. However, that is exactly what is happening with the customs duty on gold, with the result that gold has started being imported via illegal channels.

The problem with gold is that world over it now behaves like a commodity (after the abolition of the gold standard). In India, however, it behaves more like a currency. Because it internationally behaves like a commodity, standard modern economics treats it as one, and the Indian regulations also treat it such. However, given that gold is (I agree with Gurumurthy) more of a currency than a commodity in India, none of these regulations have worked.

It is time regulators started thinking of gold as a currency and financial asset.

Tata Sky’s Revenue Management (or the lack of it)

Last week Direct To Home (DTH) television provider Tata Sky launched a new channel called “Star World Premiere HD”. As the name suggests it is a high definition channel, and it broadcasts TV shows from the United States simultaneously in India (as opposed to the usual lag of about a year before a show goes on air in India). The business case for this channel itself is particularly strong – for people are nowadays unwilling to wait for a TV series’ release in India and instead download it from illegal torrents. The availability of the TV Series now through a legal channel will discourage torrents, and also allow the producers to make some money via selling advertisements.

The problem, though, is with the pricing. With great fanfare, Tata Sky had launched this “Mega Pack” a few years ago, where for a fixed annual fee (of the order of Rs. 6000) you would get all channels for an entire year. Built in to this plan was the promise that any channels launched during the year would be available for free for all Mega Pack subscribers. It was like the subscribers were paying a premium for the option of not having to pay for any newly introduced channels.

For the last few years, this plan has been going well. Numbers are not available publicly but based on a small sample of people I know, I know that the Mega Pack has a large number of subscribers (Disclosure: I too subscribe to this). Given the “bundled pricing” (which is what the Mega Pack essentially is), Tata Sky is making more money than it would have with subscribers picking only the channels they wanted.

Unfortunately their handling of Star World Premiere HD is likely to change this. With this channel, they have reneged on their promise to provide any new channels for free to Mega Pack subscribers and have asked subscribers to pay Rs. 60 per month for the channel. Over the course of the last week, there has been considerable outrage on Twitter about this. There are reports of people (Mega Pack subscribers) calling up the call centre and talking of failed promises and getting the channel for free. There are other reports of subscribers who last updated their Mega Pack after February 1st 2013 getting this channel free for the remainder of their current Mega Pack subscription period. Tata Sky CEO Harit Nagpal has been on Twitter, and has explained that he wasn’t able to offer the channel for free due to some fee arrangements with the broadcaster (Star TV India). In all, the entire handling of the launch of this new channel has been a mess.

So far, there has been a cozy relationship between Tata Sky and Mega Pack subscribers. Subscribers pay up once a year, no questions asked. In return, they get all channels for free. So far, the renewal process has been fairly mindless – given the seamless experience customers are renewing without much thought.

What this mis-handling of the Star World Premiere HD launch by Tata Sky will do is to get the customers thinking. With this one breakage of promise on the part of the distributor, customers whose Mega Packs come up for renewal will soon start asking themselves if Tata Sky will not renege on their promise with respect to other channels also. While porting across DTH operators is still not easy, what is likely to happen is that customers will critically evaluate whether they need all the channels they are paying for, and opt for a package which only includes channels they need (after a regulatory change last year, all channels have to be offered a la carte).

In short, Tata Sky’s mishandling of this episode is likely to reduce the number of their Mega Pack subscribers, and thus a decrease in their overall revenue. The additional amount they are charging for Star World Premiere HD is Rs. 720 per subscriber per year. The price gap between the price of a Mega Pack and any other pack is much more than that. In short, it has been a process of extremely bad revenue management by Tata Sky.

PS: We would be interested in offering revenue management consulting to Tata Sky or any of its competitors. If you are from any of these companies, get in touch with us through the contact form in the Contact page.

Largest crop by state

We will continue to stick with the state-wise data on agriculture. In this edition, we will look at the largest crop by state, by year. We define this as the crop with the biggest acreage in the state.

No fancy visualizations here. Just data presented in a table. Two tables, actually, one for kharif and one for rabi. For each year these two tables show the biggest crop per state.

Offered without comment.

Major Crops in India, by State: Kharif Season

(click on images for larger size)

Has Manchester United Really Been a Disaster This Season?

The talk of this English Premier League season so far has been the poor performance of defending champions Manchester United. After six rounds of matches, the Red Devils lie twelfth, with only seven points from six games. While we are barely one sixth our way into the season (where each team plays 38 games), people are talking about the loss of the United “magic” following the departure of its long-standing coach Sir Alex Ferguson last season break. Other analysts, however, are quick to point out that United started off with a rather tough fixture list this year, having visited Liverpool and Manchester City and hosted Chelsea already.

A snapshot of the Premier League Table, thus, does not paint a particularly accurate picture. It is possible that at a particular interval in the season you can go through a series of tough games, or easy games. The fixture schedule each year is different and thus early league positions can be deceptive.

On this page, we will try to adjust for that. This post is going to be updated every week, and what we will do is to compare this season to the earlier one and see how teams are performing relative to the same set of fixtures last year. Thus far this season, Manchester United have played Chelsea, West Bromwich and Crystal Palace, all at home and have traveled to Swansea, Liverpool and Manchester City. What we do is to compare the performance of Manchester United in these six games to the corresponding six fixtures last season.

To adjust for relegation and promotion, the teams that placed 18th to 20th last season are respectively replaced by the three qualifiers (in order) from the Championship. Thus, we will assume that Cardiff City will replicate Wigan’s performance, Hull City Reading’s and Crystal Palace has taken QPR’s place.

Thus, we get table 1 – the “points change graph”, which shows how many additional points each team has got so far relative to the corresponding fixtures last year.

This table confirms that irrespective of the fixture list, Manchester United’s performance so far this season is significantly inferior to that of last year. At the other end, Southampton and Tottenham Hotspur have vastly improved from last season.

Next, we will assume that the rest of the season would go as it did last season, and see how the table has changed taking into account this season’s performance.

Again, it is early in the season yet, but if the rest of this season were to go as it did last year, Manchester United is likely to still win the title, but only just. Interestingly, Tottenham will be second if the rest of the season goes as per last season’s performances.

Growing wheat and rice in India

One of the most massive data sets on data.gov.in is district-wise data on the total area under cultivation and production of various crops for each season for each year from 1998 to 2010. In this post we will look at which states utilize the most amount of land growing each crop.

First, a note on the data. The data is district-wise and season-wise. The irritating thing is that the seasons are not mutually exclusive. The seasons in the data set are “Summer”, “Kharif”, “Autumn”, “Winter”, “Rabi” and “Whole Year”. First of all, I don’t know what “Autumn” means in India – as far as I know India doesn’t have one such season. Granting some liberties, it is irritating that seasons overlap.

Here is how I’ve consolidated the data. For either crop, for each year, I took the total area under cultivation for each state for each season. Next, I looked at the maximum area under cultivation in a particular state at a particular point in time (any time in the 12 years of data I have). So the data I present in this post is the maximum area in a particular state that was under a particular crop at some point of time in the 1998-2010 time period.

So, who grows wheat in India? The graph here shows the states with the maximum area under wheat:

Notice that Uttar Pradesh and Madhya Pradesh have had much more area under wheat than Punjab or Haryana. Also notice that only eight states in India have ever had more than 5000 square kilometers of land growing wheat. To put this in perspective let us look at rice:

Here I have put the cutoff (for entry to the graph) at 10000 square kilometer, and yet fourteen states make the cut. In terms of area under cultivation at least, we can say that we are a predominantly rice growing country. Again, in rice, notice that Uttar Pradesh and Madhya Pradesh have more area under cultivation of rice than more “traditional” rice growing areas like Orissa or West Bengal.

Which state has the biggest proportion of its land area under wheat? And rice? The next two graphs show the proportion of land under wheat and under rice in each state (note again that these are maximum values over a decade).

There is much more information in this particular data set. We will revisit it in subsequent posts.

Cartels, good and bad

This post is about two professional cartels in India, and why one is better than the other. The “better” cartel is the Institute of Chartered Accountants of India (ICAI). The “worse” is the Medical Council of India (MCI). As the names suggest, they regulate the profession of chartered accountants and doctors respectively. And the way the former works is better than the latter.

First of all, let me convince you that these two are cartels. The basic concept is that in order to practice as a Chartered Accountant in India, you need certification from the ICAI. And who does the ICAI consist of? Other CAs. So it is nothing but a trade guild, which tries to control who gets to join the guild. It is a similar case with the MCI and doctors. Doctors trying to control who else can be doctors. As the more perceptive of you might have figured out, it is in the interest of both these guilds to not admit too many new members, since that would lead to supply of their profession to a level that significantly affects profit margins for the incumbents.

Now that we have established why these two are cartels, and that they both have an interest in restricting membership, let us see how they go about the process.

The ICAI follows what can be described as “exit level testing”. There are no restrictions on anyone wanting to be a CA – all you need is a high school (12th standard) degree with mathematics as one of your subjects. They have three levels of examination – “basic”, “intermediate” and “final”, and one needs to clear all of these in order to become a member of the guild. And how does the guild control membership? By making these examinations super-tough, so that only a select few pass these exams every year. There are several “CA institutes” who train students for these examinations. And there is no restriction (AFAIK) on anyone opening one such institute.

The MCI does it differently. Anyone with a recognized degree in medicine is automatically a member of the MCI. They regulate the numbers instead by controlling the number of medical colleges, so that only a select few can even aspire to get into the MCI. More importantly, the entry to medical colleges is not strictly on “merit” – colleges are free to allocate a certain portion of the seats on discretion, and they do so based on recommendations, donations, etc. I’m not really saying any of this is wrong. Just describing the situation as it is. However, when you combine this with the fact that an entry into a medical college guarantees membership of the MCI (provided you pass your college exams, which shouldn’t be too hard), it effectively means that you can buy your way into the MCI.

Actually, thinking about it, this option of creating additional membership of the MCI “upon payment” is a masterstroke by that guild. The concept is that when people pay large sums of money to gain entry, they are not going to be in a hurry to look to slash profit margins (key here is the fact that the amount of work a doctor can do is constrained by his/her time, so doctors cannot play the “volume game”). So the pricing of these seats ensure additional revenue for the MCI and their constituent colleges, while not compromising on the members’ margins.

Note, however, that it is not possible to buy your way into the ICAI. Yes, aspiring members who seeks to buy their way in might buy “training” and “apprenticeship” under the best CAs who are members of the cartel, but still, to get a membership they need to pass the examination, which is not easy at all. Contrast this with the MCI where either money or the right connections get you straight in.

I’m not saying that the ICAI is a wonderful guild. Cases such as the Price Waterhouse – Satyam case or the Deloitte-FTIL case have shown that the profession is deeply flawed, and doesn’t regulate itself adequately. All I’m saying that the entry criteria it uses, as opposed to the one used by the MCI, ensures a higher quality in terms of the ability of its members.

As for me, I’m happy that I’m involved in a profession (or professions) that don’t need any certification or guild membership.

The Raghuram Rajan Committee report on Composite Development Index of States

In July this year, at a resort near Bangalore (yes, we at Takshashila do sometimes play resort politics) I got the fifth batch of the GCPP to work on the problem of building an index which measures the development of various Indian states in the last 10 years. I used this case as a reference while doing my module on Analytical Methods in Public Policy. This was as part of one of the weekend workshops which are part of the GCPP. As part of this exercise I taught them how to pick variables, how to measure them, procure data, look for interactions between variables and then combine them to form an index.

It is interesting that a couple of months after that session, the report of the Raghuram Rajan Committee on Composite Development Index of States has been published. I will use this blog post to give my comments on that report as I go through it. Since I’m going to be effectively “live-blogging” my reading of the report, the rest of this post is in bullet points.

Also, in keeping up with my title of “resident quant” I will try as much as possible to restrict my comment to the data and methodology, and not comment on economic issues. However, it is likely that I might go on economic rants here or there.

The first paragraph of the executive summary states that the reason we adopted a command and control model after independence was so that we don’t increase the inequalities across regions and states. This is the first time I’m hearing this story
The index is based entirely on publicly available data. I think this is a good thing.
Each state gets 0.3% of the total available pool, irrespective of its size. Of the remaining 91.6% (28 states => 8.4% fixed payment), 3/4th will be distributed based on “need” and 1/4 on “performance”. Nearly seventy years since independence, I’m of the opinion that this ratio should be less skewed towards “need”
Arbitrary cutoffs have been drawn at scores of 0.6 and 0.4 to classify states as “least developed” and “less developed”. While these are round numbers, I’m not yet sure they make sense.
The report alludes to the “resource curse”, which is a good thing.
Quote: “The Normal Central Assistance (NCA) grant, which is distributed to states as per Gadgil-Mukherjee formula based on categorization of “Special Category” and “General Category” states, constituted only about 3.8 per cent of total resources transferred to States and 8.2 per cent of plan transfers.” (emphasis mine)
The underdevelopment index has ten components. I won’t comment on the wisdom of the number of quality of the components chosen.
It is a good thing that Mean Per Capita Consumption Expenditure is used as a measure of richness rather than per capita Net State Domestic Product. As the report argues, the latter can include economic activity that doesn’t really reach the people, and is hence not as good a measure as consumption expenditure.
Table 1 (on page 17 of the report) gives the correlations between the metrics chosen. I think it is a fantastic thing that they have chosen to present the correlations in the first place (something ripe to be pushed under the carpet). As expected, a number of chosen variables are highly correlated.
Correlation between Consumption Expenditure and Urbanisation is 75%!! Similarly, correlation between expenditure and female literacy is 58%.
Then comes the damp squib – the excitement induced by presenting the correlation table is doused by the statement that each of the ten parameters are going to be accorded equal weight. This is disappointing on several counts: firstly, the sheer arbitrariness (remember that ‘equal allocation’ is as arbitrary as any other distribution). Next, that the correlations are thrown out of the window and certain factors are likely to get more weight. Then, the fact that this makes it easily manipulable by adding or deleting factors of choice. I’m so disappointed by this one decision that I’m putting this entire point in boldface. Apologies.
The report acknowledges that broadly categorizing states into “developed” and “under developed” creates issues of moral hazard. However, rather than fully doing away with the division, the committee (again, disappointingly) takes a “middle path” by splitting two categories into three. I suspect some mathematical brain is involved here, in that the next committee will increase number of categories to four, and the one after to five, until a time when each state (finally!!) becomes its own category
To convert per capita allocation to state-wide allocation, the formula uses a combination of population and area. I agree that it is tougher to provide infrastructure to thinly populated areas, so this combination is fair. It reminds me of my days in airline cargo pricing when we would similarly adjust between the weight and volume of a piece of cargo.
Performance index is computed based on changes in the development index over time. This is a good thing. Shows the committee is “eating its own dog-food”
This is the first time “performance” is being used as a criterion for fund distribution. So the 25% weight is a good start. I retract my earlier abuse of this ratio.
The committee recommends that this analysis be carried out every five years, since a good amount of data used in calculating indices are published at that frequency. Also considering that’s the frequency of finance commissions, it is a good thing.
The report tries to bolster its credibility by showing that the index is highly correlated with the UN Human Development Index. I like it that a scatter plot and regression line have been presented
The allocation based on performance is again skewed in favour of less developed states. So you are likely to get more if 1. You are underdeveloped and 2. You have shown an improvement. I think this is fair.
One good thing is that the formula is plug and play. It is “timeless” in a sense. At any given future point in time, you can simply look up the data points that are required and just construct the index. There is no human intelligence required for that effort
There is heavy reliance on NSSO data, and I’m not sure that’s a good thing since it is “survey data”. I think it might have been better to have used data from census.
The committee actually examined the option of weighting factors based on squared factor loadings from a Principal Component Analysis (*applause*) and found that the index thus constituted was 99% correlated with the one using simple arithmetic averages, and thus decided to go with the simpler formula. I’ll still continue to keep the earlier point in bold, though
Each “sub-component” was normalized between 0 and 1 using a simple linear formula (higher number indicating greater under-development). I like it that they used this rather than a rank ordering metric.
The report includes a sensitivity analysis to show that the ranking and index values are robust. Again, applause
A dissent note from Committee member Shaibal Gupta indicates that there are problems in using a simple weighted average rather than data from the PCA

Finally, despite all the talk of transparency and ease of calculation, the report itself does not contain either the index number or the component values for various states. I hope the data has been released (and if it has, please help me by giving me the link). If not, we should campaign for the data to be given out to the public in a CSV (or equivalent) format through the government data portal http://data.gov.in

Nate Silver Interview At HBR Blogs

HBR Blogs has interviewed Nate Silver on analytics, building a career in analytics and how organizations should manage analytics. I agree with his views on pretty much everything. Some money quotes:

HBR: Say an organization brings in a bunch of ‘stat heads’ to use your terminology. Do you silo them in their own department that serves the rest of the company? Or is it important to make sure that every team has someone who has the analytic toolkit to pair with expertise?

Silver: I think you want to integrate it as much as possible. That means that they’re going to have some business skills, too, right? And learn that presenting their work is important. But you need it to be integrated into the fabric of the organization.

And this:

Silver: If you can’t present your ideas to at least a modestly larger audience, then it’s not going to do you very much good. Einstein supposedly said that I don’t trust any physics theory that can’t be explained to a 10-year-old. A lot of times the intuitions behind things aren’t really all that complicated. In Moneyball that on-base percentage is better than batting average looks like ‘OK, well, the goal is to score runs. The first step in scoring runs is getting on base, so let’s have a statistic that measures getting on base instead of just one type of getting on base.’ Not that hard a battle to fight.

And this:

Silver: A lot of times when data isn’t very reliable, intuition isn’t very reliable either. The problem is people see it as an either/or, when it sometimes is both or neither, as well. The question should be how good is a model relative to our spitball, gut-feel approach.

Go on and read the whole interview.