correlation – Page 2 – Pertinent Observations

Correlation in defence purchases

Nitin Pai has a nice piece on defence procurement in Business Standard today. He writes:

Even if the planning process works as intended, it still means that the defence ministry merely adds up the individual requirements and goes about buying them. This is sub-optimal: consider a particular emerging threat that everyone agrees India needs to be prepared for. The army, navy and air force then prepare their own strategies and operational plans, for which they draw up a list of requirements. At the back of their minds, they know that the defence budget is more-or-less divided in a fixed ratio among them.

What he is saying, in other words, is that the defence ministry simply takes the arithmetic sum of demands from various components of the military, rather than taking correlation into account.

Let me explain using a toy example.

Let’s say that the Western wing of the Indian army (I’m making this up), the one that guards the border with Pakistan, wants 100 widgets that will come useful in case of a war. Let’s say that the Eastern wing of the Indian army, which guards the China border, wants 150 such widgets for the same purpose. The question is how many you should purchase.

According to Nitin, the defence ministry now doesn’t think. It simply adds up and buys 250. The question is if we actually need 250.

Let’s assume that these widgets are easily transportable, and let’s assume that the probability of a simultaneous conventional conflict with Pakistan and China is zero (given all three are nuclear states, this is a fair assumption). Do we still need 250 widgets? The answer is no, we only need 150, since we can quickly swing them over to where they are most required, and at the maximum, we need 150!

This is a case of negative correlation. There could be a case of positive correlation also – perhaps the chance of an India-China conventional conflict actually goes up when an India-Pakistan conventional conflict is on, and this might lead to more prolonged battles, meaning we might need more than 250 widgets! Or we have positive correlation.

The most famous example of ignoring correlation was the 2008 financial crisis, when ignored positive correlation led to mortgage backed securities and their derivatives blowing up. The Indian defence ministry can’t afford such a mistake.

Narendra Modi and the Correlation Term

In a speech in Canada last night, Prime Minister Narendra Modi said that the relationship between India and Canada is like the “2ab term” in the formula for expansion of $(a+b)^2$ .

India and Canada are like that “2ab” that comes from the formula of (a+b)square: PM #ModiinCanada pic.twitter.com/1IpCJ0aQK8

— ANI (@ANI_news) April 16, 2015

Unfortunately for him, this has been widely lampooned on twitter, with some people seemingly not getting the mathematical reference, and others making up some unintended consequences of it.

In my opinion, however, it is a masterstroke, and brings to notice something that people commonly ignore – what I call as the “correlation term”. When any kind of break up or disagreement happens – like someone quitting a job, or a couple breaking up, or a band disbanding, people are bound to ask the question of whose fault it was. The general assumption is that if two entities did not agree, it was because both of them sucked.

However, considering the frequency at which such events (breakups or disagreements ) happen, and that people who are generally “good” are involved in such events, the badness of one of the parties involve simply cannot explain them. So the question arises – if both parties were flawless why did the relationship go wrong? And this is where the correlation term comes in!

It is rather easy to explain using vector calculus. If you have two vectors $A$ and $B$ , the magnitude of the sum of the two vectors is given by $\sqrt{|A|^2 + |B|^2 + 2 |A||B| cos \theta}$ where $|A|,|B|$ are the magnitudes of the two vectors respectively and $\theta$ is the angle between them. It is easy to see from the above formula that the magnitude of the sum of the vectors is dependent not only on the magnitudes of the individual vectors, but also on the angle between them.

To illustrate with some examples, if A and B are perfectly aligned ( $\theta = 0, cos \theta = 1$ ), then the magnitude of their vector sum is the sum of their magnitudes. If they oppose each other, then the magnitude of their vector sum is the difference of their magnitudes. And if A and B are orthogonal, then $cos \theta = 0$ or the magnitude of their vector sum is $\sqrt{|A|^2 + |B|^2}$ .

And if we move from vector algebra to statistics, then if A and B represent two datasets, the “ $cos \theta$ ” is nothing but the correlation between A and B. And in the investing world, correlation is a fairly important and widely used concept!

So essentially, the concept that the Prime Minister alluded to in his lecture in Canada is rather important, and while it is commonly used in both science and finance, it is something people generally disregard in their daily lives. From this point of view, kudos to the Prime Minister for bringing up this concept of the correlation term! And here is my interpretation of it:

At first I was a bit upset with Modi because he only mentioned “2ab” and left out the correlation term ( $\theta$ ). Thinking about it some more, I reasoned that the reason he left it out was to imply that it was equal to 1, or that the angle between the a and b in this case (i.e. India and Canada’s interests) is zero, or in other words, that India and Canada’s interests are perfectly aligned! There could have been no better way of putting it!

So thanks to the Prime Minister for bringing up this rather important concept of correlation to public notice, and I hope that people start appreciating the nuances of the concept rather than brainlessly lampooning him!

How my IIMB Class explains the 2008 financial crisis

I have a policy of not enforcing attendance in my IIMB class. My view is that it’s better to have a small class of dedicated students rather than a large class of students who don’t want to be there. One of the upsides of this policy is that there has been no in-class sleeping. Almost. I caught one guy sleeping last week, in what was session 16 (out of 20). Considering that my classes are between 8 and 9:30 am on Mondays and Tuesdays, I like to take credit for it.

I also like to take credit for the fact that despite not enforcing attendance, attendance has been healthy. There have usually been between 40 and 50 students in each class (yes, I count, when I’ve bamboozled them with a question and the class has gone all quiet), skewed towards the latter number. Considering that there are 60 students registered for the course, this translates to a pretty healthy percentage. So perhaps I’ve been doing something right.

The interesting thing to note is that where there are about 45 people in each class, it’s never the same set of 45. I don’t think there’s a single student who’s attended all of my classes. However, people appear and disappear in a kind of random uncoordinated fashion, and the class attendance has remained in the forties, until last week that is. This had conditioned me into expecting a rather large class each time I climbed up that long flight of stairs to get into class.

While there were many causes of the 2008 financial crisis, one of the prime reasons shit hit the fan then was that CDOs (collateralised debt obligations) blew up. CDOs were an (at one point in time) innovative way of repackaging receivables (home loans or auto loans or credit card bills) so as to create a set of instruments of varying credit ratings.

To explain it in the simplest way, let’s say I’ve lent money to a 100 people and each owes me a rupee each month. So I expect to get a hundred rupees each month. Now I carve it up into tranches and let’s say I promise Alice the “first 60 rupees” I receive each month. In return she pays me a fee. Bob will get the “next 20 rupees”, again for a fee. Note that if fewer than 60 people pay me this month, Bob gets nothing. Let’s say Eve gets the next 10 rupees, so in case less than 80 people pay up, Eve gets nothing. So this is very risky, and Eve pays much less for her tranche than Bob pays for his which is in turn much less than what Alice pays for hers. The last 10 rupees is so risky that no one will buy it and so I hold it.

Let’s assume that about 85 to 90 people have been paying on their loans each month. Not the same people, but different, like in my class. Both Alice and Bob are getting paid in full each month, and the return is pretty impressive considering the high ratings of the instruments they hold (yes these tranches got rated, and the best tranche (Alice’s) would typically get AAA, or as good as government bonds). So Alice and Bob make a fortune. Until the shit hits the fan that is.

The factor that led to healthy attendance in my IIMB class and what kept Alice and Bob getting supernormal returns was the same – “correlation”. The basic assumption in CDO markets was that home loans were uncorrelated – my default had nothing to do with your default. So both of us defaulting together is unlikely. When between 10 and 15 people are defaulting each month, that 40 (or even 20) people will default together in a given month has very low probability. Which is what kept Alice and Bob happy. It was similar in my IIMB class – the reason I bunk is uncorrelated to the reason you bunk, so lack of correlation in bunking means there is a healthy attendance in my class each day.

The problem in both cases, as you might have guessed, is that correlations started moving from zero to one. On Sunday and Monday night this week, they had “club selections” on IIMB campus. Basically IIMB has this fraud concept called clubs (which do nothing), which recruiters value for reasons I don’t know, and so students take them seriously. And each year’s officebearers are appointed by the previous year’s officebearers, and thus you have interviews. And so these interviews went on till late on Monday morning. People were tired, and some decided to bunk due to that. Suddenly, there was correlation in bunking! And attendance plummeted. Yesterday there were 10 people in class. Today perhaps 12. Having got used to a class of 45, I got a bit psyched out! Not much damage was done, though.

The damage was much greater in the other case. In 2008, the Federal Reserve raised rates, thanks to which banks increased rates on home loans. The worst borrowers defaulted, because of which home prices fell, which is when shit truly hit the fan. The fall in home prices meant that many homes were now worth less than the debt outstanding on them, so it became rational for homeowners to default on their loans. This meant that defaults were now getting correlated! And so rather than 85 people paying in a month, maybe 45 people paid. Bob got wiped out. Alice lost heavily, too.

This was not all. Other people had bet on how much Alice would get paid. And when she didn’t get paid in full, these people lost a lot of money. And then they defaulted. And it set off a cascade. No one was willing to trade with anyone any more. Lehman brothers couldn’t even put a value on the so-called “toxic assets” they held. The whole system collapsed.

It is uncanny how two disparate events such as people bunking my class and the 2008 financial crisis are correlated. And there – correlation rears its ugly head once again!

The problem with precedence

One common bureaucratic practice across bureaucracies and across countries is that of “precedence”. If a certain action has “precedence” and the results of that preceding actions have been broadly good, that action immediately becomes kosher. However, from the point of view of logical consistency, there are several problems with this procedure.

The first issue is that of small samples – if there is a small number of times a certain action has been tried in the past, the degree of randomness associated with the result of that action is significant. Thus, relying on the result of a handful of instances of prior action is not likely to be reliable.

The second, and related, issue is that of correlation and causation. That the particular action in the past was associated with a particular result doesn’t necessarily mean that the result, whether good or bad, was a consequence of the action. The question we need to ask in this case is whether the result was because of or in spite of the action. It is well possible that a lousy policy in the past led to good results thanks to a favourable market environment. It is also equally possible that a fantastic policy led to lousy results because of a lousy environment.

Thus, when we evaluate precedence, we should evaluate the process and methodology, rather than result. We should accept that the action alone can never fully explain the result of the action, and thus evaluate the action in light of the prevailing conditions, etc. rather than just by the result.

It is going to take significant bureaucratic rethinking to accept this, but unless this happens it is unlikely that a bureaucracy can function effectively.

Correlated judgment

When you judge people about something, you do not normally judge them on that thing alone. You also judge them on “correlated traits”. For example, there is this popular adage (that was popular when I was in IIT) that goes “beauty times brains equals constant”. This implies that anyone who is above average in terms of looks is likely to be below average in terms of mental capabilities. Whether such a correlation exists is not known, but by instinct if we someone beautiful, we assume that the person is not great in terms of mental ability (in my later years at IIT, we recognized this limitation of the model and proposed “beauty times brains times availability equals constant”, acknowledging that beautiful intelligent people exist, but are most likely taken).

There is no end to such correlations, which usually make rounds around college campuses. For example, there is the “he is the partying types, so is unlikely to be a good worker”. Now, while it is true that the amount of time available to most people is constant, and that heavy partying can come at the cost of working, such an adage discounts the fact that some people could simply be better time managers, or don’t care much for some axis apart from partying and working (sleeping, for example!), which allows them to be good at two things that people are normally not good at!

It is common for people to judge people. However, thanks to implicit correlations of traits that are built into people’s minds, when you get judged on one thing, that is not the only thing you are judged on – you are also judged on the things that are correlated with that!

Time for more examples. Once my parents saw a friend of mine very evidently flirting with a girl. They immediately judged him as being “a flirt” and branded him thus. While judgmental, there is no mistake in that judgment – he was indeed a flirt, and would gladly admit to it. But then my parents, using their inbuilt correlation filters, went one step ahead. “He is such a flirt”, they told me, “We don’t think he is a good person. You should not hang out with him any more”!!

Back in 2005, in IIMB, I had stood for elections to the Academic Council. At a party a week before the elections I happened to get wasted, and ended up talking to people inappropriately. The next morning, as I’m trying to get over my hangover, I heard “dude, how could you get wasted if you are standing for elections?” I have no clue how getting wasted at one party would make me a bad Academic Councillor! I must mention I lost the elections.

It was at a discussion yesterday with Bharati and my wife Priyanka that this topic of correlated judgment came up, when we were discussing how life in a business school can be unforgiving. A few minutes later, Priyanka popped up “that baby was so cute, I expected him to be dumb!”

Drivers in India

One of the recent data sets in data.gov.in is the number of driving licenses issued in different states of India until March 2012. Based on that, it is interesting to see which states have more drivers. The first chart here shows the proportion of population of each state that has a driving license (states for which data is unavailable have been left out). Note that this proportion is an overestimate since the number of licenses given includes people who have subsequently died, and thus not been counted in the state’s population as of 2011. Nevertheless, as a relative measure, this is useful:

Notice that the highest numbers are in Goa, Tamil Nadu and Maharashtra, all of them among the more prosperous states. States at the bottom include Assam, Uttarakhand and Jammu and Kashmir. The latter two are extremely hilly, thus discouraging driving. Nevertheless it would be an interesting correlation between proportion of drivers in a state and its per capita GSDP. Which is what we do next:

We see that apart from Delhi (where presumably a large portion of the population gets its licenses from other states?) and Sikkim (a hilly area where not too many are expected to drive), there is a strong correlation between the proportion of drivers and the per capita GSDP!

Finally, what proportion of drivers in each state are women? The following graph shows that:

Manipur, where over 30% of licenses have been handed out to women, stands way ahead of other states. The other state that stands out is Andhra Pradesh, where a measly 1.5% of driving licenses belong to women. Contrast this with neighbouring states such as Odisha (12%), Karnataka (15%) and Tamil Nadu (8%)!

Commute Distance and Prosperity

There is an interesting report on The Hindu Blogs about commute distance and prosperity. Referring to a World Bank report in 2005, the blog post talks about richer people commuting longer distances to work. Rukmini S, who has written the piece, also finds from the latest NSSO data that richer states in India have a higher proportion of people commuting more than 5 km to work.

I didn’t like the visualization (or the lack of it) in Rukmini’s article, and hence this post. I thought the point about long commutes to work and richer states would be better made in a scatter plot, and that is what I produce here:

On the X axis is the proportion of the Urban population in each state that commutes over 5 km to work each way. The data is from the latest NSSO Survey (page 28-29). On the Y axis I have a measure of the level of economic activity in a state – the per capita Gross State Domestic Product. The advantage of this measure is that it takes out from the equation the size of the state itself, and instead focuses on the level of economic activity per person. The figures are from 2011-12 and the numbers are based on 2004-05 prices. The data is from the RBI website.

The correlation is clear – barring a few small states, the above plot clearly shoes that more the proportion of people that commute long distances to work, the greater the economic activity in that state. The question, however, is whether there is a causal effect and if so, in which direction – does people traveling longer distances cause greater economic activity or does greater economic activity lead to people commuting longer distances?

The world bank paper proposes that the more well to do commute longer distances only because the cost of local transport in Mumbai is high and the poor cannot afford that. This is a view that Rukmini endorses in her piece in the Hindu. The argument doesn’t particularly make sense, though. Do the world bank researchers intend to say that transport costs outstrip housing costs in prime areas in Mumbai? If so, it is extremely hard to believe.

At the state level, one possible reason why people in richer states travel more is because greater economic activity happens in bigger urban agglomerations. The economic activity of a town or village is a super-linear function of the number of people living there. And when you have larger urban agglomerations, people tend to live farther from their workplaces, and thus commute more.

Again – this is a chicken and egg problem – a level of economic activity in a town or village leads to increase in population, which results in greater commutes. Increase in population leads to even greater economic activity, and this sets off a virtuous cycle. The 20-fold increase in Bangalore’s population in the last 70 years can be attested to this cycle, and it is hard to put a direction of causation to it.

The above explanation, however, doesn’t explain the following graph. This graph is identical to the one above except that here we look at the proportion of rural residents who commute over 5 km to work. And this is again positively correlated with economic activity!

What can possibly explain this? One way to explain this is that when people stay close to a town or city with high economic activity, they might prefer to participate in that rather than working in the village itself, and thus they might be commuting longer distances. States with high economic activity are likely to have a larger number of villages close to urban/semi-urban centres of high economic activity, and thus people are likely to travel longer distances.

When more people are willing to travel longer distances for work, it leads to people coming together to work at a higher rate than it normally happens in a village, and this leads to higher economci activity! Again, it is hard to put a directionality to the causation!

Correlation and the 1987 Stock Market Crash

Recently on this blog I had talked about the phenomenon of correlations, and how that can send financial models topsy-turvy. I had taken the example of additional cars on the road on a rainy day and had explained how in 2008 CDOs went bust as a fall in house prices led to mortgages defaulting together. Today I read this interesting post by JP Koning which attributes the stock market crash of 1987 (Black Monday) also to correlation, but of a different kind.

It basically have to do with how bubbles behave. When you know that the stock market is overheated, there are two things you can do. You can either choose to ride whatever is left of the bubble, and thus go long, or short the market and hope that the bubble has come towards its end. There are problems with both approaches – if you are long and the bubble bursts, you stand to lose significant money. On the other hand if you are short and the bubble continues, you can end up getting wiped out before the bubble bursts and offers you an opportunity to profit (as Keynes supposedly said – the market can remain irrational for longer than you can remain solvent).

Trading is difficult business during the times of a bubble. Every good trader knows that a bubble is on. Yet, they are faced with the above dilemma. They want to participate in the party as long as it lasts but leave before the house comes crashing down. But nobody knows when the house will crash. Some smart traders such as Taleb (no doubt backed by their banks’ deep pockets) simply buy put options and wait it out for the bubble to burst and make their money. Some get out of the market. But most remain, taking directional bets (in either direction) and not sure of whether they are going to get wiped out.

Suppose you are a trader in one such bubble, and you decide to use a mixed strategy of whether you go long or short. Let us assume that on four out of five days (randomly chosen) you are long the market, and you short the fifth day. Let us assume every trader follows a similar strategy, but strategies of no two traders is correlated. So on a given day, for every trader going short, there are four traders going long and thus the bubble continues (let us assume that each trader plays with the same amount of money). You can see where this is going. What if there is a day when for some reason more than the usual 20% of the traders decide to go short?

Let us briefly revisit the house party analogy. There is a party on and you want to enjoy it for as long as possible. However, the house in which the party is going on is unstable, and as soon as the number of people in the house falls below a certain number, the house will collapse, crushing anyone still in there (yes, this is a weird house, but never mind). You go near the house and you see a large number of people having a gala time. You see that the number of people in the house far exceeds the threshold, and so you join the party. And thus the party swells.

Suppose you are now in the party, and you see a large number of people leaving. Suddenly, you realize that following their exit, the number of people left in the house will be not too much more than the threshold. If you stay on, you might end up holding up the house, you might reason, and you will want to leave with the large group. The only problem is that you are not alone in thinking such. Most other guests have also seen this large group leave, and want to accompany them on their way out.

Traders were aware that the crash of 1929 had also occurred in late October, and on a Monday. On the 19th of October 1987, Koning mentions in his blog, the Wall Street Journal published a graph of the stock market in the 1980s and superimposed it with a graph of the stock market in the 1920s, leading up to the stock market crash in 1929 (which led to the Great Depression). The two graphs looked similar, as you can see below.

This was all the trigger that the market needed. Suddenly, you have a day when every trader reads about the bursting of the 1929 bubble in the newspaper, and how the current market is similarly poised. Suddenly every trader is doubly conscious of the stock market bubble, and wants to get away. Instead of every trader playing a random strategy, where only 20% will want to short, on this particular day a much larger number of traders want to short. As they collectively short, the market falls significantly enough to tell everyone that the bubble is busting. Everyone else tries to join them as they try to rush out of the party house. The house duly crashes.

Once again, notice that this was a random system being held up by low correlation. Traders knew there was a bubble, but didn’t know when it would burst and thus played uncorrelated mixed strategies, which kept the market afloat. All it took was one newspaper article, which every trader happened to read. The correlation suddenly jumped, and the market moved decisively.

As an exercise at the end of this blog post, think of other systems which are similarly “held up” because of low correlation in people’s behaviour. It need not only be financial – remember the road on rainy day example I gave in my previous post. Then think of what might result in correlations that hold up these systems to collapse to 1, and how those systems will then respond. Please don’t, however, blame me for scaring you.

Rupee and dollar volatility of Nifty

Note: This is not a particularly policy related post; just an interesting chart I want to present here.

Out at capitalmind, Deepak Shenoy writes that measured in US Dollars, the NSE Nifty has actually lost 8% in the last 6 years, a period in which the rupee value of the Nifty has gone up by 36%. This is on account of the depreciation of the Indian Rupee against the US Dollar.

Now, it would be interesting to see the volatility of the index as measured in the two currencies. Does the volatility in the USD/INR exchange rate add to the volatility of the Nifty or does it subtract from it? (note that when you multiply two volatile indices, the resultant can be less volatile than either of the components, if the components move in opposite directions).

As you can see from the following graph, the two volatilities actually add up, meaning the dollar volatility of the Nifty has for most part been much higher than the rupee volatility! And to add that the dollar returns have also been lower than the rupee returns. Makes you wonder why FIIs are still invested in India.

(please disregard the absolute values on the graph. In order to make the graph, I index both nifty and the dollar value of nifty to 100 on the first day of the time series I had and appropriately scaled down both series. The point to notice here is that in most parts the red line (dollar volatility) is above the blue line (rupee volatility). As earlier, I use 30-day quadratic variation as a measure of volatility )

The Bangalore Advantage

Last night, Pinky and I had this long conversation discussing aunts and uncles and why certain aunts and uncles were “cooler” or “more modern” compared to other aunts or uncles. I put forward my theory that in every family there is one particular generation with a large generation gap, and while in families like mine or Pinky’s this large gap occurred at our generation, these “cooler” aunts’ and uncles’ families had the large gap one generation earlier. Of course, this didn’t go far in explaining why the gap was so large in that generation in the first place.

Then Pinky came up with this hypothesis backed by data that was hard to refute, and the rest of the conversation simply went in both of us trying to confirm the hypotheses. Most of these “cool” aunts and uncles, Pinky pointed out, had spent most of their growing up years in Bangalore, and this set them apart from the more traditional relatives, who spent at least a part of their teens outside the city. The correlation was impeccable, and in an effort to avoid the oldest mistake in statistics, we sought to identify reasons that might explain this difference.

While some of the more “traditional” relatives had grown up in villages, we discovered that a large number of them had actually gone to high school/college in rather large but second-tier towns of Karnataka (this includes Mysore). So the rural-urban angle was out. Of course Bangalore was so much larger than these other towns so size alone might have been enough to account for the difference, but the rather large gap in worldviews between those that grew up in Bangalore, and those that grew up in Mysore (which, then, wasn’t so much smaller), and the rather small gap between the Mysoreans and those that grew up in small towns (like Shimoga or Bhadravati) meant that this big-city hypothesis was unfounded.

We then started talking about the kind of advantages that Bangalore (specifically) offered over other towns of Karnataka, and the real reason was soon staring us in the face. Compared to any other town in Karnataka (then, and now), Bangalore was significantly more cosmopolitan. I’ve spoken on this blog before about Bangalore having been two cities (I’ve put the LJ link rather than the NED link so that you can enjoy the comments) but the important thing was that after independence and the Britishers’ flight, the two cities got combined into one big heterogeneous city.

Relatives growing up in Mysore or Shimoga typically went to college with people from large similar backgrounds. Everyone there spoke Kannada, and the dominance of Brahmins in those towns was so overwhelming that these relatives could get through their college lives hanging out solely with other people from largely similar family backgrounds. This meant there was no new “cultural education” that college offered, and the same world views that had been prevalent in these peoples’ homes while they were growing up persisted.

It was rather different for people who grew up in Bangalore. Firstly, people from East Bangalore didn’t speak Kannada (at least, not particularly fluently), which meant English was the lingua franca. More importantly, there was greater religious, casteist and cultural diversity in the classroom, which made it so much more likely for people to interact and make friends with classmates from backgrounds rather different from one’s own. Back in those days of extreme cultural conservatism, this simple exposure to other cultures was invaluable in changing one’s world view and making one more liberal.

It is in the teens that one’s cultural norms are shaped, and exposure to different cultures at that age is critical to formation of one’s world-view. In our generation, this difference has probably played out in the kind of schools one goes to. However, the distinction in conservatism (based on school/college/ area) isn’t so stark as to come up with a unified theory like the one we’ve come up here. Sticking on to the previous generation, what other reasons can you think of that makes certain aunts and uncles “cooler” than others?