SK – Page 142 – Pertinent Observations

Journal of Bad Statistics: Road Accidents in India

Occasionally, this blog takes a break from presenting interesting data to critiquing data-related journalism in the media. Our object of attention for this post is a report in the Hindustan Times that states that “Maharashtra has highest number of road accidents in the country”. The headline is factually correct, if you go by the data on the website of the Ministry of Road Transportation and Highways. The problem, however, is that it is a meaningless statistic.

It might be intuitive to you that one cannot compare the number of accidents in a large state like Maharashtra to that of a small state of Manipur – the former is so much larger than the latter that it is bound to have more accidents. Extending this argument, does it makes sense to compare states on the basis of sheer number of accidents? Does the statistic of “state with highest number of accidents” make sense? If not, what is a good metric to compare road safety in various states?

Comparing values that are measured in ‘absolute numbers’ across geographies makes no sense, for it doesn’t take into account the difference in size of the various geographies. In order to get a good comparison we need to “normalize” the measure that takes into account the relative sizes of the geographies. And it is important that we use the right metric in order to normalize the measures.

So how do we compare the accident rates in Maharashtra and Manipur, given their different sizes? An intuitive normalizing factor is the state population. Population might be a good metric for comparing birth rates or disease incidence rates, but road accidents? Population doesn’t account for people in one state driving more than in another state. We need a better metric.

Going back to the basics, what are we trying to achieve here by comparing accident rates across states? The accident rates is probably going to be used as a proxy for road safety. So how would you compare road safety across two different regions? A good metric, I would argue, is the likelihood of having an accident if you were to drive 1 kilometer. Or the number of accidents per vehicle kilometer. Notice that this at once takes care of both problems we have discussed above – sizes of states as well as propensity of people in various states to drive.

However, whether this is the best metric is debatable. For example, this metric ignores the “vehicle mix” in various states – so would “passenger kilometer” (rather than “vehicle kilometer”) be better? Perhaps. Again, this metric assumes that all kinds of roads are similar, and treats traveling along a kilometer of a highway as equivalent to traveling a kilometer on a village road. There are no “perfect” metrics or “normalizing factors” – so we have to choose one that is “good enough” and go with it.

Now, let us compare states based on their likelihood of accidents. Unfortunately, data on “vehicle kilometers” is hard to come by – in the absence of tolled roads, no one really keeps track of this. So we need to use a proxy. Again, it is debatable about what is the best proxy (remember there was already a debate on what is the best measure), but for ease of data capture (if not anything else) let us use “accidents per total road length” as a metric here. Drawbacks of this metric is that it doesn’t capture how busy these roads are, and are only a loose proxy for how much people drive.

The graph below shows the relative safety of roads in Indian states. Based on accidents per 10000 kilometers of roads, we see that Maharashtra (green) is quite close to the national average (blue). It turns out that it is the union territory of Lakshadweep that is the clear outlier on number of accidents per kilometer of road.

Road accidents per 10000 Km of roads, per state (2008). Source: Ministry of Road Transportation and Highways

Based on this, we can say that the article in the Hindustan Times quoted at the beginning of this piece, while factually correct, does not present a correct picture.

Fundraising

The growth of a new company usually consists of one short period of high growth preceded and followed by rather long periods of steady growth. Sometimes there might be more than one period of high growth, but for most companies, it is that one period when there is a point of inflexion and growth goes to a new trajectory.

Now, my point is that if you want to raise venture funding, you better do it when you think you are on the cusp of one such inflexion. Usually points of inflexion are associated with some increase in “leading” investment, and a small chance that the company will get on to a new trajectory, and a big chance that the company will go under.

This crude chart shows the typical trajectory of a young company. The beginning of the red zone is when you should raise venture money

If you look at the picture here, the beginning of the red region is the state where you need to get venture funding. The thing with the black regions is that irrespective of how you fund those, at best you can expect steady growth. Now, venture capital funds, the way they are structured, are not set out to fund steady growth. The way venture funds make money is when one out of a number of their investments makes shockingly great returns, while the rest go under. They are not in the business of funding steady returns.

Hence, when they fund your company they value you assuming that in case your company is successful there will be steep growth, which will enable them to recover their investment. And if your company is in steady growth phase, it is never going to be able to do that. And you will have a case of your investors pushing you to do more or something different from what you had planned doing. The problem here lies in the fact that you raised the wrong kind of funding!

In times like this or at the turn of the millennium, when venture capital is big, it can sometimes become the preferred mode of fundraising for a lot of companies. The problem, however, is that most of them don’t realize that venture funding is probably not the best form of funding for them at their size and scale, and then get weighed down by investors.

On a similar note, you should go public once you know that there are no really big points of inflexion coming up, and that your company is set on a path to steady growth. Again that follows from the fact that investors in the stock market (where they pick up tiny shares in each company) are usually in it for long-term steady growth. And if you happen to take undue risks and they don’t pay off, your stock will get hammered unnecessarily.

Foreign Policy Should Be National

So we might have this weird situation in the forthcoming IPL where Sri Lankan players are not allowed to play in Chennai. While the merits of whether India should continue to have diplomatic relations with Sri Lanka in view of the alleged genocide is debatable (I personally think we should continue to have these relations), I think the “solution” of giving visa to Sri Lankan players and then not allowing them to play in a particular city shows India in bad light.

Regional leaders are entitled to, and should, have their opinions when it comes to foreign policy. However, these opinions should be discussed in Parliament or Cabinet or some similar forum, and as far as the outside world is concerned, India should have a single foreign policy. It might be in the Tamil Nadu politicians’ own political interests to take a hard stand on this Sri Lanka issue, but it is the job of the Union Government (and the Prime Minister) to hear these voices, debate them and make a decision which the regional leaders are bound to obey.

It is well known that Tamil Nadu politicians don’t want the Sri Lankans to participate in the IPL, and this might be a corollary of their stand that the Indian government should not engage with the current political establishment in Sri Lanka. Taking that into consideration, the Union Government should do one of two things – accept the stand of the TN politicians and deny work permits to the Sri Lankan cricketers, or allow the Sri Lankans to participate in the entire tournament, including in Chennai.

It is important that India presents a coherent face when it comes to foreign policy. We have already seen one international deal (on the Farakka barrage, with Bangladesh) being scuttled at the last minute because of last-minute reservations expressed by regional politicians. If we present a divided stance on this IPL issue, it could send out a signal that Indian foreign policy is hostage to regional leaders, and that it is difficult to do business with India (since that entails doing deals with regional leaders also).

At a time when doing business with other countries is paramount (given our energy security concerns) , it is important that we send a signal that we are easy to do business with. And for that, we need to signal that we have one foreign policy.

Tailpiece: I wonder whether under their current stance the TN politicians will allow Muthiah Muralitharan, a Tamil Sri Lankan who is married to a Madrasi, to play in Chennai.

Importance of candidate’s caste in voting

Not-for-profit Daksh recently conducted a massive survey in Karnataka which tried to understand voter preferences, evaluate MLA performance, etc. This was a comprehensive survey covering over 12000 voters across all districts in Karnataka. Apart from capturing demographic information, the survey asks questions on what candidates look for in a candidate and what issues they think are important for an MLA.

One of the questions asked was the importance of a candidate’s caste when it comes to voting. Voters were asked to indicate if it was “very important”, “important” or “not important”. For purpose of my analysis I’ve given a score of 1 for “very important” and 0.5 for “important” and 0 for “not important”. The relationship between a voter’s annual family income with his perception on the importance of caste is extremely interesting, as this graph indicates.

Data Source: Daksh Survey on Perceptions about Karnataka MLAs. Thanks to Harish Narasappa of Daksh for sharing data with me

Indians are killing themselves, while farmers are dying in Vidarbha

Palagummi Sainath of the Hindu has, over the years, brought attention to the problem of farmer suicides in India, particularly in Vidarbha. While it is true that in 2011 over 14000 farmers killed themselves in Maharashtra (data not available at a finer level) , this focus on farmer suicides masks the larger problem of suicide in India.

As you can see from the following graph, the suicide rate among non-farmers is an order of magnitude greater than the suicide rate among farmers.

Suicide data from http://ncrb.nic.in/CD-ADSI2011/table-2.11.pdf . Population data from Census 2011. Since occupation-wise population data is not yet available, we have assumed that 60% of the population of each state is engaged in agriculture.

How urban and upper-class are Bangalore constituencies?

There are 28 assembly constituencies in the general Bangalore area. General discourse is that these are urban middle class areas. This is going to be a bit controversial but I use the proportion of SC/ST population as a measure of how "middle-class" a constituency is. Population density is a proxy for how urban a constituency is. Data is taken from http://bbmpelections.in/wards — There are 28 assembly constituencies in the general Bangalore area. General discourse is that these are urban middle class areas. This is going to be a bit controversial but I use the proportion of SC/ST population as a measure of how “middle-class” a constituency is. Population density is a proxy for how urban a constituency is. Data is taken from http://bbmpelections.in/wards

Push and Pull Teaching

I’m writing this in the context of the Right to Education Act coming into force this year. The reason I use a musical example upfront is that music is the only thing I’ve tried to learn formally in recent times. While I use the example to illustrate the problem with the traditional Indian learning system, I refer to more basic and general education in this post.

So about a month back I decided I need to add to my education in Carnatic and Western Classical Music and decided to learn Hindustani Classical. I decided it was time to learn a new instrument (so far I’d been trained only in playing the violin) and after some facebook queries, found a teacher who lived close by. After a lecture in how he teaches to take forward a “parampara” and not for money, and that he expects extreme devotion from students, and that he likes to begin classes for a new student only on a Monday, classes began in right earnest.

Classes soon hit a roadblock, though. As the more perceptive of you here might be aware, I have (I don’t want to use the word “suffer”) ADHD (Attention Deficit and Hyperactivity Disorder), thanks to which my attention span is grossly lower than that of the normal human being. Weeks together of simply going up and down the (Bilawal) scale soon got to me and I lost interest in practicing. Soon I realized I had started to look for excuses to bunk classes. I decided to cut my losses and decided to discontinue class.

Before I discontinued class, however, I thought long and hard about telling my teacher about my ADHD, and that his methods of teaching weren’t working out for me. I wanted to tell him about the Suzuki method which my Western Classical teacher had adopted a year ago, which kept me interested in the music without relaxation of rigour. The Suzuki Method had worked fantastically well for me. Each class I would learn a new (simple) song – for example, I started my Western Classical learning by learning to play Twinkle Twinkle Little Star.

There are times when I think that I should have given my sitar teacher a fairer chance and explained to him about the Suzuki method and adopt something like it for the Sitar. However, from my knowledge of him based on my intereaction with him for a month or so, it didn’t seem like it would work, and I ended up (regretfully) quitting without giving him a chance to push the education on me.

The thing with traditional Indian learning is that it is fundamentally “pull”. The onus is on the student to convince the teacher to take him on as a student, and then to extract knowledge and wisdom from the teacher. In the traditional Indian context, it is absolutely okay for the guru to be aloof and disinterested, for it is not his duty to teach – it is the student’s duty to extract knowledge from the teacher. In fact my friend and colleague Nitin Pai informs me that according to the Upanishads, it is the duty of the teacher to reject a student the first three times he “applies”, and accept a student only after he has sucked up considerably.

While there might have been good reasons for such teaching practices back in the Vedic and Puranic ages (for example, the caste system forbid considerable sections of the population from learning the scriptures), these practices are wholly unsuited for the modern age where the focus is on increasing the reach of education and and ensuring that more people have access to education.

With the onus being on universal education and on getting every child to learn, we need to get rid of the “Acharya Devo Bhava” (teacher is god) paradigm and instead shift to a framework of professional teachers where it is the teacher’s duty to reach out to the student. We need to get to a paradigm where the students can demand that the teacher reach out to them and teach them, and where students don’t need to suck up to the teacher.

The “acharya devo bhava” concept might have served us well in the pre-writing age and ensured that our most important scriptures were transmitted down to an era where they could be written down. This paradigm, however, is not scalable, and definitely not suited to a situation where the objective is to provide education to everybody.

Flawed though it may be, the Right to Education Act is a good step by the Union Government to ensure greater learning among kids and to maximize our chances of making good of the demographic dividend. The measure, however, will be dead on arrival unless the mindset of teaching and learning is changed.

Are there really more mobile phones than toilets in slums?

Rural Development Minister Jairam Ramesh had once commented that there are more toilets than mobile phones in India. With the census board putting out numbers for the "slum census" of 2011, it might be useful to look at penetration of toilets and mobile phones in slums! The data on the website is only state-wise but it suggests that the Minister's comment is untrue in the slums of most states. — Data Source: Census 2011

Rural Development Minister Jairam Ramesh had once commented that women prefer to have mobile phones than have toilets. With the census board putting out numbers for the “slum census” of 2011, it might be useful to look at penetration of toilets and mobile phones in slums! The data on the website is only state-wise but it suggests that in most states, more slum-dwellers have access to toilets than to mobile phones.

PS: The original post has quoted the Minister incorrectly. This has been remedied.

Exponential* Growth in Toilet Ownership in Karnataka

How rural sanitation has improved in Karnataka's districts and taluks between 2001 and 2011. The graph shows that there is a strong link between the magnitude of a district or a taluk's improvement in the last decade to its starting position in 2001. While not conclusive, this is evidence for a possible peer effect in adoption of toilets in rural areas — How rural sanitation has improved in Karnataka’s districts and taluks between 2001 and 2011. The graph shows that there is a strong link between the magnitude of a district or a taluk’s improvement in the last decade to its starting position in 2001. While not conclusive, this is evidence for a possible peer effect in adoption of toilets in rural areas

Contributed by Pavan Srinath

Cross-posted on Catalyst, with more detailed analysis.

* Time for some wonkish gyaan. People normally tend to describe any superlinear growth as “exponential”. However, that is incorrect. Exponential growth is one that follows a particular formula, like this one.

Here, we see that the growth in toilets in a particular time period is proportional to the number of toilets at the beginning of the time period. So the growth of toilets can be described by the following equation, which as you can see represents exponential growth.

Change in number of toilets is proportional to initial number of toilets, thus representing exponential growth

Religion and Probability

If only people were better at mathematics in general and probability in particular, we may not have had religion

Last month I was showing my mother-in-law the video of the meteor that fell in Russia causing much havoc, and soon the conversation drifted to why the meteor fell where it did. “It is simple mathematics that the meteor fell in Russia”, I declared, trying to show off my knowledge of geography and probability, arguing that Russia’s large landmass made it the most probable country for the meteor to fall in. My mother-in-law, however, wasn’t convinced. “It’s all god’s choice”, she said.

Recently I realized the fallacy in my argument. While it was probabilistically most likely that the meteor would fall in Russia than in any other country, there was no good scientific reason to explain why it fell at the exact place it did. It could have just as likely fallen in any other place. It was just a matter of chance that it fell where it did.

Falling meteors are not the only events in life that happen with a certain degree of randomness. There are way too many things that are beyond our control which happen when they happen and the way they happen for no good reason. And the kicker is that it all just doesn’t average out. Think about the meteor itself for example. A meteor falling is such a rare event that it is unlikely to happen (at least with this kind of impact) again in most people’s lifetimes. This can be quite confounding for most people.

Every time I’ve studied probability (be it in school or engineering college or business school), I’ve noticed that most people have much trouble understanding it. I might be generalizing based on my cohort but I don’t think it would be too much of a stretch to say that probability is not the easiest of subjects to grasp for most people. Which is a real tragedy given the amount of randomness that is a fixture in everyone’s lives.

Because of the randomness inherent in everyone’s lives, and because most of these random events don’t really average out in people’s lifetimes, people find the need to call upon an external entity to explain these events. And once the existence of one such entity is established, it is only natural to attribute every random event to the actions of this entity.

And then there is the oldest mistake in statistics – assuming that if two events happen simultaneously or one after another, one of the events is the cause for the other. (I’m writing this post while watching football) Back in 2008-09, the last time Liverpool FC presented a good challenge for the English Premier League, I noticed a pattern over a month where Liverpool won all the games that I happened to watch live (on TV) and either drew or lost the others. Being rather superstitious, I immediately came to the conclusion that my watching a game actually led to a Liverpool victory. And every time that didn’t happen (that 2-2 draw at Hull comes to mind) I would try to rationalize that by attributing it to a factor I had hitherto left out of “my model” (like I was seated on the wrong chair or that my phone was ringing when a goal went in or something).

So you have a number of events which happen the way they happen randomly, and for no particular reason. Then, you have pairs of events that for random reasons happen in conjunction with one another, and the human mind that doesn’t like un-explainable events quickly draws a conclusion that one led to the other. And then when the pattern breaks, the model gets extended in random directions.

Randomness leads you to believe in an external entity who is possibly choreographing the world. When enough of you believe in one such entity, you come up with a name for the entity, for example “God”. Then people come up with their own ways of appeasing this “God”, in the hope that it will lead to “God” choreographing events in their favour. Certain ways of appeasement happen simultaneously with events favourable to the people who appeased. These ways of appeasement are then recognized as legitimate methods to appease “God”. And everyone starts following them.

Of course, the experiment is not repeatable – for the results were purely random. So people carry out activities to appease “God” and yet experience events that are unfavourable to them. This is where model extension kicks in. Over time, certain ways of model extension have proved to be more convincing than others, the most common one (at least in India) being ‘”God” is doing this to me because he/she wants to test me”. Sometimes these model extensions also fail to convince. However, the person has so much faith in the model (it has after all been handed over to him/her by his/her ancestors, and a wrong model could definitely not have propagated?) that he/she is not willing to question the model, and tries instead to further extend it in another random direction.

In different parts of the world, different methods of appeasement to “God” happened in conjunction with events favourable to the appeasers, and so this led to different religions. Some people whose appeasements were correlated with favourable events had greater political power (or negotiation skills) than others, so the methods of appeasement favoured by the former grew dominant in that particular society. Over time, mostly due to political and military superiority, some of these methods of appeasement grew disproportionately, and others lost their way. And we had what are now known as “major religions”. I don’t need to continue this story.

So going back, it all once again boils down to the median man’s poor understanding of concepts of probability and randomness, and the desire to explain all possible events. Had human understanding of probability and randomness been superior, it is possible that religion didn’t exist at all!