Duckworth Lewis and Sprinting a Marathon

How would you like it if you were running a marathon and someone were to set you targets for every 100 meters? “Run the first 100m in 25 seconds. The second in 24 seconds” and so on? It is very likely that you would hate the idea. You would argue that the idea of the marathon would be to finish the 42-odd km within the target time you have set for yourself and you don’t care about any internal targets. You are also likely to argue that different runners have different running patterns and imposing targets for small distances is unfair to just about everyone.

Yet, this is exactly what cricketers are asked to do in games that likely to be affected by rain. The Duckworth Lewis method, which has been in use to adjust targets in rain affected matches since 1999 assumes an average “scoring curve”. The formula assumes a certain “curve” according to which a team scores runs during its innings. It’s basically an extension of the old thumb-rule that a team is likely to score as many runs in the last 20 overs as it does in the first 30 – but D/L also takes into accounts wickets lost (this is the major innovation of D/L. Earlier rain-rules such as run-rate or highest-scoring-overs didn’t take into consideration wickets lost).

The basic innovation of D/L is that it is based on “resources”. With 50 overs to go and 10 wickets in hand, a team has 100% of its resource. As a team utilizes overs and loses wickets, the resources are correspondingly depleted. D/L extrapolates based on the resources left at the end of the innings. Suppose, for example, that a team scores 100 in 20 overs for the loss of 1 wicket, and the match has to be curtailed right then. What would the team have scored at the end of 50 overs? According to the 2002 version of the D/L table (the first that came up when I googled), after 20 overs and the loss of 1 wicket, a team still has 71.8% of resources left. Essentially the team has scored 100 runs using 28.2% (100 – 71.8) % of its resources. So at the end of the innings the team would be expected to score 100 * 100 / 28.2 = 354.

How have D/L arrived at these values for resource depletion? By simple regression, based on historical games. To simplify, they look at all historical games where the team had lost 1 wicket at the end of 20 overs, and look at the ratio of the final score to the 20 over score in those games, and use that to arrive at the “resource score”.

To understand why this is inherently unfair, let us take into consideration the champions of the first two World Cups that I watched. In 1992, Pakistan followed the principle of laying a solid foundation and then exploding in the latter part of the innings. A score of 100 in 30 overs was considered acceptable, as long as the team hadn’t lost too many wickets. And with hard hitters such as Inzamam-ul-haq and Imran Khan in the lower order they would have more than doubled that score by the end of the innings. In fact, most teams followed a similar strategy in that World Cup (New Zealand was a notable exception, using Mark Greatbatch as a pinch-hitter. India also tried that approach in two games – sending Kapil Dev to open).

Four years later in the subcontinent the story was entirely different. Again, while there were teams that followed the approach of a slow build up and late acceleration, but the winners Sri Lanka turned around that formula on its head. Test opener Roshan Mahanama batted at seven, with the equally dour Hashan Tillekeratne preceding him. At the top were the explosive pair of Sanath Jayasuriya and Romesh Kaluwitharana. The idea was to exploit the field restrictions of the first 15 overs, and then bat on at a steady pace. It wasn’t unlikely in that setup that more runs would be scored in the first 25 overs than the last 25.

Duckworth-Lewis treats both strategies alike. The D/L regression contains matches from both the 1992 and 1996 world cups. They have matches where pinch hitters have dominated, and matches with a slow build up and a late slog. And the “average scoring curve” that they have arrived at probably doesn’t represent either – since it is an average based on all games played. 100/2 after 30 overs would have been an excellent score for Pakistan in 1992, but for Sri Lanka in 1996 the same score would have represented a spectacular failure. D/L, however, treats them equally.

So now you have the situation that if you know that a match is likely to be affected by rain, you (the team) have to abandon your natural game and instead play according to the curve. D/L expects you to score 5 runs in the first over? Okay, send in batsmen who are capable of doing that. You find it tough to score off Sunil Narine, and want to simply play him out? Can’t do, for you need to score at least 4 in each of his overs to keep up with the D/L target.

The much-touted strength of the D/L is that it allows you to account for multiple rain interruptions and mid-innings breaks. At a more philosophical level, though, this is also its downfall. Because now you have a formula that micromanages and tells you what you should be ideally doing on every ball (as Kieron Pollard and the West Indies found out recently, simply going by over-by-over targets will not do), you are now bound to play by the formula rather than how you want to play the game.

There are a few other shortcomings with D/L, which is a result of it being a product of regression. It doesn’t take into account who has bowled, or who has batted. Suppose you are the fielding captain and you know given the conditions and forecasts that there is likely to be a long rain delay after 25 overs of batting – after which the match is likely to be curtailed. You have three excellent seam bowlers who can take good advantage of the overcast conditions. Their backup is not so strong. So you now play for the rain break and choose to bowl out your best bowlers before that! Similarly, D/L doesn’t take into account the impact of power play overs. So if you are the batting captain, you want to take the batting powerplay ASAP, before the rain comes down!

The D/L is a good system no doubt, else it would have not survived for 14 years. However, it creates a game that is unfair to both teams, and forces them to play according to a formula. We can think of alternatives that overcome some of the shortcomings (for example, I’ve developed a Monte Carlo simulation based system which can take into account power plays and bowling out strongest bowlers). Nevertheless, as long as we have a system that can extrapolate after every ball, we will always have an unfair game, where teams have to play according to a curve. D/L encourages short-termism, at the cost of planning for the full quota of overs. This cannot be good for the game. It is like setting 100m targets for a marathon runner.

PS: The same arguments I’ve made here against the D/L apply to its competitor the VJD Method (pioneered by V Jayadevan of Thrissur) also.

Numbers and management

I learnt Opeations Research thrice. The first was when I had just finished school and was about to go to IIT. My father had just started on a part-time MBA, and his method of making sure he had learnt something properly was to try and teach it to me. And so, using some old textbook he had bought some twenty years earlier, he taught me how to solve the transportation problem. I had already learnt to solve 2-variable linear programming problems in school (so yes, I learnt OR 4 times then). And my father taught my how to solve 3-variable problems using the Simplex table.

I got quite good at it, but by not using it for the subsequent two years I forgot. And then I happened to take Operations Research as a minor at IIT. And so in my fifth semester I learnt the basics again. I was taught by the highly rated Prof. G Srinivasan. He lived up to his rating. Again, he taught us simplex, transportation and assignment problems, among other things. He showed us how to build and operate the simplex table. It was fun, and surprisingly (in hindsight) never once did I consider it to be laborious.

This time I didn’t forget. OR being my minor meant that I had OR-related courses in the following three semesters, and I liked it enough to even considering applying for a PhD in OR. Then I got cold feet and decided to do an MBA instead, and ended up at IIMB. And there I learnt OR for the fourth time.

The professor who taught us wasn’t particularly reputed, and she lived up to her not-so-particular-reputation. But there was a difference here. When we got to the LP part of the course (it was part of “Quantitative Methods 2”, which included regression and OR), I thought I would easily ace it, given my knowledge of simplex. Initially I was stunned to know that we wouldn’t be taught the simplex. “What do they teach in an OR course if they don’t teach Simplex”, I thought. Soon I would know why. Computer!

We were all asked to install this software called Lindo on our PCs, which would solve any linear programming problem you would throw at it, in multiple dimensions. We also discovered that Excel had the Solver plugin. With programs like these, what use of knowing the Simplex? Simplex was probably useful back in the day when readymade algorithms were not available. Also, IIT being a technical school might have seen value in teaching us the algorithm (though we always solved procedurally. I never remember writing down pseudocode for simplex). The business school would have none of it.

It didn’t matter how the problem was actually solved, as long as we knew how to use the solver. What was more important was the art of transforming a real-life problem into one that could be solved using Solver/Lindo. In terms of formulation, the problems we got in our assignments and exams were  tough – back in IIT when we solved manually such problems were out of bounds since Simplex would take too long on those.

I remember taking a few more quant electives at IIM. They were all the same – some theory would be taught where we knew something about the workings of some of the algorithms, but the focus was on applications. How do you formulate a business problem in a way in which you can use the particular technique? How do you decide what technique you use for what problem? These were some of the questions I learnt to answer through the course of my studies at IIM.

I once interviewed with a (now large) marketing analytics firm in Bangalore. They expected me to know how to measure “feelings” and other BS so I politely declined after one round. From what I understood, they had two kinds of people. First they had experienced marketers who would do the “business end” of the problem. Then they had stats/math grads who actually solved the problem. I think that is problematic. But as I have observed in a few other places, that is the norm.

You have tech guys doing absolutely tech stuff and reporting to business guys who know very little of the tech. Because of the business guy’s disinterest in tech, he is unlikely to get his hands dirty with the data. And is likely to take what the tech guy gives him at face value. As for the tech guy doing the data work, he is unlikely to really understand the business problem that he is solving, and so he invariably ends up solving a “tech problem”, which may or may not have business implications.

There are times when people ask me if I “know big data”. When I reply in the negative, they wonder (sometimes aloud) how I can call myself a data scientist. Then there are times when people ask me about a particular statistical technique. Again, it is extremely likely I answer in the negative, and extremely likely they wonder how I call myself a data scientist.

My answer is that if I deem a problem to be solvable by a particular technique, I can then simply read up on the technique! As long as you have the basics right, you don’t need to mug up all available techniques.

Currently I’m working (for a client) on a problem that requires me to cluster data (yes, I know that much stats to know that now the next step is to cluster). So this morning I decided to read up on some clustering algorithms. I’m amazed at the techniques that are out there. I hadn’t even heard of most of them. Then I read up on each of them and considered how well they would fit my data. After reading up, and taking another look at the data, I made what I think is an informed choice. And selected a technique which I think was appropriate. And I had no clue of the existence of the technique two hours before.

Given that I solve business problems using data, I make sure I use techniques that are appropriate to solve the business problem. I know of people who don’t even look at the data at hand and start implementing complex statistical techniques on them. In my last job (at a large investment bank), I know of one guy who suggested five methods  (supposedly popular statistical techniques – I had never heard of them; he had a PhD in statistics) to attack a particular problem, without having even seen the data! As far as he was concerned he was solving a technical problem.

Now that this post is turning out to be an advertisement for my consulting services, let me go all the way. Yes, I call myself a “management consultant and data scientist”. I’m both a business guy and a data guy. I don’t know complicated statistical techniques, but don’t see the need to know either – since I usually have the internet at hand while working. I solve business problems using data. The data is only an intermediary step. The problem definition is business-like. As is the solution. Data is only a means.

And for this, I have to thank the not-so-highly-reputed professor who taught me Operations Research for the fourth time – who taught me that it is not necessary to know Simplex (Excel can do it), as long as you can formulate the problem properly.

Rupee and dollar volatility of Nifty

Note: This is not a particularly policy related post; just an interesting chart I want to present here.

Out at capitalmind, Deepak Shenoy writes that measured in US Dollars, the NSE Nifty has actually lost 8% in the last 6 years, a period in which the rupee value of the Nifty has gone up by 36%. This is on account of the depreciation of the Indian Rupee against the US Dollar.

Now, it would be interesting to see the volatility of the index as measured in the two currencies. Does the volatility in the USD/INR exchange rate add to the volatility of the Nifty or does it subtract from it? (note that when you multiply two volatile indices, the resultant can be less volatile than either of the components, if the components move in opposite directions).

As you can see from the following graph, the two volatilities actually add up, meaning the dollar volatility of the Nifty has for most part been much higher than the rupee volatility! And to add that the dollar returns have also been lower than the rupee returns. Makes you wonder why FIIs are still invested in India.

Data source: Oanda and Yahoo Finance
Data source: Oanda and Yahoo Finance

(please disregard the absolute values on the graph. In order to make the graph, I index both nifty and the dollar value of nifty to 100 on the first day of the time series I had and appropriately scaled down both series. The point to notice here is that in most parts the red line (dollar volatility) is above the blue line (rupee volatility). As earlier, I use 30-day quadratic variation as a measure of volatility )

 

India More Corrupt Than Pakistan?

Yes, if you go by the data put out by Transparency International. They question they asked was whether the respondent had ever paid a bribe. In India, 54% said yes, while in Pakistan only 34% did. Sierra Leone tops the list with 84% of people claiming they’ve paid a bribe.

I was planning to not put any visualization, since the one on the BBC website (linked above) is pretty good, but then I thought it makes sense to show the illustrious company India has been put in according to this report.

Source: Transparency International
Source: Transparency International

Interestingly, if you look at their report, their sample size is consistent across countries. While they interviewed 1025 people in India, they interviewed 1000 in tiny Fiji. This means that the results for the larger countries in the survey should be taken with more salt than that of the smaller countries.

There is a lot more in the report (linked above). Do read it if you are more interested in this.

Bayesian Recognition

We don’t meet often, but every time we talk, she reminds me that I had failed to recognize her the first time we had met after graduating together from school. Yes, I could claim in my defence that I was seeing her for the first time in over six years. While that might be a valid excuse for most people, it doesn’t apply to me, since I normally claim to have superior long-term memory. If I’ve seen you somewhere before, I ought to recognize you. The only times I don’t I’m pretending, since I don’t want to embarrass you (and myself) by recognizing you while you don’t recognize me (see this incident for an example of this).

The reason for my failure that cold Bangalore evening in December 2006 was that my Bayesian system had failed me. Let me explain, in the process giving you an insight into my Bayesian system which I use to recognize you when I meet you.

About a month or two back, I was at a friend’s wedding, which is where I hit upon this term “Bayesian recognition” to explain this phenomenon  (which I’ve been practicing for ages). Now, this friend whose wedding I was attending was one year my junior at two different schools. As you might expect at an event where you and the host share more than one social network, there were a lot of familiar faces. Some people I knew fairly well, and could easily recognize. But the others had to go through a “Bayesian search”.

So when I saw someone who was one of three people I know – let’s say X, Y and Z. In order to determine which of these this person is, I would ask myself two questions – firstly, what were the prior odds that the person I saw could be each of X, Y or Z. Secondly, what were the odds of each of X, Y and Z being there at that event. Note that the latter is important. For example, if someone at the event looks like you and I know (for example) that you are currently in another country, despite the strong resemblance I can discount the possibility that that person is you, and go ahead with my search.

Note that this differs from “frequentist recognition”, where I only look at the person’s face and try and understand who he/she most resembles, without any thought to the odds that that person is there. Frequentist recognition can lead to a large number of false positives, and after a few rounds of embarrassment, you start giving up on recognizing, and many a possible reunion thus gets missed. Bayesian recognition, on the other hand, restricts your field of search (to the people who you give good odds of being there), prevents you from being distracted and increases your chances of making a good recognition.

So why did Bayesian recognition fail me when I met this former classmate back in 2006? The problem was her company. She had come for this Deep Purple concert with another friend of mine, who was my classmate in another school (and who I had been in touch with, and so easily recognized). I had no clue that these two were friends (it turned out they didn’t know each other that well – they had come there with a common friend). So when this girl (the one I didn’t recognize) popped up with “Hey SK! Do you remember me?” I assumed that she was someone I knew from the same school as the other girl I was meeting, and that wrongly restricted my search space. And so my mind was trying to map her to my friends from school 1, while she happened to be a friend from school 2. And my search returned a blank, and my legendary long-term memory skills were embarrassed.

I must mention here, though, that this is possibly the only time that my Bayesian recognition model has acted up, and refused to recognize someone I know. There have been 2-3 false positives, but this has been the only negative. And when you consider the sample size to be all the people I have recognized in different places, this is small indeed.

Oh, and after failing to recognize her then, I’ve kept in touch with this friend.

H1B visas in 2013

It is amazing how much of the annual quota of H1B (worker) visas that the US issues goes to IT outsourcing companies.  The top 20 beneficiary companies are shown in this graph.

Source: http://h1b-visas.findthecompany.com/
Source: http://h1b-visas.findthecompany.com/

As you can see, Infosys is by far the biggest beneficiary of this. I wonder if it is a result of the lawsuit by an American employee last year against the company, which alleged that the company was misusing B1 (business) visa, which has led the company to play it safe by taking H1B visas instead.

Indian companies have been shaded blue, while non-Indian companies have been shaded red. The amount of blue on this plot tells you that India is the biggest beneficiary of the H1B visa system of the US.

The data also gives the mean salary paid by each of these companies to their H1B workers.

Source: http://h1b-visas.findthecompany.com/
Source: http://h1b-visas.findthecompany.com/

Apart from Intel, all non-Indian companies pay their H1B employees well over $90,000 per annum. None of the Indian companies even come close to that number. This might help you understand why H1B visas are such a contentious point in American domestic politics.

 

Sales and marketing

On Saturday evening, I drank a Pepsi.

You might wonder why I’m making such a big deal about it. Because it is a big deal. Because I don’t normally drink pepsi. My preferred choice of cola is Thums Up, and if it’s not available I have a Coke. The only time when I have a pepsi is when both Thums Up and Coke are not available. There are times when I end up at PepsiFoods only stores, and sometimes I even have dew instead of pepsi.

You might think I’m extrapolating based on one data point. But I know more people who swear by thums up. For whom Pepsi is only a third choice cola.

The reason I’m bringing this up now is that Pepsi has spent a bombshell on sponsoring the IPL. Yes, despite being on HD, I managed to see a number of their ads. Pepsi Atom seems cool but they didn’t seem to have had its distribution in place when I wanted to try one. I reverted to my old faithful thums up. Now, I hear news that the India head of Pepsi has been sacked because he was deemed to have over spent on the IPL.

Why someone like Pepsi would spend so much on advertising is beyond me. Yes, they need to be on the top of people’s minds. But considering that everyone they advertise to has tried each of the major colas once, and loyalties to cola brands being rather heavy, I don’t see how they seek to influence sales by advertising. That Shah Rukh Khan drinks pepsi doesn’t alter my opinion one bit – I’m loyal to my thums up. I would think the same to be true to a loyal pepsi fan.

After having said so many times that I’m a loyal Thums Up customer, you might want to know why I drank Pepsi on Saturday. Because that little shop in Malleswaram I went to stocked only pepsi products. And he didn’t have dew. Faced with the choice of Pepsi or Mirinda or 7Up, I opted for the first. It was that exclusive agreement that PepsiCo had with that shopkeeper that made me consume their product.

Pepsi should invest more in this. Give higher margins to retailers who are willing to stock only pepsi products. Cola is something in which people have loyalties, but those loyalties are typically not so strong that the shop tends to lose business if the customer’s favourite brand is not available. Given lack of choice, customers will switch.

But then I guess the problem is that Pepsi is a “marketing-driven” rather than “sales-driven” company (we used to hear a lot about this distinction during recruitment time at business school). And the thing with marketing everywhere is that they are not measured. Like this friend who markets phones once gleefully told me that an advertisement he put out had a million likes on facebook. I asked him how many extra phones his company sold as a function of that ad. He had no answer. Marketing is like that everywhere. It is not judged based on real tangible numbers. And I hear that marketers like to keep it that way!

The last time I was in this guru mode I had commented that Nokia’s strategy of promoting Lumia by the strength of its camera was doomed to fail – for people don’t buy phones because they want a camera. Nokia seems to have learnt. The latest ad for the 520 talks about the apps that are available. This time they seem to have got it right.

 

Minimum Support Prices

In India, we have this concept of “Minimum Support Price” for agricultural commodities. It is basically an unlimited put option written by the Government in order to protect farmers against not getting “appropriate remuneration” for their produce. In that sense it can be thought of as an implicit subsidy towards agriculture. There is merit in the argument in favour of such a measure – agriculture is a fundamentally high risk business and in the absence of such safety nets, not enough people might take the risk to sow a particular crop, leading to shortages.

On the other hand, it can be distortionary too. If the MSP is set too high, it can lead to a glut in that particular crop in that year, at the cost of other crops, leading to shortages in the latter. Hence, it is a tool that is necessary but one that should be used with care.

Now, the MSP has to be set in advance – so the MSP for the 2013-14 season has already been set.  This is again a risky move but a necessary move – farmers need to know the minimum amount they can get for each crop before they make their sowing decision.

Source: Commision for Agricultural Costs and Prices
Source: Commission for Agricultural Costs and Prices

The figure on shows the Compounded Annual Growth Rate (CAGR) in the MSP of a few important agricultural commodities between 2007-08 and 2013-14. Notice that the CAGR is lowest for crops such as wheat or rice, and high for crops such as Tur Dal or Moong Dal. Under the current Public Distribution System (PDS), families below the poverty line get rice and wheat at subsidized rates, but not pulses. Note that I’m only mentioning facts and not trying to suggest any causation here.

Interestingly, the MSP for coarse grains such as Ragi and Bajra has also grown significantly faster than that of rice or wheat. Also note that prices of cotton and jute have grown rather slowly over the period of consideration.

Now, while this tells us by how much prices have changed in the last six years, it is also pertinent to see how the prices have changed – did the price rise consistently over the last 5-6 years or were there some discontinuities? The next figure tries to address this issue.

Source: Commision for Agricultural Costs and Prices
Source: Commission for Agricultural Costs and Prices

The figure on the left here charts the actual year by year growth in the Minimum Support price of the crops under consideration. To me, two things jump out from this graph – apart from sugarcane, there was a steep increase in the minimum support prices of all commodities between 2007-8 and 2008-9. You might want to be reminded that India went to polls in the summer of 2009 and Maharashtra, a prime sugarcane growing state, went to polls in the winter of the same year. Again, I don’t want to claim any causation.

Then, from 2009 to 2012, minimum support prices of these commodities remained largely constant – perhaps compensating for the large jump from 2008 to 09? And then again there was a spike from 2012 to 2013. There is no such jump from 2013 to 2014, though. Note that the nation goes to the polls in 2014.

Tur and Moong dal, however, have seen a rather secular increase in prices in the last five-six years.

How the proposed Food Security Bill will affect the MSP is left as an exercise to the reader. Comments are open.

PS: Data that I’ve used for this post is available at the website of the Commission for Agricultural Costs and Prices.

 

Value of skill in rural India

Earlier today I had blogged about wage rates for unskilled workers in rural India. Now, we will use the same dataset and see what premium people pay for skills. The same data gives wages for certain occupations – carpenters, masons, cobblers, blacksmiths, etc. There are also wages given for various types of farm labour, and for the purpose of this exercise I’ve used ploughing to be representative of farm labour.

The following plot shows the wage rates for different skills in different states. A note on how to read this graph. The x axis represents the state and the y axis represents the daily wage for that particular skill. The skill itself is represented in text form. So for example a carpenter in rural Kerala gets about Rs. 600 per day while a sweeper in Bihar gets about Rs. 100.

Source: Labour Bureau. Numbers for April 2013
Source: Labour Bureau. Numbers for April 2013
  • Notice that even skilled jobs in other states don’t fetch as much as an unskilled job in Kerala. Tamil Nadu and Punjab come closest
  • The skills most in demand in rural areas across states are carpentry and masonry, if you go by this data
  • In most states, cobblers earn lower than “unskilled workers”. This is interesting because there is skill involved in making and repairing shoes. The low wages for cobblers indicates a caste bias. It is also possible that since cobblers are mostly self-employed their wage rate is inaccurate
  • Blacksmiths are again not too highly valued in villages
  • The high numbers for Kerala could be a function of the state’s lower urban-rural divide compared to the rest of India. Kerala is generally described as a semi-urban continuum with no strongly delineated urban and rural areas. Rural workers could be expensive since they are in demand for urban jobs also, unlike in other states.

 

 

The same caveats that apply to the previous post apply to this. We don’t know the sample size or the accuracy of the survey. Nevertheless, some interesting insights come out.

Wage rates in rural India

The Labour Bureau, affiliated to the Union Ministry of Labour, does a monthly survey on wages in Rural India. Wages of men and women in select occupations are polled (data is collected by the NSSO) and published on the website of the labour bureau. In this post we will look at the average daily wages of unskilled male workers (as reported by this survey) in the 20 states for which it is published (your guess is as good as mine as to why it is not published for other states).

Source: Labour Bureau statistics
Source: Labour Bureau statistics

It is interesting to note that the daily wage of the average unskilled man in Kerala is almost five times that of the average unskilled man in either Gujarat or Madhya Pradesh (states that are at the bottom of the list). Some states known to be “progressive” such as Punjab, Haryana and Tamil Nadu are also towards the top of the list while other so-called “progressive” states such as Maharashtra, Karnataka and Gujarat are close to the bottom.

Like any other data put out by the government, this should be taken with some salt. First of all, the sample sizes is not mentioned. Secondly, only the average number is reported and no measure of dispersion is given. For example, it is hypothetically possible that in Kerala they interviewed ten workers, nine of whom received Rs. 100 and the tenth received Rs. 4000 leading to an average of Rs. 490! As a thumb rule, when you put out survey data, you should always include sample size and a measure of variation (such as the standard deviation), else it is hard to conclude anything from the data.