Author: SK

Why data scientists should be comfortable with MS Excel

Most people who call themselves “data scientists” aren’t usually fond of MS Excel. It is slow and clunky, can only handle a million rows of data (and nearly crash your computer if you go anywhere close to that), and despite the best efforts of Visual Basic, is not very easy to program for doing repeatable tasks.

In fact, some data scientists may consider Excel to be “too downmarket” for them to use. At one firm I worked for, I had heard a rumour that using Excel for modelling was a fire-able offence, though I’m glad to report that I flouted this rule without much adverse effect. Yet, in my years as a “data science” and analytics consultant, and having done several modelling jobs before, I think Excel is an extremely necessary tool in a data scientist’s arsenal. There are several reasons for this.

The main one is communication. “Business types” love Excel – they use it for pretty much every official activity (I know of people who write documents in Excel). If you ask for a set of numbers, you are most likely to find it in an Excel sheet. I know of fairly large organisations which use Excel to store and transmit data (admittedly poor usage). And even non-quantitaive business types understand some of the basic quantitative functions thanks to Excel, such as joining (VLookup), pivoting, basic data cleaning (TRIM, VALUE, etc.), averaging, visualisation and sometimes even basic statistics such as correlation and regression.

One of the main problems that organisations face is lack of communication between data scientists and the business side (I mentioned this in a talk I gave last month: video here and slides here). Excel is an excellent middle ground, since it is reasonably quantitative and business people know how to use it.

In fact, in my consulting experience I’ve found that when working with clients, using Excel can make your client (usually a business person) feel more comfortable and involved in the analysis, speeding up the process and significantly improving collaboration. They’ll feel more empowered to intervene, which means they can add value, and they can feel especially happy if you occasionally let them enter some simple quantitative formulae.

The next advantage of Excel is that it puts the numbers out there. A long time back, when I was still doing full time jobs, I was asked to build a forecasting model (using a programming language) and couldn’t get it right for several months. And then on a whim I decided to use Excel, and when I saw the data in front of me, it was clear why the forecasts were so useless – because the data was so random.

Excel also allows you to quickly try things and iterate, again by putting the data and the analysis in front of you. Admittedly, the toolkit available is limited compared to what programming languages or statistical softwares can offer, but through clever usage (especially with Visual Basic), there is a lot you can achieve.

Then, Excel sometimes nudges you towards finding simple solutions. It is possible when you’re using a programming language to veer towards overly complicated solutions, and possibly use the proverbial nuclear weapon against the sparrow.

When I was working on the forecasting work a decade ago, I found that the forecasts would feed into a fairly complicated-looking model that had been developed over several years by several developers. On a whim, I decided to “do more” in Excel and managed to replicate the entire model in Excel (using VB and Solver). The people leading the product weren’t particularly happy, but using Excel was critical in ultimately moving to a simpler solution.

A similar thing occurred recently as well. I had been building a fairly complex optimisation model, which I tried replicating in Excel for communication purposes (so I could work on it together with the client). And it turned out there was a far simpler solution that I had missed all this time, and the simpler solution became apparent only because I used Excel.

I’m sure this is not an exhaustive list. So, if you’re a data scientist, you will do well to be at least conversant with Excel. I know it may only serve limited needs in terms of analysis, but the effort in learning  will get more than compensated for in the communication and collaboration and simplicity.

Tailpiece:
A long time ago, a co-worker passed by my desk and saw me work on Excel. He saw my spreadsheet and remarked, “oh, so many numbers! it must be very complicated” and went on his way. I don’t know if he is a data scientist now.

Meaningful and meaningless variables (and correlations)

A number of data scientists I know like to go about their business in a domain-free manner. They make a conscious choice to not know anything about the domain in which they are solving the problem, and instead treat a dataset as just a set of anonymised data, and attack it with the usual methods.

I used to be like this as well a long time ago. I remember in my very first job I had pissed off some clients by claiming that “I don’t care if this is a nut or a screw. As far as I’m concerned this is just a part number”.

Over time, though, I’ve come to realise that even a little bit of domain knowledge or intuition can help build significantly superior models. To use a framework I had introduced a few months back, your domain knowledge can be used to restrict the degrees of freedom in your model, thus increasing how much the machine can learn with the available data.

Then again, some problems lend themselves better to domain-based intuition than others, and this has to do with the meaning of a data point.

Consider two fairly popular problem statements from data science – determining whether a borrower will pay back a loan, and determining whether there is a cat in a given picture. While at the surface level, both are binary decisions, to be made by looking at large dimensional data (the number of data points that can be used for credit scoring can be immense), there is an important distinction between the two problems.

In the cat picture case, a single data point is basically the colour of a single pixel in an image, and it doesn’t really mean anything. If we were to try and build a cat recognition algorithm based on a single pre-chosen pixel in an image, it is unlikely we can do better than noise. Instead, the information is encoded in groups of pixels near each other – a bunch of pixels that look like cat ears, for example. In this case, whether you are training to model to identify cats or cinnamon buns is immaterial, and the domain-free approach works well.

With the credit scoring problem, the amount of information in each explanatory variable is significant. Unless we are looking at some extremely esoteric or insignificant variables (trust me, these get used fairly often in credit scoring models), it is possible to build a decision model based on just one explanatory variable and still have significant predictive power. There is definitely information in correlation between explanatory variables, but that pales compared to the information in the variables themselves.

And the amount of information captured by each explanatory variable means that it makes sense in these cases to invest some human effort to understand the variables and the impact it is having. In some cases, you might decide to use a mathematical transformation of a variable (square or log or inverse) instead of the variable itself. In other cases, you might determine based on logic that some correlations are spurious and drop the variables altogether. You might see a few explanatory variables with largely similar information and decide to drop some of them or use dimension reduction algorithms. And you can do a much better job of this if you have some experience or intuition about the domain, and care to understand what each variable means. Because variables have meanings.

Unlike in the image recognition problem, where most of the intuition is in the correlation term, because of which the “variables” don’t have any meaning, where domain doesn’t matter that much (though it can – in that some kinds of algorithms are superior at some kinds of images. I don’t have much experience in this domain to comment 🙂 ).

Again like in all the two-by-twos that I produce (and there are many, though this is arguably the most famous one), the problem is where you take people from one side and put them in a situation from the other side.

If you come from a background where you’ve mostly dealt with datasets where each individual variable is meaningless, but there is information in the collective, you are likely to “stir the pile” rather than using intuition to build better models.

If you are used to dealing with datasets with “meaning”, where variables hold the information, you might waste time doing your jiggery-pokery when you should be looking to apply models that get information in the collective.

The problem is this is a rather esoteric classification, so there is plenty of chance for people to be thrown into the wrong end.

Yet another way of classifying data scientists

There are many axes along which we can classify data scientists.

We can classify based on the primary specialty, in terms “analytics”, “business intelligence” and “machine learning”. We can classify based on domain, into “financial data scientists” and “retail data scientists” and “industrial data scientists”. We can classify by the choice of primary software tool, into “R data scientists” and “Python data scientists” and “SAS data scientists”. We can also classify by expertise, such as “deep learning” and “statistics” and “stochastic calculus”. The axes are endless.

Here is my not-so-humble attempt to contribute yet another such axis based on my observations in the industry – “technology facing” and “business facing” data scientists.

Technology facing data scientists put the software first. You’ll see them building pipelines, making sure their solutions can be easily integrated into the software stack, and worrying about how quickly their analysis can run. They will spend a lot of time on data engineering and infrastructure works, and their first concern when designing a solution is that it should be easy to implement. They are highly process oriented and not so fond of hacks.

Business facing data scientists, on the other hand, are primarily concerned with insights, and don’t care much about technological niceties. The technological feasibility and ease of implementation of a solution is an afterthought. Their data is messy, and the process is not easily repeatable (might even involve some manual processes). But they make sure that the insights they draw can be easily understood by a human, and invest time and effort in communication and visualisation. They might even build tools to help the business side of the organisation understand what is happening in the model.

This distinction is actually unsurprising if you look at who the primary clients of these respective types are. The business facing data scientists are more likely to be employed in generating insights, and building models to try and understand what is happening. The technology-facing data scientists will have spent most of their careers building production systems, and are thus very well acquainted with the software engineering process.

It is important, however, to recognise this distinction, and employ the data scientists as per their specialisation. A technology-facing data scientist in a business-facing role might be seen as spending way too much effort in getting the technology right, and doing her own thing while being unmindful of the business clients. A business-facing data scientist in a technology facing role will end up producing messy solutions that may be insightful, but will be a nightmare to implement.

This was first posted on LinkedIn

Stocks and flows

One common mistake even a lot of experienced analysts make is comparing stocks to flows. Recently, for example, Apple’s trillion dollar valuation was compared to countries’ GDP. A few years back, an article compared the quantum of bad loans in Indian banks to the country’s GDP. Following an IPL auction a few years back, a newspaper compared the salary of a player the market cap of some companies (paywalled).

The simplest way to reason why these comparisons don’t make sense is that they are comparing variables that have different dimensionality. Stock variables are usually measured in dollars (or pounds or euros or whatever), while flows are usually measured in terms of currency per unit time (dollars per year, for example).

So to take some simple examples, your salary might be $100,000 per year. The current value of your stock portfolio might be $10,246. India’s GDP is 2 trillion dollars per year.  Liverpool FC paid £67 million to buy out Alisson’s contract at AS Roma, and will pay him a salary of about £77,000 per week. Apple’s market capitalisation is 1.05 trillion dollars, and its sales as per the latest financials is 229 billion dollars per year.

Get the drift? The simplest way to avoid confusing stocks and flows is to be explicit about the dimensionality of the quantity being compared – flows have a “per unit time” suffixed to their dimensions.

Following the news of Apple’s market cap hitting a trillion dollars, I put out a tweet about the fallacy of comparing it to the GDP of the United States.

A lot of the questions that followed came from stock market analysts, who are used to looking at companies in terms of financial ratios, most of which involve both stocks and flows. They argued that because these ratios are well-established, it is legitimate to compare stocks to flows.

For example, we get the Price to Earnings ratio by dividing a company’s stock price (a stock) by the company’s annual earnings per share (a flow). The asset turnover ratio is derived by dividing the annual revenues (a flow) by the amount of assets (a stock). In fact, barring simple ratios such as gross margin, most ratios in financial analysis involve dividing a stock by a flow or the other way round.

To put it simply, financial ratios are not a case of comparing stocks to flows because ratios by themselves don’t mean a thing, and their meaning is derived from comparing them to similar ratios from other companies or geographies or other points in time.

A price to earnings ratio is simply the ratio of price per share to (annual) earnings per share, and has the dimension of “years”. When we compute the P/E ratio, we are not comparing price to earnings, since that would be nonsensical (they have different dimensions). We are dividing one by the other and comparing the ratio itself to historic or global benchmarks.

The reason a company with a P/E ratio of 25 (for example) is seen as being overvalued is because this value lies at the upper end of the distribution of historical P/E ratios. So we are comparing one ratio to the other (with both having the same dimension).

In conclusion, when you take the ratio of one quantity to another, you are just computing a new quantity – you are not comparing the numerator to the denominator. And when you compare quantities, always make sure that you are being dimensionally consistent.

 

 

Bridge!

While I have referred to the game of contract bridge multiple times on this blog, today was the first time ever since I started blogging that I actually played the game. I mean, i’ve played a few times with my computer, but today was the first time in nearly fifteen years that I actually “played”, with other humans in a semi-competitive environment.

It happened primarily thanks to the wife, who surprised me yesterday by randomly sending me links of two bridge clubs close to home. I found that one of them was meeting this evening, and welcomed newcomers (even those without partners), and I needed no further information.

One small complication was that it had been very many years since I had even played the game with my computer, or read bridge columns, and I needed to remember the rules. Complicating matters was the fact that most players at this club use four-card major bidding systems, while at IIT and with my computer I was used to playing five card majors.

I installed a bridge app on my phone and played a few games, and figured that I’m not too rusty. And so after an early dinner, and leaving a wailing Berry behind (she hates it when I go out of home without her), I took the 65 bus to the club.

The club has a “host” system, where members can volunteer to play with “visitors” without partners. My host tonight was Jenny, a retired school teacher and librarian. We quickly discussed the bidding system she uses, and it was time to play.

There were some additional complications, though. For example, they use bidding boxes to convey the bids here (so that you don’t give out verbal signals while bidding), and I had never seen one before. And then on the very first hand, I forgot that bidding takes place clockwise, and bid out of turn. That early mishap apart, the game went well.

We were sitting East-West in the pairs event, which meant we moved tables after every couple of hands. Jenny introduced me to our opponents at each table, helpfully adding in most cases that I was “playing after fifteen years. He had never seen a bidding box before today”.

I think I played fairly well, as people kept asking me where I play regularly and I had to clarify that today was the first time ever I was playing in England. Jenny was a great partner, forever encouraging and making me feel comfortable on my “comeback”.

At about three fourth of the session though, I could feel myself tiring. Hard concentration for three hours straight is not something I do on a regular basis, so it was taxing on my nerves. It came to a head when a lapse in my concentration allowed our opponents to make a contract they should have never made.

Thankfully, I noticed then that there was coffee and tea available in a back room. I quickly made myself a cup of tea with milk and sugar and was soon back to form.

Jenny and I finished a narrow second among all the East-West pairs. If my concentration hadn’t flagged three fourths of the way in, I think we might have even won our half of the event. Not a bad comeback, huh? After the event, someone told me that he would introduce me to “a very strong player who is looking for a partner”.

Oh, and did I mention that I was probably by far the youngest player there?

I’ll be back. And once again, thanks to the wife for the encouragement, and finding me this club, and taking care of Berry while I spent the evening playing!

Vacation Shopping

This is yet another of those questions whose answer seems rather obvious to everyone, and to me in full hindsight, but which has taken me a long time to appreciate

For a long time I never understood why people shop during vacations, when both time and luggage space are precious commodities. With global trade, I reasoned that most clothes should be available at reasonably comparable prices worldwide, and barring some special needs (such as a certain kind of shoes, for example), there was no real need to shop on vacations.

The last day of our trip to Munich in June convinced me otherwise. That was the only day on the trip that the wife was free from work, and we could go out together before our afternoon flight. The only place we ended up going out to turned out to be a clothing store, where the wife freaked out shopping.

It didn’t make sense to me – she was shopping at a chain store which I was pretty certain that I had seen in London as well. So why did she shop while travelling? And she shopped far more than she does in a normal shopping trip in London.

In hindsight, the answer is rather simple – diversity. While the same stores might exist in various countries or cities, each is adapted to local tastes and prevailing fashions. And while everyone watches the same “runways” in Milan and Los Angeles, there is always a subtle difference in prevailing styles in different places. And clothes in the stores in the respective places are tailored (no pun intended) to these styles.

And it can happen that the local prevailing styles are not something that you particularly agree with. For example, for years together in Bangalore I struggled to find plain “non-faded” jeans – most people there seemed to demand faced or torn jeans, and stores responded to serve that demand (interestingly, jeans shopping in my last Bangalore trip was brilliantly simple, so I guess things have changed).

Similarly, the wife finds it hard to appreciate most dresses in the shops in London (and I appreciate why she doesn’t appreciate them – most of the dresses are a bit weird to put it mildly), and as a result hasn’t been able to shop as much in recent times. She had taken to claim that “they don’t seem to be making normal clothes any more”.

But the styles in London aren’t correlated with the styles in Munich (or elsewhere), with the result that in that one chain store in Munich, she found more nice dresses than she had in some 20 shopping trips over a year in London.

Fashion suffers from the “tyranny of the majority“. It makes eminent sense for retailers to only stock those styles and models that have a reasonably high demand (or be compensated for stocking low-demand items with a high enough margin – I have a chapter on this in my book). So if your styles don’t match with those of people around you, you are out of luck.  But when you travel, you have the chance to align yourself to another majority. And if that alignment happens, you’re in luck!

PS: On a separate note, I’m quite disappointed with the quality of clothes in London. Across brands, they seem to wear much faster than those bought in continental Europe or even in India.

Mental health triggers

My ADHD seems to have become much worse over the last couple of days. Like this morning I had this episode where I couldn’t decide whether to go back home to get an umbrella, and thus turned around twice while I was in the middle of crossing a road.

In part, I blame this on having just read a book on ADHD – the second such book I’ve read in the last week (I found this book from the bibliography of previous one). While this book told me the impact of ADHD on relationships, and helped me understand what someone married to someone with ADHD goes through, in the course of doing so it reminded me of all the problems that one faces when you have ADHD.

So in some way, as I read through and “revised” the list of problems that one has with ADHD, all these problems have started surfacing (more likely I have noticed these issues every time they’ve come up). And this has led to a positive feedback loop, and thus much shorter attention spans and massive distractions and even mild addiction (to online chess).

This is not an isolated incident. In the past as well, when I’ve read material related to mental health problems that have affected me as well, the precise problem gets triggered. So when I read some stuff about depression, I’m likely to have a depressive episode after that. Similar with anxiety.

Interestingly, there is no impact when I read something related to a problem that I myself have never faced – like I once started reading Siddhartha Mukherjee’s essay on bipolar disorder and it had no effect whatsoever on me.

It wasn’t always this way. Long back, before I got diagnosed, reading stuff about mental health issues which I later got diagnosed with would make me feel hopeful – hopeful perhaps that there was in fact a diagnosis for what I was going through and it wasn’t simply “laziness” or “ineptitude” on my part that was causing me all that I was going through. But once I got the diagnosis, and figured out lifestyle changes to deal with my issues, reading more has only triggered the respective issue.

I guess the solution for this is simple – unless absolutely necessary (say there is a specific issue for which I seek help on) I shouldn’t read stuff about mental health issues that I might be facing.

I won’t spare you, though – here is an essay about ADHD that I had written three years ago (which I dug up after a conversation on ADHD with a friend yesterday).

 

Social Capital and Caste

Conventional wisdom is that social capitalin India is low because of our historical caste system. By placing people in a rigid hierarchy, and giving some people privileges over others just because of the families they were born into, the caste system prevented people from cooperating as well as they would in a more equitable society – that is what conventional wisdom says.

However, a point that we cannot miss is that despite the caste system placing a hierarchy on people, people from different castes did regularly cooperate and trade with each other. In fact, with caste being tied to hereditary professions, people had little choice but to regularly interact and trade with people from other castes. And this inevitably created social capital.

Putting it differently, the result of the caste system was an unequal but stable society, and this stability led to reasonably good social capital (history might be biased given it was written by people from certain castes, but we don’t see many instances of caste riots or clashes from over 200 years ago). You can think of it as a stable society with “handicaps”, where some people were privileged over others (in fact, there was a hierarchy of privilege), to the extent that it was okay for some people to abuse others in various ways.

Over the last 150 years or so, the caste system has been (rightly) challenged, and we are seeing various movements towards a more equal society. One side effect of this has been that the (unequal) equilibrium that had existed has been disturbed, leading to caste-based antagonism and a fall in social capital.

We are in the process of moving from one (unequal) equilibrium to another (more equal) equilibrium, but until we get there, existing beliefs and biases will continue to be challenged, which means some sets of people will continue to be suspicious of others, and there will be mistrust and thus low social capital.

Originally posted at Pragati Express

Why Indian Classical Music is Superior to Western Classical Music

I’ve been half-watching this atrocious movie called “Thank You“. Rather, the wife has been watching and I’ve been eavesdropping once in a while. Apart from the odd lame joke, it’s a horrible movie, so I wouldn’t recommend you to watch it.

But there’s one scene in that that illustrates that Indian Classical Music is superior to Western Classical Music. So the plot of the movie is that there are three stupid guys who are trying to find a conman who has been messing with them. Despite mostly obvious clues, they fail to identify him.

Until this day when they are all in his office, and one of them finds some sheet music and starts playing the notes on a conveniently located keyboard. This piece of music is something associated with the conman through the movie, and the three stupid guys immediately figure the identity of the conman.

So what does this have to do with Western Classical music? One of the key differences between Indian and Western Classical music is that in the former the performers mug up the notes of the songs – at least the parts where they don’t have to improvise. Once you know a song, you can dispense with the book. It is almost unknown for professionals to look at notes while performing.

Western Classical, on the other hand, spares performers of using up valuable memory space in their heads from remembering music, and has performers read the music from a sheet as they play it. While this has its advantages – notes are never “forgotten”, and all performers are easily in sync, and valuable memory space in the brain is not wasted – there are disadvantages as well.

Like if you have a signature tune, and if you play it often, you are likely to leave the sheet music of the tune lying around in a convenient location – which can then be found by your pursuers who can then identify you. If Akshay Kumar’s signature tune in the movie was Indian classical, he is unlikely to have had sheet music lying around in his office, and thus not got caught!

Now, if this is the way that stupid guys identify a conman, you can imagine how bad the rest of the movie might be. As if it wasn’t absurd enough, they’ve even tried to shoehorn some senti-max social messaging into the movie, making it utterly bizarre.

And once again I must point out that I didn’t really watch the movie – I just occasionally  eavesdropped as the wife watched it!

Notes from Scandinavia

Two weeks back (1st to 4th August), we visited Scandinavia – primarily Copenhagen, but also a day trip to Malmö, to give ourselves the satisfaction of having visited Sweden. A few pertinent observations (check the wife’s pertinent observations from the trip here).

  • True to reputation, we concluded our visit to Sweden without having seen a single Swedish Krona. Admittedly, our only expenses there were in places that you would normally expect to take plastic money – a restaurant, a coffee shop and a memento shop at the railway station, but this fact deserves mention given Sweden’s reputation
  • The same cannot be said of Denmark, though. A friend who used to live there had told us that we don’t need cash there, but on the first afternoon itself we found out otherwise. We hadn’t bothered drawing cash on the way to our apartment, and when the wife started craving Thai food, I found two takeaways close to the apartment but neither accepted Mastercard – they only accepted “Dankort“. The widespread use of Dankort in Copenhagen means that there aren’t too many ATMs around either. And so that day the wife was forced to make do with Cup Noodles.
  • In that sense, payment in Denmark is like the proverbial washerman’s dog . Widespread use of Dankort means few ATMs. And foreigners’ cards don’t work in a lot of places. Oh – when we did find an ATM and withdraw money, the bank (Danske Bank) had the temerity to charge us an ATM withdrawal fee
  • Relative to London at least, Copenhagen is a dense city. Based on my limited data set (from a few random walks around our apartment in the Amager suburb), the dominant form of housing is the short (3-4 storeyed) apartment building. Some of them are pretty old – the one we stayed in was built in the 1930s, and there was no lift
  • The apartment buildings are also pretty close to each other, and given that this summer in Scandinavia has been especially hot, most people kept windows open, which meant that we could see into each other’s houses. People also didn’t seem to be that concerned about privacy – in the building opposite ours (which is a kind of college hostel, I think) we saw a naked couple making out. Also – I kinda don’t write this kind of stuff on my blog nowadays, but that also told us why people like to sunbathe topless. Contact me directly if you want to know more 😛
  • This liberal attitude doesn’t preclude creeps, though. On the other side of our apartment, we saw an old man sitting in the balcony (yes, a lot of houses in Copenhagen actually have balconies, unlike in London; also windows easily open out wide. Again unlike London) with a pair of binoculars, looking towards the hostel. Go figure!
  • Thanks to the heat, we didn’t walk that much. In a way, we did the trip “in reverse”, looking for comfort food on the day we landed and walking around the old town just before heading to the airport on the last day. All the while, we kept wondering how the city would be in winter
  • Berry absolutely hated the Copenhagen metro (she would start screaming as soon as we got into a train), and for good reason. Despite being built only in the last decade, the thing lacks air conditioning, and with  a part of the line overground, there is a massive greenhouse effect. And unlike the ancient London Metro, there are no windows on the trains either
  • The metro is strange otherwise as well. The structure of the stations means the lift takes far far lesser time than the escalators, which are rather complex. The size of the lifts, however, means that most people have to take the elevator. Also I never figured out where to tap in and out at the metro (didn’t affect us since we had taken a 72 hour tourist pass)
  • We went to the beach on the day we went there. And I did my usual “beach thing” – walked into the water until it was up to my knees. Surprisingly there was no one else doing that – they were all either sunbathing (the sun was pretty intense) or swimming. The city authorities had set up “ghats” on one section of the beach from where people could dive in and swim.

There were plenty more pertinent observations i made through the trip, but the delay in documentation means that I’ve forgotten the rest. Overall, it was an interesting place, though I think we would’ve enjoyed the trip better if not for the heat.