Tag: algorithms

Compression Stereotypes

One of the most mindblowing things I learnt while I was doing my undergrad in Computer Science and Engineering was Lempel-Ziv-Welch (LZW) compression. It’s one of the standard compression algorithms used everywhere nowadays.

The reason I remember this is twofold – firstly, I remember implementing this as part of an assignment (our CSE program at IITM was full of those), and feeling happy to be coding in C rather than in the dreaded Java (which we had to use for most other assignments).

The other is that this is one of those algorithms that I “internalised” while doing something totally different – in this case I was having coffee/ tea with a classmate in our hostel mess.

I won’t go into the algorithm here. However, the basic concept is that as and when we see a new pattern, we give it a code, and every subsequent occurrence of that pattern is replaced by its corresponding code. And the beauty of it is that you don’t need to ship a separate dictionary -the compressed code itself encapsulates it.

Anyway, in practical terms, the more the same kind of patterns are repeated in the original file, the more the file can be compressed. In some sense, the more the repetition of patterns, the less the overall “information” that the original file can carry – but that discussion is for another day.

I’ve been thinking of compression in general and LZW compression in particular when I think of stereotyping. The whole idea of stereotyping is that we are fundamentally lazy, and want to “classify” or categorise or pigeon-hole people using the fewest number of bits necessary.

And so, we use lazy heuristics – gender, caste, race, degrees, employers, height, even names, etc. to make our assumptions of what people are going to be like. This is fundamentally lazy, but also effective – in a sense, we have evolved to stereotype people (and objects and animals) because that allows our brain to be efficient; to internalise more data by using fewer bits. And for this precise reason, to some extent, stereotyping is rational.

However, the problem with stereotypes is that they can frequently be wrong. We might see a name and assume something about a person, and they might turn out to be completely different. The rational response to this is not to beat oneself for stereotyping in the first place – it is to update one’s priors with the new information that one has learnt about this person.

So, you might have used a combination of pre-known features of a person to categorise him/her. The moment you realise that this categorisation is wrong, you ought to invest additional bits in your brain to classify this person so that the stereotype doesn’t remain any more.

The more idiosyncratic and interesting you are, the more the number of bits that will be required to describe you. You are very very different from any of the stereotypes that can possibly be used to describe you, and this means people will need to make that effort to try and understand you.

One of the downsides of being idiosyncratic, though, is that most people are lazy and won’t make the effort to use the additional bits required to know you, and so will grossly mischaracterise you using one of the standard stereotypes.

On yet another tangential note, getting to know someone is a Bayesian process. You make your first impressions of them based on whatever you find out about them, and go on building a picture of them incrementally based on the information you find out about them. It is like loading a picture on a website using a bad internet connection – first the picture appears grainy, and then the more idiosyncratic features can be seen.

The problem with refusing to use stereotypes, or demonising stereotypes, is that you fail to use the grainy pictures when that is the best available, and instead infinitely wait to get better pictures. On the other hand, failing to see beyond stereotypes means that you end up using grainy pictures when more clear ones are available.

And both of these approaches is suboptimal.

PS: I’ve sometimes wondered why I find it so hard to remember certain people’s faces. And I realise that it’s usually because they are highly idiosyncratic and not easy to stereotype / compress (both are the same thing). And so it takes more effort to remember them, and if I don’t really need to remember them so much, I just don’t bother.

Record of my publicly available work

A few people who I’ve spoken to as part of my job hunt have asked to see some “detailed descriptions” of work that I’ve done. The other day, I put together an email with some of these descriptions. I thought it might make sense to “document” it in one place (and for me, the “obvious one place” is this blog). So here it is. As you might notice, this takes the form of an email.

I’m putting together links to some of the publicly available work that i’ve done.
1. Cricket
I have a model to evaluate and “tell the story of a cricket match”. This works for all limited overs games, and is based on a dynamic programming algorithm similar to the WASP. The basic idea is to estimate the odds of each team winning at the end of each ball, and then chart that out to come up with a “match story”.
And through some simple rules-based intelligence, the key periods in the game are marked out.
The model can also be used to evaluate the contributions of individual batsmen and bowlers towards their teams’ cause, and when aggregated across games and seasons, can be used to evaluate players’ overall contributions.
Here is a video where I explain the model and how to interpret it:
The algorithm runs live during a game. You can evaluate the latest T20 game here:
Here is a more interactive version , including a larger selection of matches going back in time.
Related to this is a cricket analytics newsletter I actively wrote during the World Cup last year. Most Indians might find this post from the newsletter interesting:
2. Covid-19
At the beginning of the pandemic (when we had just gone under a national lockdown), I had built a few agent based models to evaluate the risk associated with different kinds of commercial activities. They are described here.
Every morning, a script that I have written parses the day’s data from covid19india.org and puts out some graphs to my twitter account  This is a daily fully automated feature.
Here is another agent based model that I had built to model the impact of social distancing on covid-19.
A tweetstorm based on Bayes Theorem that I wrote during the pandemic went viral enough that I got invited to a prime time news show (I didn’t go).
3. Visualisations
I used to collect bad visualisations.
I also briefly wrote a newsletter analysing “good and bad visualisations”.
4. I have an “app” to predict which single malts you might like based on your existing likes. This blogpost explains the process behind (a predecessor of ) this model.
5. I had some fun with machine learning, using different techniques to see how they perform in terms of predicting different kinds of simple patterns.
6. I used to write a newsletter on “the art of data science”.
In addition to this, you can find my articles for Mint here. Also, this page on my website  as links to some anonymised case studies.

I guess that’s a lot? In any case, now I’m wondering if I did the right thing by choosing “skthewimp” as my Github username.

Algorithms and the Turing Test

One massive concern about the rise of artificial intelligence and machine learning is the perpetuation of human biases. This could be racism (the story, possibly apocryphal, of a black person being tagged as a gorilla) or sexism (see tweet below) or any other forms of discrimination (objective looking data that actually represents certain divisions).

In other words, mainstream concern about artificial intelligence is that it is too human, and such systems should somehow be “cured” of their human biases in order to be fair.

My concern, though, is the opposite. That many of the artificial intelligence and machine learning systems are not “human enough”. In other words, that most present day artificial intelligence and machine learning systems would not pass the Turing Test.

To remind you of the test, here is an extract from Wikipedia:

The Turing test, developed by Alan Turing in 1950, is a test of a machine’s ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. Turing proposed that a human evaluator would judge natural language conversationsbetween a human and a machine designed to generate human-like responses. The evaluator would be aware that one of the two partners in conversation is a machine, and all participants would be separated from one another. The conversation would be limited to a text-only channel such as a computer keyboard and screen so the result would not depend on the machine’s ability to render words as speech.[2] If the evaluator cannot reliably tell the machine from the human, the machine is said to have passed the test. The test does not check the ability to give correct answers to questions, only how closely answers resemble those a human would give.

The test was introduced by Turing in his paper, “Computing Machinery and Intelligence“, while working at the University of Manchester (Turing, 1950; p. 460).

Think of any recommender system, for example. With some effort, it is easy for a reasonably intelligent human to realise that the recommendations are being made by a machine. Even the most carefully designed recommender systems give away the fact that their intelligence is artificial once in a while.

To take a familiar example, people talk about the joy of discovering books in bookshops, and about the quality of recommendations given by an expert bookseller who gets his customers. Now, Amazon perhaps collects more data about its customers than any such bookseller, and uses them to recommend books. However, even a little scrolling reveals that the recommendations are rather mechanical and predictable.

It’s similar with my recommendations on Netflix – after a point you know the mechanics behind them.

In some sense this predictability is because the designers possibly think it’s a good thing – Netflix, for example, tells you why it has recommended a particular video. The designers of these algorithms possibly think that explaining their decisions might given their human customers more reason to trust them.

(As an aside, it is common for people to rant against the “opaque” algorithms that drive systems as diverse as Facebook’s News Feed and Uber’s Surge Pricing. So perhaps some algorithm designers do see reason in wanting to explain themselves).

The way I see it, though, by attempting to explain themselves these algorithms are giving themselves away, and willingly failing the Turing test. Whenever recommendations sound purely mechanical, there is reason for people to start trusting them less. And when equally mechanical reasons are given for these mechanical recommendations, the desire to trust the recommendations falls further.

If I were to design a recommendation system, I’d introduce some irrationality, some hard-to-determine randomness to try make the customer believe that there is actually a person behind them. I believe it is a necessary condition for recommendations to become truly personalised!

Does facebook think my wife is my ex?

The “lookback” video feature that Facebook has launched on account of its tenth anniversary is nice. It flags up all the statuses and photos that you’ve uploaded that have been popular, and shows you how your life on facebook has been through the years.

My “lookback” video is weird, though, in that it contains content exclusively from my “past life”. There is absolutely no mention of the wife, despite us having been married for over three years now! And it is not like we’ve hidden our marriage from Facebook – we have a large number of photos and statuses in the recent past in which both of us have been mentioned.

Now, the danger with an exercise such as the lookback is that it can dig up unwanted things from one’s past. Let’s say you were seeing someone, the two of you together were all over Facebook and then you broke up. And then when you tried to clean up Facebook and get rid of the remnants of your past life, you miss cleaning up some stuff. And Facebook picks that up and puts that in you lookback video, making it rather unpleasant.

I’m sure the engineers at Facebook would have been aware of this problem, and hence would have come up with an algorithm to prevent such unpleasantness. Some bright engineer there would have come up with a filter such that ex-es are filtered out.

Now, back in January 2010, the (now) wife and I announced that we were in a relationship. Our respective profiles showed the names of the other person, and we proudly showed we were in a relationship. Then in August of the same year, the status changed to “Engaged’, and in November to “Married”. Through this time we we mentioned on each other’s profiles as each other’s significant others.

Then, a year or two back -I’m not sure when, exactly – the wife for some reason decided to remove the fact that she is married from facebook. I don’t think she changed her relationship status, but didn’t make the fact that she’s married public. As a consequence, my relationship status automatically changed from “Married to Priyanka Bharadwaj” to just “Married”.

So, I think facebook has this filter that if someone has once been your significant other, and is not that (according to your Facebook relationship status) anymore, he/she is an ex. And anyone who is your ex shall not appear in your lookback video – it doesn’t matter if you share status updates and photos after your “break up”.

Since Priyanka decided to hide the fact that she’s married from Facebook, facebook possibly thinks that we’ve broken up. The algorithm that created the lookback video would have ignored that we still upload pictures in which both of us are there – probably that algorithm thinks we’ve broken up but are still friends!

So – you have my lookback video which is almost exclusively about my past life (interestingly, most people who appear in the video are IIMB batchmates, and I joined Facebook two years after graduation), and contains nothing of my present!

Algorithms can be weird!

Exponential increase

“Increasing exponentially” is a common phrase used by analysts and commentators to describe a certain kind of growth. However, more often than not, this phrase is misused and any fast growth is termed as “exponential”.

If f(x) is a function of x, f(x) is said to be exponential if and only if it can be written in the form:

f(x) = K alpha ^x

So if your growth is quick but linear, then it is not exponential. While on the topic, I want to point you to two posts I’ve written on my public policy blog for Takshashila: one on exponential growth in bank transfers that I wrote earlier today and another on exponential growth in toilet ownership. The former shows you how to detect exponential growth and in the latter I use what I think is an innovative model to model toilet growth as exponential.

The Trouble with Management Consulting

While I was pumping iron (I know, I know!!) at the gym on Wednesday evening, I got a call from a client seeking to confirm our meeting yesterday afternoon. “Why don’t you put together a presentation with all the insights you’ve gathered so far?”, he suggested, adding that he was planning to call a few more stakeholders to the meeting and it would be good to give them an insight into what is happening.

Most of my morning yesterday was spent putting together the presentation, and I’m not usually one who bothers that much about the finer details in a presentation. As long as the insights are in place I’m good, I think. I had also worked late into the night on Wednesday trying to refine some algorithms, the result of which were to go into the presentation. In short, the client’s request for the presentation had turned the 18 hours between the phone call and the meeting topsy-turvy.

It is amazing how many people expect you to have a powerpoint (or Keynote) presentation every time you walk into a meeting with them. For someone like me, who doesn’t like to prepare power points unless there are specific things to show, it can get rather irritating. Some presentations are necessary, of course, like the one to the CEO of another client that I made last Thursday. What gets my goat is when people start expecting powerpoints from you even at status update meetings.

Preparing presentations is a rather time-consuming process. You need to be careful about what you present and how you present it. You need to make sure that your visualizations are labeled well and intuitive. You need to sometimes find words to fill slides that would otherwise appear empty. And if you are not me, you will need to spend time with precise and consistent formatting and dotting the is and crossing the Ts (I usually don’t bother about this bit, even in presentation to the CEO. As long as content is there and is presentable I go ahead).

So when you have to make presentations to your clients regularly, and at every status update meeting, you can only imagine how much of your time goes into just preparing the presentations rather than doing real work!

The other resource drain in the consulting business is working from client site. While it is true that you get massive amount of work done when you are actually there and have a much shorter turn around time for your requests, spending all your time there can lead to extreme inefficiency and lack of thought.

When you spend all your time at the client site, it invariably leads to more frequent status updates, and hence more presentations and thus more time spent making presentations rather than doing real work. The real damage, though, is significantly more. When you spend all your time at your client’s site, it is easy to get drawn into what can be called as “client servicing mode”. Since you meet the client often, you will have to update him often, and you are always looking for something to update him every time you need to meet him.

Consequently, you end up putting on yourself a number of short deadlines, and each day, each hour, you strive to simply meet the next short deadline you’ve set for yourself. While this might discipline you in terms of keeping your work going and make sure you deliver the entire package on time, it also results in lack of real thinking time.

Often when you are working on a large project, you need to take a step back and look at the big picture and look at where it is all going. There will be times when you realize that some time invested in simply thinking about a problem and coming up with a “global” solution  is going to pay off in the long run. You will want to take some time away from the day-to-day running so that you can work on your “global” solution.

Unfortunately a client servicing environment doesn’t afford you this time. Due to your constant short deadlines, you will always end up on a “greedy” path of chasing the nearest local optimum. There is little chance of any kind of pathbreaking work that can be done in this scenario.

In my work I have taken a conscious decision to not visit my client’s office unless it is absolutely necessary. Of course, there are times when I need to expedite something and think being there will increase my own efficiency also and spend time there. But at other times, when I”m away, here in Bangalore, the fact that there are times when there are no immediate deadlines also means that I get the time to invest on “global” thought and on developing ideas that are long-term optimal.

The long-term productivity that emerges from spending time working off-site never ceases to amaze me!