Aamir Khan and Alcohol Buddies

Over the weekend I was watching Koffee with Karan, the episode featuring Aamir Khan and Kareena Kapoor. It was one of the better episodes in the season, along with the one featuring Ranveer Singh and Alia Bhatt (I did not finish watching any of the others, they were damn boring).

The thing with Koffee With Karan is that it is highly dependent on how interesting the guests are, and not all bollywood stars are equally interesting. Even in this episode, Kareena Kapoor came off as a bit of a bore, refusing to answer most questions, but Aamir Khan was great.

In the early part of the episode, both Kareena and Karan accused Aamir of being “boring”. “You come to a party stand alone and just leave; You catch one or two people and just hang out only with them for the full party”, they said. And then a bit later, one of them (I now forget who – possibly Kareena) said “when I meet you in small groups of 5-6 or less you talk a lot and you are such an interesting person, but why is it that you are such a bore at parties?”

Then Aamir went on to talk about a party at Karan’s house where the music was so loud everyone had to shout to be heard. Nobody was dancing to the music. Nothing was happening. “What is the point of such a party?” he asked.

My friend Hari The Kid has this concept of “alcohol buddies”. These are basically people who you can hang out with only if at least one of you is drunk (there are some extreme cases who are so difficult to hang out with that the only way to do it is for BOTH of you to be drunk). The idea is that if both of you are sober there is nothing really to talk about and you will easily get bored. But hey, these are your friends so you need to hang out with them, and the easiest way of doing so is to convert them into alcohol buddies.

Bringing together this concept and Aamir Khan being “boring”, we can classify people into two kinds – those that are fun when drunk, and those that are fun when sober (some, I think, are both). And people who prefer to have fun when drunk consider the sober sorts boring, and people who prefer to have fun sober think the “alcohol buddies” are boring.

Aamir, for example, appears to be a “have fun when sober” guy, who likes to hang out in small groups and make interesting conversation. Most of Bollywood, however, doesn’t seem to operate that way, hanging out in large groups and not really bothering about conversation.

Yesterday, my wife and I were talking, after an event, about how if you are the sort that likes to hang out in small groups and make conversations, large parties can be rather boring. The problem is that you would have just about started making a nice conversation with someone, when someone else will butt in (hey, this is a party, so this is allowed) and change the topic massively or massively bring down the interest level in the conversation. Every conversation ultimately goes down to its lowest common denominator, leaving you rather frustrated.

And if you are the types who likes large parties and alcohol buddies, small conversations will drain you. You struggle to find things to talk about, and there are only so many people to talk to.

PS: Alcohol and good conversations are not mutually exclusive. Some of my best conversations have happened in very small groups, massively fuelled by alcohol. That said, these have largely been with people I can have great conversations with even when everyone is sober.

Mo Salah and Machine Learning

First of all, I’m damn happy that Mo Salah has renewed his Liverpool contract. With Sadio Mane also leaving, the attack was looking a bit thin (I was distinctly unhappy with the Jota-Mane-Diaz forward line we used in the Champions League final. Lacked cohesion). Nunez is still untested in terms of “leadership”, and without Salah that would’ve left Firmino as the only “attacking leader”.

(non-technical readers can skip the section in italics and still make sense of this post)

Now that this is out of the way, I’m interested in seeing one statistic (for which I’m pretty sure I don’t have the data). For each of the chances that Salah has created, I want to look at the xG (expected goals) and whether he scored or not. And then look at a density plot of xG for both categories (scored or not). 

For most players, this is likely to result in two very distinct curves – they are likely to score from a large % of high xG chances, and almost not score at all from low xG chances. For Salah, though, the two density curves are likely to be a lot closer.

What I’m saying is – most strikers score well from easy chances, and fail to score from difficult chances. Salah is not like that. On the one hand, he creates and scores some extraordinary goals out of nothing (low xG). On the other, he tends to miss a lot of seemingly easy chances (high xG).

In fact, it is quite possible to look at a player like Salah, see a few sitters that he has missed (he misses quite a few of them), and think he is a poor forward. And if you look at a small sample of data (or short periods of time) you are likely to come to the same conclusion. Look at the last 3-4 months of the 2021-22 season. The consensus among pundits then was that Salah had become poor (and on Reddit, you could see Liverpool fans arguing that we shouldn’t give him a lucrative contract extension since ‘he has lost it’).

It is well possible that this is exactly the conclusion Jose Mourinho came to back in 2013-14 when he managed Salah at Chelsea (and gave him very few opportunities). The thing with a player like Salah is that he is so unpredictable that it is very possible to see samples and think he is useless.

Of late, I’ve been doing (rather, supervising (and there is no pun intended) ) a lot of machine learning work. A lot of this has to do with binary classification – classifying something as either a 0 or a 1. Data scientists build models, which give out a probability score that the thing is a 1, and then use some (sometimes arbitrary) cutoff to determine whether the thing is a 0 or a 1.

There are a bunch of metrics in data science on how good a model is, and it all comes down to what the model predicted and what “really” happened. And I’ve seen data scientists work super hard to improve on these accuracy measures. What can be done to predict a little bit better? Why is this model only giving me 77% ROC-AUC when for the other problem I was able to get 90%?

The thing is – if the variable you are trying to predict is something like whether Salah will score from a particular chance, your accuracy metric will be really low indeed. Because he is fundamentally unpredictable. It is the same with some of the machine learning stuff – a lot of models are trying to predict something that is fundamentally unpredictable, so there is a limit on how accurate the model will get.

The problem is that you would have come across several problem statements that are much more predictable that you think it is a problem with you (or your model) that you can’t predict better. Pundits (or Jose) would have seen so many strikers who predictably score from good chances that they think Salah is not good.

The solution in these cases is to look at aggregates. Looking for each single prediction will not take us anywhere. Instead, can we predict over a large set of data whether we broadly got it right? In my “research” for this blogpost, I found this.

Last season, on average, Salah scored precisely as many goals as the model would’ve predicted! You might remember stunners like the one against Manchester City at Anfield. So you know where things got averaged out.

Pirate organisations

It’s over 20 years now since I took a “core elective” (yeah, the contradiction!) in IIT on “design and analysis of algorithms”. It was a stellar course, full of highly interesting assignments and quotable quotes. The highlight of the course was a “2 pm onwards” mid term examination, where we could take as much time as we wanted.

Anyway, the relevance of that course to this discussion is one of the problems in our first assignment. It was a puzzle .

It has to do with a large number of pirates who have chanced upon a number of gold coins. There is a strict rank ordering of pirates from most to least powerful (1 to N, with 1 being the most powerful). The problem is about how to distribute the coins among the pirates.

Pirate 1 proposes a split. If at least half the pirates (including himself) vote in favour of the split, the split is accepted and everyone goes home. If (strictly) more than half vote against the split, the pirate is thrown overboard and Pirate 2 proposes a split. This goes on until the split has been accepted. Assuming all the pirates are perfectly rational, how would you split the coins if you were Pirate 1? There is a Wikipedia page on it.

I won’t go into the logic here, but the winning play for Pirate 1 is to give 1 coin to each of the other odd numbered pirates, and keep the rest for himself. If he fails to do so and gets thrown overboard, the optimal solution for Pirate 2 is to give 1 coin to each of the other even numbered pirates, and keep the rest for himself.

So basically you see that this kind of a game structure implies that all odd numbered pirates form a coalition, and all the even numbered pirates form another. It’s like if you were to paint all pirates in one coalition black, you would get a perfectly striped structure.

Now, this kind of a “alternating coalition” can sometimes occur in corporate settings as well. Let us stick to just one path in the org chart, down to the lowest level of employee (so no “uncles” (in a tree sense) in the mix).

Let’s say you are having trouble with your boss and are unable to prevail upon her for some reason. Getting the support of your peers is futile in this effort. So what do you do? You go to your boss’s boss and try to get that person onside, and together you can take on your boss. This can occasionally be winning.

Similarly, let us say you seek to undermine (in the literal sense) one of your underlings who is being troublesome. What do you do? You ally with one of their underlings, to try and prevail upon your underling. Let’s say your boss and your underling have thought similarly to you – they will then ally to try and take you down.

Now see what this looks like – your boss’s boss, you and your underling’s underling are broadly allied. Your boss and your underling (and maybe your underling’s underling’s underling) are broadly allied. So it is like the pirate problem yet again, with people alternate in the hierarchy allying with each other!

Then again, in organisations, alliances and rivalries are never permanent. For each piece of work that you seek to achieve, you do what it takes and ally with the necessary people to finish it. And so, in the broad scheme of all alliances that happen, this “pirate structure” is pretty rare. And so it hasn’t been studied well enough.

PS: I was wondering recently why people don’t offer training programs in “corporate game theory”. The problem, I guess, is that no HR or L&D person will sponsor it – there is no point in having everyone in your org being trained in the same kind of game theory – they will nullify each other and the training will do down the drain.

I suppose this is why you have leadership coaches – who are hired by individual employees to navigate the corporate games.

40 and growing old

Recently (less than a month ago) my daughter came to me and said “appa, this December you’ll be turning 40. Then you will start becoming old”. Instinctively I got a little upset, and then gave her a little lecture on how aging is a continuous process, and not a discrete one.

That how much I age between 38 and 39, and between 39 and 40, and between 40 and 41 is not so different. You age just a little more each year, but well at a faster rate (aging is nonlinear). And so using an arbitrary cutoff like 40 is not proper, I told her.

But then, thinking about it, I realised that my daughter is not alone in feeling this way. I actually remember, back in the day, calling my father “old” when he turned 40. Maybe it was due to his grey hair. Maybe because most sportspersons retired well before 40 (that said, Martina Navratilova and John McEnroe were both very much active then (1992-93) ).

I don’t think my father gave me a lecture on continuous aging, but I remember him feeling rather annoyed that I had called him “old”.

And then recently an aunt sent a photo to one of my family WhatsApp group. It featured my parents, and they were 42 when the photo was taken. And in that, my father visibly looks old.

Now, we had bought our “family camera” by then (a Canon SnappyQ), but we seldom took photos, so I don’t have too many recollections of what my father looked like at that age. I frequently see family albums from 1990 and 1992 (some vacations), and from much later in the 90s, and there is a discontinuity in how my father looks in both (grey and thinning hair, paunch, etc.).

What this 1995 photo that my aunt sent recently showed me is that by then my father already looked much closer to what he looked like in his late forties and early fifties (he didn’t live much longer beyond that) than what he looked like in his thirties.

I would be lying if I were to say that the picture didn’t scare me. And instinctively I felt a bit better about calling him “old” when he was 40. And I felt a bit better about my daughter saying that “this december I will start becoming old”.

Then again I’m starting to wonder what I can do to not suddenly start aging now. Hair volume and colour I have no control over. General fitness I guess I do. Or maybe not – I have too much of a sweet tooth.

On which point I need to go full bimodal about food – as things stand I end up having “a little” junk food and “a little” alcohol on most days, but in terms of returns in terms of feeling good, I’m not sure if this is the best strategy. Should I go barbell instead?

 

PS: In most places where I need to submit a photo, I use one that was taken when I was 36, when an old friend was trying to build a career in portrait photography and used me as a guinea pig. I wonder how long I can use that.

Hybrid events

In general I’m short tempered and have a short attention span. One thing that annoys me more than anything else is if someone I’m talking to gets a phone call and moves away from the conversation.

In fact if I think about it more than 90% of my fights with my wife have been triggered by phone calls she gets while she’s taking to me, as a result of which she abandons me for the moment.

I’m writing this from a “hybrid event”. My wife is giving a talk at the Goa project, and this event is happening both online and offline. I’m offline, as are some twenty others. Another dozen people are online.

As an offline audience member I’m finding this damn annoying. The most annoying thing is that the moderator is online. And the way the event has been set up, online seems to take precedence over offline. The online moderator can interrupt. He can ask a speaker to repeat the last five minutes of her talk. And as a live audience member I find this insanely irritating.

The other problem with hybrid events is there is no scope for banter. Small offline events with 20 people can be rather intimate and have a high scope for banter. Like I cracked a wisecrack a few minutes back. People around me seemed to like it. And then one of the local moderators had to repeat the wisecrack to the zoom audience.

I wrote until this point in the first of the three talks. After that I decided writing this blog is not enough and protested (a tad too) loudly that the hybrid format was boring.

Then someone figured a simple nudge. They muted the zoom while talks were on. The remote people couldn’t interrupt as much as they used to. And the event became so much better.

So I guess, like everything else in design, its just about the defaults. Then again I don’t know if the online people were happy with the new default. Not that I care.

Though: the quality of CP is far superior from people in the room than from those who can hide behind a screen without camera on

It’s not just about status

Rob Henderson writes that in general, relative to the value they add to their firms, senior employees are underpaid and junior employees are overpaid. This, he reasons, is because senior employees trade off money for status.

Quoting him in full:

Robert Frank suggests the reason for this is that workers would generally prefer to occupy higher-ranked positions in their work groups than lower-ranked ones. They’re forgoing more earnings to hold a higher-status position in their organization.

But this preference for a higher-status position can be satisfied within any given organization.

After all, 50 percent of the positions in any firm must always be in the bottom half.

So the only way some workers can enjoy the pleasure inherent in positions of high status is if others are willing to bear the dissatisfactions associated with low status.

The solution, then, is to pay the low-status workers a bit more than they are worth to get them to stay. The high-status workers, in contrast, accept lower pay for the benefit of their lofty positions.

I’m not sure I agree. Yes, I do agree that higher productivity employees are underpaid and lower productivity employees are overpaid. However, I don’t think status fully explains it. There are also issues of variance and correlation and liquidity (there – I’m talking like a real quant now).

One the variance front – the higher you are in the organisation and the higher your salary is, the more the variance of your contribution to the organisation. For example, if you are being paid $350,000 (the number Henderson hypothetically uses), the actual value you are bringing to your firm might have a mean of $500,000 and a standard deviation of $200,000 (pulling all these numbers out of thin air, while making some sense checks that broadly risk pricing holds).

On the other hand, if you are being paid $35,000, then it is far more likely that the average value you bring to the firm is $40,000 with a standard deviation of $5,000 (again numbers entirely pulled out of thin air). Notice the drastic difference in the coefficient of variation in the two cases.

Putting it another way, the more productive you are, the harder it is for any organisation to put a precise value on your contribution. Henderson might say “you are worth 500K while you earn 350K” but the former is an average number. It is because of the high variance in your “worth” that you are paid far lower than what you are worth on average.

And why does this variance exist? It’s due to correlation.

More so at higher ranked positions (as an aside – my weird career path means that I’ve NEVER been in middle management) the value you can add to a company is tightly coupled with your interactions with your colleagues and peers. As a junior employee your role can be defined well enough that your contributions are stable irrespective of how you work with the others. At senior levels though a very large part of the value you can add is tied to how you work with others and leverage their work in your contributions.

So one way a company can get you to contribute more is to have a good set of peers you like working with, which increases your average contribution to the firm. Rather paradoxically, because you like your peers (assuming peer liking in senior management is two way), the company can get away with paying you a little less than your average worth and you will continue to stick on. If you don’t like working with your colleagues, there is the double whammy that you will add less to the company and you need to be paid more to stick on. And so if you look at people who are actually successful in their jobs at a senior level, they will all appear to be underpaid relative to their peers.

And finally there is liquidity (can I ever theorise about something without bringing this up?). The more senior you go, the less liquid is the market for your job. The number of potential jobs that you want to do, and which might want you, is very very low. And as I’ve explained in the first chapter of my book, when a market is illiquid, the bid-ask spread can be rather high. This means that even holding the value of your contribution to a company constant, there can be a large variation in what you are actually paid. And that is a gain why, on average, senior employees are underpaid.

So yes, there is an element of status. But there are also considerations of variance, correlation and bid-ask. And selection bias (senior employees who are overpaid relative to the value they add don’t last very long in their jobs). And this is why, on average, you can afford to underpay senior employees.

A day at an award function

So I got an award today. It is called “exemplary data scientist”, and was given out by the Analytics India Magazine as part of their MachineCon 2022. I didn’t really do anything to get the award, apart from existing in my current job.

I guess having been out of the corporate world for nearly a decade, I had so far completely missed out on the awards and conferences circuit. I would see old classmates and colleagues put pictures on LinkedIn collecting awards. I wouldn’t know what to make of it when my oldest friend would tell me that whenever he heard “eye of the tiger”, he would mentally prepare to get up and go receive an award (he got so many I think). It was a world alien to me.

Parallelly, I used to crib about how while I’m well networked in India, and especially in Bangalore, my networking within the analytics and data science community is shit. In a way, I was longing for physical events to remedy this, and would lament that the pandemic had killed those.

So I was positively surprised when about a month ago Analytics India Magazine wrote to me saying they wanted to give me this award, and it would be part of this in-person conference. I knew of the magazine, so after asking around a bit on legitimacy of such awards and looking at who had got it the last time round, I happily accepted.

Most of the awardees were people like me – heads of analytics or data science at some company in India. And my hypothesis that my networking in the industry was shit was confirmed when I looked at the list of attendees – of 100 odd people listed on the MachineCon website, I barely knew 5 (of which 2 didn’t turn up at the event today).

Again I might sound like a n00b, but conferences like today are classic two sided markets (read this eminently readable paper on two sided markets and pricing of the same by Jean Tirole of the University of Toulouse). On the one hand are awardees – people like me and 99 others, who are incentivised to attend the event with the carrot of the award. On the other hand are people who want to meet us, who will then pay to attend the event (or sponsor it; the entry fee for paid tickets to the event was a hefty $399).

It is like “ladies’ night” that pubs have, where on a particular days of the week, women who go to the pub get a free drink. This attracts women, which in turn attracts men who seek to court the women. And what the pub spends in subsidising the women it makes back in terms of greater revenue from the men on the night.

And so it was at today’s conference. I got courted by at least 10 people, trying to sell me cloud services, “AI services on the cloud”, business intelligence tools, “AI powered business intelligence tools”, recruitment services and the like. Before the conference, I had received LinkedIn requests from a few people seeking to sell me stuff at the conference. In the middle of the conference, I got a call from an organiser asking me to step out of the hall so that a sponsor could sell to me.

I held a poker face with stock replies like “I’m not the person who makes this purchasing decision” or “I prefer open source tools” or “we’re building this in house”.

With full benefit of hindsight, Radisson Blu in Marathahalli is a pretty good conference venue. An entire wing of the ground floor of the hotel is dedicated for events, and the AIM guys had taken over the place. While I had not attended any such event earlier, it had all the markings of a well-funded and well-organised event.

As I entered the conference hall, the first thing that struck me was the number of people in suits. Most people were in suits (though few wore ties; And as if the conference expected people to turn up in suits, the goodie bag included a tie, a pair of cufflinks and a pocket square). And I’m just not used to that. Half the days I go to office in shorts. When I feel like wearing something more formal, I wear polo T-shirts with chinos.

My colleagues who went to the NSE last month to ring the bell to take us public all turned up company T-shirts and jeans. And that’s precisely what I wore to the conference today, though I had recently procured a “formal uniform” (polo T-shirt with company logo, rather than my “usual uniform” which is a round neck T-shirt). I was pretty much the only person there in “uniform”. Towards the end of the day, I saw one other guy in his company shirt, but he was wearing a blazer over it!

Pretty soon I met an old acquaintance (who I hadn’t known would be at the conference). He introduced me to a friend, and we went for coffee. I was eating a cookie with the coffee, and had an insight – at conferences, you should eat with your left hand. That way, you don’t touch the food with the same hand you use to touch other people’s hands (surprisingly I couldn’t find sanitiser dispensers at the venue).

The talks, as expected, were nothing much to write about. Most were by sponsors selling their wares. The one talk that wasn’t by a sponsor was delivered by a guy who was introduced as “his greatgrandfather did this. His grandfather did that. And now this guy is here to talk about ethics of AI”. Full Challenge Gopalakrishna feels happened (though, unfortunately, the Kannada fellows I’d hung out with earlier that day hadn’t watched the movie).

I was telling some people over lunch (which was pretty good) that talking about ethics in AI at a conference has become like worshipping Ganesha as part of any elaborate pooja. It has become the de riguer thing to do. And so you pay obeisance to the concept and move on.

The awards function had three sections. The first section was for “users of AI” (from what I understood). The second (where I was included) was for “exemplary data scientists”. I don’t know what the third was for (my wife is ill today so I came home early as soon as I’d collected my award), except that it would be given by fast bowler and match referee Javagal Srinath. Most of the people I’d hung out with through the day were in the Srinath section of the awards.

Overall it felt good. The drive to Marathahalli took only 45 minutes each way (I drove). A lot of people had travelled from other cities in India to reach the venue. I met a few new people. My networking in data science and analytics is still not great, but far better than it used to be. I hope to go for more such events (though we need to figure out how to do these events without that talks).

PS: Everyone who got the award in my section was made to line up for a group photo. As we posed with our awards, an organiser said “make sure all of you hold the prizes in a way that the Intel (today’s chief sponsor) logo faces the camera”. “I guess they want Intel outside”, I joked. It seemed to be well received by the people standing around me. I didn’t talk to any of them after that, though.

The “intel outside” pic. Courtesy: https://www.linkedin.com/company/analytics-india-magazine/posts/?feedView=all

 

Proof of work

I like to say sometimes that one reason I never really get crypto is that it involves the concept of “proof of work”. That phrase sort of triggers me. It reminds me of all the times when I was in school when I wouldn’t get full marks in maths despite getting all the answers correct because I “didn’t show working”.

In any case, I spent about fifteen minutes early this morning drinking my aeropress and deleting LinkedIn connection requests. Yeah, you read that right. It took that long to refuse all the connection requests I had got since yesterday, when I put a fairly innocuous post saying I’m hiring.

I understand that the market is rather tough nowadays. Companies are laying employees off ($) left right and centre (in fact, this (paywalled) article prompted my post – I’m hoping to find good value in the layoff market). Interest rates are going up. Stock prices are going down. Startup funding has slowed. The job market is not easy. And so you see an innocuous post like this getting such a massive reaction.

In any case, the reason I was thinking about “proof of work” is that the responses to my post reminded me of my own (unsuccessful) job hunts from a few years back. I remember randomly applying through LinkedIn. I remember using easy apply. And I remember pretty much not hearing back from anyone.

Time for a bollywood break:

Yes, the choice of where I’ve started this video is deliberate. As i was spending time this morning refusing all the LinkedIn connection requests (some 500+ people I have no clue about had simply added me without any matter of introduction or purpose), I was thinking of this song.

I followed a simple strategy – I engaged with people who had cared to write a note (or InMail) to me along with the connection request, and I just ignored the rest. As I kept hitting “ignore ignore ignore … ” on my phone (while sipping coffee with the other hand), I realised that I almost hit “ignore” on one of my company HRs who had added me. A few minutes later, I actually hit ignore on a colleague who I’ve actually worked with (I made amends by sending him back a connection request that he accepted).

Given the flood of requests that I had got, I was forced to use a broad brush. I was forced to use simple heuristics rather than evaluating each application on its true merit. I’m pretty sure I’ve made plenty of errors of omission today (that said, my heuristic has thrown up a bunch of fairly promising candidates).

In any case, if you think about it, the heuristic I used can pretty well be described as “proof of work”. And what the proof of work achieved here was to help people stand out in a crowded market. That there was some work showed a certain minimum threshold of interest, and that was sufficient to get my attention, which is all that mattered here. And on a related note, during normal times (when I get a maximum of one or two LinkedIn requests each day), I do take the effort to evaluate each request on its own merit. No proof of work is necessary.

And if you think about it, “proof of work” is rather prevalent in the natural world. A peacock’s feathers are the most commonly quoted example of this one. The beautiful tail comes at a huge cost in terms of agility and ability to fly, and the tail is a way for the peacock to show off to potential mates that “I can carry this thing and yet stay alive so imagine how fit my genes are. Mate with me”.

Anyway, back to the hiring market, you need a way to stand out. Maybe a nicely written cover letter. Maybe a referral (or “influence” as we used to pejoratively call this back in the 90s). Maybe a strong github profile. (Ok the last one is literally a proof of work!)

Else you will just get swept away with the tide.

 

PS: In general, I was also thinking of the wisdom of writing to someone at a time when you know he/she will be flooded with other messages. The bar for you to stand out is much much higher. Being contrarian helps i guess.

So many numbers! Must be very complicated!

The story dates back to 2007. Fully retrofitting, I was in what can be described as my first ever “data science job”. After having struggled for several months to string together a forecasting model in Java (the bugs kept multiplying and cascading), I’d given up and gone back to the familiarity of MS Excel and VBA (remember that this was just about a year after I’d finished my MBA).

My seat in the office was near a door that led to the balcony, where smokers would gather. People walking to the balcony, with some effort, could see my screen. No doubt most of them would’ve seen my spending 90% (or more) of my time on Google Talk (it’s ironical that I now largely use Google Chat for work). If someone came at an auspicious time, though, they would see me really working, which was using MS Excel.

I distinctly remember this one time this guy who shared my office cab walked up behind me. I had a full sheet of Excel data and was trying to make sense of it. He took one look at my screen and exclaimed, “oh, so many numbers! Must be very complicated!” (FWIW, he was a software engineer). I gave him a fairly dirty look, wondering what was complicated about a fairly simple dataset on Excel. He moved on, to the balcony. I moved on, with my analysis.

It is funny that, fifteen years down the line, I have built my career in data science. Yet, I just can’t make sense of large sets of numbers. If someone sends me a sheet full of numbers I can’t make out the head or tail of it. Maybe I’m a victim of my own obsessions, where I spend hours visualising data so I can make some sense of it – I just can’t understand matrices of numbers thrown together.

At the very least, I need the numbers formatted well (in an Excel context, using either the “,” or “%” formats), with all numbers in a single column right aligned and rounded off to the exact same number of decimal places (it annoys me that by default, Excel autocorrects “84.0” (for example) to “84” – that disturbs this formatting. Applying “,” fixes it, though). Sometimes I demand that conditional formatting be applied on the numbers, so I know which numbers stand out (again I have a strong preference for red-white-green (or green-white-red, depending upon whether the quantity is “good” or “bad”) formatting). I might even demand sparklines.

But send me a sheet full of numbers and without any of the above mentioned decorations, and I’m completely unable to make any sense or draw any insight out of it. I fully empathise now, with the guy who said “oh, so many numbers! must be very complicated!”

And I’m supposed to be a data scientist. In any case, I’d written a long time back about why data scientists ought to be good at Excel.

Recruitment and diversity

This post has potential to become controversial and is related to my work, so I need to explicitly state upfront that all opinions here are absolutely my own and do not, in any way, reflect those of my employers or colleagues or anyone else I’m associated with.

I run a rather diverse team. Until my team grew inorganically two months back (I was given more responsibility), there were eight of us in the team. Each of us have masters degrees (ok we’re not diverse in that respect). Sixteen degrees / diplomas in total. And from sixteen different colleges / universities. The team’s masters degrees are in at least four disjoint disciplines.

I have built this part of my team ground up. And have made absolutely made no attempt to explicitly foster diversity in my team. Yet, I have a rather diverse team. You might think it is on accident. You might find weird axes on which the team is not diverse at all (masters degrees is one). I simply think it is because there was no other way.

I like to think that I have fairly high standards when it comes to hiring. Based on the post-interview conversations I have had with my team members, these standards have percolated to them as well. This means we have a rather tough task hiring. This means very few people even qualify to be hired by my team. Earlier this year I asked for a bigger hiring budget. “Let’s see if you can exhaust what you’ve been given, and then we can talk”, I was told. The person who told me this was not being sarcastic – he was simply aware of my demand-supply imbalance.

Essentially, in terms of hiring I face such a steep demand-supply imbalance that even if I wanted to, it would be absolutely impossible for me to discriminate while hiring, either positively or negatively.

If I want to hire less of a certain kind of profile (whatever that profile is), I would simply be letting go of qualified candidates. Given how long it takes to find each candidate in general, imagine how much longer it would take to find candidates if I were to only look at a subset of applicants (to prefer a category I want more of in my team). Any kind of discrimination (apart from things critical to the job such as knowledge of mathematics and logic and probability and statistics, and communication) would simply mean I’m shooting myself in the foot.

Not all jobs, however, are like this. In fact, a large majority of jobs in the world are of the type where you don’t need a particularly rare combination of skills. This means potential supply (assuming you are paying decently, treating employees decently, etc.) far exceeds demand.

When you’re operating in this kind of a market, cost of discrimination (either positive or negative) is rather low. If you were to rank all potential candidates, picking up number 25 instead of number 20 is not going to leave you all that worse off. And so you can start discriminating on axes that are orthogonal to what is required to do the job. And that way you can work towards a particular set of “diversity (or lack of it) targets”.

Given that a large number of jobs (not weighted by pay) belong to this category, the general discourse is that if you don’t have a diverse team it is because you are discriminating in a particular manner. What people don’t realise is that it is pretty impossible do discriminate in some cases.

All that said, I still stand by my 2015 post on “axes on diversity“. Any externally visible axis of diversity – race / colour / gender / sex / sexuality – is likely to diminish diversity in thought. And – again this is my personal opinion – I value diversity in thought and approach much more than the visible sources of diversity.