Ranga and Big Data

There are some meeting stories that are worth retelling and retelling. Sometimes you think it should be included in some movie (or at least a TV show). And you never tire of telling the stories.

The way I met Ranga can qualify as one such story. At the outset, there was nothing special about it – both of us had joined IIT Madras at the same time, to do a B.Tech. in Computer Science. But the first conversation itself was epic, and something worth telling again and again.

During our orientation, one of the planned events was “a visit to the facilities”, where a professor would take us around to see the library, the workshops, a few prominent labs and other things.

I remember that the gathering point for Computer Science students was right behind the Central Lecture Theatre. This was the second day of orientation and I’d already met a few classmates by then. And that’s where I found Ranga.

The conversation went somewhat like this:

“Hi I’m Karthik. I’m from Bangalore”.
“Hi I’m Ranga. I’m from Madras. What are your hobbies?”
“I play the violin, I play chess…. ”
“Oh, you play chess? Me too. Why don’t we play a blindfold game right now?”
“Er. What? What do you want to do? Now?”
“Yeah. Let’s start. e4”.
(I finally managed to gather my senses) “c5”

And so we played for the next two hours. I clearly remember playing a Sicilian Dragon. It was a hard fought game until we ended up in an endgame with opposite coloured bishops. Coincidentally, by that time the tour of the facilities had ended. And we called it a draw.

We kept playing through our B.Techs., mostly blindfold in the backbenches of classrooms. Most of the time I would get soundly thrashed. One time I remember going from our class, with the half-played game in our heads, setting it up on a board in Ranga’s room, and continued to play.

In any case, chess apart, we’ve also had a lot of nice conversations over the last 21 years. Ranga runs a big data and AI company called TheDataTeam, so I thought it would be good to record one of our conversations and share it with the world.

And so I present to you the second episode of my new “Data Chatter” podcast. Ranga and I talk about all things “big data”, data architectures, warehousing, data engineering and all that.

As usual, the podcast is available on all podcasting platforms (though, curiously, each episode takes much longer to appear on Google Podcasts after it has released. So this second episode is already there on Spotify, Apple Podcasts, CastBox, etc. but not on Google yet).

Give it a listen. Share it with whoever you think might like it. Subscribe to my podcast. And let me know what you think of it.

Social Skills Decelerator

This post is a consequence of a conversation I had with my wife this morning.

I was telling her about how a friend, who also went to IIT Madras with me, recently said something like “I’m surprised XXXXX (another mutual friend) has so much confidence in himself even though he went to <local college>. A lot of my self-confidence comes from IIT”.

“It is precisely because XXXXX went to <local college> that he has so much confidence”, my wife countered (she studied in an engineering college that is very similar to the aforementioned <local college>). And then we started talking about respective lives, and experiences, and all that.

The big insight (I’m writing about this because I think my wife is too lazy to blog about this – though it’s her idea) is that in different schools, you build reputation based on different things. In 11th and 12th standards in schools where people are mugging for JEE, for example, the primary currency for reputation is academics (both my wife and I went to such schools – different branches of the same school, to be precise).

In other schools, it’s due to extracurricular achievement. Some schools place a premium on sport. Others on gadgets and shoes. Yet others simply on good looks. And some on who you know. It’s not a finite list, except that each school has its own “portfolio” of things that help build its students’ reputations.

Coming back to my wife’s big insight, it’s that in IIT, at the prime ages of 17/18 to 21-22, the primary currency to build reputation still remains academics, and that is not the case in other colleges. And in some sense, in terms of building social skills and generally increasing one’s confidence in life, these are the prime years.

And the other thing is that academics is a zero-sum status game – there might be infinitely many “nine pointers” but there can only be one class topper. And if reputation is based on academics, it is likely to be based on relative grading. So at the prime years of your life, when you are looking to build social skills (social skills are positive sum), you are playing a zero-sum academic game for marks. You are not only not building your social skills at an age when you should, but you are actively destroying them by indulging in too many zero-sum status games.

The other corollary, of course, is that by indulging in status games where there can only be one winner, we are all (most of us at least) effectively turning ourselves into losers.  By being in an environment where reputation is built only on one (zero sum) dimension, our confidence is shattered.

Some of us come back from this shattered confidence, and at some point in life (albeit belatedly) start building social skills. Others, unfortunately, never recover.

My wife is right – people are not “confident despite not going to IIT”. They are “confident BECAUSE THEY DIDN’T GO TO IIT”!

 

Does alma matter?

I just spent the holiday afternoon massively triggering myself by watching the just-released Netflix documentary Alma Matters, about life in IIT Kharagpur. Based on the trailer itself, I thought I could relate to it, thanks to my four years at IIT Madras. And so my wife and I sat, and spent three hours on the documentary. Our daughter was with us for the first half hour, and then disappeared for reasons mentioned below.

I have too many random thoughts in my head right now, so let me do this post in bullet points.

  • I have always had mixed feelings about my time at IIT Madras. On the one hand, I found it incredibly depressing. Even now, the very thought of going to Chennai depresses me. On the other, I have a lot of great memories from there, and built a strong network.

    Now that I think of it, having watched the documentary, a lot of those “great memories” were simply about me making the best use of a bad situation I was in. I don’t think I want to put myself, or my daughter, through that kind of an experience again

  • My basic problem at IIT was that I just couldn’t connect with most people there. I sometimes joke that I couldn’t connect with 80% of the people there, but remain in touch with the remaining 20%. And that is possibly right.

    The problem is that most people there were either “too fighter”, always worried about and doing academics, or “too given up”, not caring about anything at all in life. I couldn’t empathise with either and ended up having a not so great time.

  • My wife intently watched the show with me, even though she got bored by the end of the first episode. “It’s all so depressing”, she kept saying. “Yes, this is how life was”, I kept countering.

    And then I think she caught the point. “Take out the cigarettes and alcohol, and this is just like school. Not like college at all”, she said. And I think that quite sums up IIT for me. We were adults (most of us for most of the time – I turned eighteen a few months after I joined), but were treated like children for the most part. And led our lives like children in some ways, either being too regimented, or massively rebelling.

    “Now I can see why people don’t grow up when they go to IIT”, my wife said. After I had agreed, she went on, “this applies to you as well. You also haven’t grown up”. I couldn’t counter.

  • The “maleness” of the place wasn’t easy to notice. After one scene, my wife mentioned that we spend such a long time in the prime years of our lives dealing only with other men, that it is impossible to have normal relationships later on. It’s only a few who have come from more liberal backgrounds, or who manage to unlearn the IIT stuff, who manage to have reasonably normal long-term relationships.
  • The maleness of IITs was also given sort of ironic treatment by the show. There is a segment in the first episode about elections, which shows a female candidate, about how girls have a really bad time at IIT due to the massively warped sex ratio (in my time it was 16:1), and so it is difficult for girls to get respect.

    And then that turned out to be the “token female segment” in the show, as girls were all but absent in the rest of the three hours. That girls hardly made it to the show sort of self-reinforced the concept that girls aren’t treated well at IITs,

  • After intently watching for half an hour, my daughter asked, “if this is a movie about IIT, why aren’t you in it?”. I told her that it’s about a different IIT. “OK fine. I’ll watch it when they make a movie about your IIT”, she said and disappeared.
  • While the second and third episodes of the show were too long-drawn and sort of boring, I did manage to finish the show end-to-end in one sitting, which has to say something about it being gripping (no doubt to someone like me who could empathise with parts of it).
  • Finally, watch this trailer to the show. Watch what the guy says about people with different CGPA ranges.
  • He talks about “respect 8 pointers and don’t like 9 pointers”. That sort of made me happy since I finished (I THINK) with a CGPA of 8.9

If you’re from an IIT, you are likely to empathise with the show. If you are close to someone from IIT, you might appreciate them better when you watch it. Overall three episodes is too long-drawn. The first episode is a good enough gist of life at IIT.

And yeah, trigger warnings apply.

The Puritan Topper

This was an idea that sort of got ingrained in my head at the turn of the millennium – around the time I was transitioning from school to undergrad. That you would be a topper if and only if you led an otherwise diligent and disciplined life.

For starters you needed to be a nice person (among the things this entailed for a potential topper was to liberally share notes and clarify people’s doubts when called upon). You weren’t allowed to have any character flaws. You weren’t supposed to get distracted with things like hitting on someone or being in a romantic relationship. You would talk to, and be polite with, people of the opposite sex, but “nothing more than necessary”. “Bad habits” like smoking and drinking were out of the question.

These were just the necessary conditions. On top of this, of course, you had to work with single-minded devotion towards becoming the topper. You needed to be diligent, be rigorous with all your assignments, study more than anyone else and all that.

I don’t know how this view of the “puritan topper” got formed in my head. Maybe it was pattern recognition based on the profile of people who used to top in my schools (this was after I had all but given up on doing well academically, apart from entrance exams), especially in undergrad.

I’m also wondering if this image of the puritan topper had something to do with my own giving up – while I might have had the enthu to work hard at academics and do well, this sort of a puritan lifestyle that I had come to associate with toppers (I didn’t smoke or drink, but being nice to everyone all the time was well beyond me) seemed rather daunting.

In any case, this image of the puritan topper didn’t last long. At IIMB, for example, there was this guy who lived a few doors away from me who spent most of his time drinking and hardly any time studying, but aced all exams. Another guy quickly found himself a girlfriend, but continued to top. Suddenly, I found that “normal people” could be toppers as well, and that my view of the puritan topper had been formed mainly on a small number of data points and didn’t hold.

Yet, the number of years that this puritan topper image stayed in my head means that it’s one that has been hard to shake off. A couple of years back, for example, the all india topper in the IIT-JEE, while talking to the press, expressed tribute to his girlfriend for her support. While it’s normal for a class 12 person to have a girlfriend, this comment sort of threw me off – it didn’t fit my mental image of the puritan topper.

Sometimes it is possible to form an irrational belief based on a small number of data points, and irrespective of the number of data points you see to the contrary, it becomes hard to let go of these beliefs. And that makes you more irrational. But I guess, there’s no logic to a lot of these beliefs. Maybe as Rory Sutherland puts it, it’s all “psycho-logic”.

Gults and Grammar

Back in IIT, it was common to make fun of people from Andhra Pradesh for their poor command over the English language. It was a consequence of the fact that JEE coaching is far more institutionalised in that (undivided) state, because of which people come to IIT from less privileged backgrounds (on average) than their counterparts in Karnataka or Tamil Nadu or Maharashtra.

Now, in hindsight, making fun of people’s English doesn’t sound particularly nice, but sometimes stories come up that make it incredibly hard to resist.

This one is from Matt Levine’s newsletter. And it is about an insider trading ring. This is a quote that Levine has quoted in his newsletter (pay attention to the names):

According to the SEC’s complaint, Janardhan Nellore, a former IT administrator then at Palo Alto Networks Inc., was at the center of the trading ring, using his IT credentials and work contacts to obtain highly confidential information about his employer’s quarterly earnings and financial performance. As alleged in the complaint, until he was terminated earlier this year, Nellore traded Palo Alto Networks securities based on the confidential information or tipped his friends, Sivannarayana Barama, Ganapathi Kunadharaju, Saber Hussain, and Prasad Malempati, who also traded.

The SEC’s complaint alleges that the defendants sought to evade detection, with Nellore insisting that the ring use the code word “baby” in texts and emails to refer to his employer’s stock, and advising they “exit baby,” or “enter few baby.” The complaint also alleges that certain traders kicked back trading profits to Nellore in small cash transactions intended to avoid bank scrutiny and reporting requirements. After the FBI interviewed Nellore about the trading in May, he purchased one-way tickets to India for himself and his family and was arrested at the airport.

You can look at Levine’s newsletter to understand his take on the story (it’s towards the bottom), but what catches my eye is the grammar. I think it is all fine to refer to the insider-traded stock as a “baby”, but at least be grammatically correct about it!

“Enter few baby” is so obviously grammatically incorrect (it’s hard to even be a typo) that when intercepted by someone like the SEC, it would immediately send alarm bells ringing. Which is what I suppose possibly happened.

So my take on this case is – don’t insider  trade, but even if you do, be grammatical about your signals. If you’re so obviously grammatically wrong, it is easy for whoever intercepts your chats to know you’re up to something fishy.

But then if you’re gult..

Wimpy

This was posted by the dean of Alumni Affairs on LinkedIn.

The comments so far have been boilerplate – some just listing out some nicknames, and others saying a lot of them are not for public consumption.

In any case, for me, my nickname is a source of identity. I think there are a few criteria that it satisfies that make it so.

  • My “real name”, as I write it in most places (“Karthik S”), is incredibly common. So identifying myself as “Karthik” or “Karthik S” or “S Karthik” is hard. Not that many people (especially those I mostly interacted with more than 10 years back) easily identify me by my “full name”
  • My nickname is not obscene (I’ve found that names such as “cock”, “dildo”, “condom”, which are rather common in places like IITM don’t stick after graduation)
  • My nickname is rather unique. I don’t know anyone else (at all – either at IITM or otherwise) named Wimpy (apart from the Popeye character). So among people who know that Wimpy is one of my names, identifying myself thus means they can instantly recognise me

Guests at a party over the weekend included a couple of guys from IITM. My wife primarily know them by their nicknames (since that’s how I’ve always referred to them), and proceeded to introduce them as such to her friends (it helps again that both of them have rather unique nicknames, and common first names) using their nicknames.

So “seniors” at elite institutions – when you go through the rituals of giving “freshies” their names – keep in mind that if you make your choice well, the names you endow will continue to be used for decades. So eschew the lunds and the condoms and the dildos, and get creative. And a nice back story for the name helps as well!

The popularity of nicknames and political correctness

It is a rite of passage in an institution such as IIT (Indian Institute of Technology) that a first year student be given a potentially embarrassing nickname following “interaction” with senior students. The profundity of these nicknames varies significantly, with some people simply being given names that correspond to body parts in different languages, which others have more involved names.

Based on a conversation yesterday, the hypothesis is that more profound nicknames which are embarrassing only in a particular context are more likely to propagate, and thus stick, while the more crass names are likely to die out more easily.

The logic is simple – the crass names (a few examples being “lund”, “condom” and “dildo” – there is at least one person with each of these names in every hostel of every batch at IIT Madras) are potentially embarrassing for an “outsider” to use, and to be used in public. So when the bearer of such a name graduates and moves on to a new setting, the new people he encounters make a prudent choice to not use the embarrassing word, and the nickname dies a quick death.

When the nickname is embarrassing or derogatory for more contextual reasons, though, the name quickly loses its context and becomes incredibly simple for people to use. Take my own name “Wimpy”, for example – not too many people know it has an embarrassing origin, and it is a perfectly respectable word to shout out in public, or even in an office setting. And so it has propagated – in at least two offices I worked in, everyone called me “Wimpy”.

It is similar for lots of other “benign” names. But it is unlikely that a name like “condom” or “dildo” will propagate, and it is in fact more likely that even the people who bestowed such names upon the unsuspecting will stop using them once everyone graduates and moves on to a more formal environment.

There are exceptions, of course, a notable one being “Baada“. It is a cuss-word representing a body part, except that it is in a non-standard (though not small by any means) language, but everyone I know calls Baada Baada. He used to be my colleague, and people at work also called him Baada. It is unlikely that his nickname would’ve propagated, though, had it been the synonym in English or Hindi.

Thanks to Katpadi Katsa for discussions leading up to this post. In a future post, I’ll talk about models for propagation of nicknames across institutions.

 

 

Varamahalakshmi and Organic Chemistry

Today is Varamahalakshmi Vrata, a minor festival for South Indian Hindus. It is major enough, however, for a sufficiently large/influential proportion of the population, that schools declare a holiday on this day. It is not major enough, however, for the day to be declared a public holiday.

Mine is one of those families where this festival is not major enough to be celebrated. “It’s not an important festival for people of our caste”, my mother told me, though this now confounds me since this is a rather major festival in my wife’s family, and she belongs to the same caste as me.

The fact that this festival has been rather minor has meant that I don’t have much memories of past occurrences of this festival. There is one exception, though, which is what I want to talk about in this post. Varamahalakshmi Vrata of 1999 played an important part in shaping my performance in the IIT-JEE ten months hence.

In 1999, I was in class 12, and had spent the holidays between classes 11 and 12 attending the International Maths Olympiad Training Camp (IMOTC) in Mumbai. While I didn’t ultimately get selected to represent India, I had an overall great time at the camp, and learnt a lot of maths.

By the time I returned to Bangalore, though, class 12 had already started in school, and classes were also underway at my JEE factory, which I had joined just prior to my travel to Mumbai.

With the school teachers intending to finish the entire academic year’s portions by November, classes had been scheduled for Saturdays as well. This, combined with my JEE factory having classes on Friday and Saturday evenings and all day on Sunday, this left me little time to do pretty much anything.

It wasn’t that I wanted to do too many things – my focus that academic year had been to simply focus on the IIT-JEE and (to a much lesser extent) my class 12 board exams. Yet the near non-stop schedule at both school and factory had meant that I was constantly “running” to catch up, with little time for independent study outside of school, factory and their assignments. I desperately needed a holiday to slow down, grab my breath and catch up.

It is a quirk of the Indian festival calendar that there are few holidays between May Day and Independence Day (August 15). If one of the Muslim festivals (which move around the year) doesn’t occur in this time period, it is possible to not have any holidays at all. 1999 was one such year. And this is where Varamahalakshmi Vrata came to the rescue.

I don’t remember the exact date it occurred on in 1999, but it was a Friday (it always is). I had been especially struggling with organic chemistry in the past month, totally unable to grasp the concepts.

Now, the thing with class 12 organic chemistry is that there are lots of patterns, which you need to learn to recognise. Simply mugging is an option, of course (and I suppose a lot of people take that path), but the syllabus is so voluminous that you rather take a more scalable approach. Learning to recognise patterns, however, means that you be able to spend a sufficient amount of time on the concept without distractions. It takes a special kind of focus to be able to do that.

And so I sat down on the morning of Varamahalakshmi Vrata 1999 with “Tata McGraw Hill guide to IIT JEE Chemistry” (forget precise name), and started doing problems. I didn’t intend to discover patterns that day – simply to solve lots of problems so that I’d somehow get a hang. The fact that the festival wasn’t celebrated in my family meant there was no disturbance (of bells and prayers).

So it happened sometime around noon, or a bit later. I had started the morning mostly struggling with the problems, and having to put major fight to be able to solve them. Over time I had gotten better steadily, but slowly. Now, suddenly I found myself being able to solve most problems rather easily. I had to only look at a problem before I could recognise the pattern and apply the appropriate framework. Organic chemistry would be a breeze for the rest of that academic year.

It’s funny how learning happens sometimes. There is usually a moment, which usually comes after you’ve spent sufficient time on the problem, when there is a flash of inspiration and it all falls into place. It has happened to me several times hence. So much so that I fundamentally believe this is how all learning happens!

Or at least so I believed back in 2004 when I had to give a lecture on “Quality takes time” (this was part of a communications course at IIMB). Watch the video:

Dominant affiliation groups

I was writing an email to connect two friends, when I realised that when you know someone through more than one affiliation group, one of the affiliation groups becomes “dominant”, and you will identify with them through that group at the cost of others. And sometimes this can lead you to even forgetting that you share other affiliation groups with them.

In social networking theory, affiliation groups refer to entities such as families, communities, schools or workplaces through which people get connected to other people. It is not strictly necessary for two connected people to share an affiliation group, but it is commonly the case to share one or more such groups. Social networking companies such as Facebook and LinkedIn sometimes suggest connections to you based on commonly identified affiliation groups.

So my hypothesis is that when you share multiple affiliation groups with someone, you are likely to have been more strongly connected to them through one than through others. For example, you might have gone to the same school and then worked together, but your interaction in school would have been so little that it almost doesn’t count. Yet, the school  remains as a common affiliation group.

Does it happen to you as well? Do you forget that you share an affiliation group with someone because it is not the “dominant” one, since you share another? And due to that do you miss out on making connections, and thus on opportunities?

I had this hilarious incident two weeks back where I was meeting this guy W with whom I share three affiliation groups – BASE (the local JEE coaching factory), IIT Madras and IIM Bangalore. Due to the extent of overlap and degree of interaction, I know him fundamentally as an “IITM guy”. And there’s this other guy X who I also know through three affiliation groups – BASE (again), IIM Bangalore (again) and a shared hobby (the strongest).

So I was talking to W and was going to bring up the topic of X’s work, and suddenly wondered if W knows X, so I said “do you remember X, he was in your batch at BASE?”. And then a minute later “oh yeah, you guys were classmates at IIMB also!”.

The rather bizarre thing is that I had completely stopped associating both W and X with IIMB, since I have much stronger affiliation groups with them. And then when I had to draw a connection between them, I even more bizarrely picked BASE, where I hadn’t interacted with either of them, rather than IIMB, where I interacted with both of them to a reasonable degree (X much more than W).

I know I didn’t do much damage, but in another context, not realising connections that exist might prove costly. So I find this “interesting”!

Is there anybody else in here who feels the way I do?

Educating at scale

You can’t run a high-quality business school with 20 faculty members

In the course of a twitter discussion yesterday, journalist Mathang Seshagiri quoted numbers from a parliamentary reply by the ministry of HRD (on the 24th of November 2014) on the sanctioned faculty strength and vacancies in “institutes of national importance”. While his purpose was to primarily show that even the older IITs and IIMs have massive vacancies, what struck me was the sanctioned faculty strength of the newer IIMs. Here is the picture posted by Mathang:

Source: Parliamentary Proceedings (Rajya Sabha). November 24th 2014. Reply by MHRD

Look at the second column which shows the sanctioned faculty strength in each IIM. Once you go beyond the six older IIMs, the drop is stark. The seven newer IIMs have a sanctioned faculty strength of about 20! The question is how one can run a business school with such a small faculty base.

About ten years back, when I was a student at IIM Bangalore, I had gone for an event where I met someone from another business school in Bangalore whose name I can’t remember now. During the course of the conversation he asked me how many electives he had. I replied that we had about 80-100 courses from which we had to pick about 15. This he found shocking for in his college (from what I remember) there were only three or four electives!

The purpose of an MBA is to provide broad-based education and broaden one’s horizons. Thus, after a set of core courses in the first year (usually about fifteen courses), one is exposed to a wide variety of electives in the second year. It is a standard practice among most top B-schools to fill the entire second year with electives. In fact, in IIM Bangalore, electives start towards the end of the first year itself.

With 20 faculty members, there are only so many electives that can be offered each year. For example, in the coming trimester, IIM Bangalore is offering students (about 400 in the batch) a choice of about 40-50 electives, of which each student can pick four to six. This gives students massive choice, and a good chance to tailor the second year of their MBA and mould themselves as per their requirements.

By having 20 faculty members, the number of electives that can be theoretically offered itself is smaller (given research requirements, most IIM professors have a requirement to teach no more than three courses a year, and they have core and graduate courses to teach, too), which gives students an extremely tiny bouquet of choices – if there is any choice at all. This significantly limits the scope of what a student in such a school can do. And the student has no option but to accept the straitjacket offered by the lack of choice in the school.

In the ensuing twitter conversation this morning, Mathang contended that it is okay to have a faculty strength of 20 in schools with 60 students per batch. While this points to an extremely healthy faculty-student ratio, the point is that for broad-based education such as MBA, faculty-student ratio is not a good metric. What makes sense is the choice that the student is offered and that comes only at scale.

Thus, the new IIMs (Shillong “onwards”) are flawed in their fundamental design. It is impossible to run a quality business school with only 20 faculty. One way to supplement this is by using visiting faculty and guest lectures, but some of the new IIMs are located in such obscure places (where there is little local business, and which are not easily accessible by flight) that this is also not an option.

Merging some of these smaller IIMs (a very hard decision politically) might be the only way to make them work.

PS: Here is the sanctioned faculty strength and actual faculty strength numbers for IITs (same source as above). I might comment upon that at a later date.

Source: Parliamentary reply by Ministry of HRD; November 24th 2014