work – Pertinent Observations

The team around you

Back in 2016, footballer Oscar, then of Chelsea, was bought by Shanghai SIPT for a (then whopping) GBP 60M, with a salary of about GBP 20M a year.

Around the time the deal got announced, we were having our 10th year reunion at IIMB. During that a professor told us that one reason Shanghai had to pay Oscar so heavily was the quality (or lack thereof) of teammates he would have to deal with in Shanghai. He was then playing for Chelsea, who had won the Premier League in 2015, and would win it again that season (2016-17). And he was leaving that quality of teammates to join unknown teammates in China, and that meant he would have to be compensated heavily.

During and after last nights’ Manchester Derby, a friend and I were talking about Andre Onana, now the Manchester United goalie. Onana has been an extremely promising goalkeeper, excelling for Ajax and Inter (apart from one doping ban). He is a brilliant sweeper keeper (one reason he got chucked out of the Cameroon national team during last year’s World Cup), adept at playing with his feet and with great positioning sense.

And who does he have in front of him at ManYoo? The former Leicester defensive pair of Harry Maguire and Jonny Evans! They absolutely lack pace which means they can’t play a high line. That means Onana’s sweeping skills are grossly underutilised, and he ends up getting judged based on his shot-stopping skills, where he is nowhere in the same league as his predecessor David De Gea.

When we get into organisations, things we evaluate are the kind of work we do and what we are getting compensated for it. The thing we tend to overlook is who we need to work with, and whether they will elevate us or drag us down. Sometimes, the organisation (like Shanghai SIPT) recognises that you have to put up with suboptimal colleagues, and thus pay you a premium for your services.

Often, though, the organisation will be more like ManYoo, which doesn’t really recognise that the team around you may not be optimal for your playing style. And not everyone is willing to accept a premium in exchange for suboptimal colleagues. So, if you end up like Onana, you are not only frustrated because of the quality (or lack thereof) of your colleagues and peers, but you also end up getting judged on axes that are not your strengths (and what you have NOT been hired for).

Over the last decade, hiring at Manchester United has been curious, to say the least. There have been half-hearted attempts at changing the playing style, and almost everyone brought in to play the new style (I assume Erik Ten Haag wanted to play a more high-press style when he bought Onana) has been frustrated and unable to perform to potential.

Related to this, going back to something I’d written earlier this year, every company has an optimal rate of attrition, which is non-zero. If you end up paying too much of a premium to loyalty, you risk stagnation. If your Onana has to put up with Maguire and Evans, he won’t perform to potential. And then you go back to setting up the way it is optimal for Maguire and Evans.

Status and money

Over the last week or so, I’ve been discussing this post by Robin Hanson with just about anyone. The first paragraph is the one that caught my attention.

Having a romantic partner is useful in many ways. You won’t be as lonely, you can ask them for advice, you can do activities together, and you can share transport and even a household with them. But if you look carefully, you will notice that many people don’t choose such partners mainly for their promise in such roles. They instead seek high status partners, who make them look good by association. Partners who are hot, funny, rich, powerful, etc.

Nevertheless, I urge you to read the whole thing. Hanson goes on to talk about status in several other fields, such as politics or in organisations.

Broadly paraphrasing (you should still read the whole thing), he says that people want to be associated with people with high status, or people who add status to them. So politicians who can project higher status will get elected. Organisations will appoint people who can further increase the status of the organisation.

I was thinking about this today from the point of view of last night’s post, where I had compared my life in my (current) full time job to that of a consultant, which I had been for nine years prior.

Sometimes it is common for us to comment, or gossip, that someone got hired purely on the strength of their reputation, and that their abilities are not extraordinary. Sometimes, reputations can be self-fulfilling – if you can somehow get the reputation of being good at something, more people will start with the Bayesian prior that you’re good at that, and as long as you don’t suck at that thing, the prior will continue to hold. And so more people will think you’re good at it, and so on.

So when I think of my own career, basically I realise the way to go is to get into a position that my sheer presence adds status to the organisation I’m associated with. That way, they will be more forgiving of the work that I do (or don’t do). At the same time, from my own perspective, the organisation also needs to (at least marginally) add to my status – at some level I may not want to join a club that wants me as a member.

I remember back in the day when I was consulting – one of my clients, during the negotiations prior to the engagement, had wanted me to put on LinkedIn that I was working for them. Now when I think of it from the point of view of Hanson’s post, this was the client leveraging my then reputation in data to further their own status.

This is what I need to bring to my employers as well (I have no clue if I do already with my current ones – though I’m not so popular within my (data science) domain in india). The target, if I were to think of it, is to get into that self-fulfilling space when it comes to status – that people want me just because I’m me and bring along a certain (positive) status.

Now that I’ve identified the target, I need to figure out how to get there. I know in his famous podcast, Naval said that we should optimise for wealth (a positive sum game) rather than for status (a zero sum game). But Hanson’s post, and my analysis of it, suggests that status can also lead to wealth. I need to figure out the tradeoff now!

Coasean notes

I’m well over two and a half years into my current job, easily making this my longest unbroken spell of employment ever. This is a random set of pertinent observations, more a set of notes to myself rather than for any reader, regarding how the job has been playing out.

The Nature of The Firm is real. For nine years, as a consultant, I enjoyed market pricing (adjusting for illiquidity and and other distortions) for all the work that I did, but also suffered from the transaction costs that Coase writes about in his famous paper.
This meant that unless the work was reasonably well defined, or of a certain minimum size, I wouldn’t take it up – the transaction costs involved in doing the deal would far outweigh any benefits that my counterparty and I would achieve from the deal. This meant I added less value than I could have to my clients
“Going deep” has its benefits. If I look at some of the work that I’ve done in the last few months here, and compared that to my work in my first year here, there is an absolutely marked difference. The difference is the two years of compounded extreme domain knowledge (about the company and its business).
From that perspective, consulting can sometimes suffer from a limitation of domain knowledge
Countering the above point is that I’ve “been internalised” after two plus years here. The things that excited me at the time I joined don’t excite me any more. There are times when I get what I think are interesting insights, and then just don’t bother about showing them to anyone, based on the historical reaction to such insights.
A fresh consultant, on the other hand, would share more, and would thus get more done
The biggest advantage of being “in house” is the data – I have access to pretty much ALL data in the company, and if I don’t have access to something, there is a good chance that the data doesn’t exist. This means I’m able to craft better hypotheses and do better analysis, compared to the time when I relied on clients to share specific datasets with me (pretty much nobody opened up full live access to their database to me)
In a way I also miss the novelty of being a consultant – because you work with a company for a short period of time, you are bringing in new ideas and insights in that period of time, and people pay you attention for it. As an in-house employee, you become a part of the furniture. And a lot of the time, it is a good thing if nobody notices you
Lack of friction in terms of taking up work means average quality of work can suffer. If you are very particular about the kind of work you want to do, it’s good if you can be a consultant – the friction means it’s easier to say no there.
As a consultant, by definition, I was a “hybrid worker”, working by myself for long periods of time and then visiting the client for meetings and discussions. That had worked out brilliantly well for me.
However, I realise “that hybrid” is different from “this hybrid” (the job), since here people have access to my calendar and are able to schedule meetings even at times when I’m not in office. Rather, since my company has a multiple-headquarter setup, I even prefer to take meetings with colleagues not in Bangalore on days when I’m at home.
The biggest difference between monogamy (one employer) and polyamory (two or more “clients”) is that in the latter, no one owns your time. Because they know that they are “one of several” (even if at some point in time they are “one of one” it doesn’t matter, since that’s a special case), they can’t take your time for granted. And that gives you immensely more control over your time.
This was possibly the hardest part for me getting back to a full time job – the lack of control over my time since I had now sold ALL of it to one company.
The flip side of this is that, at least for someone like me, not having to keep selling myself constantly is a brilliant feeling. Though, there is some amount of “within the company selling” that has to happen from time to time.
Apart from control over my time, the thing I miss the most about my consulting life are the “semi work meetings” – these are meetings with prospective clients, people who can lead you to prospective clients, old clients, etc. Where there is a tinge of work to the meeting, but you also catch up on several other things.
Now that I’m in a job, and one that is entirely internal facing, there is no concept of “pseudo work meetings”. It is either proper work meetings (or “water cooler conversations”) with colleagues, or proper socialisation with others. That means I’m meeting far fewer people on average, nowadays
I admit that having become a sort of a “company man“, I’ve started taking myself more seriously than I would like to. Of late I’ve started making a conscious effort to dial this back a little bit, and I think it’s already making me happier.
Oh, and game theory rocks. Not a day goes by without me thinking about “saama daana bhEda danDa“

I can go on and on and on, but I think this is enough for now. If I have more, I’ll write another post.

Bad Data Analysis

This is a post tangentially related to work, so I must point out that all views here are my own, and not views of my employer or anyone else I’m associated with

The good thing about data analysis is that it’s inherently easy to do. The bad thing about data analysis is also that it’s inherently easy to do – with increasing data democratisation in companies, it is easier than ever than pulling some data related to your hypothesis, building a few pivot tables and charts on Excel and then presenting your results.

Why is this a bad thing, you may ask – the reason is that it is rather easy to do bad data analysis. I’m never tired of telling people who ask me “what does the data say?”, “what do you want it to say? I can make it say that”. This is not a rhetorical statement. As the old saying goes, you can “take data down into the basement and torture it until it confesses to your hypothesis”.

So, for example, when I hire analysts, I don’t check as much for the ability to pull and analyse data (those can be taught) as I do for their logical thinking skills. When they do a piece of data analysis, are they able to say that it makes sense or not? Can they identify that some correlations data shows are spurious? Are they taking ratios along the correct axis (eg. “2% of Indians are below the poverty line”, versus “20% of the world’s poor is in India”)? Are they controlling for instrumental variables?

This is the real skill in analytics – are you able to draw logical and sensible conclusions from what the data says? It is no coincidence that half my team at my current job has been formally trained in economics.

One of the externalities of being a head of analytics is that you come across a lot of bad data analysis – you are yourself responsible for some of it, your team is responsible for some more and given the ease of analysing data, there is a lot from everyone else as well.

And it becomes part of your job to comment on this analysis, to draw sense from it, and to say if it makes sense or not. In most cases, the analysis itself will be immaculate – well written queries and logic / code. The problem, almost all the time, is in the logic used.

I was reading this post by Nabeel Qureshi on puzzles. There, he quotes a book on chess puzzles, and talks about the differences between how experts approach a problem compared to novices.

The lesson I found the most striking is this: there’s a direct correlation between how skilled you are as a chess player, and how much time you spend falsifying your ideas. The authors find that grandmasters spend longer falsifying their idea for a move than they do coming up with the move in the first place, whereas amateur players tend to identify a solution and then play it shortly after without trying their hardest to falsify it first. (Often amateurs, find reasons for playing the move — ‘hope chess’.)

Call this the ‘falsification ratio’: the ratio of time you spend trying to falsify your idea to the time you took coming up with it in the first place. For grandmasters, this is 4:1 — they’ll spend 1 minute finding the right move, and another 4 minutes trying to falsify it, whereas for amateurs this is something like 0.5:1 — 1 minute finding the move, 30 seconds making a cursory effort to falsify it.

It is the same in data analysis. If I think about the amount of time I spend in analysing data, a very very large percentage of it (can’t put a number since I don’t track my time) goes in “falsifying it”. “Does this correlation make sense?”; “Have I taken care of all the confounding variables?”; “Does the result hold if I take a different sample or cut of data?”. “Has the data I’m using been collected properly?”; “Are there any biases in the data that might be affecting the result?”; And so on.

It is not an easy job. One small adjustment here or there, and the entire recommendations might flip. Despite being rigorous with the whole process, you can leave in some inaccuracy. And sometimes what your data shows may not conform to the counterparty (who has much better domain knowledge)’s biases – and so you have a much harder job selling it.

And once again – when someone says “we have used data, so we have been rigorous about the process”, it is more likely that they are more wrong.

Hybrid work

I’m in a job that can broadly be described as “hybrid”. The mandate from HR is that we are are “expected to be in office three days a week, and live in the same city as the office”. Nobody really checks how often people go in to office, though I do end up going three times a week on average.

Of late, some tech “gurus” have taken on dunking on hybrid work. DHH of 37signals / Basecamp (I quite like his blog, in general) wrote that “hybrid combines the worst of in-person and remote“. Then, Paul Graham wrote some tweets on remote work. I quite like this one:

Why were all these smart people fooled? Partly I think because remote work does work initially, if you start with a system already healthy from in-person work. (It's like communism in that respect.) And partly because it seemed to solve recruiting, which is always a bottleneck.

— Paul Graham (@paulg) June 10, 2023

Back to hybrid work – I’m in a hybrid role now, where I go into office about three days a week on average, and stay home the other two days (in general, because Monday is crowded with long online meetings, and another day to do some “thinking work”). Different people in my company have different such strategies, and all come into office on their own schedules.

This is not the first time I’m doing “hybrid”. During my rather long independent consulting career, I largely worked from home but travelled to clients’ offices ever so often (once a week if in Bangalore; one week a month if not; on average). It was about getting the best combination of focussed work and collaboration. It worked then, and it works now.

In fact, as far back as 2007 I was in a hybrid office. I was in what is now called a “global capability centre”, and interacting with headquarters in Texas meant being available for calls later in the evening. Consequently, we could work from home a few days a week as long as we were available for these calls.

Coming as it did at the beginning of my career, it was a disaster. I slacked like nobody’s business. Less time spent in office meant less time understanding parts of the business not directly concerned with what I was working on. Most of my development in that period happened due to my independent reading and writing, rather than due to my work.

Now, once again, I’m in a company with “multiple headquarters”. This means that irrespective of where you are, you end up spending a considerable amount of time on video calls with people in other locations. According to DHH, video calls when you are in office is a waste of office time. I agree with him there. The way I manage is through my schedule.

Of course, it helps that I have a reputation in office that I don’t like to do unnecessary meetings – and all matters need to be resolved to the extent possible in text messages or email. This means I spend less time on video calls than many of my colleagues, and when I find a lot of them appearing on a day, i spend that day at home.

Also, I have an unspoken agreement with my (rather small) team on days of the week when we’ll meet in office, and so the technical discussions I find so difficult to have online can be had in person.

Hybrid primarily works because of optionality (a rather underappreciated concept). In my line of business, things can get so technical that there is a limit on the complexity of discussions that can be had online. Similarly, things can get so technical that we need undisturbed alone time to think through some of the solutions.

Hybrid works because it allows for both – it allows you to have your me time for your deep thinking, and the optionality of summoning a teammate to office “tomorrow” for some deep collaboration. The former is unavailable in an all-in office; the latter is not possible if you’re fully remote (I’ve experienced this during the pandemic years).

Yes, hybrid means you need to live within commuting distance of office (sometimes during interviews, I see candidates furiously googling for “richmond circle” or “residency road” when I tell them our office is there. It’s a strong signal that they’re not going to join ). However, that you only need to commute twice a week (rather than 5 times a week) means you can choose to live a little bit farther.

Yes, it does make hiring harder (compared to all-remote), but once hired, people can be far more productive in a hybrid model. With the option of doing deep work without the danger / fear of someone poking you (this literally happened to me yesterday) when you’re in the middle of deep work!

So yes, put me down as someone who likes the hybrid model of work.

New blood joins this team

I intended to write this a year ago, when Sadio Mane left Liverpool after six brilliant years at the club. There was much heartbreak among the club fan base about Mane leaving, and a lot of people saw it as a failure on the part of the management and ownership in terms of not being able to keep him.

Now, a year on, I admit that Darwin Nunez hasn’t quite set the club on fire (though I personally quite like him), but as a general principle, this kind of “freshening up” is a highly necessary process in a team, if you need to avoid stagnation.

A month or two back, I was watching some YouTube video on “Liverpool’s greatest Premier League goals against Manchester City” (this was just before the 4-1 hammering at the Etihad). As the goals were shown one by one, I kept trying to guess which season and game it was in.

There were important clues – whether Firmino wore 9 or 11, whether Mane wore 19 or 10, the identity of some players, the length of Trent Alexander Arnold’s hair, my memory of the scoreline from that game, etc. (Liverpool always wear the home Red at the Etihad, so the colour of the away kit wasn’t a clue).

However, for one goal I simply wasn’t able to figure out which season it was. There was TAA wearing 66, Fabinho, Henderson, the fab front three (Firmino-Mane-Salah, wearing 9-10-11 respectively) and Robertson. That’s when it hit me that for a fairly long time, a large part of Liverpool’s team had stayed constant! There was very little change at the club.

Now, there are benefits to having a consistently settled team (as the fabulous 2021-22 season showed), but there is also the danger of stasis. In something like football where careers are short, you don’t want the whole team “getting old together”. In the corporate world, people can get into too much of a comfort zone. And cynicism can set in.

Good new employees are always buzzing with ideas, fearless about what has been rejected before and who thinks how. As people spend longer in the organisation, though, colleagues become predictable and certain ways of doing things become institutionalised. Sooner than you know it, you would have become a “company man”, (figuratively) wearing the same white shirt and blue suits as your fellow company men, and socialising with your colleagues at the (figurative) company club.

There can be different kinds of companies here – some companies allow people to retain a lot of their individuality; and there the “decay” into company-manhood is slower. In this kind of a place, the same set of people can stay together for longer and still continue to innovate and add significant value to one another.

Other companies are less forgiving, and you very quickly assimilate, and lose part of your idiosyncrasy. Insofar as innovation comes out of fresh ideas and thinking and unusual connections, these companies are not very good at it. And in such companies, pretty much the only way to keep the innovative wheel going and continue to add value is by bringing in fresh blood well-at-a-faster-rate.

Putting it another way, if you are a cohesive kind of company, some attrition may not actually be a bad thing (unless you are growing rapidly enough to expand your team rapidly). To grow and innovate, you need people to think different.

And you get there either by having the sort of superior culture where existing employees continue to think different long after they’ve been exposed to one another’s thoughts; or by continuing to bring in fresh employees.

There is no other way.

Round Tables

One of the “features” of being in a job is that you get invited to conferences and “industry events”. I’ve written extensively about one of them in the past – the primary purpose of these events is for people to be able to sell their companies’ products, their services and even themselves (job-hunting) to other attendees.

Now, everyone knows that this is the purpose of these events, but it is one of those things that is hard to admit. “I’m going to this hotel to get pitched to by 20 vendors” is not usually a good enough reason to bunk work. So there is always a “front” – an agenda that makes it seemingly worthy for people to attend these events.

The most common one is to have talks. This can help attract people at two levels. There are some people who won’t attend talks unless they have also been asked to talk, and so they get invited to talk. And then there are others who are happy to just attend and try to get “gyaan”, and they get invited as the audience. The other side of the market soon appears, paying generous dollars to hold the event at a nice venue, and to be able to sell to all the speakers and the audience.

Similarly, you have panel discussions. Organisers in general think this is one level better than talks – instead of the audience being bored by ONE person for half an hour, they are bored by about 4-5 people (and one moderator) for an hour. Again there is the hierarchy here – some people won’t want to attend unless they have been put on the panel. And who gets to be on the panel is a function of how desperate one or more sponsors is to sell to the potential panelists.

The one thing most of these events get right is to have sufficient lunch and tea breaks for people to talk to each other. Then again, these are brilliant times for sponsors to be able to sell their wares to the attendees. And it has the positive externality that people can meet and “network” and talk among themselves – which is the best value you can get out of an event like this one.

However, there is one kind of event that I’ve attended a few times, but I can’t understand how they work. This is the “round table”. It is basically a closed room discussion with a large number of invited “panellists”, where everyone just talks past each other.

Now, at one level I understand this – this is a good way to get a large number of people to sell to without necessarily putting a hierarchy in terms of “speakers” / “panellists” and “audience”. The problem is that what they do with these people is beyond my imagination.

I’ve attended two of these events – one online and one offline. The format is the same. There is a moderator who goes around the table (not necessarily in any particular order), with one question to each participant (the better moderators would have prepared well for this). And then the participant gives a long-winded answer to that question, and the answer is not necessarily addressed at any of the other participants.

The average length of each answer and the number of participants means that each participant gets to speak exactly once. And then it is over.

The online version of this was the most underwhelming event I ever attended – I didn’t remember anything from what anyone spoke, and assumed that the feeling was mutual. I didn’t even bother checking out these people on LinkedIn after the event was over.

The offline version I attended was better in the way that at least we could get to talk to each other after the event. But the event itself was rather boring – I’m pretty sure I bored everyone with my monologue when it was my turn, and I don’t remember anything that anyone else said in this event. The funny thing was – the event wasn’t recorded, and there was hardly anyone from the organising team at the discussion. There existed just no point of all of us talking for so long. It was like people who organise Satyanarayana Poojes to get an excuse to have a party at home.

I’m wondering how this kind of event can be structured better. I fully appreciate the sponsors and their need to sell to the lot of us. And I fully appreciate that it gives them more bang for the buck to have 20 people of roughly equal standing to sell to – with talks or panels, the “potential high value customers” can be fewer.

However – wouldn’t it be far more profitable to them to be able to spend more time actually talking to the lot of us and selling, rather than getting all of us to waste time talking nonsense to each other? Like – maybe just a party or a “lunch” would be better?

Then again – if you want people to travel inter-city to attend this, a party is not a good enough excuse for people to get their employers to sponsor their time and travel. And so something inane like the “round table” has to be invented.

PS: There is this school of thought that temperatures in offices and events are set at a level that is comfortable for men but not for women. After one recent conference I attended I have a theory on why this is the case. It is because of what is “acceptable formal wear” for men and women.

Western formal wear for men is mostly the suit, which means dressing up in lots of layers, and maybe even constraining your neck with a tie. And when you are wearing so many clothes, the environment better be cool else you’ll be sweating.

For women, however, formal wear need not be so constraining – it is perfectly acceptable to wear sleeveless tops, or dresses, for formal events. And the temperatures required to “air” the suit-wearers can be too cold for women.

At a recent conference I was wearing a thin cotton shirt and could thus empathise with the women.

Shrinking deadlines

I’m reminded of this old joke/riddle, which also happened to feature in Gowri Ganesha. “If a 1 metre long sari takes 1 hour to dry in the sun, how long will and 8 metre long sari take to dry?”.

The instinctive answer, of course, is 8 hours, while if you think about it (and assume that you have enough clothesline space to not need to fold), the correct answer is likely to be 1 hour.

Now this riddle is completely unconnected to do with the point of the post, except that both have to do with time.

And then one day you find, ten years have got behind you.
No one told you when to run. You missed the starting gun.

Ok enough distractions. I’m now home, home again.

Modern workspaces are synonymous with tight deadlines. Even when you give a conservative estimate on how long something will take, you get asked to compress the timelines further. If you protest too much and say that there is a lot to be done, sometimes you might get asked to “put one more person on the job and get it done quickly”.

This might work for routine, or “fighter” jobs – for example, if your job is to enter and copy data for (let’s say) 1000 records, you can easily put another person on the job, and the entire job will be done in about half the time (allowing for a little time for the new person to learn the job and for coordination).

As the job gets more complex, the harder it gets. At one level, there is more time to be spent by the new person coming into the job. Then, as the job gets more complex, it gets harder to divide and conquer, or to “specialise”. This means there is lesser impact to the new person coming in.

And then when you get closer and closer to the stud end of the spectrum, the advantage of putting more people to get the work done faster get lesser and lesser. There comes a point when the extra person actively becomes a liability. Again – I’m reminded of my childhood when occasionally I would ask my mother if she needed help in cooking. “Yes, the best way for you to help is for you to stay out of the kitchen”, she would say.

And then when the job gets really creative, there is a further limit on compression – a lot of the work is done “offline”. I keep telling people about how I finally discovered the proof of Ramsey’s numbers (3,3) while playing table tennis in my hostel, or how I had solved a tough assignment problem while taking a friend’s new motorcycle for a ride.

When you want to solve problems “offline” (to let the insight come to you rather than going hunting for it – I had once written about this) – there is no way to shorten the process. You need to let the problem stew in your head, and hope that some time it will get solved.

There is nothing that can be done here. The more you hurry up, the less the chances you give yourself of solving the problem. Everything needs to take its natural course.

I got reminded of it when we missed a deadline last Friday, and I decided to not think about it through the weekend. And then, an hour before I got to work on Monday, an idea occurred in the shower which fixed the problem. Even if I’d stressed myself (and my team) out on Friday, or done somersaults, the problem would not have been solved.

As I’d said in 2004, quality takes time.

Average skill and peak skill

One way to describe how complex a job is is to measure the “average level of skill” and “peak level of skill” required to do the job. The more complex the job is, the larger this difference is. And sometimes, the frequency at which the peak level of skill is required can determine the quality of people you can expect to attract to the job.

Let us start with one extreme – the classic case of someone turning screws in a Ford factory. The design has been done so perfectly and the assembly line so optimised that the level of skill required by this worker each day is identical. All he/she (much more likely a he) has to do is to show up at the job, stand in the assembly line, and turn the specific screw in every single car (or part thereof) that passes his way.

The delta between the complexity of the average day and the “toughest day” is likely to be very low in this kind of job, given the amount of optimisation already put in place by the engineers at the factory.

Consider a maintenance engineer (let’s say at an oil pipeline) on the other hand. On most days, the complexity required of the job is very close to zero, for there is nothing much to do. The engineer just needs to show up and potter around and make a usual round of checks and all izz well.

On a day when there is an issue however, things are completely different – the engineer now needs to identify the source of the issue, figure out how to fix it and then actually put in the fix. Each of this is an insanely complex process requiring insane skill. This maintenance engineer needs to be prepared for this kind of occasional complexity, and despite the banality of most of his days on the job, maintain the requisite skill to do the job on these peak days.

In fact, if you think of it, a lot of “knowledge” jobs, which are supposed to be quite complex, actually don’t require a very high level of skill on most days. Yet, most of these jobs tend to employ people at a far higher skill level than what is required on most days, and this is because of the level of skill required on “peak days” (however you define “peak”).

The challenge in these cases, though, is to keep these high skilled people excited and motivated enough when the job on most days requires pretty low skill. Some industries, such as oil and gas, resolve this issue by paying well and giving good “benefits” – so even an engineer who might get bored by the lack of work on most days stays on to be able to contribute in times when there is a problem.

The other way to do this is in terms of the frequency of high skill days – if you can somehow engineer your organisation such that the high skilled people have a reasonable frequency of days when high skills are required, then they might find more motivation. For example, you might create an “internal consulting” team of some kind – they are tasked with performing a high skill task across different teams in the org. Each time this particular high skill task is required, the internal consulting team is called for. This way, this team can be kept motivated and (more importantly, perhaps) other teams can be staffed at a lower average skill level (since they can get help on high peak days).

I’m reminded of my first ever real taste of professional life – an internship in an investment bank in London in 2005. That was the classic “high variance in skills” job. Having been tested on fairly extreme maths and logic before I got hired, I found that most of my days were spent just keying in numbers in to an Excel sheet to call a macro someone else had written to price swaps (interest rate derivatives).

And being fairly young and immature, I decided this job is not worth it for me, and did not take up the full time offer they made me. And off I went on a rather futile “tour” to figure out what kind of job has sufficient high skill work to keep me interested. And then left it all to start my own consultancy (where others would ONLY call me when there was work of my specialty; else I could chill).

With the benefit of hindsight (and having worked in a somewhat similar job later in life), though, I had completely missed the “skill gap” (delta between peak and average skill days) in my internship, and thus not appreciated why I had been hired for it. Also, that I spent barely two months in the internship meant I didn’t have sufficient data to know the frequency of “interesting days”.

And this is why – most of your time might be spent in writing some fairly ordinary code, but you will still be required to know how to reverse a red-black tree.

Most of your time might be spent in writing SQL queries or pulling some averages, but on the odd day you might need to know that a chi square test is the best way to test your current hypothesis.

Most of your time might be spent in managing people and making sure the metrics are alright, but on the odd day you might have to redesign the process at the facility that you are in charge of.

In most complex jobs, the average day is NOT similar to the most complex day by any means. And thus the average day is NOT representative of the job. The next time someone I’m interviewing asks me what my “average day looks like”, I’ll maybe point that person to this post!

Chat GPT and Information Content

For the last week or so I’ve been fooling around with Chat GPT. It’s been a lot of fun, if you can endure the verbosity of its answers. Yesterday, at work, I used it for tech support when some people I was talking to were using too much jargon.

Before I came home from work yesterday, my last conversation was with a couple of recruiters, about a role that I’m hiring for. So when I got home, on a whim, I wondered if I could check out of Chat GPT could write job descriptions for me.

I asked:

write a job description for a senior data scientist at India’s largest logistics company. 4-7 years exp. Needs to have experience in ML, OR and statistics, and be able to work close to business and on ambiguous problems

It answered (pasting in full since the screenshot above is limited):

Job Title: Senior Data Scientist

Company: India’s Largest Logistics Company

Location: India

Experience: 4-7 years

We are seeking an experienced and highly skilled Senior Data Scientist to join our team at India’s largest logistics company. In this role, you will be responsible for using machine learning, operations research, and statistics to drive business insights and decision making.

As a Senior Data Scientist, you will have a deep understanding of data science techniques and technologies, and will be able to apply them to complex and ambiguous problems. You will be able to work closely with business teams to understand their needs and to develop solutions that drive value for the company.

To be successful in this role, you should have at least 4-7 years of experience in data science, and should have a strong background in machine learning, operations research, and statistics. You should be able to work independently and be comfortable working on complex and ambiguous problems.

Responsibilities:

Use machine learning, operations research, and statistics to drive business insights and decision making

Work closely with business teams to understand their needs and develop solutions

Develop and maintain data pipelines and systems

Collaborate with other data scientists and cross-functional teams to develop and implement solutions

Qualifications:

Bachelor’s or Master’s degree in a quantitative field, such as computer science, statistics, or engineering

At least 4-7 years of experience in data science

Strong background in machine learning, operations research, and statistics

Ability to work independently and on complex and ambiguous problems

Excellent communication and collaboration skills

Experience with Python, R, or other programming languages is a plus

We offer a competitive salary and benefits package, as well as opportunities for growth and advancement. If you are an experienced data scientist with a passion for driving business insights and decision making, we encourage you to apply.

In the past when I’ve been hiring, I’ve been a bit annoyed by people asking for job descriptions. I’ve largely found them to be fluff and boilerplate, and I’ve tried to get away with a sentence or two. However, due to the insistence of recruiters and candidates alike, I’ve ended up writing elaborate job descriptions. I don’t need to any more.

This post, however, is not about hiring (but yeah – I am hiring, and the above job description is reasonably accurate). It’s about information content in general.

Among other things, this tells you how little information content there usually is in job descriptions!

— Karthik S (@karthiks) December 7, 2022

Traditionally, information content has been measured in bits. A bit is the amount of information contained in an equi-probable binary event.

Sometimes when we find that someone is too verbose and using too many words when fewer would suffice, we say that their bit rate is low. We also use “low bit rate” to describe people such as former Prime Minister Atal Behari Vajpayee, who would speak incredibly slowly.

However, beyond the bit, which is a fairly technical concept, it has been difficult to quantify information content. Sometimes you read an article or a story and find that there is nothing much to it. But given the natural language, and the context of various words, it is impossible to quantify the information content.

Now, with Chat GPT, maybe it becomes a bit easier (though one would need a “reverse chat GPT algo”, to find the set of prompts required for Chat GPT to churn out a particular essay). Above, for example, I’ve shown how much fluff there generally is to the average job description – a fairly short prompt generated this longish description that is fairly accurate.

So you can define the information content of a piece or essay in terms of the number of words in the minimum set of prompts required for Chat GPT (or something like it) to come up with it. If you are a boring stereotypical writer, the set of prompts required will be lower. If you are highly idiosyncratic, then you will need to give a larger number of prompts for Chat GPT to write like you. You know where I’m going.

This evening, in office, a colleague commented that now it will be rather easy to generate marketing material. “Even blogs might become dead, since with a few prompts you can get that content”, he said (it can be a legit service to build off the Chat GPT API to take a tweet and convert it into an essay).

I didn’t tell him then but I have decided to take it up as a challenge. I consider myself to be a fairly idiosyncratic writer, which means I THINK there is a fair bit of information content in what I write, and so this blog will stay relevant. Let’s see how it goes.

PS: I still want to train a GAN on my blog (well over a million words, at last count) and see how it goes. If you know of any tools I can use for this, let me know!