arbit – Page 12 – Pertinent Observations

Uncertainty and Anxiety

A lot of parenting books talk about the value of consistency in parenting – when you are consistent with your approach with something, the theory goes, the child knows what to expect, and so is less anxious about what will happen.

It is not just about children – when something is more deterministic, you can “take it for granted” more. And that means less anxiety about it.

From another realm, prices of options always have “positive vega” – the higher the market volatility, the more the price of the option. Thinking about it another way, the more the uncertainty, the more people are willing to pay to hedge against it. In other words, higher uncertainty means more anxiety.

However, sometimes the equation can get flipped. Let us take the case of water supply in my apartment. We have both a tap water connection and a borewell, so historically, water supply has been fairly consistent. For the longest time, we didn’t bother thinking about the pressure of water in the taps.

And then one day in the beginning of this year the water suddenly stopped. We had an inkling of it that morning as the water in the taps inexplicably slowed down, and so stored a couple of buckets until it ground to a complete halt later that day.

It turned out that our water pump, which is way deep inside the earth (near the water table) was broken, so it took a day to fix.

Following that, we have become more cognisant of the water pressure in the pipes. If the water pressure goes down for a bit, the memory of the day when the motor conked is fresh, and we start worrying that the water will suddenly stop. I’ve panicked at least a couple of times wondering if the water will stop.

However, after this happened a few times over the last few months I’m more comfortable. I now know that fluctuation of water pressure in the tap is variable. When I’m showering at the same time as my downstairs neighbour (I’m guessing), the water pressure will be lower. Sometimes the level of water in the tank is just above the level required for the pump to switch on. Then again the pressure is lower. And so forth.

In other words, observing a moderate level of uncertainty has actually made me more comfortable now and reduced my anxiety – within some limits, I know that some fluctuation is “normal”. This uncertainty is more than what I observed earlier, so in other words, increased (perceived) uncertainty has actually reduced anxiety.

One way I think of it is in terms of hidden risks – when you see moderate fluctuations, you know that fluctuations exist and that you don’t need to get stressed around them. So your anxiety is lower. However, if you’ve gone a very long time with no fluctuation at all, then you are concerned that there are hidden risks that you have not experienced yet.

So when the water pressure in the taps has been completely consistent, then any deviation is a very strong (Bayesian) sign that something is wrong. And that increases anxiety.

Simpson’s Paradox for Levitt’s Measure

Some of you might know that I do this daily covid-19 update on twitter (not linking since I delete each day’s posts the next morning). A couple of weeks back I revamped it, in advance of which I asked what people wanted to see.

A lot of people suggested I use “Levitt’s metric”. I ignored it. Then, after I had revamped the output last week, two people I know very well got in touch asking me to report that metric every morning in my update. This time I decided to do it, and added it to my update on Monday.

My daily update has the smoothed line using a loess smoothing, but I also wanted to see if I can “predict” when the pandemic might end in different places. And so I did a linear fit as well (using 1 month of data – the slope of the line is highly sensitive to how far back you go), and posted it on Twitter.

I’ve extended the X axis of the graph until the end of the year. The idea is that when the blue line (the regression line based on the last 30 data points) hits the red line, the pandemic in that place is “effectively over”. So we can predict when the pandemic might end in different places.

Now, if you slightly contort your neck and try and extend the “india” graph here rightwards, you might see that the pandemic might end (for all practical purposes) around February. The funny thing is that while on average the pandemic might end in India in February, we see that for specific regions the slope is actually increasing (which suggests the pandemic might never end).

And this creates confusion. When you have a bunch of regions with upward slopes, and then suddenly for the aggregate (India) it is a downward slope, it doesn’t make intuitive sense. It is similar to Simpson’s paradox, where a trend disappears when you aggregate data. This graph possibly represents the most famous example of Simpson’s paradox.

Back to the Levitt’s metric, my only explanation is that the curve can’t be infinitely upward sloping – the number of people in any place is finite and so the disease is bound to die out some time or the other. The upward sloping lines are only a figment of the arbitrary linear extrapolation, and are likely to turn down sooner rather than later.

3 x 4 = 6 x 2

I’ll get to the “weird” title of this post soon.

Over at The Paper, which Suprio Guha Thakurta and I have been writing for two months now, one of our ongoing themes (in the context of the pandemic) is that “people will continue to do the same things, but do them in a different way”. We have corollaries to this and all that.

Here is one corollary that is suited more for this blog than it is to The Paper. Basically, when people do things in a different way, they do more of and less of certain smaller things, and this more and less balance out (that explains the title). OK I don’t think you would have understood any of that so let me clarify with some examples.

People are going to commute less (more working from home, less going out and all that), but when they commute, they are far more likely to use cars than using public transport. So the amount of traffic on the road remains a constant.

There will be far fewer “casual restaurant visits”, so when people want to go out to eat, they want to make sure it counts. So they go to really nice places. The “mass luxury” mid-tier places might lose out.

There will be fewer guests at weddings, since in some places the law mandates that now, and people won’t want to go to very crowded events. However, since the number of guests is going to be smaller, people can afford more lavish weddings “per guest”. So they’ll book fancier (if smaller) halls than they would earlier. Fancier (if fewer) meals. Put up guests in hotels rather than in crowded choultries.

In all this there will be winners and losers. The wedding caterer who charges per guest is a loser. The guy supplying the more fancy stuff (or the hotel guy) might be the winner. The large wedding hall guy is a loser. The fancy small hall guy is a winner.

And so on and so forth.

So this post was triggered by two things I saw during a walk yesterday. I first passed by a small-ish (but nice) hall that used to be used for small functions back in the day. It was hosting a wedding yesterday, and the few people who were there seemed rather well dressed up. Far better dressed than people dress for weddings in Bangalore.

Two minutes later, I paused while crossing the road to make way for a bus, and started thinking about when the next time would be when I would take public transport. And then decided to write this.

The Mint Way and the NED Way

I wrote for Mint for six and a half years. I loved writing for them. The editors were fantastic, the copy desk was understanding, and for the most part (until the editor who hired me moved on), they published most of the stuff I wrote.

What I wrote for Mint also helped open doors, as I ended up striking up many conversations based on that (though I’ve forgotten if any of them converted to revenue generation). At least in my initial year of writing for them, when I did a data-based take on elections in the run up to Modi’s first national election, I seemed to get a lot of “footage” and attention.

However, one person who was definitely unimpressed with my writing for Mint was my wife. Apart from two or three articles (I remember this and this for sure), she considers most of my writing for Mint as being rather boring. And when I go back to read some of the stuff I’ve written for them, I must agree.

The sort of flow that is there to every post I write here (or at least most posts) is completely missing there. A lot of pieces seems to simply be a collection of facts, and a small dose of analysis based on those facts. I hesitated to state my opinions and “take risks” in my writing. Barring one or two pieces I even hesitated to use a personal voice, appearing rather impersonal in most of my writing.

I had started writing for them at a time when I was starting to be known as a sort of data guy (I mainly wrote data related stuff for them). And from somewhere I had picked up this notion that it is honourable to be faithful to only the data, and to describe it as it is and to the extent possible simply state facts without taking sides.

And the fact that my pieces mostly appeared in the news pages (rather than the opinion pages) made me even more hesitant to use my personal voice in the writing – if it’s going to be news, I need to be as impersonal as possible, I thought. And so I wrote. The editors seemed to like it, since they kept me for six and a half years. Social media feedback tells me that at least occasionally the readers liked it. My wife never liked it.

Nowadays, from time to time, I find myself getting into “the Mint frame of mind” when I’m writing something. This happens when I need to get something out by a deadline, and I try to become too careful about what I’m stating and not bring in a personal opinion. So I try to find links to support every piece of information I put in. I try to be careful to not appear taking political sides. In other words, I get into “Mint mode”. And when I write in Mint mode, I end up writing stuff that, in hindsight, is not very interesting to read.

I guess my blog gives me the freedom that when I’m not writing well, I simply abandon the post. In that sense, the quality of my writing that you see has some selection bias – if I’m not happy with how something is going, I simply abandon it. However, my writing elsewhere doesn’t have that luxury, and so I sometimes end up “delivering shit”.

I really don’t know what I can do to prevent this from happening on a consistent basis. Maybe I should just blog more. And try and be myself when I write elsewhere as well. Maybe I should just write like I write a blog and then edit it to take out any personal touches, rather than trying to write impersonally in the first place.

OK I know i’ve rambled here 😛

Writing and monetisation

I started writing this blog, or its predecessor, in 2004. For nine years I made zero money off it. In fact, in 2008, after I moved to this website, I started paying money to run this blog, in terms of an annual domain name and hosting fee.

And then in 2013, I became part of a “big bundle”, as Mint offered me the opportunity to write for them. I had a contract to write at least three pieces a month around a particular topic, in return for which I would be paid a reasonable sum of money.

That sum of money was “reasonable” enough that it sort of provided me “tenure” until 2017, when I moved to London (I continued to write for Mint, and get paid, but the money wasn’t enough for “tenure” in London where expenses were higher). The Mint editor changed in early 2018, and the tenure ended in late 2018. I briefly got another tenure with the same editor at his new digs in 2019, but I decided to end that after a few months.

(By tenure, I mean steady stable income out of work that doesn’t take too much of my time. So I never had to struggle for basic expenses and every business deal was a bonus. Wonderful times)

In other words, I built my reputation as a writer by myself, writing this blog (and its predecessor), and then monetised it by joining a large bundle.

Recent trends in the media seem to be reversing the process. Recently, for example, Andrew Sullivan, a journalist with the New York magazine, quit his job and started his own newsletter. And this seems to be the part of a larger trend.

Columnist Matt Taibbi left Rolling Stone in April to write on Substack full time. Andrew Sullivan did the same last week, leaving New York Magazine to resurrect his blog the Dish. Joan Niesen, a Sports Illustrated staff writer who was laid off in October, shortly after the magazine’s sale, started a free Substack newsletter last week.

Essentially, journalists who made their names as being part of big bundles, are leaving these bundles and instead trying to monetise on their own platforms. This is exactly the opposite of the route that I, and many other bloggers of the 2000s wave, took – build reputation independently and monetise as part of a bundle.

At the outset, I’m sceptical about lifelong bundlers leaving their bundles. Essentially, once you’ve gotten used to working as part of a large professional setup, you would have started taking a large number of things for granted, and replicating those things are not going to be easy once you go indie.

As a writer, for example, who will edit your copy? Who will make and design your graphics? Who will write your headlines? Who will tell you lots to write about? While you might have experience as a journalist and be very good at your core job, being part of an institution means that you will find it very difficult to do everything by yourself (I THINK I’ve written something on these lines for non-journalism jobs as well, but I can’t be bothered to find that post now. I actually searched for it and found that like an idiot I’d written it on LinkedIn, and now I can‘t find it).

Moreover, people will face “subscription fatigue”, and won’t want to subscribe to too many individual writers. A case can be made for bundling again (get a bunch of writers who write about sort of complementary stuff, and then bundle their newsletters for an integrated subscription). (after all, all disruption is about either bundling or unbundling).

Breaking up sentences in the absence of punctuation

From time to time, a joke goes around that makes the value of punctuation clear. Check out this picture, for example.

Recently, I saw this on my twitter timeline (though here it’s an issue of spacing apart from punctuation).

Though Punctuations are Important….But Space is much more IMPORTANT..???? pic.twitter.com/Exwf4EFPeP

— Anjali S Vishnoi ™ ®?? (@anansavishnoi) July 22, 2020

Someone actually wrote an entire book about the value of punctuation.

In any case, I have a pretty bad track record in terms of reading sentences that don’t have punctuation. I can think of two examples right away.

Firstly, my school diary was filled with quotes from Sri Aurobindo and “The Mother“. By turns, we would have to say “thought for the day” in the school assembly. And we had a reliable way of finding such thoughts – just look in the diary and spout out the nuggets.

One of those went:

Always do what you know to be the best even if it the most difficult thing to do.

Yes, I remember that. Because I had spouted this not once but twice. Now, this is a long sentence without any punctuation. How would you read it?

For the longest time I read it like this.

Always do what you know, to be the best, even if it is the most difficult thing to do.

So if there are many things that you can do, and you know one of the things, you do that thing even if it is harder than everything else that you might do (but don’t know how to do).

Clearly that doesn’t make that much sense. It was only when I was about to graduate that I figured that it was actually:

Always do what you know to be the best even if it is the most difficult thing to do.

So there are many things you can do. You know one of them is the “best thing to do”. So even if it is the most difficult thing to do, you do it because you know it is “the best” (not because you “know it the best”).

Another example is from this store near my house. I don’t think the store existed for too long, but it had an interesting and quirky name (in Kannada).

“ellAdEvarakrupe stores“, it said. For the longest time I read it as “ellAdEvara krupe”, or “grace of all gods”. And I thought it was a fascinating name in terms of recognising all religions. And I’ve quoted it many times.

When I was quoting this on Twitter earlier today, I realised that I had got the name of the store all wrong. It’s “ellA dEvarakrupe”, meaning “everything is god’s grace”. It says nothing about which gods are included or excluded, or how many gods there are.

What are your favourite examples of sentences that you’ve misread thanks to the lack of punctuation or other visible sentence markers?

Conductors and CAPM

Recently I watched this video that YouTube recommended to me about why orchestras have conductors.

The basic idea is that an orchestra needs a whole lot of coordination, in terms of when to begin and end, when to slow down or speed up, when to move to the next line and so on. And in case there is no conductor, the members of the orchestra need to coordinate among themselves.

This is easy enough when there is a small number of members, so you don’t see bands (for example) needing conductors. However, notice that if the orchestra has to coordinate among themselves, coordination is an $O(n^2)$ problem. By appointing an external conductor whose only job is to conduct and not play, this $O(n^2)$ problem is reduced to an $O(n)$ problem.

When I saw this, this took me back to my Investments course in IIMB, where the professor one day introduced what he called the “Sharpe single index model“, which is sort of similar to the CAPM.

Just before learning the Sharpe Single Index Model, we had been learning about Markowitz’s portfolio theory. And then, as he introduced the Sharpe Single Index Model, Vaidya said something to the effect that “instead of knowing so many correlation terms” (which is an $O(n^2)$ problem), “we only need to know the correlation of each stock to the market index” (makes it an $O(n)$ problem).

As someone who has studied computer science formally, converting $O(n^2)$ problems to $O(n)$ problems is a massive fascination. It is interesting how I connected two such reductions from completely different fields.

In other words, conductors are the “market of the orchestra”.

Too cheap to cancel

One of the great philosophical battles of our times is the “cancel culture“. This culture dictates that if you have ever done something reprehensible in the past (I’m sure in my case you can find lots of incriminating blogposts and tweets), then you deserve to be “cancelled”.

This is how it works, according to Vox:

A celebrity or other public figure does or says something offensive. A public backlash, often fueled by politically progressive social media, ensues. Then come the calls to cancel the person — that is, to effectively end their career or revoke their cultural cachet, whether through boycotts of their work or disciplinary action from an employer.

In 2019 alone, the list of people who’ve faced being canceled included alleged sexual predators like R. Kelly; entertainers like Kanye West, Scarlett Johansson, and Gina Rodriguez, who all had offensive foot-in-mouth moments; and comedians like Kevin Hartand Shane Gillis, who each faced public backlash after social media users unearthed homophobic and racist jokes they’d made in the past.

In any case, recently the New York Post found that some ancestors of their great city rivals New York Times were slaveholders and supported the confederacy.

While it is pretty certain that any white American who had an ancestor who lived in the US in the early 1800s is likely to be the descendent of slaveholders, maybe this is good reason enough to “cancel the New York Times”?

Cancel the New York Times then? https://t.co/zQynap6aXE

— Karthik (@karthiks) July 20, 2020

Anyway the point of this post can be seen in the replies to the tweet, and you can think of the sole purpose of this post being to save that idea for posterity.

Essentially, New York Times is a subscription-based newspaper, and a more conventional meaning of “cancel” applies to it – you can simply cancel your subscription. I’m a subscriber, having taken advantage of a ?25 per week offer they ran a few months back (this is less than half of what I pay for a print edition of the Times of India; that is how zero marginal cost products work).

Now, through my twitter timeline I’ve been seeing several people make a case that the NYT is not what it used to be, and that it is a partisan rag now, and that it is not worth subscribing to, and hence deserves to be (in the conventional sense) cancelled.

Every time I come across such an argument I briefly consider cancelling my NYT subscription and then I think “what the hell, it’s just ?25 per week. The option value of a few good articles here and there is worth more than that”, and I move on.

I have mentally set myself to cancel my subscription in March next year, when my cheap offer ends, but until then, as far as I’m concerned the “NYT is too cheap to be cancelled”.

So that led to this thought – you can only be cancelled if you are not “cheap”. As long as you are cheap enough, people will see no benefit in cancelling you.

Now I’m reminded of the time when at the New Year’s Eve celebrations, I got the “cheap guy of the year” award for 2004 at IIM Bangalore. I suppose that’s insurance enough against getting cancelled?

WhatsApp Export Chat

There was a tiny controversy on one WhatsApp group I’m part of. This is a “sparse” WhatsApp group, which means there aren’t too many messages sent. Only around 1000 in nearly 5 years (you’ll soon know how I got that number).

And this morning I wake up to find 42 messages (many members of the group are in the US). Some of them I understood and some I didn’t. So the gossip-monger I am (hey, remember that Yuval Noah Harari thinks gossip is the basis of human civilisation?), I opened up half a dozen backchannel chats.

Like the six blind men of Indostan, these chats helped me construct a picture of what had happened. My domain knowledge had gotten enhanced. However, there was one message that had made a deep impression on me – that claimed that some people were monopolising whatever little conversation there was on that group.

I HAD to test that hypothesis.

The jobless guy that I am, I figured out how to export a chat from WhatsApp. With iOS, it’s rather easy. Go to the info page of a chat or a group, and near “delete chat/group”, you see “export chat/group”. If you say you don’t want media (like I did), you get a text file (I airdropped mine immediately into my Mac).

The formatting of the WhatsApp export file is rather clean, making it easy to parse. The date is in square brackets. The sender’s name (or number, if they’re not in your contact list) is before a colon after the square brackets. A couple of “separate” functions later you are good to go (there are a couple of other nuances. If you can read R code, check mine here).

chat <- read_lines('~/Downloads/_chat.txt')
tibble(txt=chat) %>% 
separate(txt, c("Date", "Content"), '\\] ') %>%
separate(Content, c("Sender", "Content"), ': ') %>%
mutate(
Content=coalesce(Content, Date),
Date=str_trim(str_replace_all(Date, '\\[', '')),
Date2=as.POSIXct(Date, format='%d/%m/%y, %H:%M:%S %p')
) %>%
fill(Date2, .direction = 'updown') %>%
fill(Sender, .direction = 'downup') %>%
filter(!str_detect(Sender, "changed their phone number to a new number") ) %>%
filter(!str_detect(Sender, ' added ') & !str_detect(Sender, ' left')) %>%
filter(!str_detect(Sender, " joined using this group's invite link"))->
mychat

That’s it. You are good to go. You have a nice data frame with sender’s name, message content and date/time of sending. And as one of the teachers at my JEE coaching factory used to say, you can now do “gymnastics”.

And so for the last hour or so I’ve been wasting my time doing such gymnastics. Number of posts sent on each day. Testing the hypothesis that some people talk a lot on the group (I turned out to be far more prolific than I’d imagined). People who start conversations. Whether there are any long bilateral conversations on the group. And so on and so forth (this is how I know there are ~1000 messages on this group).

Now I want to subject all my conversations to such analysis. For bilaterals it won’t be that much fun – but in case there is some romantic or business interest involved you might find it useful to know who initiates more and who closes more conversations.

You can subject the conversations to natural language processing (with what objective, I don’t know). The possibilities are endless.

And the time wastage can be endless as well. So I’ll stop here.

Unbundling Higher Education

In July 2004, I went to Madras, wore fancy clothes and collected a laminated piece of paper. The piece of paper, formally called “Bachelor of Technology”, certified three things.

First, it said that I had (very likely) got a very good rank in IIT JEE, which enabled me to enrol in the Computer Science B.Tech. program at IIT Madras. Then, it certified that I had attended a certain number of lectures and laboratories (equivalent to “180 credits”) at IIT Madras. Finally, it certified that I had completed assignments and passed tests administered by IIT Madras to a sufficient degree that qualified me to get the piece of paper.

Note that all these three were necessary conditions to my getting my degree from IIT Madras. Not passing IIT JEE with a fancy enough rank would have precluded me from the other two steps in the first place. Either not attending lectures and labs, or not doing the assignments and exams, would have meant that my “coursework would be incomplete”, leaving me ineligible to get the degree.

In other words, my higher education was bundled. There is no reason that should be so.

There is no reason that a single entity should have been responsible for entry testing (which is what IIT-JEE essentially is), teaching and exit testing. Each of these three could have been done by an independent entity.

For example, you could have “credentialing entities” or “entry testing entities”, whose job is to test you on things that admissions departments of colleges might test you on. This could include subject tests such as IIT-JEE, or aptitude tests such as GRE, or even evaluations of extra-curricular activities, recommendation letters and essays as practiced in American universities.

Then, you could have “teaching entities”. This is like the MOOCs we already have. The job of these teaching entities is to teach a subject or set of subjects, and make sure you understood your stuff. Testing whether you had learnt the stuff, however, is not the job of the teaching entities. It is also likely that unless there are superstar teachers, the value of these teaching entities comes down, on account of marginal cost pricing, close to zero.

To test whether you learnt your stuff, you have the testing entities. Their job is to test whether your level of knowledge is sufficient to certify that you have learnt a particular subject. It is well possible that some testing entities might demand that you cleared a particular cutoff on entry tests before you are eligible to get their exit test certificates, but many others may not.

The only downside of this unbundling is that independent evaluation becomes difficult. What do you make of a person who has cleared entry tests mandated by a certain set of institutions, and exit tests traditionally associated with a completely different set of institutions? Is the entry test certificate (and associated rank or percentile) enough to give you a particular credential or should it be associated with an exit test as well?

These complications are possibly why higher education hasn’t experimented with any such unbundling so far (though MOOCs have taken the teaching bit outside the traditional classroom).

However, there is an opportunity now. Covid-19 means that several universities have decided to have online-only classes in 2019-20. Without the peer learning aspect, people are wondering if it is worth paying the traditional amount for these schools. People are also calling for top universities to expand their programs since the marginal cost is slipping further, with the backlash being that this will “dilute” the degrees.

This is where unbundling comes into play. Essentially anyone should be able to attend the Harvard lectures, and maybe even do the Harvard exams (if this can be done with a sufficiently low marginal cost). However, you get a Harvard degree if and only if you have cleared the traditional Harvard admission criteria (maybe the rest get a diploma or something?).

Some other people might decide upon clearing the traditional Harvard admission criteria that this credential itself is sufficient for them and not bother getting the full degree. The possibilities are endless.

Old-time readers of this blog might remember that I had almost experimented with something like this. Highly disillusioned during my first year at IIT, I had considered dropping out, reasoning that my JEE rank was credential enough. Finally, I turned out to be too much of a chicken to do that.