SK – Page 54 – Pertinent Observations

TV Punditry

Those of you who might be following me on social media (Twitter/Facebook/LinkedIN) might know that I’ve started a career in TV Punditry over the last week. Well, it’s not that much of a career – I still need to figure out how to get paid for it.

Anyway, so I was on News9 once on Saturday (analysing exit polls) and again on Tuesday (analysing the election results). It happened pretty much at random, from a random twitter conversation:

yeah there till 28th. fix one election TV appearance for me 😛

— Karthik Shashidhar (@karthiks) May 1, 2018

And so Mathang (who I’d first met in 2004 when he had interviewed me for Education Times) set me up with Anil Kumar from News9, who presently asked me for my number. A couple of twitter DMs, a couple of emails and a couple of phone calls later, I had been asked to come to the News9 studio at 5pm on Saturday.

Saturday’s session was really enjoyable, and I spoke a fair bit on the process of conducting an exit poll, the importance of sample sizes and representative samples, the process of converting votes to seats, etc. A 5 minute monologue on sampling process got the anchors interested in me, and they kept coming back to me. As is my wont, I summarised the import of my arguments for Mint.

And so I got invited again for Tuesday’s post-counting session, and I’m not sure I enjoyed it that much. As the elections threw up a hung assembly, the politicians on the panel spent their time shouting at each other. I was seated in an inappropriate place – right between a loud JDS spokesperson and a loud BJP spokesperson. I recused myself from much of the discussion and was only brought in because the anchors probably thought I should be “given some lines” – an opportunity I used to comment on the parties’ election strategies.

So two TV appearances later, I must say I quite like the format – it’s good footage (literally) if not anything else, but it can be a bit painful. Writing is easy in the sense that you just collect your thoughts and deliver them at a time.

Video means that you are virtually participating in a group discussion, and need to butt in to make your point. You might have something insightful to say, but need to wait for an opportune time to interject. You might be in the middle of a long point but get interrupted by another panellist. You might wait for ages to say something but the opportunity never comes. At other times, you might get a question that you’re not prepared for.

The worst thing as an analytical guy on TV is that you need to keep referring to your data, and your analysis. So there was one occasion on each session when the anchors asked me a question to answer which I’d to write some code to answer. So each time I mumbled something and bent down to my laptop, and got bailed out by the anchor who got someone else’s view in the time I took to get the requisite data.

In any case, I want to do more of this. I also hope that like with my writing, I can some day hope to get paid for TV appearances – this is a hard job since panellists representing political parties don’t charge anything – it’s in their parties’ interests to be represented on the show.

But, some day..

A journey back to civilisation

Earlier this evening, I was at a coffee shop in Whitefield with a friend when it started raining cats and dogs. I got a message from a wife stating that it was raining insanely in her part of town, and that I should be careful while coming back. I promised her that I would wait it out before returning, and returned to my conversation.

I made my first attempt at booking a cab at 1845, by which time the rain had stopped. Uber showed that the nearest cab was 8 minutes away, except that when I tried to book it it failed to find me a ride. Ola was no better – except that it showed that the nearest cab was 20 minutes away when I opened the app.

I continued waiting, and continued checking on both platforms. No cabs materialised. And after some 45 minutes of waiting thus, I decided to get out and find a bus. My friend was surprised that I was willing to change buses to get home. “I would never do that”, he declared, adding that it would be easier for me to move back to India.

I walked up Varthur main road looking for a bus stop. It had stopped raining but there were huge puddles on the roadside, and mosquitoes buzzed all around. There was a huge crowd at the bus stop. The first two buses came at an interval of five minutes each. Both were jam packed.

It was clear that Varthur main road wasn’t a great place to be, since the bus frequency there was low – most buses would be coming from the other side of Whitefield, so it was clear that I should get to Kundalahalli gate.

Presently an “illegal bus” (an office bus picking up passengers for some extra income for the driver) materialised, and it was a good opportunity to get to Kundalahalli gate. The bus sped there, and charged 10 bucks.

As expected, there were plenty of buses, including Volvos, at Kundalahalli Gate, except that there was no room to get into any of them. Once again, there was no luck to be had on the Uber or Ola front. I even tried UberPool and Ola Share (stuff I normally never use), but nothing materialised. The only result of all that was that my phone battery drained like crazy. And it started raining as well – I was happy I had behaved like a rich man this morning and bought a new umbrella when I realised I’d forgotten mine at home.

An airport bus appeared as a sort of a saviour. It was empty, and the conductor said passengers not headed to the airport weren’t allowed on it. I offered to buy a ticket till the airport, and was allowed on. The conductor said I best get off at the next stop (Marathahalli bridge) given where I was headed. He charged me Rs. 16.

So at every step I got closer and closer to civilisation. Kundalahalli Gate was civilisation compared to Varthur Main Road. Marathahalli was civilisation compared to Kundalahalli Gate. Another illegal bus there dropped me to Domlur (Rs. 20), and under normal circumstances that should count as “proper civilisation”. Except that the design of the Domlur flyover means that it’s rather desolate and dark and unwalkable under it. So I needed to reach the next stage of civilisation, which I did when yet another illegal bus dropped me to Richmond Circle (the driver demanded Rs. 15, but I gave only Rs. 10 since I didn’t have change).

At all stages, I continuously tried to get cabs and autos, but perhaps due to tomorrow’s state elections, none materialised. Most of the time I was on one road (Old Airport Road), and most sections of it are rather badly lit and seem unsafe and “rural”. This was a journey I would have never done if I had been with family.

And the mode of transport was bimodal – three of the five buses I took to reach home were “illegal”. Two others were the most expensive Volvos. The last leg of the journey was completed on yet another airport Volvo, where the conductor made no fuss of letting people in, and not only gave me change for Rs. 100 (ticket cost Rs. 37), but also gave me 5 100 rupee notes for a 500 rupee note I handed him.

The entire journey, from the time I started hailing the cab to when I opened my door, took exactly three hours. A cab would have cost me upwards of Rs. 500, but my bimodal transport cost me Rs. 105. Frankly I would’ve been more than happy to spend the former amount for the pleasure of getting home an hour and half earlier, and being able to do something productive on the ride home.

But then it’s not often that an NRI has an adventure such as this!

NRI Diaries: Volume 2, Number 2

I’m writing this while chomping on a bar of Amul Colombia Classique Black Single Origin Dark Chocolate. You read that right. Amul now produces single origin dark chocolate (55%) using chocolate from Colombia, Ecuador and Venezuela (!!!).

It was only six months back that I first came across the concept of single origin dark chocolate, when a (then) colleague in London offered me some Haitian dark chocolate he had bought at Waitrose. I had been bloody impressed, and fallen in love with this brand of chocolate then, but now I’m reminded of what my father used to say – that there’s nothing that is not available in Bangalore “nowadays” (to put that “nowadays” in context, he died in 2007).

The said bar of chocolate was purchased at Namdhari Fresh, which I visited with the express purpose of checking out what exotic foods are available in Bangalore nowadays (and Namdhari is hardly the place where you get the most exotic stuff in Bangalore nowadays – it just happens to be within walking distance of my house).

And now I see that several kinds of cheeses are available here – apart from the “usual” Cheddar and Mozzarella, you get Feta and Halloumi as well. Well, Feta we used to get back in 2016 as well, though not always (which prompted an entire chapter in my book), but this is the first time I’m finding Halloumi in Bangalore. It’s expensive, though – INR 640 for a 150 gram block (if I remember correctly). To compare, a 250 gram block of Halloumi costs GBP 2.25 (~INR 200) in London.

On my long walk to “flowth block” today, I felt like an NRI. I felt like someone who’s visiting India from abroad, and who is very impressed with the “energy” and growth. A couple of my old favourite restaurants had shut down, I saw (La Casa and Gramina Thindi, for those who want to know), but there were also loads (and loads) of small cutesy places that had opened up. There is plenty of construction activity, and plenty of investment in the city. Some of the investment will surely go under, but a lot of it will produce outsized returns. In a way, compared to when I left last year, based on very tiny anec-data, it seems like risk-taking ability of people in Jayanagar has gone up.

Another piece of evidence of the vibrancy of India is in the mobile internet space – I was telling a friend I met today that before I moved last year, I had a package that provided me with 3GB of data per month. The pay as you go plan that I use now offers me 2GB of data PER DAY! And I pay a fifth of what I used to pay in 2016.

And this has set off another wave of entrepreneurship, since this has resulted in a massive spike in the amount of video consumption (the friend I was talking to today quoted some impressive numbers in terms of this growth).

Earlier in the day, I took my wife’s old scooter to move all round town for a series of meetings. Based on anec-data (once again), traffic actually seems to have better, at least in parts of old Bangalore that are now served by the metro.

I might have a different opinion of Bangalore’s traffic tomorrow, though, since I’m headed to Whitefield (not taking the scooter there, though).

The Government Should Regulate Cooks

I wrote this for Pragati Express. Reposting it here since my general readers might find it interesting as well. This follows from my old blog post that bathrooms should be banned.

Yet another wedding, yet another truckload of wasted food. If, in reality TV show style, we were to try to identify the “root cause” in this instance, it was the cook (or the team of cooks, rather). Each of the seven respondents this correspondent surveyed expressed their displeasure at the quality of the food. One even called it her “worst ever Indian wedding dinner”.

This wedding was only one isolated instance – it is all too common an occurrence in these parts for copious amounts of food to be wasted all because of a cook who ended up cooking badly. And it is all the fault of the cooks, most of whom have never gone to culinary school (we don’t have too many of those in India), and many of whom haven’t gone to school either.

When thousands of people in India die of hunger everyday, and farmers continue to kill themselves in Vidarbha(and elsewhere), this wastage of food is indeed criminal. It comes at a high human cost. And that it comes out of sheer incompetence of unregulated cooks makes it indeed tragic.

There is only one solution to this – the government should regulate cooks. Not just wedding cooks – since wastage of food at weddings and other parties are only part of the problem – the government should regulate anyone who wants to cook. The other day my daughter refused to eat an idli. We decided to salvage our karma by feeding it (the idli) to the neighbourhood street dog, who took one bite and promptly ran away.

Whether you want to make yourself a 2-minute Maggi, or Shantavva in Santemarahalli wants to make a ragi ball, or chef Madhu Menon (hope he doesn’t edit this bit out) wants to make bloggers’ b***, you should need a licence from the government, which certifies that you are a cook of a high enough quality that what you cook will not go waste.

That’s the only way we can save millions of our population from hunger. There is already enough wastage of food because farmers cannot coordinate on what to grow, and because of inefficiencies in the food supply chain, and because of the way agricultural markets are regulated. We don’t want badly cooked food to add to the wastage. And the only way to ensure that is by having the government regulate cooks.

PS: As Ravikiran Rao, a former editor of the former avatar of this publication, likes to put it, “#thatzwhy we need strong regulation”

PS2: Some readers might be advised to consume irony supplements along with this article

NRI Diaries: Volume 2, Number 1

We were welcomed with the mildly warm mildly humid air of Bangalore as we walked out of the airport early this morning. It was that nice kind of humidity, that makes you feel good without breaking a sweat.

The exit from the airport had been rather smooth, except for a bit of a wait at the baggage carousel. Passport control, which had taken 30 mins during our December visit, took 3. No questions asked anywhere.

The airport taxis seemed rather disorganised, though, with random (non-licensed) taxis standing in the same rank as the licenced taxis, and drivers shouting to attract customers. This is a departure from the usual practice at Bangalore airport where the taxi rank is rather well organised!

First order of business after landing at the in-laws’ house in Rajajinagar was to go for breakfast, which was at CTR. The driver of the first auto rickshaw we sat in insisted on filling up gas before he dropped us. So we jumped out and let him have his gas instead.

The vaDe at CTR was average. The masaldose was good but not great. Based on recent samples, I’m once again in the Vidyarthi Bhavan camp in the great north-south dose battle. The dose there during my last visit in December had been brilliant. I’m going there again next week.

I had a work meeting in the afternoon. Over the last year or so in London I’ve consistently eschewed the big chains in favour of the numerous independent hipster cafes the city is littered with. Suddenly, here I was at a loss in terms of meeting venues – not finding anything beyond the chains in the area of town I’d to go to. I finally picked Coffee Day Square at the Tiffany’s junction, and I wasn’t disappointed.

Then, for the first time this season, I’m getting to watch the IPL live. Living in the UK, where cable TV subscriptions are not a thing, I’ve hardly watched live sport. I occasionally buy day passes on Sky TV to watch football, but taking a specific subscription just to watch IPL doesn’t seem worth it.

And as it happens, my in-laws have only once channel that shows live IPL – Star Suvarna Plus, with the channel’s specialty being Kannada commentary! While it’s pleasant to hear fresh voices describe the cricket, somehow it feels weird listening to Vijay Bharadwaj and B Akhil talk about the game. And I didn’t feel this way when listening to Kannada commentary of Ranji trophy matches on radio in the early 1990s. As I write this, I’m watching on mute.

Oh, and I’m still yet to adjust to the time zone change. At 3:30 this afternoon I was wondering why the IPL game hadn’t yet begun, and while writing this at 8:30 I’m wondering why the Guardian hasn’t started its MBM on the Atletico-Arsenal game yet!

Finally maybe it’s because it was a shorter gap between India visits this time (4 months), but I feel less like an NRI. I’m not too fazed by the heat, and most of the day has felt “rather normal”. Maybe I’m getting used to being an NRI!

Statistics and machine learning

So a group of statisticians (from Cyprus and Greece) have written an easy-to-read paper comparing statistical and machine learning methods in time series forecasting, and found that statistical methods do better, both in terms of accuracy and computational complexity.

To me, there’s no surprise in the conclusion, since in the statistical methods, there is some human intelligence involved, in terms of removing seasonality, making the time series stationary and then using statistical methods that have been built specifically for time series forecasting (including some incredibly simple stuff like exponential smoothing).

Machine learning methods, on the other hand, are more general purpose – the same neural networks used for forecasting these time series, with changed parameters, can be used for predicting something else.

In a way, using machine learning for time series forecasting is like using that little screwdriver from a Swiss army knife, rather than a proper screwdriver. Yes, it might do the job, but it’s in general inefficient and not an effective use of resources.

Yet, it is important that this paper has been written since the trend in industry nowadays has been that given cheap computing power, machine learning be used for pretty much any problem, irrespective of whether it is the most appropriate method for doing so. You also see the rise of “machine learning purists” who insist that no human intelligence should “contaminate” these models, and machines should do everything.

By pointing out that statistical techniques are superior at time series forecasting compared to general machine learning techniques, the authors bring to attention that using purpose-built techniques can actually do much better, and that we can build better systems by using a combination of human and machine intelligence.

They also helpfully include this nice picture that summarises what machine learning is good for, and I wholeheartedly agree:

The paper also has some other gems. A few samples here:

Knowing that a certain sophisticated method is not as accurate as a much simpler one is upsetting from a scientific point of view as the former requires a great deal of academic expertise and ample computer time to be applied.

[…] the post-sample predictions of simple statistical methods were found to be at least as accurate as the sophisticated ones. This finding was furiously objected to by theoretical statisticians [76], who claimed that a simple method being a special case of e.g. ARIMA models, could not be more accurate than the ARIMA one, refusing to accept the empirical evidence proving the opposite.

A problem with the academic ML forecasting literature is that the majority of published studies provide forecasts and claim satisfactory accuracies without comparing them with simple statistical methods or even naive benchmarks. Doing so raises expectations that ML methods provide accurate predictions, but without any empirical proof that this is the case.

At present, the issue of uncertainty has not been included in the research agenda of the ML field, leaving a huge vacuum that must be filled as estimating the uncertainty in future predictions is as important as the forecasts themselves.

Chasing Dhoni

Former India captain Mahendra Singh Dhoni has a mixed record when it comes to chasing in limited overs games (ODIs and T20s). He initially built up his reputation as an expert chaser, who knew exactly how to pace an innings and accelerate at the right moment to deliver victory.

Of late, though, his chasing has been going wrong, the latest example being Chennai Super Kings’ loss at Kings XI Punjab over the weekend. Dhoni no doubt played excellently – 79 off 44 is a brilliant innings in most contexts. Where he possibly fell short was in the way he paced the innings.

And the algorithm I’ve built to represent (and potentially evaluate) a cricket match seems to have done a remarkable job in identifying this problem in the KXIP-CSK game. Now, apart from displaying how the game “flowed” from start to finish, the algorithm is also designed to pick out key moments or periods in the game.

One kind of “key period” that the algorithm tries to pick is a batsman’s innings – periods of play where a batsman made a significant contribution (either positive or negative) to his team’s chances of winning. And notice how nicely it has identified two distinct periods in Dhoni’s batting:

The first period is one where Dhoni settled down, and batted rather slowly – he hit only 21 runs in 22 balls in that period, which is incredibly slow for a 10 runs per over game. Notice how this period of Dhoni’s batting coincides with a period when the game decisively swung KXIP’s way.

And then Dhoni went for it, hitting 36 runs in 11 balls (which is great going even for a 10-runs-per-over game), including 19 off the penultimate over bowled by Andrew Tye. While this brought CSK back into the game (to right where the game stood prior to Dhoni’s slow period of batting), it was a little too late as KXIP managed to hold on.

Now I understand I’m making an argument using one data point here, but this problem with Dhoni, where he first slows down and then goes for it with only a few overs to go, has been discussed widely. What’s interesting is how neatly my algorithm has picked out these periods!

Beer and diapers: Netflix edition

When we started using Netflix last May, we created three personas for the three of us in the family – “Karthik”, “Priyanka” and “Berry”. At that time we didn’t realise that there was already a pre-created “kids” (subsequently renamed “children” – don’t know why that happened) persona there.

So while Priyanka and I mostly use our respective personas to consume Netflix (our interests in terms of video content hardly intersect), Berry uses both her profile and the kids profile for her stuff (of course, she’s too young to put it on herself. We do it for her). So over the year, the “Berry” profile has been mostly used to play Peppa Pig, and the occasional wildlife documentary.

Which is why we were shocked the other day to find that “Real life wife swap” had been recommended on her account. Yes, you read that right. We muttered a word of abuse about Netflix’s machine learning algorithms and since then have only used the “kids” profile to play Berry’s stuff.

Since then I’ve been wondering what made Netflix recommend “real life wife swap” to Berry. Surely, it would have been clear to Netflix that while it wasn’t officially classified as one, the Berry persona was a kid’s account? And even if it didn’t, didn’t the fact that the account was used for watching kids’ stuff lead the collaborative filtering algorithms at Netflix to recommend more kids’ stuff? I’ve come up with various hypotheses.

Since I’m not Netflix, and I don’t have their data, I can’t test it, but my favourite hypothesis so far involves what is possibly the most commonly cited example in retail analytics – “beer and diapers“. In this most-likely-apocryphal story, a supermarket chain discovered that beer and diapers were highly likely to appear together in shopping baskets. Correlation led to causation and a hypothesis was made that this was the result of tired fathers buying beer on their diaper shopping trips.

So the Netflix version of beer-and-diapers, which is my hypothesis, goes like this. Harrowed parents are pestered by their kids to play Peppa Pig and other kiddie stuff. The parents are so stressed that they don’t switch to the kid’s persona, and instead play Peppa Pig or whatever from their own accounts. The kid is happy and soon goes to bed. And then the parent decides to unwind by watching some raunchy stuff like “real life wife swap”.

Repeat this story in enough families, and you have a strong enough pattern that accounts not explicitly classified as “kids/children” have strong activity of both kiddie stuff and adult content. And when you use an account not explicitly mentioned as “kids” to watch kiddie stuff, it gets matched to these accounts that have created the pattern – Netflix effectively assumes that watching kid stuff on an adult account indicates that the same account is used to watch adult content as well. And so serves it to Berry!

Machine learning algorithms basically work on identifying patterns in data, and then fitting these patterns on hitherto unseen data. Sometimes the patterns make sense – like Google Photos identifying you even in your kiddie pics. Other times, the patterns are offensive – like the time Google Photos classified a black woman as a “gorilla“.

Thus what is necessary is some level of human oversight, to make sure that the patterns the machine has identified makes some sort of sense (machine learning purists say this is against the spirit of machine learning, since one of the purposes of machine learning is to discover patterns not perceptible to humans).

That kind of oversight at Netflix would have suggested that you can’t tag a profile to a “kiddie content AND adult content” category if the profile has been used to watch ONLY kiddie content (or ONLY adult content). And that kind of oversight would have also led Netflix to investigate issues of users using “general” account for their kids, and coming up with an algorithm to classify such accounts as kids’ accounts, and serve only kids’ content there.

It seems, though, that algorithms run supreme at Netflix, and so my baby daughter gets served “real life wife swap”. Again, this is all a hypothesis (real life wife swap being recommended is a fact, of course)!

More on interactive graphics

So for a while now I’ve been building this cricket visualisation thingy. Basically it’s what I think is a pseudo-innovative way of describing a cricket match, by showing how the game ebbs and flows, and marking off the key events.

Here’s a sample, from the ongoing game between Chennai Super Kings and Kolkata Knight Riders.

As you might appreciate, this is a bit cluttered. One “brilliant” idea I had to declutter this was to create an interactive version, using Plotly and D3.js. It’s the same graphic, but instead of all those annotations appearing, they’ll appear when you hover on those boxes (the boxes are still there). Also, when you hover over the line you can see the score and what happened on that ball.

When I came up with this version two weeks back, I sent it to a few friends. Nobody responded. I checked back with them a few days later. Nobody had seen it. They’d all opened it on their mobile devices, and interactive graphics are ill-defined for mobile!

Because on mobile there’s no concept of “hover”. Even “click” is badly defined because fingers are much fatter than mouse pointers.

And nowadays everyone uses mobile – even in corporate settings. People who spend most time in meetings only have access to their phones while in there, and consume all their information through that.

Yet, you have visualisation “experts” who insist on the joys of tools such as Tableau, or other things that produce nice-looking interactive graphics. People go ga-ga over motion charts (they’re slightly better in that they can communicate more without input from the user).

In my opinion, the lack of use on mobile is the last nail in the coffin of interactive graphics. It is not like they didn’t have their problems already – the biggest problem for me is that it takes too much effort on the part of the user to understand the message that is being sent out. Interactive graphics are also harder to do well, since the users might use them in ways not intended – hovering and clicking on the “wrong” places, making it harder to communicate the message you want to communicate.

As a visualiser, one thing I’m particular about is being in control of the message. As a rule, a good visualisation contains one overarching message, and a good visualisation is one in which the user gets the message as soon as she sees the chart. And in an interactive chart which the user has to control, there is no way for the designer to control the message!

Hopefully this difficulty with seeing interactive charts on mobile will mean that my clients will start demanding them less (at least that’s the direction in which I’ve been educating them all along!). “Controlling the narrative” and “too much work for consumer” might seem like esoteric problems with something, but “can’t be consumed on mobile” is surely a winning argument!