Author: SK

Human, Animal and Machine Intelligence

Earlier this week I started watching this series on Netflix called “Terrorism Close Calls“. Each episode is about an instance of attempted terrorism that has been foiled in the last 2 decades. For example, there is one example of the plot to bomb a set of transatlantic flights from London to North America in 2006 (a consequence of which is that liquids still aren’t allowed on board flights).

So the first episode of the series involves this Afghani guy who drives all the way from Colorado to New York to place a series of bombs in the latter’s subways (metro train system). He is under surveillance through the length of his journey, and just as he is about to enter New York, he is stopped for what seems like a “routine drugs test”.

As the episode explains, “a set of dogs went around his car sniffing”, but “rather than being trained to sniff drugs” (as is routine in such a stop), “these dogs had been trained to sniff explosives”.

This little snippet got me thinking about how machines are “trained” to “learn”. At the most basic level, machine learning involves showing a large number of “positive cases” and “negative cases” based on which the program “learns” the differences between the positive and negative cases, and thus to identify the positive cases.

So if you want to built a system to identify cats in an image, you feed the machine a large number of images with cats in them, and a large(r) number of images without cats in them, each appropriately “labelled” (“cat” or “no cat”) and based on the differences, the system learns to identify cats.

Similarly, if you want to teach a system to detect cancers based on MRIs, you show it a set of MRIs that show malignant tumours, and another set of MRIs without malignant tumours, and sure enough the machine learns to distinguish between the two sets (you might have come across claims of “AI can cure cancer”. This is how it does it).

However, AI can sometimes go wrong by learning the wrong things. For example, an algorithm trained to recognise sheep started classifying grass as “sheep” (since most of the positive training samples had sheep in meadows). Another system went crazy in its labelling when an unexpected object (an elephant in a drawing room) was present in the picture.

While machines learn through lots of positive and negative examples, that is not how humans learn, as I’ve been observing as my daughter grows up. When she was very little, we got her a book with one photo each of 100 different animals. And we would sit with her every day pointing at each picture and telling her what each was.

Soon enough, she could recognise cats and dogs and elephants and tigers. All by means of being “trained on” one image of each such animal. Soon enough, she could recognise hitherto unseen pictures of cats and dogs (and elephants and tigers). And then recognise dogs (as dogs) as they passed her on the street. What absolutely astounded me was that she managed to correctly recognise a cartoon cat, when all she had seen thus far were “real cats”.

So where do animals stand, in this spectrum of human to machine learning? Do they recognise from positive examples only (like humans do)? Or do they learn from a combination of positive and negative examples (like machines)? One thing that limits the positive-only learning for animals is the limited range of their communication.

What drives my curiosity is that they get trained for specific things – that you have dogs to identify drugs and dogs to identify explosives. You don’t usually have dogs that can recognise both (specialisation is for insects, as they say – or maybe it’s for all non-human animals).

My suspicion (having never had a pet) is that the way animals learn is closer to how humans learn – based on a large number of positive examples, rather than as the difference between positive and negative examples. Just that the limit of the animal’s communication being limited means that it is hard to train them for more than one thing (or maybe there’s something to do with their mental bandwidth as well. I don’t know).

What do you think? Interestingly enough, there is a recent paper that talks about how many machine learning systems have “animal-like abilities” rather than coming close to human intelligence.

For millions of years, mankind lived, just like the animals.
And then something happened that unleashed the power of our imagination. We learned to talk
– Stephen Hawking, in the opening of a Roger Waters-less Pink Floyd’s Keep Talking

Single Malt Recommendation App

Life is too short to drink whisky you don’t like.

How often have you found yourself in a duty free shop in an airport, wondering which whisky to take back home? Unless you are a pro at this already, you might want something you haven’t tried before, but don’t want to end up buying something you may not like. The names are all grand, as Scottish names usually are. The region might offer some clue, but not so much.

So I started on this work a few years back, when I first discovered this whisky database. I had come up with a set of tables to recommend what whisky is similar to what, and which single malts are the “most unique”. Based on this, I discovered that I might like Ardbeg. And I ended up absolutely loving it.

And ever since, I’ve carried a couple of tables in my Evernote to make sure I have some recommendations handy when I’m at a whisky shop and need to make a decision. But then the tables are not user friendly, and don’t typically tell you what you should buy, and what your next choice should be and so on .

To make things more user-friendly, I have built this app where all you need to enter is your favourite set of single malts, and it gives you a list of other single malts that you might like.

The data set is the same. I once again use cosine similarity to find the similarity of different whiskies. Except that this time I take the average of your favourite whiskies, and then look for the whiskies that are closest to that.

In terms of technologies, I’ve used this R package called Shiny to build the app. It took not more than half an hour of programming effort to build, and most of that was in actually building the logic, not the UI stuff.

So take it for a spin, and let me know what you think.

 

Caffeine kick

Until June or July this year, I firmly believed that well-made South Indian filter coffee was the best form of coffee ever. This belief possibly had to do with my conditioning, having been exposed this to this coffee form from an extremely early age, and the belief sustained even in the face of pretty excellent coffees from quite a few artisanal “Aussie style” cafes here in London.

Then, around then, I decided to embark on “intermittent fasting”, which meant no calorie consumption from 8 in the night to the next noon (each day). The diet permitted me to drink coffee or tea in the mornings as long as no milk or sugar was added to it, and that presented a problem.

For South Indian filter coffee can’t be drunk black. The addition of the chicory, which slows down the pace through which water/steam filters through the beans in order to maximise flavour, adds its own flavour, which when unmasked by milk can be pretty revolting. Though I must mention that chicory powder is sold as a separate “health drink” here in the UK (maybe it needs to be marketed such because its taste is most revolting).

That I couldn’t add milk to my coffee meant that I needed to explore other ways of making good black coffee. Counter top space (or the lack of it) ruled out contraptions such as an espresso machine or even a Nespresso machine. There was an old Braun “coffee maker” (which my mother-in-law reportedly procured two decades ago) at home, but that dished out pretty bad coffee (which only Americans might appreciate).

And so I started exploring, asking around coffee-geek friends (not to be confused with the cafe of a similar name in Victoria). The French Press was quickly ruled out on account of taste. I strongly considered the Aeropress and the Hario V60, and in the spirit of “try before you buy” or even “learn before you buy”, I asked baristas at my favourite local artisanal cafe to show me how to brew in these methods.

I quite liked the output of both methods, but found the aeropress apparatus a bit cumbersome and hard to clean (one reason I didn’t want to use my trusty Bialetti Moka Pot to make non-South Indian coffee as well). The V60 on the other hand offered simplicity of making process as well as extreme ease of cleaning. So quickly after I had tried, I had bought the pourover cup from Amazon, and a bag of beans from Electric (they ground it for me) and I was ready to go.

I’ve since fallen in love with this form of coffee, though when I go to a cafe I order an espresso-based drink (Cortado/Piccolo or Flat White depending on the cafe). And though I gave up on intermittent fasting a month and half after I started it, I continue to make this (I’m sipping on one such cup as I type this). And this is because of the caffeine kick.

I think I had this realisation for the first time back when I was still fasting – I drank a cup of pourover coffee just before I hit the gym (on an otherwise empty stomach), and I was astounded by my own energy levels that day. And I have since tested this in several other situations – before meetings, while doing an important piece of work or simply to stay awake. The caffeine kick from pourover coffee is simply unparalleled compared to any other kind of coffee I’ve had (though espresso-based coffees in cafes come very close).

South Indian filter coffee optimises for flavour at the cost of the caffeine. The decoction is frequently stored for a long time, even overnight. The large amount of milk added means that a given amount of beans can be used to make several more cups. And the chicory addition means that brewing is slower and more flavour gets extracted from the beans, though it’s unlikely that the amount of caffeine extracted is proportionally large.

And all this together means you get incredibly tasty coffee, but not something you can get that much of a caffeine kick out of. And that is possibly why we are conditioned to drinking so many cups of coffee a day – you need so many cups to get the level of caffeine your body “needs” to function.

And this explains why South Indian filter coffee in the evenings has never interfered with my sleep, buy any coffee bought in a good cafe after 5pm has invariably led to sleepless nights!

Do you have anything else to add to this theory?

PS: The first time I made pourover coffee, I used Indian beans from Chickmaglur (that I bought here in the UK), so it’s not to do with the beans. It’s the extraction method.

Voice assistants and traditional retail

Traditionally, retail was an over-the-counter activity. There was a physical counter between the buyer and the seller, and the buyer would demand what he wanted, and the shopkeeper would hand it over to him. This form of retail gave greater power to the shopkeeper, which meant that brands could practice what can be described as “push marketing”.

Most of the marketing effort would be spent in selling to the shopkeeper and then providing him sufficient incentives to sell it on to the customer. In most cases the customer didn’t have that much of a choice. She would ask for “salt”, for example, and the shopkeeper would give her the brand of salt that benefited him the most to sell.

Sometimes some brands would provide sufficient incentives to the shopkeeper to ensure that similar products from competing brands wouldn’t be stocked at all, ensuring that the customer faced a higher cost of getting those products (going to another shops) if they desired it. Occasionally, such strategies would backfire (a client with extremely strong brand preferences would eschew the shopkeeper who wouldn’t stock these brands). Mostly they worked.

The invention of the supermarket (sometime in the late 1800s, if I remember my research for my book correctly – it followed the concept of set prices) changed the dynamics a little bit. In this scenario, while the retailer continues to do the “shortlisting”, the ultimate decision is in the hands of the customer, who will pick her favourite among the brands on display.

This increases the significance of branding in the minds of the customer. The strongest incentives to retailers won’t work (unless they result in competing brands being wiped out from the shelves – but that comes with a risk) if the customer has a preference for a competing product. At best the retailer can offer these higher-incentive brands better shelf space (eye level as opposed to ankle level, for example).

However, even in traditional over-the-counter retail, branding matters to an extent when there is choice (as I had detailed in an earlier post written several years ago). This is in the instance where the shopkeeper asks the customer which brand she wants, and the customer has to make the choice “blind” without knowing what exactly is available.

I’m reminded of this issue of branding and traditional retail as I try to navigate the Alexa voice assistant. Nowadays there are two ways in which I play music using Spotify – one is the “direct method” from the phone or computer, where I search for a song, a list gets thrown up and I can select which one to play. The other is through Alexa, where I ask for a song and the assistant immediately starts playing it.

With popular songs where there exists a dominant version, using the phone and Alexa give identical results (though there are exceptions to this as well – when I ask Alexa to play Black Sabbath’s Iron Man, it plays the live version which is a bit off). However, when you are looking for songs that have multiple interpretations, you implicitly let Alexa make the decision for you, like a shopkeeper in traditional retail.

So, for example, most popular nursery rhymes have been covered by several groups. Some do the job well, singing the rhymes in the most dominant tunes, and using the most popular versions of the lyrics. Other mangle the tunes, and even the lyrics (like this Indian YouTube channel called Chuchu TV has changed the story of Jack and Jill, to give a “moral” to the story. I’m sure as a teenager you had changed the lyrics of Jack and Jill as well :P).

And in this situation you want more control over which version is played. For most songs I prefer the Little Baby Bum version, while for some others I prefer the Nursery Rhymes 123 version, but there is no “rule”. And this makes it complicate to order songs via Alexa.

More importantly, if you are a music publisher, the usage of Alexa to play on Spotify means that you might be willing to give Spotify greater incentives so that your version of a song comes up on top when a user searches for it.

And when you factor in advertising and concepts such as “paid search” into the picture, the fact that the voice assistants dictate your choices makes the situation very complicated indeed.

I wonder if there’s a good solution to this problem.

Elegant and practical solutions

There are two ways in which you can tie a shoelace – one is the “ordinary method”, where you explicitly make the loops around both ends of the lace before tying together to form a bow. The other is the “elegant method” where you only make one loop explicitly, but tie with such great skill that the bow automatically gets formed.

I have never learnt to tie my shoelaces in the latter manner – I suspect my father didn’t know it either, because of which it wasn’t passed on to me. Metaphorically, however, I like to implement such solutions in other aspects.

Having been educated in mathematics, I’m a sucker for “elegant solutions”. I look down upon brute force solutions, which is why I might sometimes spend half an hour writing a script to accomplish a repetitive task that might have otherwise taken 15 minutes. Over the long run, I believe, this elegance will pay off, in terms of scaling easier.

And I suspect I’m not alone in this love for elegance. If the world were only about efficiency, brute force would prevail. That we appreciate things like poetry and music and art and what not means that there is some preference for elegance. And that extends to business solutions as well.

While going for elegance is a useful heuristic, sometimes it can lead to missing the woods for the trees (or missing the random forests for the decision trees if you may will). For there are situations that simply don’t, or won’t, scale, and where elegance will send you on a wild goose chase while a little fighter work will get the job done.

I got reminded of this sometime last week when my wife asked me for some Excel help in some work she was doing. Now, there was a recent article in WSJ which claimed that the “first rule of Microsoft Excel is that you shouldn’t let people know you’re good at it”. However, having taught a university course on spreadsheet modelling, there is no place to hide for me, and people keep coming to me for Excel help (though it helps I don’t work in an office).

So the problem wasn’t a simple one, and I dug around for about half an hour without a solution in sight. And then my wife happened to casually mention that this was a one-time thing. That she had to solve this problem once but didn’t expect to come across it again, so “a little manual work” won’t hurt.

And the problem was solved in two minutes – a minor variation of the requirement was only one formula away (did you know that the latest versions of Excel for Windows offer a “count distinct” function in pivot tables?). Five minutes of fighter work by the wife after that completely solved the problem.

Most data scientists (now that I’m not one!)  typically work in production environments, where the result of their analysis is expressed in code that is run on a repeated basis. This means that data scientists are typically tuned to finding elegant solutions since any manual intervention means that the code is not production-able and scalable.

This can mean finding complicated workarounds in order to “pull the bow of the shoelaces” in order to avoid that little bit of manual effort at the end, so that the whole thing can be automated. And these habits can extend to the occasional work that is not needed to be repeatable and scalable.

And so you have teams spending an inordinate amount of time finding elegant solutions for problems for which easy but non-scalable “solutions exist”.

Elegance is a hard quality to shake off, even when it only hinders you.

I’ll close with a fairytale – a deer looks at its reflection and admires its beautiful anchors and admonishes its own ugly legs. Lion arrives, the ugly legs help the deer run fast, but the beautiful antlers get stuck in a low tree, and the lion catches up.

 

I’m not a data scientist

After a little over four years of trying to ride a buzzword wave, I hereby formally cease to call myself a data scientist. There are some ongoing assignments where that term is used to refer to me, and that usage will continue, but going forward I’m not marketing myself as a “data scientist”, and will not use the phrase “data science” to describe my work.

The basic problem is that over time the term has come to mean something rather specific, and that doesn’t represent me and what I do at all. So why did I go through this long journey of calling myself a “data scientist”, trying to fit in in the “data science community” and now exiting?

It all started with a need to easily describe what I do.

To recall, my last proper full-time job was as a Quant at a leading investment bank, when I got this idea that rather than building obscure models for trading obscure corner cases, I might as well use use my model-building skills to solve “real problems” in other industries which were back then not as well served by quants.

So I started calling myself a “Quant consultant”, except that nobody really knew what “quant” meant. I got variously described as a “technologist” and a “statistician” and “data monkey” and what not, none of which really captured what I was actually doing – using data and building models to help companies improve their businesses.

And then “data science” happened. I forget where I first came across this term, but I had been primed for it by reading Hal Varian saying that the “sexiest job in the next ten years will be statisticians”. I must mention that I had never come across the original post by DJ Patil and Thomas Davenport (that introduces the term) until I looked for it for my newsletter last year.

All I saw was “data” and “science”. I used data in my work, and I tried to bring science into the way my clients thought. And by 2014, Data Science had started becoming a thing. And I decided to ride the wave.

Now, data science has always been what artificial intelligence pioneer Marvin Minsky called a “suitcase term” – words or phrases that mean different things to different people (I heard about the concept first from this brilliant article on the “seven deadly sins of AI predictions“).

For some people, as long as some data is involved, and you do something remotely scientific it is data science. For others, it is about the use of sophisticated methods on data in order to extract insights. Some others conflate data science with statistics. For some others, only “machine learning” (another suitcase term!) is data science. And in the job market, “data scientist” can sometimes be interpreted as “glorified Python programmer”.

And right from inception, there were the data science jokes, like this one:

It is pertinent to put a whole list of it here.

‘Data Scientist’ is a Data Analyst who lives in California”
“A data scientist is someone who is better at statistics than any software engineer and better at software engineering than any statistician.”
“A data scientist is a business analyst who lives in New York.”
“A data scientist is a statistician who lives in San Francisco.”
“Data Science is statistics on a Mac.”

I loved these jokes, and thought I had found this term that had rather accurately described me. Except that it didn’t.

The thing with suitcase terms is that they evolve over time, as they start getting used differentially in different contexts. And so it was with data science. Over time, it has been used in a dominant fashion by people who mean it in the “machine learning” sense of the term. In fact, in most circles, the defining features of data scientists is the ability to write code in python, and to use the scikit learn package – neither of which is my distinguishing feature.

While this dissociation with the phrase “data science” has been coming for a long time (especially after my disastrous experience in the London job market in 2017), the final triggers I guess were a series of posts I wrote on LinkedIn in August/September this year.

The good thing about writing is that it helps you clarify your mind, and as I ranted about what I think data science should be, I realised over time that what I have in mind as “data science” is very different from what the broad market has in mind as “data science”. As per the market definition, just doing science with data isn’t data science any more – instead it is defined rather narrowly as a part of the software engineering stack where problems are solved based on building machine learning models that take data as input.

So it is prudent that I stop using the phrase “data science” and “data scientist” to describe myself and the work that I do.

PS: My newsletter will continue to be called “the art of data science”. The name gets “grandfathered” along with other ongoing assignments where I use the term “data science”.

Wheels of the bus went swimming one day

One story I like to tell is about how Mozart charged so much for setting Twinkle Twinkle to tune (if he did set it to tune, that is) that propagators of nursery rhymes decided to use the same tune for several other popular songs – most prominently for ABCDEFG and with a small variation for Ba Ba Black Sheep. It’s confusing, not just for kids but also for the parents. I’d written here a month or so back about how I would play tunes on the keyboard and Berry would try to guess the song and sing along. As someone who sets quizzes occasionally, the lack of “a unique answer” drives me nuts. And it possibly drives Berry nuts as well, since she changes from twinkle twinkle to ABCD within the course of one stanza. I wonder why this is the case. Using my one data point (Berry) kids can catch on to tunes pretty quickly (she was barely a year old when she started humming the tune of Black Sabbath’s Iron Man. Now she knows the full lyrics). And having unique tunes for songs means that kids are able to make easy associations between music and words – always a desirable thing. And the lack of one-to-one correspondence doesn’t just run one way – sometimes there are multiple ways in which the same song can be sung. For some songs, such as Happy Birthday, this is due to copyright issues. I’m not sure why other songs are sung in different tunes. For example, there are two clearly different ways in which the third line of itsy-bitsy/incy-wincy (depending on which side of the pond you’re on) spider is sung, and it gets especially confusing when I’m playing on the keyboard, since I don’t know which version Berry is expecting ( we invariably sing/play the “other” way). The usage of voice controlled players has made things worse. In fact, the first time I appreciated Siri on my phone was when Berry was just born, and I needed both hands to hold her and put her to sleep, and then someone turn on a lullaby (“Hey Siri, play iron man by rockabye baby”). Now, the problem with voice-controlled playing is that when there are multiple versions of the same song you don’t know which one will get played. An extreme case is like earlier today we asked Alexa to use Amazon Unlimited (we have a 3-month free trial, possibly because of my Prime membership) to play “london bridge”. It belted out some dhinchak EDM song! Within the realms of nursery rhymes itself there are songs that are sung to completely new tunes (like I had never expected that there exists a version of Jack and Jill sung to the tune of Yankee Doodle. It is most annoying). It is extremely disorienting for me – and I guess it is for kids as well, for I’m told they like predictability. I don’t know what can be done to restore the sanity of one-song-one-tune.  Yes, I can record a set of songs in unique and popular tunes, but there is no guarantee that it will take off. And with the increase of voice controlled music playing, there is no guarantee that the “bad tunes” won’t get any air time. The title, for those that didn’t get it, is a portmanteau of two songs that share a name. I must mention I have no intention of popularising these two precise renditions of these songs – they were simply on top of the search engine results.
Mommy duck said quack quack quack, all day long! PS: There are differing versions in lyrics as well. One version says “all day long”; another says “all the way to town”. As Aditya Narayan sang in Rangeela Re 23 years ago, it’s complicated being a kid.

Programming Languages

I take this opportunity to apologise for my prior belief that all that matters is thinking algorithmically, and language in which the ideas are expressed doesn’t matter.

About a decade ago, I used to make fun of information technology company that hired developers based on the language they coded in. My contention was that writing code is a skill that you either have or you don’t, and what a potential employer needs to look for is the ability to think algorithmically, and then render ideas in code. 

While I’ve never worked as a software engineer I find myself writing more and more code over the years as a part of doing data analysis. The primary tool I use is R, where coding doesn’t really feel like coding, since it is a rather high level language. However, I’m occasionally asked to show code in Python, since some clients are more proficient in that, and the one thing that has done is to teach me the value of domain knowledge of a programming language. 

I take this opportunity to apologise for my prior belief that all that matters is thinking algorithmically, and language in which the ideas are expressed doesn’t matter. 

This is because the language you usually program in subtly nudges you towards thinking in a particular way. Having mostly used R over the last decade, I think in terms of tables and data frames, and after having learnt tidyverse earlier this year, my way of thinking algorithmically has become in a weird way “object oriented” (no, this has nothing to do with classes). I take an “object” (a data frame) and then manipulate it in various ways, changing it, summarising stuff, calculating things on the fly and aggregating, until the point where the result comes out in an elegant manner. 

And while Pandas allows chaining (in fact, it is from Pandas that I suspect the tidyverse guys got the idea for the “%>%” chaining operator), it is by no means as complete in its treatment of chaining as R, and that that makes things tricky. 

Moreover, being proficient in R makes you think in terms of vectorised operations, and when you see that python doesn’t necessarily offer that, and and operations that were once simple in R are now rather complicated in Python, using list comprehension and what not. 

Putting it another way, thinking algorithmically in the framework offered by one programming language makes it rather stressful to express these thoughts in another language where the way of algorithmic thinking is rather different. 

For example, I’ve never got the point of the index in pandas dataframes, and I only find myself “resetting” it constantly so that my way of addressing isn’t mangled. Compared to the intuitive syntax in R, which is first and foremost a data analysis tool, and where the data frame is “native”, the programming language approach of python with its locs and ilocs is again irritating. 

I can go on… 

And I’m guessing this feeling is mutual – someone used to doing things the python way would find R’s syntax and way of doing things rather irritating. R’s machine learning toolkit for example is nowhere as easy as scikit learn is in python (this doesn’t affect me since I seldom need to use machine learning. For example, I use regression less than 5% of the time in my work). 

The next time I see a job opening for a “java developer” I will not laugh like I used to ten years ago. I know that this posting is looking for a developer who can not only think algorithmically, but also algorithmically in the way that is most convenient to express in Java. And unlearning one way of algorithmic thinking and learning another isn’t particularly easy. 

The Crane-Mongoose Theory of Public Policy

I have several favourite stories from the Panchatantra (which perhaps explains my lack of appreciation of modern children’s fiction). One of them involves a crane and a mongoose. And I think it is a good lesson on when and where to call for regulation, and government or legal intervention.

So the story goes like this. A snake lives at the bottom of the tree where a crane has built its nest. Each time the crane lays eggs, the snake slithers up the tree and devours them. And the crane doesn’t know what to do. Ultimately it receives some “brilliant advice”.

There is a mongoose living somewhere nearby, and the crane lays out a Hansel-and-Gretel like path of fish from the mongoose’s house to the snake’s house. The mongoose duly follows the trail of fish and finishes off the snake. The next day, the mongoose is hungry again, and it climbs up the tree and devours the crane’s eggs.

It is common political discourse nowadays to call for the government’s or court’s intervention to solve what seems to be private problems. The governments and courts are of course happy to oblige – any new source for intervention and rent-seeking are good news for the people involved. And then you get a solution that temporarily solves the problem (slaughtering the snake). And then in the long term, what you get is a bigger problem (mongoose eating the crane’s eggs). The only difference is that in real life it is not just the crane that gets negatively affected – the regulations hurt everyone.

The examples that come to my mind at this point in time are all “local”. Some residents in Indiranagar in Bangalore weren’t happy about the noise from nearby pubs. They asked the government to “do something”. And the government “did something” – it banned the playing of live music in restaurants, killing off what was then a budding industry in Bangalore.

Some other residents somewhere else in Bangalore were unhappy that their neighbours had dogs that barked. They asked the government to do something. The government did something – coming up with an elaborate document to regulate dogs that people can own.

And there are more involved (and dangerous) examples of this as well.

Don’t be like the crane.

Investing in ETFs

So I put some money in an ETF today. This isn’t the first time I invested in one. A long time back, before my then employer had bought and essentially killed Benchmark, I had invested in a couple of their ETFs – the Nifty ETF to get invest in the broad Indian market, and GoldBees to hedge against increase in the price of gold as I was planning my wedding.

I had some Rupees lying around in my bank account for a long time, and given that the Indian markets have tanked, I thought this is a good time to get invested. In fact, this isn’t the first time in recent times I’m having such a thought – about a month back I had put in more money into the Indian markets, but had then chosen a low cost index tracking mutual fund (and I’m not tracking how my investment is doing).

Anyway, today I decided to invest in ETFs since the transaction costs (in terms of both trading, and annual expenses) are much lower. A quick chat with a friend currently trading the Indian markets revealed that the SBI Nifty ETF is the best option to go with, and I was left with the small matter of just making the investment.

I’m generally happy with ICICI Direct as my broker, since in general the interface and app are pretty nice. Last month, the purchase of the mutual fund through the same app had been pretty simple. And I imagined buying the ETF will be easy as well. It wasn’t. And if I, as a professional investor with considerable capital markets experience, find it hard to invest in ETFs, I can only imagine how hard it might be for mango people to invest in them.

So the points of pain, in order, that prevent people from investing in ETFs:

  1. Knowing that indexing exists. Most people seem to think that the only ways to invest are by researching the stocks themselves, or by paying an asset manager fairly hefty fees.
  2. Once you know you can index, the fact that you can do it through an ETF. ETFs are again not well known, and not really marketed broadly since their fees are low (with Benchmark’s demise, we don’t really have ETF-first fund houses in India, like we have Vanguard in the US).
    1. Related, even some of the more popular robo advisory funds in India largely use mutual funds, rather than ETFs.
  3. Once you know you can index, and do so through an ETF, the next task is to find out which ETF you should invest in. Literature exists, but is not easy to find. My friend sent me this page, and asked me to select the fund with highest market size. Knowing that I want to invest in the broad market, and in large caps, the choice of SBI Nifty ETF was easy for me.
    1. But it’s not so intuitive for a less sophisticated investor. For example, correlating asset size with liquidity isn’t exactly intuitive.
    2. Different ETFs track different indices, and knowing which one to invest in is again not a trivial task.
  4. Having selected an ETF to invest in, you go to your broker’s site or app (I used the app). And you need to know that ETFs are clubbed with equities, and not with mutual funds (not an intuitive classification for most people)
  5. So I go to ICICI Direct’s Equities page, allocate funds to it (from my bank account, also with ICICI), and hit “buy”. There’s a text box where I need to enter what I’m looking for, and then there’s a dropdown that pops up.

    I type “SBI”, and the first thing it shows is the SBI Bank Nifty tracker. This is followed by lots of bonds. I don’t know if it’s clever nudging on ICICI’s part to get you to invest in the Bank Nifty, since that has a significant exposure to ICICI, or if it’s something as mundane as alphabetical sorting. The latter is more likely.

  6. Scrolling down the list past all the bonds, it’s not easy to know which is the SBI Nifty ETF. Because there’s a “SBI Nifty Next 50 ETF” (smaller caps, so more volatile, not something I want), and a few others with confusing names.
  7. Then you need to enter the number of units you need to purchase. This is unlike in mutual funds where you just enter the amount you want to invest. Here I had to pull up a calculator to know exactly how many units I had to buy.
  8. I hit “market order”, and then on the next screen I got a warning that since this wasn’t a particularly liquid instrument I was only allowed to post limit orders. So I had to guess what was a reasonable spread I was willing to pay, and put that. Thankfully the ETF was fairly liquid, and I got execution close to mid.

Honestly, I felt rather daunted at the end of the exercise, and I’m what most people would classify as a sophisticated investor. So there is no wonder that more people aren’t investing in ETFs.

The advantage of ETFs is extremely low fees (the fund I purchased today charges 7 basis points a year), and one downside of it is that it doesn’t allow for more marketing budget.

I’m beginning to think that the way to “solve” this market is by having a bundled ETF and robo advisory offering. Perhaps more on that later.