Category: arbit

Ratings revisited

Sometimes I get a bit narcissistic, and check how my book is doing. I log on to the seller portal to see how many copies have been sold. I go to the Amazon page and see what are the other books that people who have bought my book are buying (on the US store it’s Ray Dalio’s Principles, as of now. On the UK and India stores, Sidin’s Bombay Fever is the beer to my book’s diapers).

And then I check if there are new reviews of my book. When friends write them, they notify me, so it’s easy to track. What I discover when I visit my Amazon page are the reviews written by people I don’t know. And so far, most of them have been good.

So today was one of those narcissistic days, and I was initially a bit disappointed to see a new four-star review. I started wondering what this person found wrong with my book. And then I read through the review and found it to be wholly positive.

A quick conversation with the wife followed, and she pointed out that this reviewer perhaps reserves five stars for the exceptional. And then my mind went back to this topic that I’d blogged about way back in 2015 – about rating systems.

The “4.8” score that Amazon gives as an average of all the ratings on my book so far is a rather crude measure – since one reviewer’s 4* rating might differ significantly from another reviewer’s.

For example, my “default rating” for a book might be 5/5, with 4/5 reserved for books I don’t like and 3/5 for atrocious books. On the other hand, you might use the “full scale” and use 3/5 as your average rating, giving 4 for books you really like and very rarely giving a 5.

By simply taking an arithmetic average of ratings, it is possible to overstate the quality of a product that has for whatever reason been rated mostly by people with high default ratings (such a correlation is plausible). Similarly a low average rating for a product might mask the fact that it was rated by people who inherently give low ratings.

As I argue in the penultimate chapter of my book (or maybe the chapter before that – it’s been a while since I finished it), one way that platforms foster transactions is by increasing information flow between the buyer and the seller (this is one thing I’ve gotten good at – plugging my book’s name in random sentences), and one way to do this is by sharing reviews and ratings.

From this perspective, for a platform’s judgment on a product or seller (usually it’s the seller, but for products such as AirBnb, information about buyers also matters) to be credible, it is important that they be aggregated in the right manner.

One way to do this is to use some kind of a Z-score (relative to other ratings that the rater has given) and then come up with a normalised rating. But then this needs to be readjusted for the quality of the other items that this rater has rated. So you can think of some kind of a Singular Value Decomposition you can perform on ratings to find out the “true value” of a product (ok this is an achievement – using a linear algebra reference given how badly I suck in the topic).

I mean – it need not be THAT complicated, but the basic point is that it is important that platforms aggregate ratings in the right manner in order to convey accurate information about counterparties.

Freelancing and transaction costs

In the six years of running my own consulting business, I’d forgotten about an essential part that you need to endure as part of a job – piecemeal work. It is fairly often when you’re working for someone else that you get work that is so tiny or insignificant that you can hardly take ownership of it. The best strategy for dealing with it is to quietly get it over with and hope you won’t get such stuff again.

However, sometimes you can get caught in a rut of continuously getting this kind of work, and start wondering what you actually signed up for. And this is one thing I hadn’t expected to encounter when I got back to full time working earlier this year.

Thinking about why I never had to encounter such stuff during my consulting life, I realised there’s a fairly simple explanation – transaction costs.

Being a consultant is high transaction cost business. Every time you need to take on a new piece of work, you need to go through the charade of negotiating specifics with the client, pricing and drawing up a contract. All put together, the effort is not insignificant.

Moreover, in the line of work that I used to do, there was this massive overhead cost of understanding, cleaning and getting comfortable with the client’s data  – the effort involved in that meant that after a particular point in time I stopped taking work that wasn’t chunky enough. For a while I started refusing such work, but then got smarter and started pricing myself out of such work (though some clients were generous enough to meet that price to get their little tasks done – effectively I’d passed on the transaction costs to them).

The downside of this, of course, was that there was a fair amount of money I could have made taking up small works which I didn’t since the transaction cost was too high – this can be thought of as potential lost revenues. The upside was that whatever work I did was of high quality and (hopefully) made a big impact on the client’s business.

In the nature of the firm, Ronald Coase wrote that the purpose of the corporation was that transaction cost of dealing with co-workers can be eliminated. But then, I realise that sometimes this transaction cost can also be a good thing!

Oh, and obligatory plug here – my book Between the buyer and the seller deals with transaction costs, among other things. It’s available for sale (both in print and digital) on Amazon.

 

The (missing) Desk Quants of Main Street

A long time ago, I’d written about my experience as a Quant at an investment bank, and about how banks like mine were sitting on a pile of risk that could blow up any time soon.

There were two problems as I had documented then. Firstly, most quants I interacted with seemed to be solving maths problems rather than finance problems, not bothering if their models would stand the test of markets. Secondly, there was an element of groupthink, as quant teams were largely homogeneous and it was hard to progress while holding contrarian views.

Six years on, there has been no blowup, and in some sense banks are actually doing well (I mean, they’ve declined compared to the time just before the 2008 financial crisis but haven’t done that badly). There have been no real quant disasters (yes I know the Gaussian Copula gained infamy during the 2008 crisis, but I’m talking about a period after that crisis).

There can be many explanations regarding how banks have not had any quant blow-ups despite quants solving for math problems and all thinking alike, but the one I’m partial to is the presence of a “middle layer”.

Most of the quants I interacted with were “core” in the sense that they were not attached to any sales or trading desks. Banks also typically had a large cadre of “desk quants” who are directly associated with trading teams, and who build models and help with day-to-day risk management, pricing, etc.

Since these desk quants work closely with the business, they turn out to be much more pragmatic than the core quants – they have a good understanding of the market and use the models more as guiding principles than as rules. On the other hand, they bring the benefits of quantitative models (and work of the core quants) into day-to-day business.

Back during the financial crisis, I’d jokingly predicted that other industries should hire quants who were now surplus to Wall Street. Around the same time, DJ Patil et al came up with the concept of the “data scientist” and called it the “sexiest job of the 21st century”.

And so other industries started getting their own share of quants, or “data scientists” as they were now called. Nowadays its fashionable even for small companies for whom data is not critical for business to have a data science team. Being in this profession now (I loathe calling myself a “data scientist” – prefer to say “quant” or “analytics”), I’ve come across quite a few of those.

The problem I see with “data science” on “Main Street” (this phrase gained currency during the financial crisis as the opposite of Wall Street, in that it referred to “normal” businesses) is that it lacks the cadre of desk quants. Most data scientists are highly technical people who don’t necessarily have an understanding of the business they operate in.

Thanks to that, what I’ve noticed is that in most cases there is a chasm between the data scientists and the business, since they are unable to talk in a common language. As I’m prone to saying, this can go two ways – the business guys can either assume that the data science guys are geniuses and take their word for the gospel, or the business guys can totally disregard the data scientists as people who do some esoteric math and don’t really understand the world. In either case, value added is suboptimal.

It is not hard to understand why “Main Street” doesn’t have a cadre of desk quants – it’s because of the way the data science industry has evolved. Quant at investment banks has evolved over a long period of time – the Black-Scholes equation was proposed in the early 1970s. So the quants were first recruited to directly work with the traders, and core quants (at the banks that have them) were a later addition when banks realised that some quant functions could be centralised.

On the other hand, the whole “data science” growth has been rather sudden. The volume of data, cheap incrementally available cloud storage, easy processing and the popularity of the phrase “data science” have all increased well-at-a-faster rate in the last decade or so, and so companies have scrambled to set up data teams. There has simply been no time to train people who get both the business and data – and the data scientists exist like addendums that are either worshipped or ignored.

Firecrackers and the Hindu religion

There was massive controversy in India last month when the Supreme Court banned the sale of firecrackers in and around Delhi, in an ostensible Move to cut pollution.

As one might expect, the move drew heavy criticism on the grounds that the court was ruling against a fundamental tenet of Hindu religion. In return, other people pointed out that bursting firecrackers on the occasion of Deepavali is a rather recent tradition, and thus has nothing to do with the “fundamental tenets of Hinduism”.

As it happened, the ban continued to stay, though reports say that both noise and air pollution levels in Delhi were unaffected by the ban. Here’s my humble attempt to argue that why modern traditions such as bursting firecrackers is important to religion,

As I’ve mentioned several times on this blog, religion in general and festivals in particular are memes, in the traditional Richard Dawkins sense of the term.

Religion and festivals are basically ideas that compete in an ideas marketplace, and people propagate the ideas that they like the most. In one sense people like what they find useful – which is why imagined orders such as democracy or public limited companies continue to propagate and thrive.

At a more personal level, though, people will choose to associate with and propagate ideas that they simply like, and at a very basic level, enjoy. In other words, the more fun people find a concept, the more heavily they’ll adopt and propagate it.

Hence religions evolve, and (in what can be seen as parallels to mutation), pick up ideas from outside that can make them more fun. So the American Christians picked up and appropriated thanksgiving from the red Indians. Even further back Christianity picked up and amalgamated the idea of Christmas. Hare Krishna people picked up wild dancing. Bombay people picked up Ganesha processions. And so on.

By incorporating fun practices from outside, religions make themselves fitter, as they open up leeway’s for new recruits (such as kids). Short of coercion, without fun practices there’s little chance that religion can pick up new recruits.

Crackers on Deepavali, or colours on Holi, are aspects that have come into the hindu religion that have made it more fun. That theee aspects make the religion more fun mean that it’s easier to co-opt new recruits, especially the young kind. This makes the meme that is the hindu religion fitter.

So it doesn’t matter how ancient a practice is – as long as it’s fun, and increases the memetic fitness of a religion, it remains a fundamental part of the religion.

Without firecrackers the idea of Deepavali might lose its identity and people might stop celebrating it. And it being one of Hinduism’s most celebrated festivals, a weakening Deepavali meme leads to a weakening hindu meme.

So the banning of firecrackers in Delhi on the occasion of Deepavali was indeed injurious to the hindu religion.

Just keep in mind that culture (using memes) evolves much much faster than organisms (which use genes)!

London’s 7D

In classes 11 and 12 i had to travel every day from Jayanagar to indiranagar to get to school. There was a direct bus that took me from just behind my house to Just behind my school. This was 7D. But despite my mother’s insistence that I take that, I seldom did. For it took such a circuitous route that it would take ages.

I’m sure that someone has done a survey of bangalores most convoluted bus routes, and if so, 7D would fall close to the top there (the only bus that I imagine could beat 7D is 201).

So rather than take 7D I’d take one of the many buses bound to Shivajinagar and get off at Richmond circle, from where I’d get 138 to take me right behind school (or the double decker 131 to take me 10 mins walk away in the other direction). The changeover at Richmond circle was rather simple (no walking involved) and this process would help me save at least 15 minutes each way every day.

Now I’ve figured that the London Underground has its own 7D, except for the fact that the route is not circuitous – it’s simply slow. I live in Ealing and my office is near Victoria so the most direct way for me to travel is to take the district line. It takes 35 minutes and runs once every 10 minutes (the line splits in two places to frequency to Ealing is low).

On most days I don’t travel directly from home to work since I drop Berry to her Nursery on the way. So taking the district line straight from Home to work is never an option.

Yesterday I was ill and so my wife took Berry to her Nursery. So I travelled directly to work. And for the first time ever since I joined this office I took the district line on the way to Office.

I reached Ealing broadway at 8:02 and Just about caught the 8:03 train. The train rolled into Victoria at 8:40 and I was in Office at 8:45.

Today once again I was traveling directly from home to work, and reached Ealing broadway station a few seconds later than yesterday, just missing the train I’d caught yesterday. I had the option to wait 10 minutes for the next district line train or using what seemed like a convoluted route. I chose the latter.

I took a great western railway train to Paddington, where I walked for about 5-7 minutes to the bakerloo line and got it. I got off the bakerloo five stops later at oxford circus where I changed to the Victoria line, and got off two stops later at Victoria. The time was 8:35!

In other words I’d left later than I had yesterday, changed trains twice (one involving a long walk) and still reached five minutes earlier. And all the time traveling in trains far less crowded than an early morning district line train headed to the city!

I hereby christen the district line as London’s 7D. Except that the route isn’t anywhere circuitous!

JEE coaching and high school learning

One reason I’m not as good at machine learning as I can possibly be is because I suck at linear algebra. I totally completely suck at it. Seven years of usage of R has meant that at least I no longer get spooked out by the very sight of vectors or matrices, and I understand the concept of matrix multiplication (an operator rotating a vector), but I just don’t get linear algebra.

For example, when I see terms such as “singular value decomposition” I almost faint. Multiple repeated attempts at learning the concept have utterly failed. Don’t even get me started on the more complicated stuff – and machine learning is full of them.

My inability to understand linear algebra runs deep, and it’s mainly due to a complete inability to imagine vectors and matrices and matrix operations. As far back as I remember, I have hated matrices and have tried to run away from it.

For a long time, I had placed the blame for this on IIT Madras, whose mathematics department in its infinite wisdom had decided to get its brilliant Graph Theory expert to teach us matrices. Thinking back, though, I remember going in to MA102 (Vectors, Matrices and Differential Equations) already spooked. The rot had set in even earlier – in school.

The problem with class 11 in my school (a fairly high-profile school which was full of studmax characters) was that most people harboured ambitions of going to IIT, and had consequently enrolled themselves in formal coaching “factories”. As a result, these worthies always came to maths, physics and chemistry classes “ahead” of people like me who didn’t go for such classes (I’d decided to chill for a year after a rather hectic class 10 when I’d been under immense pressure to get my school a “centum”).

Because a large majority of the class already knew what was to be taught, teachers had an incentive to slack. Also the fact that most students were studmax had meant that people preferred to mug on their own rather than display their ignorance in class. And so jai happened.

I remember the class when vectors and matrices were introduced (it was in class 11). While I don’t remember too many details, I do remember that a vocal majority already knew about “dot product” and “cross product”. It was similar a few days later when the vocal majority knew matrix multiplication.

And so these concepts were glossed over, and lacking a grounding in fundamentals, I somehow never “got” the concept.

In my year (2000), CBSE decided to change format for its maths examination – everyone had to attempt “Part A” (worth 70 marks) and then had a choice between “Part B” (vectors, matrices, etc.) and “Part C” (introductory statistics). Most science students were expected to opt for Part B (Part C had been introduced for the benefit of commerce students studying maths since they had little to gain from reading about vectors). For me and one other guy from my class, though, it was a rather obvious choice to do Part C.

I remember the invigilator (who was from another school) being positively surprised during my board exam when I mentioned that I was going to attempt Part C instead of Part B. He muttered something to the extent of “isn’t that for commerce students?” but to his credit permitted us to do the paper in whatever way we wanted (I fail to remember why I had to mention to him I was doing Part C – maybe I needed log tables to do that).

Seventeen odd years down the line, I continue to suck at linear algebra and be stud at statistics. And it is all down to the way the two subjects were introduced to me in school (JEE statistics wasn’t up to the same standard as Part C so the school teachers did a great job of teaching that).

Pudina family

Sometimes they say that opposites attract.

But more practically, I think it’s impossible to louvvu someone unless you have lots of similar interests, and that also means lots of similar ambitions. And in that sense my wife and I have shared quite a few ambitions.

First we wanted to become celebrity bloggers. Then (ok the order gets messed up here) there was the MBA. And before all this there is Ganeshana Maduve (which we re-watched perhaps for the 50th time this weekend).

And adding to all this, there’s the desire to write in newspapers. I remember that over a decade ago I wanted to regularly write in newspapers, and about “policy issues”. I didn’t follow up on that ambition, of course, but through lots of twists and turns and happy coincidences meant that I started writing for Mint in 2013, and some of the stuff I’ve written there are about “policy issues”.

And the wife has had similar ambitions as well, though her methods have been vastly different, and much more focussed. She’s always wanted to write a column on relationships. Rather, she first wanted to be a relationship blogger, and then a relationship columnist, and she’s gone about the process with single-minded ambition.

So, first there was the MarriageBrokerAuntie blog (now hosted here). Then it turned into a Facebook page. It even led to a business that she ran during her maternity and post-childbirth periods (imagine running a business while nursing a tiny baby). And now she’s in the papers. Yay!

It so happens that it’s the same paper that I write for. And it also happens that the edition of the paper where it was published (Mint on Sunday) also happened to carry an excerpt from my book two-three weeks back. And that also happened to be about relationships.

So a long long time ago, a couple of days after we’d first met, she had written about “Arranged Louvvu“. I don’t think it’s a coincidence that the first piece she’s written for Mint is about “Love, and other arrangements“.  It’s about dating apps, and how what they lead to is not “real love” and it’s no different from “other arrangements”. That people think arranged marriage is uncool, but dating apps lead to basically arranged relationships. And so on.

Read the whole thing, it’s damn well written. Oh, and it features 1-6-1 calls, Panchatantra and George Akerlof’s “market for lemons”, among other stud fundaes.

Now the only thing left is for Berry to start writing for Mint. They don’t have a children’s issue (where they feature drawings, poems, etc. written by kids) so I guess she’ll have to wait a while. But I’m damn hopeful!

In any case, for now massive pride is happening on account of the wife!

Poor food

Until about 1970, when the so-called Green Revolution happened, India as a country collectively didn’t have enough food (remember PL-480 and “ship to mouth existence”?). Until liberalisation in the 1990s, even people who could possibly afford it couldn’t get the food they wanted (remember lining up at ration shops?).

In other words, Indians (as a country – there are still lots of people who don’t get to decide on what to eat since they’re way too poor) have had a proper choice in terms of what to eat for just about one generation now. More than half the Indians who are currently alive spent at least some part of their lives at a time when it just wasn’t possible at all to eat what one wanted.

What this implies is that what we consider to be “traditional food” is largely “poor food” – we and our ancestors ate that not because it was what was the most nutritious, but because that is what was available, and what we could afford.

And so you have most of our traditional food being extremely heavy in carbs and light on almost everything else. I have friends who comment that most Indian vegetarian food hardly has vegetables – consider the sambar, for instance, which just has a few pieces of vegetables floating around. It is a correct comment, but that is because most of what we know as traditional Indian food evolved through times of shortages and poverty.

There are times when I attempt to give people nutrition advice, and while people listen to me politely, they end up saying something to the effect that if they start eating “traditional food”, all will be fine with their health again.

We’ve evolved to fundamentally trust the familiar, and distrust the new. And so it is with our food choices. Without really understanding why we and our ancestors ate the food that we ate, we consider “traditional food” to be good.

Now that I can afford it, I try to make sure I have balanced meals, and a lot of “traditional indian foods” that I grew up eating hardly get consumed in my house now. Consider the uppit – which is mostly carbs (semolina) with a small handful of vegetables and some fats thrown in – incredibly unbalanced stuff. Or beaten rice (avlakki/poha) – which is so light that you start feeling hungry within a couple of hours of eating. And so on – once you start looking at at the nutritional value of what you are eating, you will find yourself thoroughly dissatisfied with a lot of “traditional stuff”.

So my advice to you is this – if you can afford it, give what you are eating a thought, and make sure you get the right kind of nutrition without giving too much concern to your “priors”. And if you’re on a tight budget, optimise that to make sure it goes as far as possible in providing you a balanced diet.

On cultural appropriation

Over the last few months, I’ve come across this concept of “cultural appropriation” several times. I don’t claim to get it completely, but I think I understand it enough to comment about it.

Going by Wikipedia, cultural appropriation

is the adoption or use of the elements of one culture by members of another culture. Cultural appropriation, often framed as cultural misappropriation, is sometimes portrayed as harmful and is claimed to be a violation of the collective intellectual property rights of the originating culture

The list of celebrities who’ve been accused of cultural appropriation runs way too long to list here, but it’s basically a popular topic of outrage among the modern left, commonly described by their detractors as “social justice warriors” (SJW).

In any case, my attention to the topic was drawn by a recent essay on the topic by philosopher Kenan Malik. In “In defence of cultural appropriation“, first published in the New York Times, Malik writes:

But who does the policing? Every society has its gatekeepers, whose role is to protect certain institutions, maintain the privileges of particular groups and cordon off some beliefs from challenge. Such gatekeepers protect not the marginalized but the powerful. Racism itself is a form of gatekeeping, a means of denying racialized groups equal rights, access and opportunities.

In minority communities, the gatekeepers are usually self-appointed guardians whose power rests on their ability to define what is acceptable and what is beyond the bounds. They appropriate for themselves the authority to license certain forms of cultural engagement, and in doing so, entrench their power.

In fact, reading the rather long essay, it was hard for me to disagree with him. In fact, it started to make me wonder why cultural appropriation is a matter of debate at all – controversial enough that at least three editors who defended it have lost their jobs (per Malik). In fact, Malik himself was victim of significant online abuse and trolling following his article.

So thinking about this topic during a work break the other day, I found compelling evidence about why the concept is bullshit – basically, it’s one-sided.

The whole concept of “cultural appropriation” hinges on there being a “superior community” and a “marginalised community”, with members of the former not allowed to adopt practices of the latter. This is a one-way street – if you turn the argument around and say that a person from a traditionally “marginalised community” should not adopt cultural practices of a “superior community”, you’re essentially being racist or casteist or whatever.

Consider this, for example – “Dalits should not recite the Vedas because by doing so, they are appropriating the culture of caste Hindus“.  It is unlikely that any self-respecting SJW would condone this statement. But turn the communities around, and the outrage on cultural appropriation become legit!

This makes the entire concept problematic, since it rests on a prior of certain communities being “marginalised”. In other words, it rests on a prior of a partial ordering of “communities”, with some considered more advanced than the other. Take away any such ordering or hierarchy, and the concept of cultural appropriation falls flat.

To me, the outrage about cultural appropriation smacks of a sort of “white man’s burden” among SJWs in an attempt to seemingly protect seemingly marginalised communities. All this achieves, as Kenan Malik mentions in his essay, is to empower the self-appointed leaders of these marginalised communities.

Maths, machine learning, brute force and elegance

Back when I was at the International Maths Olympiad Training Camp in Mumbai in 1999, the biggest insult one could hurl at a peer was to describe the latter’s solution to a problem as being a “brute force solution”. Brute force solutions, which were often ungainly, laboured and unintuitive were supposed to be the last resort, to be used only if one were thoroughly unable to implement an “elegant solution” to the problem.

Mathematicians love and value elegance. While they might be comfortable with esoteric formulae and the Greek alphabet, they are always on the lookout for solutions that are, at least to the trained eye, intuitive to perceive and understand. Among other things, it is the belief that it is much easier to get an intuitive understanding for an elegant solution.

When all the parts of the solution seem to fit so well into each other, with no loose ends, it is far easier to accept the solution as being correct (even if you don’t understand it fully). Brute force solutions, on the other hand, inevitably leave loose ends and appreciating them can be a fairly massive task, even to trained mathematicians.

In the conventional view, though, non-mathematicians don’t have much fondness for elegance. A solution is a solution, and a problem solved is a problem solved.

With the coming of big data and increased computational power, however, the tables are getting turned. In this case, the more mathematical people, who are more likely to appreciate “machine learning” algorithms recommend “leaving it to the system” – to unleash the brute force of computational power at the problem so that the “best model” can be found, and later implemented.

And in this case, it is the “half-blood mathematicians” like me, who are aware of complex algorithms but are unsure of letting the system take over stuff end-to-end, who bat for elegance – to look at data, massage it, analyse it and then find that one simple method or transformation that can throw immense light on the problem, effectively solving it!

The world moves in strange ways.