confidence – Pertinent Observations

Channel Coding Theorem in Real Life

One of my favourite concepts in Computer Science is Shannon’s Channel Coding Theorem. This theorem is basically about the efficiency of communication over a noisy channel. And as I was thinking a few minutes back, this has interesting implications in real life as well, well away from the theory of communication.

I don’t have that much understanding of the rigorous explanation of the theorem. However, I absolutely love the central idea of it – that the noisier a channel is, the more the redundancy you need in your communication, and thus the slower is your communication. A corollary of this is that every channel has a “natural maximum speed”, and as long as you try to communicate within that speed, you can communicate reliably.

I won’t go into the technical details here – that involves assuming that the channel loses (or garbles) X% of bits, and then constructing a redundant code that shows that even with this loss, you can communicate effectively.

Anyway, let’s leave behind the theory communication and go on to real life.

I’ve found that I communicate badly when I’m not sure what language to talk in. If I’m talking in English with someone who I know knows good English, I communicate rather well (like my writing 😛 ) . However, if I’m not sure about the quality of language of the other person, I hesitate. I try to force myself to find simpler / more obvious words, and that disturbs my flow of thought, and I stammer.

Similarly, when I’m not sure whether to talk in Kannada or English (the two languages I’m very comfortable in), I stammer heavily. Again, because I’m not sure if the words I would naturally use will be understood by the other person (the counterparty’s comprehension being the “noise in the channel” here), I slow down, get jittery, and speak badly.

Then of course, there is the very literal interpretation of the channel coding theorem – when your internet connection (or call quality in general) is bad, you end up having to speak slower. When I was hunting for a job in 2020, I remember doing badly in a few interviews because of the quality (or lack thereof) of the internet connection (this was before I had discovered that Google Meet performs badly on Safari).

Similarly, sometime last month, I had thought I had prepared well for what I thought was going to be a key conversation at work. The internet was bad, we couldn’t hear each other and kept repeating (redundancy is how you overcome the noise in the channel), and that diminished throughput massively. Given the added difficulty in communication, I didn’t bring up the key points I had prepared for. It was a damp squib.

Related to this is when you aren’t sure if the person you are speaking to can hear clearly. This disability again clouds the communication channel, meaning you need to build in redundancy, and thus a reduction in throughput.

When you are uncertain of yourself, or underconfident, you end up tending to do badly. That is because when you are uncertain, you aren’t sure if the other person will fully understand what you are going to say. Consequently, you end up talking slower, building redundancy in your speech, etc. You are more doubtful of what you are going to say, and don’t take risks, since your lack of confidence has clouded the “communication channel”, thus depressing your throughput.

Again a lot of this might apply to me alone – I function best when I’m talking / writing at a certain minimum throughput, and operating at anywhere below that makes me jittery and underconfident and a bad communicator. It is no surprise that my writing really took off once I got a computer of my own.

That was in the beginning of July 2004, and within a month, I had started (the predecessor of) this blog. I’ve been blogging for 19 years now.

That aside aside, the channel coding theorem works in non-verbal contexts as well. Back in 2016, before my daughter was born, I remember reading somewhere that tentative mothers lead to cranky babies. The theory was that if the mum was anxious or afraid while handling her baby, the baby wouldn’t perceive the signals of touch sufficiently, and being devoid of communication, become cranky.

We had seen a few examples of this among relatives and friends (and this possibly applies to me as well – my mother had told me that I was the first newborn she ever handled, and so she was a bit tentative in handling me). This again can be explained using the Channel Coding Theorem.

When the mother’s touch is tentative, it is as if the touchy channel between mother and child has some “noise”. The tentativeness of the touch means the baby is not really sure of what the mother is “saying”. With touch, unlike language or bits, redundancy is harder. And so the child goes up insufficiently connected to its mother.

Conversely, later on in life, these tentative mothers tend to bring in redundancy in their communications with their (now jittery) children, and end up holding them too hard, and not letting them go (and some of these children go to therapists, who inevitably blame it on the mothers 😛 ). Ultimately, all of this stems from the noise in the initial communication channel (thanks to the tentativeness of the source).

Ok I’ve rambled on here, so will stop now. However, now that I’ve seeded this thought in you, you too will start seeing the channel coding theorem everywhere (oh – if you think this post is badly written, then that is again like reading this over a noisy channel. And you will get irritated with the lack of throughput and pack).

Fragility of two-sided markets

Two-sided markets are inherently fragile for participation of each side depends on a certain degree of confidence in participation on the other side. Thus, small negative shocks can lead to quick downward spirals.

Following the ill-advised ban on Uber and other taxi aggregators in four Indian states (Delhi, Karnataka, Andhra Pradesh, Telangana), business for drivers who ply their services via such apps has dropped significantly. While on first inspection you might expect it to go to zero (given their services have been banned), the fact that enforcement is tough (there is nothing to identify a cab as “belonging to Uber”) means that apart from Delhi (where Uber has pulled its services) these cabs continue to ply.

In the days after the ban, various news reports have interviewed drivers who ply for Uber who complain about drastically reduced services. While numbers vary from report to report, the general sense is that so far the number of trips per driver per day has fallen by half. And I expect this to fall further unless drastic steps are taken – such as issuance of new regulations or removal of the ban.

In a “normal” market (where the owner of the market is also a participant), when demand for a particular good drops, price is expected to fall and availability is expected to increase. If demand for a particular item that you have in stock drops, you need to take steps to get rid of the excess inventory that you have. You are most likely to indulge in discounting or other such promotional activities, in order to make it more attractive for the buyers to buy, and thus take the inventory off your shelves.

In a “two-sided market” (one where the owner of the market is not a participant), however, things work differently. It is a popular saying that in such markets “demand creates its own supply”. A corollary to that is that “lack of demand creates lack of supply”. Let us take the case of Uber itself. Over the last few days, irrespective of whether the ban on the service is official or not, legal or not, the number of people who have been requesting for the service has dropped.

Now, if you are a driver using the app, you realise that your potential revenues and profits from continuing to use the app are not as high as they used to be. Thus, if there are other avenues for you to make money, you are now more likely to take those avenues rather than logging on to Uber (since the “hurdle rate” for such a switch is now lower thanks to lower Uber revenues). As many of you take the same route, the availability of cabs on Uber also drops – something that I’ve seen anecdotally over the last few days. And when availability of Uber cabs drops beyond a point, I start questioning my trust in the service – a week ago I would be confident that I would be able to hail an Uber from anywhere in Bangalore with very high confidence; that confidence has now dropped. And when my trust in the service drops, I start using it less, and when many of us do that, drivers see less demand and more of them pull away from the market. And this results in a vicious cycle.

Notice that things would work very differently had Uber been a “traditional” taxi service which owned its cabs and employed its drivers. In that case, falling demand would have been met with a response that would have made it easier for customers to buy – price cuts, perks, etc.

The point is that platforms or two-sided markets are inherently fragile, and highly dependent on confident in the system. I leave my car at home only if I have enough trust in the taxi platforms that I’ll be able to get a cab when I need one. A driver will forsake other trips and switch on his Uber app only if he is confident that he can get enough rides through the app.

The same network effects that can lead to a rapid ramp-up in two-sided markets can also lead to its downfall. All it takes is a small trigger that leads to loss of confidence in the service from one side. Unless that loss of confidence is quickly addressed, the “positive feedback” from it can quickly escalate and the market grinds to a halt!

Another good example of lack of confidence killing two-sided markets is in the market for CDOs and associated derivatives in 2007-08. There were standardised pricing models for such products and a vibrant market existed (between sophisticated financial institutions) in 2007. When house prices started coming down, some people started expressing doubts in such models. Soon, this led to massive loss of trust in the pricing models that underpinned such markets and people stopped trading. This meant companies were unable to mark their securities to market or rationalise their portfolios, and this led to the full-blown 2008 financial crisis!

So when you build a platform, you need to make sure that both sides of the market retain confidence in your platform. For in the platforms business loss of confidence can lead to a much quicker fall than in “traditional” markets. This dependence on confidence thus makes such markets fragile.

Eroding Trust in the Indian National Government

The latest issue of The Economist carries an article which talks about the “eroding trust in national governments”. This article is based on a poll conducted by Gallup in 2007 and again in 2012 with one simple question “do you trust your national government?”. World over, the proportion of people answering “yes” to this question has dipped significantly between the two years.

Now, this graph has been sorted by the orange dots (2012 data) so India is lost somewhere in the middle. What if, however, this graph were sorted by the 2007 numbers (white dots)? Notice that the white dot for India is very close to the 90% mark – the highest ever achieved among all countries surveyed in this poll!

This just goes to show the kind of confidence the Indian National Government (UPA-1) enjoyed back in 2007, perhaps a result of the populist schemes it had launched such as the NREGA. This was before any of the scams hit, and this goodwill might have resulted in the government getting voted back into power in 2009. The interesting thing, though, is that the number for India is still higher than that of a large number of OECD countries.

PS: I would have drawn this graph differently. Rather than using a scatter-plot like this, I would have rather used a slope-graph, which would have shown the relative standings in both years and also the way the ratings have moved.

“But she is a really nice person”

That is the reply I usually get when I tell someone that someone else is dumb, or is an imbecile or is boring. And now I think I have some insight into why people who are otherwise idiots or irritating or boring are also extremely nice people, with “big hearts”.

Basically I’ve found that whenever I’m low on confidence or self esteem I end up being more sensitive, both with respect to myself and others. I display greater empathy, I care more about how people would feel and react to things I would do, and my usual buffalo skin disappears and I get affected by any adverse comments or remarks or incidents. Actually, I’ve seen a two-way implication here, but again you need to remember that I’m extrapolating from one data point here. Back in 2001, I had received extensive feedback (from various parties) that I had become too arrogant and self-centered, and that I needed to make an effort to be nicer and more sensitive towards people. I did make that effort, too successfully I think, for though I consequently became more popular, I entered into a prolonged period of low self esteem. Anyway, I digress.

So, based on the one strong data point that I have, which is myself, I hypothesize that low self esteem leads to greater empathy. People who you are likely to normally consider to be “boring” or “stupid” are likely to know that people think of them as that, and are consequently more likely to have low self esteem. And going by my hypothesis, that means they are more sensitive, have greater empathy, and “have big hearts”. And so, the remark “but she is a really nice person” in the context I mentioned largely holds true.

Fighterization of food

One of the topics that I’d introduced on my blog not so long ago was “fighterization“. The funda was basically about how professions that are inherently stud are “fighterzied” so that a larger number of people can participate in it, and a larger number of people can be served. In the original post, I had written about how strategy consulting has completely changed based on fighterization.

After that, I pointed out about how processes are set – my hypothesis being that the “process” is something that some stud would have followed, and which some people liked because of which it became a process. And more recently, I wrote about the fighterization of Carnatic music, which is an exception to the general rule. Classical music has not been fighterized so as to enable more people to participate, or to serve a larger market. It has naturally evolved this way.

And even more recently, I had talked about how “stud instructions” (which are looser, and more ‘principles based’) are inherently different from “fighter instructions” (which are basically a set of rules). Ravi, in a comment on Mohit‘s google reader shared items, said it’s like rule-based versus principles-based regulation.

Today I was reading this Vir Sanghvi piece on Lucknowi cuisine, which among other things talks about the fact that it is pulao that is made in Lucknow, and now biryani; and about the general declining standards at the Taj Lucknow. However, the part that caught my eye, which has resulted in this post with an ultra-long introduction was this statement:

The secret of good Lucknowi cooking, he said, is not the recipe. It is the hand. A chef has to know when to add what and depending on the water, the quality of the meat etc, it’s never exactly the same process. A great chef will have the confidence to improvise and to extract the maximum flavour from the ingredients.

This basically states that high-end cooking is basically a stud process. That the top chefs are studs, and can adapt their cooking and methods and styles to the ingredients and the atmosphere in order to churn out the best possible product.You might notice that most good cooks are this way. There is some bit of randomness or flexibility in the process that allows them to give out a superior product. And a possible reason why they may not be willing to give out their recipes even if they are not worried about their copyright is that the process of cooking is a stud process, and is hence not easily explained.

Publishing recipes is the attempt at fighterization of cooking. Each step is laid down in stone. Each ingredient needs to be exactly measured (apart from salt which is usually “to taste”). Each part of the process needs to be followed properly in the correct order. And if you do everything perfectly, you will get the perfect standardized product.

Confession time. I’ve been in Gurgaon for 8 months and have yet to go to Old Delhi to eat (maybe I should make amends this saturday. if you want to join me, or in fact lead me, leave a comment). The only choley-bhature that I’ve had has been at Haldiram’s. And however well they attempt to make it, all they can churn out is the standardized “perfect” product. The “magic” that is supposed to be there in the food of Old Delhi is nowhere to be seen.

Taking an example close to home, my mother’s cooking can be broadly classified into two. One is the stuff that she has learnt from watching her mother and sisters cook. And she is great at making all of these – Bisibelebhath and masala dosa being her trademark dishes (most guests usually ask her to make one of these whenever we invite them home for a meal). She has learnt to make these things by watching. By trying and erring. And putting her personal touch to it. And she makes them really well.

On the other hand, there are these things that she makes by looking at recipes published in Women’s Era. Usually she messes them up. When she doesn’t, it’s standardized fare. She has learnt to cook them by a fighter process. Though I must mention that the closer the “special dish” is to traditional Kannadiga cooking (which she specializes in), the better it turns out.

Another example close to home. My own cooking. Certain things I’ve learnt to make by watching my mother cook. Certain other things I’ve learnt from this cookbook that my parents wrote for me before I went to England four years ago. And the quality of the stuff that I make, the taste in either case, etc. is markedly different.

So much about food. Coming to work, my day job involves fighterization too. Stock trading is supposed to be a stud process. And by trying to implement algorithmic trading, my company is trying to fighterize it. The company is not willing to take any half-measures in fighterization, so it is recruiting the ultimate fighter of ’em all – the computer – and teaching it to trade.

Preliminary reading on studs and fighters theory:

Studs and Fighters

Extending the studs and fighters theory

Cribbing

When this blog was young, I used to crib a lot. Ok let me correct that. When my livejournal, which is the predecessor of this blog, was young, I used to crib a lot. At least half my posts were “crib posts”. They would go on the lines of “oh i’m feeling so crappy. everything’s awful with the world”. I’d occasionally get comments. They’d either be of the “yeah you’ve done wrong” variety or “ok i empathise” type. Most such comments didn’t get any posts at all. Sorry, I meant that most such posts didn’t get any comments at all.

I decided to obey the market and moved away from crib posts. i still do crib once in a while, and use this blog as a personal rant, i don’t crib here as much as I used to. This “adjustment to the needs of the market” has had its own problem. It sometimes makes yo ugo to the other extreme. Where you are just not able to crib at all. You feel guilty about cribbing. Everytime you crib, you think yo uare bothering someone and so you should stop. You stop.

Cribbing is an art. Not everyone is a master of cribbing. The biggest problem with the lack of ability to crib is what I call as the “two-person theory”. It is something like each of us is a superposition of several people. And at each point in time, we “collapse” to one of those people. All of us live in the form of a dynamic equilibrium as me. We share a hard disk, but we don’t share memory. And usually, there is no coexistence. Ok I suppose the name two-person theory is some kind of a misnomer. It is actually the trivial case where you are made of a superposition of just two people.

When you are low, you want to crib. You want to pour out all your woes to the world. You want to cry. You want to be cared for. You want to be cuddled. But you can’t speak. You don’t have the confidence to speak. You just don’t feel like speaking. Speaking is an effort. And you end up not cribbing, though you really wanted to crib.

You recover. And you are now not who you were when you were low. You are in a different state (no, this is not like going from Delhi to Haryana). You are able to communicate now. You remember that you needed to crib. That much has been coded into the hard disk. However, the details of that were left in the memory of the other you. You don’t know what to crib about. You don’t understand the other yourself. You trivialize your body-sharer. You decide you are better off not cribbing. And you are happy that you didn’t feel. Each time you think about it, you feel happeier that you didn’t crib; until you want to crib again, and are too low to communicate.

Now you blame the other you for not speaking out for you. Effectively you blame yourself. You feed in into the downward spiral. You want to crib even more now. And you can’t communicate. You wait till you get better. And then you wait till you get worse. You cycle. You oscillate.

One of you is dumb and cna’t communicate. The other of you can’t understand the other of you. This other doesn’t want to speak for the other other. One of the others is unable to communicate. And like the mythical Bherunda birds (also the state bird of Karnataka) one of the other consumes poison, taking the other other down with him.

The Enhanced DTPH Theory

Dil To Pagal Hai (DTPH) was a nice movie. I really enjoyed it when I first saw it some eleven years ago. The only problem was the message it imprinted on my 15-year-old mind: someone, somewhere is made for you. It ended up completely messing me up for the next 3-4 years. I took it at face value. Every time I met a new girl, I would start asking the question “is she the someone, somewhere who is made for me?”. I would be lucky if the answer was an immediate no.

As I had explained in a blog post almost three years ago, these kind of questions never give “yes” as an answer. They either say “no” or they say “maybe”. And the maybes are a problem, since a few years down the line they might be converted to a “no”. They will never turn into a “yes”, mind you, and if you are forced to make a decision, you’ll have to make do with the number of occurrences of maybe and the confidence bounds that it produces. So the maybes were a problem for me 10 years ago. I didn’t know how to handle them. And in the one case where the answer consistently came out to be “maybe” (even when i ask the question now, it comes out to be “maybe”), I royally messed up the blade. Disaster was an overstatement.

Now that the digression is done, DTPH similarly messed up thousands of young minds all over the country. It didn’t even spare the married. Everyone started asking themselves the question “is this the someone somewhere that is made for me?” I think Yash Chopra (he directed it, didn’t he?) should shoulder a large part of the blame for the spurt in suicides in the late 90s.

The theory I’m going to state now was first stated by Neha a couple of years back. Back then, I’d thought I need to blog it, since she wasn’t blogging then. She has started blogging recently, but still I think I’ll write about this. As you might have figured out from the title, I call this the Enhanced DTPH Theory. It is quite ironical that this is coming from the fairly irreligious me, since it somewhat endorses creationism. I know the inherent contradictions here, but I think I should write it anyway.

The theory states that there are several people, in several places, who are “made for you” (if you are religious) or are “inherently compatible with you” (if you are not). The key is in finding at least one of them and making things work.

I think this is easier on people’s minds. The constant quest to find “the best partner” should be laid to rest, I think, mainly because it is unlikely that you’ll find a “dominating partner” (someone who is better, in your eyes, than everyone else that you could’ve gotten married to). Instead, what you will get is what I can call as a “dominating set” – a set of people who are collectively dominating over the rest of the population but cannot really be compared to each other.

Each person has his/her own different evaluation criteria. And based on that, each person has his/her own dominating set. And it is this dominating set that is the “several people who are made for you”. I suppose you are getting the drift. I know this is a bit confusing.

Then, you need to understand that the universe doesn’t obey the Hall’s Marriage Theorem. This is trivial to prove since the total number of men exceeds the total number of women. Actually, as a corollary to this, we can establish that the original DTPH theory is false, unless of course it assumes that the population of gays is significantly higher than the population of lesbians, or if it takes into account animal sex.

Some hand-waving here, but my next hypothesis is that Hall’s Theorem doesn’t hold for local smaller populations also. I’ll probably try give an explanation of this in a subsequent post (else there would be no reason for women to remain single).

Tailpiece: The cost of not marrying the “right person” is significantly lower than the cost of marrying the wrong person.

PS: I also acknowledge Baada’s contribution to the development of this theory.