Channel Coding Theorem in Real Life

One of my favourite concepts in Computer Science is Shannon’s Channel Coding Theorem. This theorem is basically about the efficiency of communication over a noisy channel. And as I was thinking a few minutes back, this has interesting implications in real life as well, well away from the theory of communication.

I don’t have that much understanding of the rigorous explanation of the theorem. However, I absolutely love the central idea of it – that the noisier a channel is, the more the redundancy you need in your communication, and thus the slower is your communication. A corollary of this is that every channel has a “natural maximum speed”, and as long as you try to communicate within that speed, you can communicate reliably.

I won’t go into the technical details here – that involves assuming that the channel loses (or garbles) X% of bits, and then constructing a redundant code that shows that even with this loss, you can communicate effectively.

Anyway, let’s leave behind the theory communication and go on to real life.

I’ve found that I communicate badly when I’m not sure what language to talk in. If I’m talking in English with someone who I know knows good English, I communicate rather well (like my writing 😛 ) . However, if I’m not sure about the quality of language of the other person, I hesitate. I try to force myself to find simpler / more obvious words, and that disturbs my flow of thought, and I stammer.

Similarly, when I’m not sure whether to talk in Kannada or English (the two languages I’m very comfortable in), I stammer heavily. Again, because I’m not sure if the words I would naturally use will be understood by the other person (the counterparty’s comprehension being the “noise in the channel” here), I slow down, get jittery, and speak badly.

Then of course, there is the very literal interpretation of the channel coding theorem – when your internet connection (or call quality in general) is bad, you end up having to speak slower. When I was hunting for a job in 2020, I remember doing badly in a few interviews because of the quality (or lack thereof) of the internet connection (this was before I had discovered that Google Meet performs badly on Safari).

Similarly, sometime last month, I had thought I had prepared well for what I thought was going to be a key conversation at work. The internet was bad, we couldn’t hear each other and  kept repeating (redundancy is how you overcome the noise in the channel), and that diminished throughput massively. Given the added difficulty in communication, I didn’t bring up the key points I had prepared for. It was a damp squib.

Related to this is when you aren’t sure if the person you are speaking to can hear clearly. This disability again clouds the communication channel, meaning you need to build in redundancy, and thus a reduction in throughput.

When you are uncertain of yourself, or underconfident, you end up tending to do badly. That is because when you are uncertain, you aren’t sure if the other person will fully understand what you are going to say. Consequently, you end up talking slower, building redundancy in your speech, etc. You are more doubtful of what you are going to say, and don’t take risks, since your lack of confidence has clouded the “communication channel”, thus depressing your throughput.

Again a lot of this might apply to me alone – I function best when I’m talking / writing at a certain minimum throughput, and operating at anywhere below that makes me jittery and underconfident and a bad communicator. It is no surprise that my writing really took off once I got a computer of my own.

That was in the beginning of July 2004, and within a month, I had started (the predecessor of) this blog. I’ve been blogging for 19 years now.

That aside aside, the channel coding  theorem works in non-verbal contexts as well. Back in 2016, before my daughter was born, I remember reading somewhere that tentative mothers lead to cranky babies. The theory was that if the mum was anxious or afraid while handling her baby, the baby wouldn’t perceive the signals of touch sufficiently, and being devoid of communication, become cranky.

We had seen a few examples of this among relatives and friends (and this possibly applies to me as well – my mother had told me that I was the first newborn she ever handled, and so she was a bit tentative in handling me). This again can be explained using the Channel Coding Theorem.

When the mother’s touch is tentative, it is as if the touchy channel between mother and child has some “noise”. The tentativeness of the touch means the baby is not really sure of what the mother is “saying”. With touch, unlike language or bits, redundancy is harder. And so the child goes up insufficiently connected to its mother.

Conversely, later on in life, these tentative mothers tend to bring in redundancy in their communications with their (now jittery) children, and end up holding them too hard, and not letting them go (and some of these children go to therapists, who inevitably blame it on the mothers 😛 ). Ultimately, all of this stems from the noise in the initial communication channel (thanks to the tentativeness of the source).

Ok I’ve rambled on here, so will stop now. However, now that I’ve seeded this thought in you, you too will start seeing the channel coding theorem everywhere (oh – if you think this post is badly written, then that is again like reading this over a noisy channel. And you will get irritated with the lack of throughput and pack).

More on Diversity and Inclusion

Diversity and Inclusion are words that are normally thrown around by people of a certain persuasion. In fact, they were among the key principles espoused by one of my earlier employers as well (to their credit, some of their diversity and inclusion sessions did a lot of help broaden my worldview).

However, as I’ve argued earlier on this blog, in a lot of cases, arguments on diversity and inclusion are (literally) only skin deep – people go big on diversity of sex, sexuality, skin colour, nationality and so on while giving short shrift to things like diversity of thought, which in my opinion plays a larger role in building a more successful team.

I’ve also mentioned earlier on this blog about how some simple acts of inclusion can go a long way – for example, I’d mentioned about how building a pedestrian walkway, or pedestrian crossing with signals, would help make one of the roads in Bangalore more inclusive towards pedestrians (a class of people the usual proponents of diversity and inclusion don’t care about).

I was reminded of diversity and inclusion when the recent hoopla about messaging apps happened. A number of my contacts said they were leaving WhatsApp and moving to Telegram or Signal. Others said they weren’t going anywhere and were sticking to WhatsApp, and that Facebook’s new privacy rules were nothing new.

From my personal point of view, since I didn’t have a view on this messaging apps issue, the best solution turned out to be “inclusion”.

I’m on all apps. I’m on Signal, and Telegram, and WhatsApp, and iMessage, and good old SMS. However you choose to reach me, I’m there to receive your message and respond to you. In that sense, when you don’t have a strong opinion, the best thing to do is to be inclusive.

Of late I’ve realised it’s the same with language. Since I now work for a company that is headquartered in Gurgaon, a number of colleagues instinctively speak in Hindi. Initially I used to be a bit snobbish, and tell them that my Hindi sucked, and when they spoke Hindi, I would reply in English.

Over time, however, I’ve realised that I’m only being an asshole by refusing to be inclusive. Since I know Hindi (I got more marks in Hindi in Class 10 board exams than I did in English – not that that says anything), I should let the people decide whether I’m worth talking to in Hindi at all. I’ll talk to them in my broken Hindi, and if they think it’s too broken they can choose to switch to a language I’m more comfortable in.

And a week ago, Pranay and Saurabh of the Puliyabaazi podcast asked me if I’m willing to go on their (Hindi) podcast to talk about logical fallacies and “how not to use data”. I immediately accepted, not only because it’s a great podcast to be on (they’re fun to talk to), but it also gives me an opportunity to show off my broken Hindi.

The episode dropped on Thursday. You can listen to it here:

I realised while I was recording that my Hindi has become really rusty, and I found myself struggling for words many times. I also realised after the episode dropped that I don’t even understand what the title means, yet I’ve been happily sharing it around in my office! (a colleague kept asking me if I knew this word and that word, and I realised the answer to all that was no. Yet I had made assumptions and gone on with the podcast – another example of my own “inclusiveness”!)

Henceforth I’m never telling a colleague that I don’t know Hindi. However, if I find that someone overestimates my level of Hindi I might inflict this podcast on them. Even then, if they choose to speak to me in Hindi, so be it! I’m going to make an attempt to be more inclusive, after all.

 

Breaking up sentences in the absence of punctuation

From time to time, a joke goes around that makes the value of punctuation clear. Check out this picture, for example.

Recently, I saw this on my twitter timeline (though here it’s an issue of spacing apart from punctuation).

Someone actually wrote an entire book about the value of punctuation.

In any case, I have a pretty bad track record in terms of reading sentences that don’t have punctuation. I can think of two examples right away.

Firstly, my school diary was filled with quotes from Sri Aurobindo and “The Mother“. By turns, we would have to say “thought for the day” in the school assembly. And we had a reliable way of finding such thoughts – just look in the diary and spout out the nuggets.

One of those went:

Always do what you know to be the best even if it the most difficult thing to do.

Yes, I remember that. Because I had spouted this not once but twice. Now, this is a long sentence without any punctuation. How would you read it?

For the longest time I read it like this.

Always do what you know, to be the best, even if it is the most difficult thing to do. 

So if there are many things that you can do, and you know one of the things, you do that thing even if it is harder than everything else that you might do (but don’t know how to do).

Clearly that doesn’t make that much sense. It was only when I was about to graduate that I figured that it was actually:

Always do what you know to be the best even if it is the most difficult thing to do. 

So there are many things you can do. You know one of them is the “best thing to do”. So even if it is the most difficult thing to do, you do it because you know it is “the best” (not because you “know it the best”).

Another example is from this store near my house. I don’t think the store existed for too long, but it had an interesting and quirky name (in Kannada).

ellAdEvarakrupe stores“, it said. For the longest time I read it as “ellAdEvara  krupe”, or “grace of all gods”. And I thought it was a fascinating name in terms of recognising all religions. And I’ve quoted it many times.

When I was quoting this on Twitter earlier today, I realised that I had got the name of the store all wrong. It’s “ellA dEvarakrupe”, meaning “everything is god’s grace”. It says nothing about which gods are included or excluded, or how many gods there are.

What are your favourite examples of sentences that you’ve misread thanks to the lack of punctuation or other visible sentence markers?

 

The difficulty of song translation

One of my wife’s favourite nursery rhymes is this song that is sung to the tune of “for he’s a jolly good fellow”, and about a bear going up a mountain.

For a long time I only knew of the Kannada version of this song (which is what the wife used to sing), but a year or two back, I found the “original” English version as well.

And that was a revelation, for the lyrics in the English version make a lot more sense. They go:

The bear went over the mountain;
The bear went over the mountain
The bear went over the mountain, to see what he could see.
And all that he could see, and all that he could see
Was the other side of the mountain,
The other side of the mountain
The other side of the mountain, was all that he could see.

Now, the Kannada version, sung to the same tune, obviously goes “???? ??????? ??????” (karaDi beTTakke hoithu). That part has been well translated. However, the entire stanza hasn’t been translated properly, because of which the song goes a bit meaningless.

The lyrics, when compared to the original English version, are rather tame. Since a large part of my readership don’t understand Kannada, here is my translation of the lyrics (btw, the lyrics used in these YouTube versions are different from the lyrics that my wife sings, but both are similar):

The bear went to the mountain.
The bear went to the mountain.
The bear went to the mountain.
To see the scenery

And what did it see?
What did it see?
The other side of the mountain.
The other side of the mountain.
It saw the scenery of the other side of the mountain.

Now, notice the important difference in the two versions, which massively changes the nature of the song. The Kannada version simply skips the “all that he could see” part, which I think is critical to the story.

The English version, in a way, makes fun of the bear, talking about how it went over the mountain thinking it’s a massive task, but “all that he could see” from there was merely the other side of the mountain. This particular element is missing in Kannada – there is nothing in the lyrics that suggests that the bear’s effort to climb the mountain was a bit of a damp squib.

And that,  I think, is due to the difficulty of translating songs. When you translate a song, you need to get the same letter and spirit of the lyrics, while making sure they can follow the already-set music as well (and even get the rhyming right). And unless highly skilled bilingual poets are involved, this kind of a translation is really difficult.

So you get half-baked translations, like the bear story, which possibly captures the content of the story but completely ignores its spirit.

After I had listened to the original English version, I’ve stopped listening to the Kannada version of the bear-mountain song. Except when the wife sings it, of course.

 

Gults and Grammar

Back in IIT, it was common to make fun of people from Andhra Pradesh for their poor command over the English language. It was a consequence of the fact that JEE coaching is far more institutionalised in that (undivided) state, because of which people come to IIT from less privileged backgrounds (on average) than their counterparts in Karnataka or Tamil Nadu or Maharashtra.

Now, in hindsight, making fun of people’s English doesn’t sound particularly nice, but sometimes stories come up that make it incredibly hard to resist.

This one is from Matt Levine’s newsletter. And it is about an insider trading ring. This is a quote that Levine has quoted in his newsletter (pay attention to the names):

According to the SEC’s complaint, Janardhan Nellore, a former IT administrator then at Palo Alto Networks Inc., was at the center of the trading ring, using his IT credentials and work contacts to obtain highly confidential information about his employer’s quarterly earnings and financial performance. As alleged in the complaint, until he was terminated earlier this year, Nellore traded Palo Alto Networks securities based on the confidential information or tipped his friends, Sivannarayana Barama, Ganapathi Kunadharaju, Saber Hussain, and Prasad Malempati, who also traded.

The SEC’s complaint alleges that the defendants sought to evade detection, with Nellore insisting that the ring use the code word “baby” in texts and emails to refer to his employer’s stock, and advising they “exit baby,” or “enter few baby.” The complaint also alleges that certain traders kicked back trading profits to Nellore in small cash transactions intended to avoid bank scrutiny and reporting requirements. After the FBI interviewed Nellore about the trading in May, he purchased one-way tickets to India for himself and his family and was arrested at the airport.

You can look at Levine’s newsletter to understand his take on the story (it’s towards the bottom), but what catches my eye is the grammar. I think it is all fine to refer to the insider-traded stock as a “baby”, but at least be grammatically correct about it!

“Enter few baby” is so obviously grammatically incorrect (it’s hard to even be a typo) that when intercepted by someone like the SEC, it would immediately send alarm bells ringing. Which is what I suppose possibly happened.

So my take on this case is – don’t insider  trade, but even if you do, be grammatical about your signals. If you’re so obviously grammatically wrong, it is easy for whoever intercepts your chats to know you’re up to something fishy.

But then if you’re gult..

Good vodka and bad chicken

When I studied Artificial Intelligence, back in 2002, neural networks weren’t a thing. The limited compute capacity and storage available at that point in time meant that most artificial intelligence consisted of what is called “rule based methods”.

And as part of the course we learnt about machine translation, and the difficulty of getting the implicit meaning across. The favourite example by computer scientists in that time was the story of how some scientists translated “the spirit is willing but the flesh is weak” into Russian using an English-Russian translation software, and then converted it back into English using a Russian-English translation software.

The result was “the vodka is excellent but the chicken is not good”.

While this joke may not be valid any more thanks to the advances in machine translation, aided by big data and neural networks, the issue of translation is useful in other contexts.

Firstly, speaking in a language that is not your “technical first language” makes you eschew jargon. If you have been struggling to get rid of jargon from your professional vocabulary, one way to get around it is to speak more in your native language (which, if you’re Indian, is unlikely to be your technical first language). Devoid of the idioms and acronyms that you normally fill your official conversation with, you are forced to think, and this practice of talking technical stuff in a non-usual language will help you cut your jargon.

There is another use case for using non-standard languages – dealing with extremely verbose prose. A number of commentators, a large number of whom are rather well-reputed, have this habit of filling their columns with flowery language, GRE words, repetition and rhetoric. While there is usually some useful content in these columns, it gets lost in the language and idioms and other things that would make the columnist’s high school English teacher happy.

I suggest that these columns be given the spirit-flesh treatment. Translate them into a non-English language, get rid of redundancies in sentences and then  translate them back into English. This process, if the translators are good at producing simple language, will remove the bluster and make the column much more readable.

Speaking in a non-standard language can also make you get out of your comfort zone and think harder. Earlier this week, I spent two hours recording a podcast in Hindi on cricket analytics. My Hindi is so bad that I usually think in Kannada or English and then translate the sentence “live” in my head. And as you can hear, I sometimes struggle for words. Anyway here is the thing. Listen to this if you can bear to hear my Hindi for over an hour.

The Comeback of Lakshmi

A few months back I stumbled upon this dataset of all voters registered in Bangalore. A quick scraping script followed by a run later, I had the names and addresses and voter IDs of all voters registered to vote in Bangalore in the state assembly elections held this way.

As you can imagine, this is a fantastic dataset on which we can do the proverbial “gymnastics”. To start with, I’m using it to analyse names in the city, something like what Hariba did with Delhi names. I’ll start by looking at the most common names, and by age.

Now, extracting first names from a dataset of mostly south indian names, since South Indians are quite likely to use initials, and place them before their given names (for example, when in India, I most commonly write my name as “S Karthik”). I decided to treat all words of length 1 or 2 as initials (thus missing out on the “Om”s), and assume that the first word in the name of length 3 or greater is the given name (again ignoring those who put their family names first, or those that have expanded initials in the voter set).

The most common male first name in Bangalore, not surprisingly, is Mohammed, borne by 1.5% of all male registered voters in the city. This is followed by Syed, Venkatesh, Ramesh and Suresh. You might be surprised that Manjunath doesn’t make the list. This is a quirk of the way I’ve analysed the data – I’ve taken spellings as given and not tried to group names by alternate spellings.

And as it happens, Manjunatha is in sixth place, while Manjunath is in 8th, and if we were to consider the two as the same name, they would comfortably outnumber the Mohammeds! So the “Uber driver Manjunath(a)” stereotype is fairly well-founded.

Coming to the women, the most common name is Lakshmi, with about 1.55% of all women registered to vote having that name. Lakshmi is closely followed by Manjula (1.5%), with Geetha, Lakshmamma and Jayamma coming some way behind (all less than 1%) but taking the next three spots.

Where it gets interesting is if we were to look at the most common first name by age – see these tables.

 

 

 

 

 

 

Among men, it’s interesting to note that among the younger age group (18-39, with exception of 35) and older age group (57+), Muslim names are the most common, while the intermediate range of 40-56 seeing Hindu names such as Venkatesh and Ramesh dominating (if we assume Manjunath and Manjunatha are the same, the combined name comes top in the entire 26-42 age group).

I find the pattern of most common women’s names more interesting. It is interesting to note that the -amma suffix seems to have been done away with over the years (suffixes will be analysed in a separate post), with Lakshmamma turning into Lakshmi, for example.

It is also interesting to note that for a long period of time (women currently aged 30-43), Lakshmi went out of fashion, with Manjula taking over as the most common name! And then the trend reversed, as we see that the most common name among 24-29 year old women in Lakshmi again! And that seems to have gone out of fashion once again, with “modern names” such as Divya, Kavya and Pooja taking over! Check out these graphs to see the trends.

(I’ve assumed Manjunath and Manjunatha are the same for this graph)

So what explains Manjunath and Manjula being so incredibly popular in a certain age range, but quickly falling away on both sides? Maybe there was a lot of fog (manju) over Bangalore for a few years? 😛

The one bit machine

My daughter is two weeks old today and she continues to be a “one bit machine”. The extent of her outward communication is restricted to a maximum of one bit of information. There are basically two states her outward communication can fall under – “cry” and “not cry”, and given that the two are not equally probable, the amount of information she gives out is strictly less than one bit.

I had planned to write this post two weeks back, the day she was born, and wanted to speculate how long it would take for her to expand her repertoire of communication and provide us with more information on what she wants. Two weeks in, I hereby report that the complexity of communication hasn’t improved.

Soon (I don’t know how soon) I expect her to start providing us more information – maybe there will be one kind of cry when she’s hungry, and another when she wants her diaper changed. Maybe she’ll start displaying other methods of outward communication – using her facial muscles, for example (right now, while she contorts her face in a zillion ways, there is absolutely no information conveyed), and we can figure out with greater certainty what she wants to convey.

I’m thinking about drawing a graph with age of the person on the X axis, and the complexity of outward information on the Y axis. It starts off with X = 0 and Y = 1 (I haven’t bothered measuring the frequency of cry/no-cry responses so let’s assume it’s equiprobable and she conveys one bit). It goes on to X = 14 days and Y = 1 (today’s state). And then increases with time (I’m hoping).

While I’m sure research exists some place on the information content per syllable in adult communication, I hope to draw this graph sometime based on personal observation of my specimen (though that would limit it to one data point).

Right now, though, I speculate what kind of shape this graph might take. Considering it has so far failed to take off at all, I hope that it’ll be either an exponential (short-term good but long-term I don’t know ) or a sigmoid (more likely I’d think).

Let’s wait and see.

Languages as memes

A while back on this blog I had compared religious and cultural practices to memes (in the original Richard Dawkins sense of the word). Back then I had written:

So if you were to look at it in terms of responsibility to society, you need to propagate only those cultural traits that you deem to be relevant and important. “So what if everyone stops celebrating Ganesh Chaturthi?” you may ask. If that would happen that would simply mean a vote of no confidence for the festival and an indication that the festival needs to be phased out. If everyone were to propagate only those cultural traits they find useful, traits that a significant proportion of society finds significant will continue to survive and thrive. For Ganesh Chaturthi to exist 30 years hence, it isn’t necessary for ALL families that have inherited it to celebrate it now. As long as a critical mass of families celebrate it, the festival will survive. If not, it probably doesn’t need to exist.

 

Now, thinking about it, you can consider language to also be a meme. When a bunch of you find that there is a concept for which the language you speak in has no word, you invent a word and add it to the language (this is like a genetic mutation). If enough people like this mutation (i.e. if it is “fit”) it will propagate, and soon become part of the language.

If there is a word in the language that is archaic and not useful for describing any of the phenomena that you are likely to encounter, you stop using it. When people stop using such words, they become “archaic” (ok I see circular reasoning in this paragraph) and effectively drop out of the language. Thus, a living language is always dynamic, receptive to new words (to describe concepts that earlier didn’t need description) and receptive to discarding words that are not useful any more. Thus, the feature that defines a living language is dynamism and change.

This has several policy implications.

1. The concept of “purity” of language is wrong. Some people want to speak in the “pure form” of a language. As long as it is a language that has been truly alive (and not kept alive mostly by ancient literature) there exists no “pure form”, for the definition of a successful language involves frequent “mutations”. So if you ask me to talk in “pure Kannada” it is nonsense. Pure Sanskrit, on the other hand, has some meaning, for the language has been so little used that it’s stopped evolving and mutating.

2. People like to appoint themselves guardians of culture and dictate top-down what words should be part of a particular language. For example, there exists a body under the Government of Karnataka (if I’m not wrong) which dictates what “Kannada words” must be used for different new concepts. This is wrong, and a recipe for such words not being used.

Instead, “memetics” must be respected and evolution must be bottom up. People find the need to describe phenomena around themselves and if they don’t find a word in their language that describes it, they will either invent or borrow one such word. Some such new words become widely used, at which point of time they can be introduced into the language dictionary. Usage should precede presence in the dictionary, not the other way round.

3. “Slang” is a part of language, and a leading indicator of how the language is going to evolve. It should be encouraged and not denounced. For it exists because the language as it stands now cannot effectively enough describe certain concepts.

I’m currently reading this book called The Information by James Gleick, which has a chapter or two dedicated to languages and dictionaries. It was while reading it that I realised how languages are memes.

 

Switching languages

I used to marvel about how whenever I was in the company of other people from IIT Madras, I would instinctively switch to speaking “IITese“. Words such as “slisha”, “peace”, “rod”, and all others that I would not normally use in normal English when speaking to normal people would suddenly appear in my vocabulary while talking to others from IITM.

I used to consider myself special that I could discriminate thus, and make best use of the languages I know while not discriminating against people who didn’t understand one of the languages, such as IITese. I used to consider this great, but this bubble got broken when my nephew started talking.

This guy is half-Kannadiga, half-Marathi, with a Gult nanny and his parents speak to each other in Hindi. He is now three years old and for over a year now he’s been very comfortable speaking Kannada and Marathi, and to an extent Telugu, Hindi and English (which he’s learning in school) !  The most remarkable thing with him, though, (as with all other multilingual kids, I would imagine) is that he has mapped people to languages. For example, he knows that I speak Kannada and he speaks to me only in Kannada. And while talking to me if his father (who is Marathi) is present, he immediately switches to Marathi to talk to him. Across languages that are very different, he is able to switch easily and seamlessly and moreover know who speaks which language!

There is a downside, though. Once when his mother, who is “supposed to speak to him in Kannada”, tried talking to him in Marathi, he got really angry and wild and asked her to speak in Kannada! Our initial thought was he was being finicky, but I now think it is to do with parsing. When his mother speaks, he has his “Kannada parser” switched on, and if she speaks Marathi, there is a parsing error and it causes great stress on his processor to switch languages. And being a small kid, that makes him cranky and wild!

In other words, this can be considered as another case of Bayesian recognition! It seems like the human mind’s parsing of speech is influenced by the prior distribution of what language the speaker is speaking in. As the first few words come out, we firm up which parser to use, and then it is smooth sailing. For a kid, though, it seems like the prior distribution of parsers is “binary” (one 1, and the rest 0s), which is what makes the wrong speaker wrong language combo annoying for them!

Us human beings are smarter than we think!