Tag: hypotheses

The Bangalore Advantage

Last night, Pinky and I had this long conversation discussing aunts and uncles and why certain aunts and uncles were “cooler” or “more modern” compared to other aunts or uncles. I put forward my theory that in every family there is one particular generation with a large generation gap, and while in families like mine or Pinky’s this large gap occurred at our generation, these “cooler” aunts’ and uncles’ families had the large gap one generation earlier. Of course, this didn’t go far in explaining why the gap was so large in that generation in the first place.

Then Pinky came up with this hypothesis backed by data that was hard to refute, and the rest of the conversation simply went in both of us trying to confirm the hypotheses. Most of these “cool” aunts and uncles, Pinky pointed out, had spent most of their growing up years in Bangalore, and this set them apart from the more traditional relatives, who spent at least a part of their teens outside the city. The correlation was impeccable, and in an effort to avoid the oldest mistake in statistics, we sought to identify reasons that might explain this difference.

While some of the more “traditional” relatives had grown up in villages, we discovered that a large number of them had actually gone to high school/college in rather large but second-tier towns of Karnataka (this includes Mysore). So the rural-urban angle was out. Of course Bangalore was so much larger than these other towns so size alone might have been enough to account for the difference, but the rather large gap in worldviews between those that grew up in Bangalore, and those that grew up in Mysore (which, then, wasn’t so much smaller), and the rather small gap between the Mysoreans and those that grew up in small towns (like Shimoga or Bhadravati) meant that this big-city hypothesis was unfounded.

We then started talking about the kind of advantages that Bangalore (specifically) offered over other towns of Karnataka, and the real reason was soon staring us in the face. Compared to any other town in Karnataka (then, and now), Bangalore was significantly more cosmopolitan. I’ve spoken on this blog before about Bangalore having been two cities (I’ve put the LJ link rather than the NED link so that you can enjoy the comments) but the important thing was that after independence and the Britishers’ flight, the two cities got combined into one big heterogeneous city.

Relatives growing up in Mysore or Shimoga typically went to college with people from large similar backgrounds. Everyone there spoke Kannada, and the dominance of Brahmins in those towns was so overwhelming that these relatives could get through their college lives hanging out solely with other people from largely similar family backgrounds. This meant there was no new “cultural education” that college offered, and the same world views that had been prevalent in these peoples’ homes while they were growing up persisted.

It was rather different for people who grew up in Bangalore. Firstly, people from East Bangalore didn’t speak Kannada (at least, not particularly fluently), which meant English was the lingua franca. More importantly, there was greater religious, casteist and cultural diversity in the classroom, which made it so much more likely for people to interact and make friends with classmates from backgrounds rather different from one’s own. Back in those days of extreme cultural conservatism, this simple exposure to other cultures was invaluable in changing one’s world view and making one more liberal.

It is in the teens that one’s cultural norms are shaped, and exposure to different cultures at that age is critical to formation of one’s world-view. In our generation, this difference has probably played out in the kind of schools one goes to. However, the distinction in conservatism (based on school/college/ area) isn’t so stark as to come up with a unified theory like the one we’ve come up here. Sticking on to the previous generation, what other reasons can you think of that makes certain aunts and uncles “cooler” than others?

Data Science and Software Engineering

I’m a data scientist. I’m good with numbers, and handling large and medium sized data sets (that doesn’t mean I’m bad at handling small data sets, of course). The work-related thing that gives me most kicks is to take a bunch of data and through a process of simple analysis, extract information out of it. To twist and turn the data, or to use management jargon “slice and dice”, and see things that aren’t visible to too many people. To formulate hypotheses, and use data to prove or disprove them. To represent data in simple but intuitive formats (i.e. graphs) so as to convey the information I want to convey.

I can count my last three jobs (including my current one) as being results of my quest to become better at data science and modeling. Unfortunately, none of these jobs have turned out particularly well (this includes my current one). The problem has been that in all these jobs, data science has been tightly coupled with software engineering, and I suck at software engineering.

Let me stop for a moment and tell you that I don’t mind programming. In fact, I love programming. I love writing code that makes my job easier, and automates things, and gives me data in formats that I desire. But I hate software engineering. Of writing code within a particular system, or framework. Or adhering to standards that someone else sets for “good code”. Of following processes and making my code usable by some dumbfuck somewhere else who wouldn’t get it if I wrote it the way I wanted. As I’d mentioned earlier, I like coding for myself. I don’t like coding for someone else. And so I suck at software engineering.

Now I wonder if it’s possible at all to decouple data science from software engineering. My instinct tells me that it should be possible. That I need not write production-level code in order to turn my data-based insights into commercially viable form. Unfortunately, in my search around the corporatosphere thus far, I haven’t been able to find something of the sort.

Which makes me wonder if I should create my own niche, rather than hoping for someone else to create it for me.

Collateralized Death Obligations

When my mother died last Friday, the doctors at the hospital where she had been for three weeks didn’t have a diagnosis. When my father died two and a half years back, the hospital where he’d spent three months didn’t have a diagnosis. In both cases, there were several hypotheses, but none of them were even remotely confirmed. In both cases, there have been a large number of relatives who have brought up the topic of medical negligence. In my father’s case, some people wanted me to go to consumer court. This time round, I had signed several agreements with the hospital absolving them of all possible complications, etc.

The relationship between the doctor and the patient is extremely asymmetric. It is to do with the number of counterparties, and with the diversification. If you take a “medical case”, it represents only a small proportion of the doctor’s total responsibility – it is likely that at any given point of time he is seeing about a hundred patients, and each case takes only a small part of his mind space. On the other hand, the same case represents 100% for the patient, and his/her family. So say 1% on one side and 100% on the other, and you know where the problem is.

The medical profession works on averages. They usually give a treatment with “95% confidence”. I don’t know how they come up with such confidence limits, and whether they explicitly state it out, but it is a fact that no disease has a 100% sure shot cure. From the doctor’s point of view, if he is administering a 95% confidence treatment, he will be happy as long as his success rate is over that. The people for whom the treatment was unsuccessful are just “statistics”. After all, given the large number of patients a doctor sees, there is nothing better he can do.

The problem on the patient’s side is that it’s like Schrodinger’s measurement. Once a case has been handled, from the patient’s perspective it collapses to either 1 or 0. There is no concept of probabilistic success in his case. The process has either succeeded or it has failed. If it is the latter, it is simply due to his own bad luck. Of ending up on the wrong side of the doctor’s coin. On the other hand, given the laws of aggregation and large numbers, doctors can come up with a “success rate” (ok now I don’t kn0w why this suddenly reminds me of CDOs (collateralized debt obligations)).

There is a fair bit of randomness in the medical profession. Every visit to the doctor, every process, every course of treatment is like a toin coss. Probabilities vary from one process to another but nothing is risk-free. Some people might define high-confidence procedures as “risk-free” but they are essentially making the same mistakes as the people in investment banks who relied too much on VaR (value at risk). And when things go wrong, the doctor is the easiest to blame.

It is unfortunate that a number of coins have fallen wrong side up when I’ve tossed them. The consequences of this have been huge, and it is chilling to try and understand what a few toin cosses can do to you. The non-linearity of the whole situation is overwhelming, and depressing. But then this random aspect of the medical profession won’t go away too easily, and all you can hope for when someone close to you goes to the doctor is that the coin falls the right way.