SK – Page 50 – Pertinent Observations

Pertinent observations on liquidity in startup markets

“Liquidity” was one of those words Wall Street people threw around when they wanted the conversation to end, and for brains to go dead, and for all questioning to cease

– Michael Lewis in Flash Boys

The quote that begins this blog post is also the quote that begins my book, which was released exactly a year ago. Despite its utility in everyday markets and economics, the concept of liquidity has not been explored too much outside of financial markets. In fact, one reason I wrote my book was that it appeared as if there was a gap in the market for material using the concept of liquidity to analyse everyday markets.

From this perspective, I was pleasantly surprised to come across a bunch of blog posts written by investors and tech analysts and startup fellows about the concept of “liquidity”. Most of these posts I came across by way of this excellent blog post by Andrew Chen of Andreessen Horowitz. It is always good to see others analysing topics in the same way as you are, so I thought I’ll share some insights from these posts here – some quotes, some pertinent observations. This is best done in bullet points. If you want to know more, I urge you to click through and read the blog posts in full. They’re all excellent.

You wonder why some startups make a big deal of how many cities they are in. This is because they usually function as within-city marketplaces, and so they need to be launched one city at a time. Uber famously started operations in San Francisco and remained there for a while.
“The best way to measure liquidity in the marketplace is to track the % of items or services that get sold/booked, and within what period of time. The higher the % and shorter period of time, the more sellers are making money and buyers are becoming loyal customers” – from here
“Where absolute pricing management makes most sense (i.e., where the marketplace operator sets prices) is where there isn’t a proper barometer for what the supply side should be charging and when the software can leverage systems should to optimize for liquidity” – from this excellent post
“In a zero sum game there, it’s most likely the marketplace with the most demand wins”. This was in the context of delivery marketplaces, and why Uber was likely to win that game (though it’s not clear if they’ve “won” it yet)
Trust is critical in building marketplaces. Both sides of the market need to trust the intermediary, and this can make marketplaces fragile. I had a recent incident where I appreciated the value of AirBnB landlord insurance (a lamp at a property I stayed at broke just after my stay, and the landlord wanted compensation). This post talks about how this insurance was critical to AirBnB’s growth
The same post talks about why even early stage businesses often make acquisitions – usually earlier stage businesses. “Marketplaces are normally winner-take-all markets. If we had lost ground to European competitors in 2012, we may have never gotten it back”
Ratings are a critical measure to build trust in a marketplace. And two-way ratings can help establish trust on both sides of the market
During the book launch function last year, there was a question on how marketplaces should build liquidity. I had given an example of the Practo/OpenTable model where you first sell a standalone service to one side of the market and then develop a marketplace. Another method (something I helped put in place for one of my current clients) is for the marketplace itself to become a “proprietary supplier”. The third, as this blog post describes, is about building markets where buyers are also sellers and the other way round (classic financial markets, for example).

For more on liquidity, and how it affects just about every market that you participate in on a daily basis, read my book!

We’ll miss sushi

One food item that my daughter and I will really miss when we move back to India is sushi. It is not that it is not available in Bangalore – restaurants such as Matsuri and Harima make excellent quality sushi, just that the transaction cost of procuring it will be far higher.

I grew up vegetarian, and didn’t eat meat until I was twenty eight. The decision to try meat was ad hoc – at a restaurant in Monastiraki square in Athens, the meat looked fantastic and the vegetables looked sad. And I decided that if I were losing my religion, I would lose it all the way and started my meat-eating career by eating beef souvlaki.

It wasn’t until a year later that I tasted fish, though – from childhood the smell of fish had put me off. As it happened, I first ate fish at a restaurant in Karwar, en route to Goa. Then, a consulting project in Mumbai happened, with a fish-loving client who took me to the best fish restaurants in that city (sometime during this time, I discovered I’m allergic to prawns).

It would take another year or two before I would have raw fish, though, in the form of sushi and sashimi. The first time was a trip to Matsuri, where my wife was treating me. I quickly grew fond of it, and would have a Japanese meal (at either Harima or Matsuri) at least once in six months (these are easily the best and most authentic Japanese restaurants in Bangalore. Edo is good but overpriced).

My love for sushi really took off during the three months I spent in Barcelona in 2016. That city has loads of sushi shops (it helped we were living in a dense district), mostly run by Korean immigrants. it is not too expensive either, which meant I would have it once a week at least (I might have eaten more often, but the wife was pregnant then, and hence off raw fish).

London doesn’t have the same density of sushi shops as Barcelona, but there are some chains that make pretty good sushi (Wasabi and Itsu, though I prefer the latter). Like other things London, it is not cheap, but we end up eating it reasonably often (it helps that the daughter loves sushi as well, though she only eats salmon nigiri – which also happens to be my favourite kind of sushi).

While craving sushi and planning a sushi run for dinner earlier this evening (finally we ended up eating at a Korean restaurant), it hit me that I won’t be able to have sushi so regularly in Bangalore. I started wondering what it would take for the likes of Freshmenu to be offering sushi on their menu. And I remembered a chapter in my book on specialty food.

The problem with low demand products is that the volatility of demand is high relative to the average demand. This means that for a retailer to stock items with low demand, either the margin needs to be high, or the inventory levels will be so low that customers might be disappointed rather often – neither of which is sustainable.

Making matters worse is the fact that fresh fish is an integral part of sushi, and it has an incredibly short shelf life. So unless demand can be aggregated to a high level (which Harima and Matsuri do, by being located in the middle of town and especially catering to the Japanese population in the city. In fact, I’m told the Chancery (where Matsuri is located) is the hotel of choice for Japanese visitors to Bangalore), it is not feasible to run a sushi restaurant in Bangalore.

Oh, and in the same chapter in the book, I discuss why people like to live with other people like themselves – others demanding the same thing you demand is the only way you can ensure that there is supply to meet your demand.

Information Technology and Large Cities

In my book Between the buyer and the seller, officially released exactly a year ago, I have a chapter on cities. In that I explain why industry clusters form, and certain cities or regions become hubs for certain types of industries.

In that, I spoke about the software industry in California’s Silicon Valley, and in Bangalore. I also mentioned how the Industrial Revolution wasn’t evenly distributed around England, and how it was clustered around textile hubs such as Birmingham and Manchester. I also used that chapter to talk about the problem with government-mandated special economic zones (this podcast with Amit Varma can help you understand the last point).

Back when Silicon Valley was still silicon valley (basically a semiconductor and hardware hub), it wasn’t as concentrated a hub as it is today. It was still fairly common for semiconductor companies to base themselves away from the valley. With the “new silicon valley” and the tech startup scene, though, there is no escaping the valley. It is almost an unwritten rule in US Tech startup circles that if you want to be successful with a tech startup, you better be in the valley.

And this is for good reason, as I explain in the book – Silicon Valley is where the ecosystem for successfully running a tech startup already exists, including access to skilled employees, subcontractors and investors, not to speak of a captive market. This, however, has meant that Silicon Valley is now overcrowded in many respects, with rents being sky high (reflected in high salaries), freeways jammed and other infrastructure under stress.

In fact, it is not just the silicon valley that has got crushed under the weight of being a tech hub – other “secondary hubs” such as Seattle (which also have a few tech majors, and where startups put off by the cost of the valley set up) are seeing their quality of life go down. The traffic and infrastructure woes in Bangalore are also rather similar.

So why is it that information technology has led to hubs that are much larger than historical hubs (based on other industries)? The simple answer lies in investment, or the lack of it.

Setting up an information technology company is “cheap” in terms of the investment in capital expenditure. No land needs to be bought, no plants need to be constructed and no machinery needs to be bought. All one needs is an office space (for which rent is paid monthly), and a set of employees (who again get paid a monthly salary). Even IT infrastructure (such as computing power and storage and communication) can be leased, and paid for periodically.

This implies that there is nothing that stops a startup company from locating itself in one of the existing hubs. This way, the company can avail all the benefits of being in the hub (supplier and customer infrastructure, employee pool, quality of life for employees and investors) without a high upfront cost.

Contrast this to “hard” industries that require manufacturing, where the benefits of being located in hubs is similar but the costs are far higher. As a hub develops, land gets expensive, which puts off further investors from locating themselves in the hub. This puts a natural limit on the size of the hubs, and if you think about it, large cities from earlier era were all “multi-purpose cities”, serving as hubs for several unrelated industries.

With information technology, though, the only impediment to the growth of a hub is the decreasing quality of life, information regarding which gets transmitted in indirect means such as higher rentals and commute times, and poor health. This indirect transmission of costs to investors results in friction, which means information technology hubs will grow larger before they stop growing. And as they go through this process, the quality of life of the hub’s residents suffers!

One axis politics

Historically, political leanings have ben described on two dimensions – economic freedom and social freedom. In the American scenario, the Republican Party has historically been supportive of economic freedom and restrictive of social freedom. The Democratic Party has been liberal on social freedom but illiberal on the economic freedom front.

While other major Western democracies occupying these two opposite quadrants, the other two quadrants have been largely empty. The libertarians occupy the “free on both fronts” quadrant, but nowhere is there a party to represent them – giving people freedom on all fronts means lesser power for the government and no politician wants that. And being restrictive of both kinds of freedom means people won’t vote for you – at least this was the way historically.

Of course things have been different in India. While we did have a series of governments between 1991 and 2004 that were reasonably economically liberal (“liberalisation” happened in this time period), all Indian political parties are required to swear by socialism, and they swear by it in spirit as well. So the difference on the economic freedom front between different Indian parties is marginal (in 2014, many of us thought the BJP might be supportive of economic freedom, given its record in the 1999-2004 period. Instead, it gave us demonetisation).

So in effect, in India we have a one-axis democracy, where parties try to differentiate themselves on one axis, which is the kind of social freedom they allow. Even there, it is not so much of an axis, but different ways in which they control social freedoms.

The BJP doesn’t want you to eat beef. The AAP doesn’t want recorded music in restaurants. The Congress and JDS don’t want live music in restaurants. The BJP puts cow welfare over human welfare. The Congress enacts and supports laws that allow suppression of Muslim women (by Muslim men). Many parties want to ban liquor, despite it having been repeatedly shown that such bans don’t work. No party wants to legalise marijuana, despite our rich tradition in the substance (heck, its scientific name is Cannabis indica). And we all seem to vote based on which of these social freedoms are more precious to us than others – economic freedom is a battle already lost.

In any case, it seems like other countries are also moving towards one axis democracy. A chart posted on Twitter today describes results from a survey in the US on voters’ attitudes towards social and economic freedoms, and how they voted in the 2016 presidential elections (which Donald Trump famously won).

Source: https://www.voterstudygroup.org/publications/2016-elections/political-divisions-in-2016-and-beyond

A large part of America seems to lie in the left half of the economic freedom spectrum. Yes, the republican voters are still more towards the centre than the democratic voter, but the bigger separation here is on the social rather than the economic dimension. And the Trump administration has been pursuing several policies cutting economic freedoms, such as tearing up trade deals and imposing tariffs.

So it seems like the world is following India in terms of enacting one axis politics – where voters vote more on the social dimension rather than the economic dimension. Then again, I don’t expect this to last – with parties moving left economically, soon you can expect economic freedoms to be crushed to the extent that it becomes advantageous for a party to signal economically right and still get votes.

PS: We don’t need to limit ourselves to two dimensions. A few years back, Nitin Pai had proposed the Niti Mandala which has three dimensions.

Source: http://acorn.nationalinterest.in/2012/03/03/nitimandala-the-indian-political-spectrum/

The missing middle in data science

Over a year back, when I had just moved to London and was job-hunting, I was getting frustrated by the fact that potential employers didn’t recognise my combination of skills of wrangling data and analysing businesses. A few saw me purely as a business guy, and most saw me purely as a data guy, trying to slot me into machine learning roles I was thoroughly unsuited for.

Around this time, I happened to mention to my wife about this lack of fit, and she had then remarked that the reason companies either want pure business people or pure data people is that you can’t scale a business with people with a unique combination of skills. “There are possibly very few people with your combination of skills”, she had said, and hence companies had gotten around the problem by getting some very good business people and some very good data people, and hope that they can add value together.

More recently, I was talking to her about some of the problems that she was dealing with at work, and recognised one of them as being similar to what I had solved for a client a few years ago. I quickly took her through the fundamentals of K-means clustering, and showed her how to implement it in R (and in the process, taught her the basics of R). As it had with my client many years ago, clustering did its magic, and the results were literally there to see, the business problem solved. My wife, however, was unimpressed. “This requires too much analytical work on my part”, she said, adding that “If I have to do with this level of analytical work, I won’t have enough time to execute my managerial duties”.

This made me think about the (yet unanswered) question of who should be solving this kind of a problem – to take a business problem, recognise it can be solved using data, figuring out the right technique to apply to it, and then communicating the results in a way that the business can easily understand. And this was a one-time problem, not something you would need to solve repeatedly, and so without the requirement to set up a pipeline and data engineering and IT infrastructure around it.

I admit this is just one data point (my wife), but based on observations from elsewhere, managers are usually loathe to get their hands dirty with data, beyond perhaps doing some basic MS Excel work. Data science specialists, on the other hand, will find it hard to quickly get intuition for a one-time problem, get data in a “dirty” manner, and then apply the right technique to solving it, and communicate the results in a business-friendly manner. Moreover, data scientists are highly likely to be involved in regular repeatable activities, making it an organisational nightmare to “lease” them for such one-time efforts.

This is what I call as the “missing middle problem” in data science. Problems whose solutions will without doubt add value to the business, but which most businesses are unable to address because of a lack of adequate skillset in solving the issue; and whose one-time nature makes it difficult for businesses to dedicate permanent resources to solve.

I guess so far this post has all the makings of a sales pitch, so let me turn it into one – this is precisely the kind of problem that my company Bespoke Data Insights is geared to solving. We specialise in solving problems that lie at the cusp of business and data. We provide end-to-end quantitative solutions for typically one-time business problems.

We come in, understand your business needs, and use a hypothesis-driven approach to model the problem in data terms. We select methods that in our opinion are best suited for the precise problem, not hesitating to build our own models if necessary (hence the Bespoke in the name). And finally, we synthesise the analysis in the form of recommendations that any business person can easily digest and action on.

So – if you’re facing a business problem where you think data might help, but don’t know how to proceed; or if you are curious about all this talk about AI and ML and data science and all that, and want to include it in your business; or you want your business managers to figure out how to use the data teams better, hire us.

Dimensional analysis in stochastic finance

Yesterday I was reading through Ole Peters’s lecture notes on ergodicity, a topic that I got interested in thanks to my extensive use of Utility Theory in my work nowadays. And I had a revelation – that in standard stochastic finance, mean returns and standard deviation of returns don’t have the same dimensions. Instead, it’s mean returns and the variance of returns that have the same dimensions.

While this might sound counterintuitive, it is not hard to see if you think about it analytically. We will start with what is possibly the most basic equation in stochastic finance, which is the lognormal random walk model of stock prices.

$dS = \mu S dt + \sigma S dW$

This can be rewritten as

$\frac{dS}{S} = \mu dt + \sigma dW$

Now, let us look at dimensions. The LHS divides change in stock price by stock price, and is hence dimensionless. So the RHS needs to be dimensionless as well if the equation is to make sense.

It is easy to see that the first term on the RHS is dimensionless – $\mu$ , the average returns or the drift, is defined as “returns per unit time”. So a stock that returns, on average, 10% in a year returns 20% in two years. So returns has dimensions $t^{-1}$ , and multiplying it with $dt$ which has the unit of time renders it dimensionless.

That leaves us with the last term. $dW$ is the Wiener Process, and is defined such that $dW^2 = dt$ . This implies that $dW$ has the dimensions $\sqrt{t}$ . This means that the equation is meaningful if and only if $\sigma$ has dimensions $t^{-\frac{1}{2}}$ , which is the same as saying that $\sigma^2$ has dimensions $\frac{1}{t}$ , which is the same as the dimensions of the mean returns.

It is not hard to convince yourself that it makes intuitive sense as well. The basic assumption of a random walk is that the variance grows linearly with time (another way of seeing this is that when you add two uncorrelated random variables, their variances add up to give the variance of the sum). From this again, variance has the units of inverse time – the same as the mean.

Finally, speaking of dimensional analysis and Ole Peters, check out his proof of the Pythagoras Theorem using dimensional analysis.

People were asking about dimensional analysis.
If a technique produces a three-line proof of Pythagoras, come on! you know you've got something good. Here's a sketch — full details and caveats in Barenblatt's "Scaling." pic.twitter.com/u9zOtLyGpC

— Ole Peters (@ole_b_peters) September 3, 2018

Isn’t it beautiful?

PS: Speaking of dimensional analysis, check out my recent post on stocks and flows and financial ratios.

Revenue management and transaction costs

So I just sent off a letter to India. To be precise, it is a document I had to sign and send to my accountant there – who sends regular “letters” any more?

The process at the post office (which, in my suburb, is located inside a large bookstore) was simple. In the first screen of the touch screen kiosk, there was an option for “worldwide < 20 grams”. A conveniently placed scale told me my letter weighed 18 grams, and one touch and one touch of my debit card later, I had my stamp. Within a minute, my letter was in the letterbox.

The story of how we pay the same amount for sending mail over large areas (“worldwide” in my case today) is interesting. Earlier, mail rates were based on distance, but as new roads kept being built in the 19th century America, and distances kept changing, figuring out how much to charge for a letter became “expensive”. A bright fellow figured out that the cost (in terms of time) of figuring out how much to charge for mail was of the same order of magnitude as the cost of the mail itself. And so the flat rate scheme for mail, that is prevalent worldwide today, was born.

Putting it in technical terms, transaction costs trumped price discrimination in this case. Price discrimination is the art (yes, it’s an art) of charging different amounts to different people based on their differential willingness to pay. Uber surge pricing is one example (I have a chapter in my book on this). Airline fares are another common example.

Until the late 18th century (well after mail prices had gone “flat”), price discrimination was rather common everywhere, a concept I have devoted a chapter to in the book. In fact, the initial motivation for fixed price retail was religious – Quakers, who owned many departmental stores in the US North-East, thought “all men are created equal before God” and so it was incorrect to charge different amounts to different people.

Soon other benefits of fixed prices became apparent (faster billing; less training for staff; in fact it was fixed prices that permitted the now prevalent supermarket format), and it took off. The concept is the same as stamps – the transaction cost of figuring out how much to charge whom is higher than the additional revenue you can make with such price differentiation (not counting possible loss of reputation, and fairness issues). Price discrimination at the shop is now confined to high value high margin businesses such as cars.

And it works in other high gross margin businesses such as airlines, hotels and telecom. These are all businesses with high fixed costs and low marginal costs for the suppliers. Low marginal costs has meant that price discrimination ha been termed as “revenue management” in the airline industry.

During the launch function of my book last year, I got asked if Uber’s practice of personalising fares for passengers is fair (I had given a long lecture on how Uber’s surge pricing is a necessary component of keeping average prices low and boosting liquidity in the taxi market). I had answered that a marketplace needs to ensure that its pricing is perceived as being “fair”, else they might lose customers to competitors. But what if all players in a market practice extreme price discrimination?

Thinking about it, transaction costs will take care of price discrimination before businesses and marketplaces start thinking of fairness. Beyond a point (the point varies by industry), the marginal revenues from price discrimination will fall below the transaction cost of executing this discrimination. And that poses a natural limit to how much price discrimination a business can practice.

Gods, Demons and Superheroes

Rule 12 Pet a cat when you encounter one on the street

That is the last of Jordan Peterson’s 12 Rules for Life (I have earlier written about Rule 11). In that chapter, he talks about why suffering exists, and what we can do to get around it. Taking the example of his daughter’s arthritis, which which she has been affected since a young age, he explains how one should draw boundaries around the time one deals with the suffering, and learn to enjoy the little joys of life (hence the cat and the petting).

In the lead up to this rule, Peterson philosophises about why there exists suffering in this world. To illustrate, he takes the example of the time when Marvel Comics and DC Comics decided to bring together their superheroes, to create stories such as “Batman versus Superman”. The two superheroes, he explains, were so mismatched that DC’s superheroes, who had a bit of a grey background, had to be given massive additional superpowers in order to match up to Superman, and even then, Superman had a huge advantage. And this made those superhero matchup stories not fun.

From this, he draws the analogy to religion (he frequently references the Bible in his book – which for me was one of the more irritating things about it) and talks about how if there was no suffering or negativity, there would be no point of life, since there would be no challenges. And so he says that god wills to bring in some suffering, that is randomly (but not evenly) distributed, which makes life challenging and gives it a purpose.

In some ways, Peterson’s view of why suffering exists in the world is remarkably similar to the Hindu philosophical view. An excellent point to understand the latter is this post by Shrikanth Krishnamachary. Shrikanth writes:

So to summarize, it is fair to conclude that Indian ideas on the “Problem of evil” are unique and their exceptionalism derives from the ideas of an Agnostic, even playful God, championed in the foundational texts of the tradition, including the Rig Veda and the Brahma Sutras.

Earlier in the post, Shrikanth elaborates about the concept of “Leela” in Hindu philosophy which is sometimes translated as “god’s play”. It is like the gods, in their “play” created evil because otherwise the world would be too one-sided and not fun.

Speaking of superheroes and “fun”, one thing that makes Hindu mythological stories fascinating and fun is that it is not just the good guys that have superpowers – the antagonists have them as well. In fact, a common motif in Hindu myth is of a character performing immense sacrifices and penances, and being rewarded with a boon in the form of a superpower. And then the character proceeds to get arrogant about this boon and starts misusing it, at which point a god with a neutralising superpower is sent to deal with the situation.

So the fact that the antagonist in Hindu myth usually has a superpower means that the task of annihilating them becomes a non-trivial task, and this makes the story fun. In fact, with most Hindu myth stories having evolved over several millennia, it is not surprising that these stories conform to the “Leela model” (god’s play) of evil in myth – both sides get superpowers and the gods have fun watching them face off.

In his post, Shrikanth also mentions about how the concept of evil has changed since the Bhakti movement. He writes:

However in the past 1000 years, the Karmic orthodoxy of the Mahabharata has been challenged by the emergence of the Bhakti movement. This movement, with its roots in Southern India, has traditionally placed a great emphasis on the extolling of divine grace. God’s grace is spontaneous and is as much a part of his Leela as the act of creation itself. Hence Karma does not always have a deterministic linkage with Phala, which is the reason why often evil things can befall the most virtuous of men.

Perhaps not coincidentally, it is also notable that Bhakti movement era stories have a vastly different model of superhero battles compared to longer evolved Hindu myth. Over the last few days I’ve accompanied the daughter in watching a few episodes of “Little Krishna”, a cartoon series based on the stories of Krishna as a child (produced by Big Entertainment an ISKCON). While the battles in that might fascinate a child, they are hardly engaging.

And this is because Little Krishna has superpowers that far exceed that of the various demons sent by his uncle Kamsa to kill him. The demons do have some superpowers, but all it takes is brute force on the part of Little Krishna to destroy them. This is quite in contrast with traditional myth, where both sides have comparable superpowers, which means that the “good guys” need to engage in some clever solutions, including the odd episode of cheating, in order to win their battles.

PS: Apart from Jordan Peterson’s analogy, one other connection of this post with superheroes is one reason I find superhero movies boring – just like in Bhakti-based stories, or stories from the Bible, or even Superman vs Batman, I find that superhero stories have superheroes possessing superpowers vastly in excess of the “bad guy”, making these stories not fun.

Statistics and machine learning approaches

A couple of years back, I was part of a team that delivered a workshop in machine learning. Given my background, I had been asked to do a half-day session on Regression, and was told that the standard software package being used was the scikit-learn package in python.

Both the programming language and the package were new to me, so I dug around a few days before the workshop, trying to figure out regression. Despite my best efforts, I couldn’t locate how to find out the R^2. What some googling told me was surprising:

There exists no R type regression summary report in sklearn. The main reason is that sklearn is used for predictive modelling / machine learning and the evaluation criteria are based on performance on previously unseen data

As it happened, I requested the students at the workshop to install a package called statsmodels, which provides standard regression outputs. And then I proceeded to lecture to them on regression as I know it, including significance scores, p values, t statistics, multicollinearity and the likes. It was only much later was I to figure out that that is now how regression (and logistic regression) is done in the machine learning world.

In a statistical framework, the data sets in regression are typically “long” – you have a large number of data points, and a small number of variables. Putting it differently, we start off with a model with few degrees of freedom, and then “constrain” the variables with a large enough number of data points, so that if a signal exists, and it is in the right format (linear relationship and all that), we can pin it down effectively.

In a machine learning framework, it is common to run a regression where the number of data points is of the same order of magnitude as, or even smaller than the number of variables. Strictly speaking, such a problem is unbounded (there are too many degrees of freedom), and so regression is not well-defined. Instead, we rely upon “regularisation methods” to “tie down” the variables and (hopefully) produce a consistent solution.

Moreover, machine learning approaches are common to problems where individual predictor variables don’t have meaning. In this scenario, knowing whether a particular variable is significant or not is of no utility. Then, the signal in machine learning lies in the combination of variables, which means that multicollinearity (correlation between predictor variables) is not really a bad thing as it is in statistics. Variables not having meanings means that there are no correlations per se to be defined, and so machine learning models are harder to interpret, and are more likely to have hidden spurious correlations.

Also, when you have a small number of variables and a large number of data points, it is easy to get an “exact solution” for regression, which is what statistical methods use. In a machine learning framework with “wide” data, though, exact solutions are computationally infeasible, and so you need to use approximate algorithms such as gradient descent – which are common across ML techniques.

All in all, while statistics and machine learning might use techniques with the same name (“regression”, for example), they are both in theory and practice, very different ways to solve the problem. The important thing is to figure out the approach most suited for a particular problem, and use it accordingly.

Dam capacity

In Mint, Narayan Ramachandran has a nice op-ed on the issue of dam capacity and damn management in the wake of the floods in Kerala last year. In that, he writes:

For dams to do their jobs in extreme situations, they should have large unfilled capacity in their reservoirs when extreme events occur

Reading this piece reminded me of Benoit Mandelbrot’s The (Mis)Behaviour of Markets, and his description of the efforts of the colonial British government in Egypt in deciding the height of the Aswan dam. The problem with the Nile was “long range dependence” – the flow in the river in a year was positively correlated with the flow in the previous few years. This meant that there would be years of high flow followed by years of low flow.

The problem was solved by a British hydrologist Harold Edwin Hurst by looking at thousands of years of data of the flooding of the Nile (yes, this data was available), and there is a nice description of it in Mandelbrot’s book.

I had taken a few insights from this chapter for my own piece on long-range dependence in stock markets that I had written for Mint a few years back.

Coming back to Narayan’s piece, one problem is that in India we have an obsession with keeping dams filled up. In Karnataka, for example, every year during the monsoons, newspapers keep track of the level of water in the major reservoirs, expressing worry in case they’re not full enough. In that sense, I guess our dams haven’t been planned for long-range water sharing, and that has contributed to problems such as sudden water release.

Also not helping matters I guess is the fact that a lot of rivers flow across states, and the level of dams is a source of negotiation between states, and this leads to further keeping them small and ill-geared to long term water management.