Tag: creativity

Creative Cycles

When you’re doing creative work, your work broadly falls into two phases – the “invention phase” and the “implementation phase”. Both imply what they mean.

There are times when you are tinkering around and experimenting to find something fundamentally new that is cool. And then, once you have made the breakthrough in finding something cool, you need to make it useful. And this can take considerable amount of work, and its own creativity.

So if you are one person doing a “creative job”, your work will alternate in these cycles – where you create and you implement. The cycles are unlikely to be periodic. Some creative solutions are so creative that implementation is a breeze. In most cases, the inspiration is only 1% of the problem – the devil in the details for which you need to perspire.

When you are part of a creative team, this cycle thing can play out in different ways. Some teams form a caste system, where one set of people work purely on the invention phase, while the other works on the implementation phase. This is especially useful when solving highly complex problems, in which case the skills required for the invention and implementation phases are different.

The big cost of having separate teams like his is the cost of communication (AGES back, when GPUs were just becoming a thing, I was part of a committee that was exploring the use of GPUs in our work. One of the findings there was that GPUs can do the work incredibly fast, but the data transfer from GPUs to CPUs was slow, and could be a bottleneck. I assume that problem is solved now). People sometimes grossly underestimate the effort involved in communicating your solution to someone else. Even if you manage to communicate, there can be significant handholding that might be required to get the other team to take forward your invention.

And so this investment in communication cost is worth it if and only if the work is complex enough. Think of large industrial projects – such as the manufacture of the iPhone, for example – they are complex enough that you need several specialist teams to perform the entire creative process. And in the larger scheme of the complexity, the cost of communication across teams is small.

On the other hand, this usage of multiple teams to perform a creative process can be massive overkill for simpler work – there the cost of communication can far overpower the gains in efficiency through specialisation.

Anyway, I’m getting distracted here.

Coming back, the alternative is to have the same people or sub-teams perform the invention and implementation stages of the creative process. Here, I’ve seen things play out in multiple ways.

Some teams are uncorrelated – this means that different members or sub-teams are in different phases of the work. As a consequence, this kind of a team constantly provides creative output. When some of the people are deep in implementation, others are inventing. And the other way round. This means that the team is constantly both coming up with new ideas and delivering stuff.

Other teams can be more correlated – either everyone is working on the same thing, or the whole team moves in sync (invention at some points in time, implementation at others). Here the issue is that there can go long periods of time without the team really producing anything – in the common invention phase, no shit is getting done. In the common implementation phase, there are no new ideas.

This can lead to stagnation in the team, and frustration outside. And so not ideal.

The other related concept is in terms of management. Some managers of creative teams are better off at managing the invention phase. Others are better off at managing the implementation phase. Given that the creative process involves both, for the team to be effective, we need managers who can manage both as well.

And this is easier said than done in a single person, and so you need a management team. And what you find is that you have a “complementary number two” (no pun intended). If you as the team leader is better off at invention, you get a number two who is better at implementation. And the two (or more) of you together manage the process.

I’ve spoken about this before – this can sometimes lead to suboptimal succession. Let’s say the inventive head leaves. The organisation promotes the implementation number two. Now, it is contingent upon this new number one to get a (inventive) number two asap. If that doesn’t happen, invention can cease. The team will carry on for a while implementing the already invented stuff, and then grind to a halt.

Similarly if an implementation head leaves, the inventive number two gets promoted. And unless a new implementation number two is hired, you’ll see lots of proofs of concept and little actual implementation. Again suboptimal.

Shrinking deadlines

I’m reminded of this old joke/riddle, which also happened to feature in Gowri Ganesha. “If a 1 metre long sari takes 1 hour to dry in the sun, how long will and 8 metre long sari take to dry?”.

The instinctive answer, of course, is 8 hours, while if you think about it (and assume that you have enough clothesline space to not need to fold), the correct answer is likely to be 1 hour.

Now this riddle is completely unconnected to do with the point of the post, except that both have to do with time.

And then one day you find, ten years have got behind you.
No one told you when to run. You missed the starting gun. 

Ok enough distractions. I’m now home, home again.

Modern workspaces are synonymous with tight deadlines. Even when you give a conservative estimate on how long something will take, you get asked to compress the timelines further. If you protest too much and say that there is a lot to be done, sometimes you might get asked to “put one more person on the job and get it done quickly”.

This might work for routine, or “fighter” jobs – for example, if your job is to enter and copy data for (let’s say) 1000 records, you can easily put another person on the job, and the entire job will be done in about half the time (allowing for a little time for the new person to learn the job and for coordination).

As the job gets more complex, the harder it gets. At one level, there is more time to be spent by the new person coming into the job. Then, as the job gets more complex, it gets harder to divide and conquer, or to “specialise”. This means there is lesser impact to the new person coming in.

And then when you get closer and closer to the stud end of the spectrum, the advantage of putting more people to get the work done faster get lesser and lesser. There comes a point when the extra person actively becomes a liability. Again – I’m reminded of my childhood when occasionally I would ask my mother if she needed help in cooking. “Yes, the best way for you to help is for you to stay out of the kitchen”, she would say.

And then when the job gets really creative, there is a further limit on compression – a lot of the work is done “offline”. I keep telling people about how I finally discovered the proof of Ramsey’s numbers (3,3) while playing table tennis in my hostel, or how I had solved a tough assignment problem while taking a friend’s new motorcycle for a ride.

When you want to solve problems “offline” (to let the insight come to you rather than going hunting for it – I had once written about this) – there is no way to shorten the process. You need to let the problem stew in your head, and hope that some time it will get solved.

There is nothing that can be done here. The more you hurry up, the less the chances you give yourself of solving the problem. Everything needs to take its natural course.

I got reminded of it when we missed a deadline last Friday, and I decided to not think about it through the weekend. And then, an hour before I got to work on Monday, an idea occurred in the shower which fixed the problem. Even if I’d stressed myself (and my team) out on Friday, or done somersaults, the problem would not have been solved.

As I’d said in 2004, quality takes time.

Creative Grit!

This, by Annie Murphy Paul is a very interesting blogpost I came across today. This one is on “creative grit”.

There are two very interesting things that the blogpost talks about.

The first is that there are two ways to creativity – the “traditional way” (the way I’ve always seen it) is to think about the problem, internalise it and then somewhere “wait for inspiration to strike”.

The other method that the author talks about, referring to artist Franz Kline, is to “just keep trying”. Kline would make hundreds of paintings every night. And he would find that one (or few) of them would be good enough to work further on. So this form of creativity comes from repeated practice.

Then later in the blogpost, she also talks about some research on when creativity hits. Again, “traditionally” we are trained to think that if creativity has to “hit” us for a particular problem, that is much more likely to occur early on in our effort. Based on this, a lot of us creative people have come up with heuristics where if solution doesn’t occur within a few iterations of trying, we just give up and move on.

Annie Murphy Paul says that this is the incorrect approach. Quoting from her post:

Lucas and Nordgren call this the “creative cliff illusion”: we imagine that, after an initial upward leap, our creativity will then fall off a cliff—when in reality our creativity capacities are just getting ready to ascend.

We also misjudge the thoroughness of our search. In one study, people estimated that they had explored 75 percent of the solution space—when in fact they had covered only 20 to 30 percent of the relevant domain.

I sure should try the second method that she recommends – keep trying and occasionally you’ll be happy with the result. Or maybe I already do that with all my writing (this blog, my newsletters, etc.) – basically “spray and pray”. The reason I’ve managed to write so much is that I have a low bar for myself. So I write a lot of rubbish. And occasionally I end up writing something people like. On the other hand when I’m paid to write, I don’t “spray and pray”. And in trying to limit my downside I limit my upside as well.

And thinking about it, the reason this method works is that in creative pursuits only the wins matter. As long as you produce sufficient wins, no one cares about your duds!

While on the topic of creativity, here is an ancient lecture (maybe my first ever recorded “speech”) I gave on why “quality takes time”. This clearly shows that at least as of mid-2004 (coincidentally just before I started this blog) I used to strongly believe in the “wait for inspiration to strike” model of creativity.

Oh and btw, read the whole post. It’s worth it.

Data Science is a Creative Profession

About a month or so back I had a long telephonic conversation with this guy who runs an offshored analytics/data science company in Bangalore. Like most other companies that are being built in the field of analytics, this follows the software services model – a large team in an offshored location, providing long-term standardised data science solutions to a client in a different “geography”.

As is usual with conversations like this one, we talked about our respective areas of work and kind of projects we take on, and soon we got to the usual bit in such conversations where we were trying to “find synergies”. Things were going swimmingly when this guy remarked that it was the first time he was coming across a freelancer in this profession. “I’ve heard of freelance designers and writers, but never freelance data scientists or analytics professionals”, he mentioned.

In a separate event I was talking to one old friend about another old friend who has set up a one-man company to do provide what is basically freelance consulting services. We reasoned that the reason this guy had set up a company rather than calling himself a freelancer given the reputation that “freelancers” (irrespective of the work they do) have – if you say you are a freelancer people think of someone smoking pot and working in a coffee shop on a Mac. If you say you are a partner or founder of a company, people imagine someone more corporate.

Now that the digression is out of the way let us get back to my conversation with the guy who runs the offshored shop. During the conversation I didn’t say much, just saying things like “what is wrong with being a freelancer in this profession”. But now that i think more about it, it is simply a function of the profession being a fundamentally creative profession.

For a large number of people, data science is simply about statistics, or “machine learning” or predictive modelling – it is about being given a problem expressed in statistical terms and finding the best possible model and model parameters for it. It is about being given a statistical problem and finding a statistical solution – I’m not saying, of course, that statistical modelling is not a creative profession – there is a fair bit of creativity involved in figuring out what kind of model to model, and picking the right model for the right data. But when you have a large team working on the problem, working effectively like an assembly line (with different people handling different parts of the solution), what you get is effectively an “assembly line solution”.

Coming back, let us look at this “a day in the life” post I wrote about a year back about a particular day in office for me. I’ve detailed in that the various kinds of problems I had to solve that day – hidden markov models and bayesian probability to writing code using dynamic programming and implementing the code in R, and then translating the solution back to the business context. Notice that when I started off working on the problem it was not known what domain the problem belonged in – it took some poking and prodding around in order to figure out the nature of the problem and the first step in solution.

And then on, it was one step leading to another, and there are two important facts to consider about each step – firstly, at each step, it wasn’t clear as to what the best class of technique was to get beyond the step – it was about exploration in order to figure out the best class of technique. Next, at no point in time was it known what the next step was going to be until the current step was solved. You can see that it is hard to do it in an assembly line fashion!

Now, you can talk about it being like a game of chess where you aren’t sure what the opponent will do, but then in chess the opponent is a rational human being, while here the “opponent” is basically the data and the patterns it shows, and there is no way to know until you try something as to how the data will react to that. So it is impossible to list out all steps beforehand and solve it – solution is an exploratory process.

And since solving a “data science problem” (as I define it, of course) is an exploratory, and thus creative, process, it is important to work in an atmosphere that fosters creativity and “thinking without thinking” (basically keep a problem in the back of your mind and then take your mind off it, and distract yourself to solve the problem). This is best done away from a traditional corporate environment – where you have to attend meetings and be liable to be disturbed by colleagues at all times, and this is why a freelance model is actually ideal! A small partnership also works – while you might find it hard to “assembly line” the problem, having someone to bounce thoughts and ideas with can have a positive impact to the creative process. Anything more like a corporate structure and you are removing the conditions necessary to foster creativity, and are in such situations more likely to come up with cookie-cutter solutions.

So unless your business model deals with doing repeatable and continuous analytical work for a client, you are better off organising yourselves in an environment that fosters creativity and not a traditional office kind of structure if you want to solve problems using data science. Then again, your mileage might vary!

The Importance of Discipline

I’ve never been a fan of discipline. I think it is a major constraint and hinders creativity, and puts too many walls within which you need to live your life. Despite constant exhortations by my father, I never wanted to join the army. Hell, I tried my best (successfully) in order to even avoid NCC when I was at IIT. I pride myself on being some sort of a free spirit who isn’t held back by any arbitrary rules that I create for myself to live my life by.

A really nice article that I read today, however, makes me think twice about this stand. So this article is about “decision fatigue” and is not very dissimilar to what I’d read a long time back (again in the NYT) about the Law of Conservation of Willpower. So this article talks about how every time you need to make a decision it consumes some part of your mental energy. Irrespective of the size of the decision that is to be made, there is some willpower that is lost, and that causes you to be suboptimal in your decision making as the day progresses.

The article really struck a chord with me, and I realize I’m also heavily prone to decision fatigue. Sometimes the smallest decisions take away so much energy from me that I simply put NED. And yeah, on a related note, I’ve got the wife upset innumerable times solely because of my indecisiveness, a part of which can be attributed to decision fatigue. I even remember not going to a wedding reception some three years back because I couldn’t decide which shirt to wear! And no, I’m not making this up.

So on that note, here’s where I think discipline has a part to play in life. By putting certain constraints on your life, you are reducing the number of decisions that you have to make. And that implies your willpower and mental energy will be reserved for those things where it’s really important that you decide carefully. By making a schedule for yourself, you are outsourcing to you-the-planner all the trivial decisions of your life. Yes, you might feel constrained at times. But it saves you so much energy by way of saving you from several trivial decisions.

Of course, feeling constrained can also affect your mental energy in a negative way, and prevent you from giving your best. Nevertheless, this decision fatigue thingy implies that discipline may not be all that bad. Or maybe I need to think about it some more.

Rajkumar Hirani Copycat

Ok this post has nothing to do wtih Five Point Someone or its related controversies. Yeah, the story is inspired by 5PS more than the claimed 3% but I’ll let Chetan Bhagat and his army of followers fight out that battle. Copying from others is honourable, at least you are taking inspiration from someone. What is just not done is copying from oneself. It simply shows lack of creativity and laziness to come up with new ideas.

Maybe when Rajkumar Hirani made 3 Idiots, he assumed that the public would have forgotten Munnabhai MBBS. He assumed that Munnabhai MBBS would be so out of circulation that it would have gone out of people’s minds, eclipsed by the more successful sequel Lage Raho. What he didn’t bargain for was that Munnabhai MBBS was on the menu on the New York JFK  to Dubai Emirates Airlines flight, and that people like me would watch it within 3 weeks of watching 3 idiots.

The similarities are uncanny. Both colleges are “Imperial”, have Boman Irani playing the “big prof” (diro here, dean there), and acting similarly in both. Both have a nerdy Tam who comes 2nd in class, 2nd to the hero. Yeah, Chatur is caricatured in 3I while Swami is given a more positive role in Munnabhai. Both are about the system, about how the larger-than-life hero fights the system and makes the big prof realize that the way he has been running the institution is wrong. The hero’s love interest is the big prof’s daughter. And so on..  Just that Munnabhai and Rancho use different methods to achieve their goals, that’s all.

I suppose most of you would have watched 3Idiots recently. I urge you to pick up a DVD or a torrent of Munnabhai MBBS and watch it, again. And keep an eye out for the similarities. You will be convinced that Rajkumar Hirani is guilty of copying, from his own stuff. It is indeed sad to see a good director such has him stooping to Anu Malik* depths.

While on the topic of 3Idiots, my esteemed colleague Baada wanted me to do a stud-fighter post on the movie. I suppose all of you who have seen the movie will easily figure out why the framework fits. I don’t think it needs any more explanation from the resident stud-fighter expert, that is me. Also, if you recall, I had taken a vow that I won’t do any more stud-fighter blogging. Though I must mention that my book on the topic is going nowhere.

* Listen to the prelude music of Ae Mere Humsafar from Baazigar, and then to the title song of Ishq. Next, listen to the interlude music of Kitaben Bahut Si, again from Baazigar, and then to the title song from Fiza. The self-copy is obvious. And I must mention that I had used this concept in a quiz question, twice. Yeah, I’ve also been guilty of “petering” my own questions.