Machine Learning Thoughts: Happiness of a scientist II: the 80/20 rule

Archives

Having tried to do scientific research at several different places, I have observed the following, which has been confirmed by several colleagues:

No matter where you are, it is almost impossible to spend more than 20% of your time doing research.

Moderating this claim:

First of all, let me define what I mean by research. I essentially mean exploratory intellectual activity. So in a way, I am not restricting myself to scientific research, and I possibly include other activities such as brainstorming about the strategy of a company for example. But, for the rest of the discussion, let us focus on scientific research.
Secondly, I shall add that the above claim is only valid provided you care about having a balanced life. Many people manage to do more research but they then have to sacrifice something (e.g., family life, material conditions...). Finding a good balance is a complex optimization problem with many local minima (and probably a multi-objective one).
Thirdly, this is of course an average value and you can find people who manage to do more than 20% or people who cannot do more than 10%. But the point is that the distribution is very short-tailed: it is easy to reach 20%, hard to reach 30% and almost impossible to reach 40%...

Justification of the claim:
To make the above claim a bit more clear, let me give some examples of activities one has to carry out in the remaining 80%:
Being a professor, you have to prepare classes, teach these classes, organize exams, grade the students, deal with scheduling, deal with various administrative duties, take care of PhD students...
Being a research scientist in academia, you possibly have to review papers, organize conferences, file grant applications, deal with administrative duties, prepare lectures...
Being a research scientist in a private company, you have to spend a lot of time working on improving existing systems or developing new ones, but the point is that you have to instantiate your research ideas into something that has impact on the company.

Of course, I am not complaining about all these extra duties, but I am just trying to give an objective description of what these positions require (sometimes implicitely). The subtle point is that often these extra duties are not fully mandatory (e.g., one could live without reviewing papers) but performing them is often a necessary condition for obtaining certain types of rewards (recognition by peers, being in good terms with colleagues, promotions...)

Consequences:
So, admitting that reader is now partially convinced about the truth of the above claim, I shall derive the following consequence about how to choose a job:

Instead of choosing a job based on the amount of time that you will be allowed to spend on research, rather choose it based on what exactly the 80% other activities are.

Indeed, you should rather pick a job whose 80% activities you find enjoyable (unless you really can stand doing things you do not like for the sake of the remaining 20%).

Comments

I guess a corollary to this is that you should go into academia as a professor only if you like teaching...

Posted by: Suresh | February 04, 2007 at 03:41 AM

Watts Humphrey says basically the same thing about software engineering and he has a lot of hard data to back it up.

His Personal (and Team) Software Processes are based on recording time spent actually designing/coding/compiling/testing versus time spent doing everything else. I don't have the book in front of me, but I'm fairly sure it was also around 20/80.

Posted by: AndrewS | February 07, 2007 at 01:13 AM

I generally agree that we can't spend too much of our time doing core research thinking. The reason is that research is hard, hard enough that we can't really keep at it during all waking hours, simply because mental exhaustion wipes out the ability to creatively think.

There are many other constraints on research: time to test ideas, time to write papers, time to travel and meet with others. These all require a substantial fraction of your time, but not necessarily your creative thinking time.

I'm not sure that I agree with optimizing for how you spend nonresearch time though. Many people choose jobs with an expectation the "I'll make time for research" but they discover that nonresearch activities can easily end up claiming 100% of the hours they can devote to creative thinking in a day. I've seen this happen to people going into both professor and industry positions.

So, if what you want to do is research, an environment that both encourages it and provides you the resources (time, computers, data, etc...) needed to succeed is very beneficial.

Posted by: John Langford | February 17, 2007 at 11:53 PM

This conclusion makes me comfortable that much of my time is not on research but playing:)

Posted by: phoenixbull | February 28, 2007 at 04:40 AM

A point approximation makes no sense. I found as a faculty member at CMU that the time I had to spend on research went from close to 100% as a first-year post-doc to close to 0% as an 8th-year associate professor. If you count talking to students about their work, then the final result was 20% -- I just had no time to do any of my own work and wasn't a good enough manager to unify my own and my student's work.

I then went to Bell Labs research, where the number went back to nearly 100%, the only drag being useless HR, department, lab and research meetings. No, I didn't have to work on real products or making the company money. It was impossible to ever get anyone in the same room who was connected to actual Lucent business. When the whole lab came crashing down and everyone was applying for jobs, the number went way down again.

Then I moved to SpeechWorks, a startup of 200 people in the well-funded pre-IPO stage. There I was working 125% (about 50 hours/week) learning how to be a real industrial C programmer (having written books on programming language theory and written quite widely used programs in Prolog). Post-IPO, that number went way down as we had HR meetings, etc. When the first round of layoffs was announced, management told us an unspecified 30% of the company would be laid off in 6 weeks. Needless to say, productive time plummeted again.

Then I went to work for Alias-i, a two-person research company on DARPA and later NIH funding. There's almost no BS in a two-person company, so my number's real close to 100% again, because the other person does all the admin. The only bummer is that now we're doing consulting for customers, which is very researchy in many ways, but may technically lower the number below 100%.

Posted by: Bob Carpenter | March 13, 2007 at 08:49 PM

Very nice post! I find it particularly relevant and researchers should send it to their non-researchers friends. I think one of the key point is to correctly distribute the heavy thinking work during the day.

Posted by: Sandro | May 08, 2007 at 11:57 AM

excellent post! but I am just trying to give an objective description of what these positions require.

Posted by: Nursing clothing | May 13, 2010 at 07:50 AM

Lovely flowers! Wish I can also arrange beautiful flowers like that. I want to learn how to arrange flowers cause it so interesting. Anyway, I enjoyed reading this post. Thanks for sharing!

Posted by: Saleenadavid | July 15, 2010 at 01:24 PM

~~~~~excellent post! but I am just trying to give an objective description of what these positions require.~~~~~

Posted by: cna classes michigan | July 16, 2010 at 08:47 AM

I am really interested in what you wrote here. This looks absolutely perfect. All these tinny details are made with lot of background knowledge.
http://www.thecnaschool.com/cna-schools-nevada

Posted by: Account Deleted | January 19, 2011 at 09:49 AM