Machine Learning Thoughts

Happiness of a scientist III: testing ideas

If you like rationalizing events (see Happiness of a scientist I), you would probably enjoy convincing yourself that your jobs actually fits with your initial expectations, and this will require a certain amount of rationalization and a posteriori reinterpretation of your expectations ;-)
So this is why I will try to do now.

Scientific research is about understanding phenomena (whether natural [e.g. physics], human [e.g. economics] or abstract [e.g. mathematics]). A scientific breakthrough can either be a discovery of a new phenomenon, a new insight about an existing phenomenon, or an explanation/justification/proof of an insight.
The point here is that it is all about having a better understanding of things.
What for? Well, besides the joy of understanding, the immediate consequence is the ability to positively impact the real world: e.g. build new systems or better ones, do things you could not do before, make a step forward technologically...
Of course, this is not necessarily the main motivation nor is it the main justification for scientific research, but it can be seen as a nice side-effect of understanding.

In any case, if you come up with a new idea, whether you care about applying this idea or not, the first thing you want do to is to test this idea. Testing can be done in two ways

• publishing it (so that your peers can comment on it, argue, verify, derive consequences...)
• experiment it (so that you can verify by yourself that the new understanding is indeed correct)

It is often difficult and time-consuming to design and execute experiments, so many scientists are satisfied by publishing.
But if one is given the necessary resources, experimenting can be a good substitute to publishing.

In private companies, not only do you often have resources for experimenting ideas, but also you are encouraged to do so (this is your part of your 80%) and even further, to go beyond experiments towards real-world applications (which give the ultimate validation of your ideas).

The conclusion I want to draw from all this is the following: one can be very happy doing research in a company because publishing is advantageously replaced by real-world testing of ideas.

[Note: I carefully avoided the topic of long-term research vs short-term engineering solutions... I might deal with this in a future post, although it is more comfortable not to face this at the moment ;)]

Happiness of a scientist II: the 80/20 rule

Having tried to do scientific research at several different places, I have observed the following, which has been confirmed by several colleagues:

No matter where you are, it is almost impossible to spend more than 20% of your time doing research.

Moderating this claim:

1. First of all, let me define what I mean by research. I essentially mean exploratory intellectual activity. So in a way, I am not restricting myself to scientific research, and I possibly include other activities such as brainstorming about the strategy of a company for example. But, for the rest of the discussion, let us focus on scientific research.
2. Secondly, I shall add that the above claim is only valid provided you care about having a balanced life. Many people manage to do more research but they then have to sacrifice something (e.g., family life, material conditions...). Finding a good balance is a complex optimization problem with many local minima (and probably a multi-objective one).
3. Thirdly, this is of course an average value and you can find people who manage to do more than 20% or people who cannot do more than 10%. But the point is that the distribution is very short-tailed: it is easy to reach 20%, hard to reach 30% and almost impossible to reach 40%...

Justification of the claim:
To make the above claim a bit more clear, let me give some examples of activities one has to carry out in the remaining 80%:
Being a professor, you have to prepare classes, teach these classes, organize exams, grade the students, deal with scheduling, deal with various administrative duties, take care of PhD students...
Being a research scientist in academia, you possibly have to review papers, organize conferences, file grant applications, deal with administrative duties, prepare lectures...
Being a research scientist in a private company, you have to spend a lot of time working on improving existing systems or developing new ones, but the point is that you have to instantiate your research ideas into something that has impact on the company.

Of course, I am not complaining about all these extra duties, but I am just trying to give an objective description of what these positions require (sometimes implicitely). The subtle point is that often these extra duties are not fully mandatory (e.g., one could live without reviewing papers) but performing them is often a necessary condition for obtaining certain types of rewards (recognition by peers, being in good terms with colleagues, promotions...)

Consequences:
So, admitting that reader is now partially convinced about the truth of the above claim, I shall derive the following consequence about how to choose a job:

Instead of choosing a job based on the amount of time that you will be allowed to spend on research, rather choose it based on what exactly the 80% other activities are.

Indeed, you should rather pick a job whose 80% activities you find enjoyable (unless you really  can stand doing things you do not like for the sake of the remaining 20%).

Happiness of a scientist I: rationalization

This post is a first of a series of somewhat personal posts. This will not be about Machine Learning in particular, nor will it be about science, but rather about scientists, their life and their quest for happiness.

I often find that rationalizing situations or events that one is going through helps a lot. It introduces (at least for me) a distance between oneself and the event, making it an object of scientific investigation rather than some inescapable fate.
This is why I intend to write this post and the few others that will follow. Indeed, having started my career as a scientist in academia, I am now working for a private company which does not quite coincide with my initial goals (or dreams if you will), but the more I think about it (and rationalize it), the more I find myself comfortable and happy about this situation.

Talking about all this surely comes from a selfish interest: psychologists would tell you that it does you good to speak about your problems. But on the other hand, I sincerely hope that these few thoughts will help those who may going through the same kind of questioning...

The Failure of AI

In the early days of AI, scientists thought they would be able to build an intelligent computer by the end of the 20th century. This raised various fears about computers eventually taking over the world, and human beings replaced by robots.
Not only this did not happen yet but we are very far from this!
But what is even worse is that we are now following a somewhat opposite trend. There are many tasks at which humans are far better than computers, but instead of trying to build better algorithms for these tasks, people are now trying to find ways to better make use of human intelligence, or rather to automate this usage!

Two examples of this: Luis von Ahn's "Artificial artificial intelligence" and Amazon's "Mechanical Turk".

Luis von Ahn has designed a couple of internet games whose purpose is to make players perform useful tasks such as labeling images. Amazon is taking this to the industrial scale (although not as a game anymore) by allowing people to design programs which include calls to web services which are actually executed by paid humans (e.g. your program calls "translate(text)" and the text is sent to someone who translates the text and return the result)!

Update (21/12/2006): I found a related article in the Boston Globe, citing other examples of the same kind, such as Mozes Mob, a cell phone Q&A service powered by humans.

Machine Learning Search Engine

Google custom search engines are a very nice idea.
I have started to create such a custom search engine for Machine Learning research. It is still preliminary, so feel free to suggest URLs or categories to add to it.
Here is the link (Update: I removed the search box from this blog as it did not work, you should use the above link to try it out)

Machine Learning Videos

I have just created a page with a selection of cool Machine Learning videos (mainly talks given at Google). Feel free to suggest some more to add to this page...

Winning The DARPA Grand Challenge

A very nice talk by Sebastian Thrun about how he and his team designed an autonomous driving vehicle for the DARPA Grand Challenge. What is interesting is the use of Machine Learning to train the program.

Making Machine Learning More Scientific

The title of this post may seem awkward as most people working on ML would consider themselves as scientists.
However, ML is still a young field, and as most scientific fields in their youth, the methods and practices are still being defined and formalized. Indeed, scientific fields usually start unorganized and, as they mature, begin to have proper definitions for their goals, proper vocabulary and proper methods for verifying their results

My feeling is that Machine Learning is still at an early age of its development. Many things are still lacking and I will try to list some of them:

• Foundations
  • Agreed-upon vocabulary: there are many different fields that concurred to the infancy of the domain, they all have different ways of calling the same object, for example: function, model, hypothesis, concept... It would be nice to speak the same language.
  • Statement and definition of the goals: what is the object under study?
  • Statement of main problems: providing a list of problems in a clear and formalized way (this has already started with the open problems sessions at the COLT conference, but there is yet to be a consensus about which ones are more important for advancing the field).
    
• Experimental methods
  • Common language for specifying algorithms and experiments so that they can be reproduced. Indeed, too often, the papers describe results that no one can reproduce because some specific ad-hoc tuning of the parameters was used. I recently found this article which describes an attempt to provide a language for automating experiments.
  • Define and agree upon a set of measures or criteria that are relevant to assess the success of an algorithm (there could be many, depending on the field of application, the idea would be to use as many as possible rather than focusing only on a handful)
  • Datasets: go beyond the UCI database, share the data, define a common standard for this data or provide tools to convert from one format into another.
  • An idea could be to run evaluations more or less like a challenge (as the ones that have recently been proposed, for example by the Pascal network), but with a way to share even the code of the algorithm: you could send your code to a server and the server runs it on many databases, and measures many different criteria, generates a report and adds the entry to a database, thus creating a big source of data for meta-learning.
  • The goal is to say which algorithm is better for which problems, but not in general, and especially to avoid the dataset selection issue.
• Knowledge management
  • Write more introductory papers, or textbooks.
  • Collect and maintain relevant bibliographic references.
  • Revisit what has been done or discovered so far with a fresh look: avoid repeating always the same things without trying to understand them or putting them under a new light (e.g. "SVM are great because they are well-founded", "Boosting is maximizing the margin", "Fisher's discriminant is valid only when the classes are distributed as Gaussians", "What if the data is not iid", "Bayes rule is optimal")

Of course, this may take some years and a lot of effort, but hopefully, as more money is poured into Machine Learning research and applications, this will happen...

Learning Theory disconnected from Practice?

Is learning theory really answering the right questions?
More precisely, does working on (statistical) learning theory bounds really help designing better algorithms?

Let me first say that this note is not against anyone in particular.
I know most of the people working on SLT bounds and I do have a lot of respect for them, and I also wrote papers that were trying to justify an algorithm from a theoretical analysis. So I am just questioning the approach but not judging anyone following it!

I have spent a lot of time thinking about what theory did really bring that could not have been obtained without it. Because this is really the question. If you consider the algorithms that people use in practice, the two important questions are whether any of those algorithms could not have been inspired by considerations of a non-SLT nature, and whether the SLT analysis brings an understanding of why they work.

Regarding the first one (inspiration), it is risky to tell what could have inspired an algorithm a posteriori. But I believe that the amount of effort spent on trying to prove bounds and then obtain a criterion to minimize for an algorithm would be better used if people were just trying to find new criteria directly. Indeed, there are surely many ideas that we (theoreticians) refrain from trying or even expressing, because we do not see how they are connected to the standard patterns of SLT analysis!

Regarding the second one (whether bounds justify an algorithm), I would be even less confident: the only thing we can infer from most learning theory bounds is that the algorithm we are studying is not meaningless, that is, it will eventually make good predictions (with enough samples), but these bounds cannot really help comparing two algorithms. Also, they rarely (if ever) can justify the use of a specific criterion.

So, I think that the theoretical analysis is mainly a way to popularize an algorithm and to raise its visibility. The effect is then that more people try it out, and streamline it. So in the end, the algorithm may be adopted, but a theoretical analysis rarely justifies the algorithm and never provides guarantees. Hence theory is a way to attract attention to an algorithm.
It should also be a way to get new insights for the development of new algorithms, but this happens much less frequently than is claimed!

Learning abilities of computers and humans

While commenting on the comments to my previous note (see here), I thought about the comparison of human and computers in terms of learning performance. Does this comparison even make sense? If it does, on which basis, or with which criteria can it be made?

If you adopt a purely theoretical point of view, the no free lunch theorem tells you that you just cannot compare two learning machines in general. So all comparisons should be based on a specific, restricted set of reference problems.

If you adopt a Bayesian point of view, learning is very easy: once you have chosen your prior (an likelihood function), it is just matter of computation to get the posterior. So that, again, you cannot compare two learning machines, you can only compare their priors. Their performance will be more or less directly related to how well the prior matches the problem at hand (i.e. how high is the prior probability of the problem to be learned).

So we may already have very efficient learning algorithms (probably even better than the brain because they can compute more precisely and much faster -- although one can discuss what computing means in this context), and we still believe computers are not able to match humans learning performance because we compare them on tasks for which humans have much better priors.

Of course I am not saying anything new here: Bayesians would tell you that this has always been clear for them, you only have to build a good prior and you are done.

But building a good prior is not an easy task: it requires to define the right features, to find the right notion of smoothness... and there is basically no guidance for this! Moreover, it is completely problem-specific. So, apart from helping to implement Bayes rule efficiently, general (i.e. application-independent) Machine Learning research cannot help much.
One could then draw the conclusion that the essence of the learning problem is not statistical but computational.

But I still think there are important statistical problems, and I will come back to this issue in a future note...