Machine Learning Thoughts

Making Machine Learning More Scientific

The title of this post may seem awkward as most people working on ML would consider themselves as scientists.
However, ML is still a young field, and as most scientific fields in their youth, the methods and practices are still being defined and formalized. Indeed, scientific fields usually start unorganized and, as they mature, begin to have proper definitions for their goals, proper vocabulary and proper methods for verifying their results

My feeling is that Machine Learning is still at an early age of its development. Many things are still lacking and I will try to list some of them:

• Foundations
  • Agreed-upon vocabulary: there are many different fields that concurred to the infancy of the domain, they all have different ways of calling the same object, for example: function, model, hypothesis, concept... It would be nice to speak the same language.
  • Statement and definition of the goals: what is the object under study?
  • Statement of main problems: providing a list of problems in a clear and formalized way (this has already started with the open problems sessions at the COLT conference, but there is yet to be a consensus about which ones are more important for advancing the field).
    
• Experimental methods
  • Common language for specifying algorithms and experiments so that they can be reproduced. Indeed, too often, the papers describe results that no one can reproduce because some specific ad-hoc tuning of the parameters was used. I recently found this article which describes an attempt to provide a language for automating experiments.
  • Define and agree upon a set of measures or criteria that are relevant to assess the success of an algorithm (there could be many, depending on the field of application, the idea would be to use as many as possible rather than focusing only on a handful)
  • Datasets: go beyond the UCI database, share the data, define a common standard for this data or provide tools to convert from one format into another.
  • An idea could be to run evaluations more or less like a challenge (as the ones that have recently been proposed, for example by the Pascal network), but with a way to share even the code of the algorithm: you could send your code to a server and the server runs it on many databases, and measures many different criteria, generates a report and adds the entry to a database, thus creating a big source of data for meta-learning.
  • The goal is to say which algorithm is better for which problems, but not in general, and especially to avoid the dataset selection issue.
• Knowledge management
  • Write more introductory papers, or textbooks.
  • Collect and maintain relevant bibliographic references.
  • Revisit what has been done or discovered so far with a fresh look: avoid repeating always the same things without trying to understand them or putting them under a new light (e.g. "SVM are great because they are well-founded", "Boosting is maximizing the margin", "Fisher's discriminant is valid only when the classes are distributed as Gaussians", "What if the data is not iid", "Bayes rule is optimal")

Of course, this may take some years and a lot of effort, but hopefully, as more money is poured into Machine Learning research and applications, this will happen...

Learning Theory disconnected from Practice?

Is learning theory really answering the right questions?
More precisely, does working on (statistical) learning theory bounds really help designing better algorithms?

Let me first say that this note is not against anyone in particular.
I know most of the people working on SLT bounds and I do have a lot of respect for them, and I also wrote papers that were trying to justify an algorithm from a theoretical analysis. So I am just questioning the approach but not judging anyone following it!

I have spent a lot of time thinking about what theory did really bring that could not have been obtained without it. Because this is really the question. If you consider the algorithms that people use in practice, the two important questions are whether any of those algorithms could not have been inspired by considerations of a non-SLT nature, and whether the SLT analysis brings an understanding of why they work.

Regarding the first one (inspiration), it is risky to tell what could have inspired an algorithm a posteriori. But I believe that the amount of effort spent on trying to prove bounds and then obtain a criterion to minimize for an algorithm would be better used if people were just trying to find new criteria directly. Indeed, there are surely many ideas that we (theoreticians) refrain from trying or even expressing, because we do not see how they are connected to the standard patterns of SLT analysis!

Regarding the second one (whether bounds justify an algorithm), I would be even less confident: the only thing we can infer from most learning theory bounds is that the algorithm we are studying is not meaningless, that is, it will eventually make good predictions (with enough samples), but these bounds cannot really help comparing two algorithms. Also, they rarely (if ever) can justify the use of a specific criterion.

So, I think that the theoretical analysis is mainly a way to popularize an algorithm and to raise its visibility. The effect is then that more people try it out, and streamline it. So in the end, the algorithm may be adopted, but a theoretical analysis rarely justifies the algorithm and never provides guarantees. Hence theory is a way to attract attention to an algorithm.
It should also be a way to get new insights for the development of new algorithms, but this happens much less frequently than is claimed!

Learning abilities of computers and humans

While commenting on the comments to my previous note (see here), I thought about the comparison of human and computers in terms of learning performance. Does this comparison even make sense? If it does, on which basis, or with which criteria can it be made?

If you adopt a purely theoretical point of view, the no free lunch theorem tells you that you just cannot compare two learning machines in general. So all comparisons should be based on a specific, restricted set of reference problems.

If you adopt a Bayesian point of view, learning is very easy: once you have chosen your prior (an likelihood function), it is just matter of computation to get the posterior. So that, again, you cannot compare two learning machines, you can only compare their priors. Their performance will be more or less directly related to how well the prior matches the problem at hand (i.e. how high is the prior probability of the problem to be learned).

So we may already have very efficient learning algorithms (probably even better than the brain because they can compute more precisely and much faster -- although one can discuss what computing means in this context), and we still believe computers are not able to match humans learning performance because we compare them on tasks for which humans have much better priors.

Of course I am not saying anything new here: Bayesians would tell you that this has always been clear for them, you only have to build a good prior and you are done.

But building a good prior is not an easy task: it requires to define the right features, to find the right notion of smoothness... and there is basically no guidance for this! Moreover, it is completely problem-specific. So, apart from helping to implement Bayes rule efficiently, general (i.e. application-independent) Machine Learning research cannot help much.
One could then draw the conclusion that the essence of the learning problem is not statistical but computational.

But I still think there are important statistical problems, and I will come back to this issue in a future note...