Machine Learning (Theory)

As for foundations, something tells me we ought to be trying to understand 'Nonparametric Information Geometry' (http://www-personal.umich.edu/~junz/zhang%20aism.pdf ) and interpreting 'Unifying Divergence Minimization and Statistical Inference via Convex Duality' (http://ttic.uchicago.edu/~altun/pubs/AltSmo-COLT06.pdf ) in its terms.

Doesn't machine learning goes back to the fifties? I don't see what is young about from a computer science perspective.

Reference:

Samuel, A.L., Some studies in machine learning using the game of checkers, IBM. J, 1959.

Also, you have to be realistic as to what can be agreed upon and what cannot be. People have been doing Fourier analysis for literally two centuries and even the basic definitions will vary, sometimes not even in trivial ways.

You could argue that once you agree on all the terms, there is nothing left do discuss and a field dies. ;-)

Anyhow, I think that Machine Learning, just like AI in general, is too far off from applications for its own goods. Abstracting problems is nice, mathematicians have been doing it for centuries... and see where it lead them.

Well, I better not get started with this topic.

Thanks for the pointers, David!

Regarding the nonparameteric information geometry, I think this is quite relevant indeed: in my opinion, information geometry is a very nice way to study the properties of statistical models and it has provided deep insights in various domains of statistics, however it has been so far restricted to finite dimensional manifolds (i.e. parametric models). I have been looking for extensions of this framework to infinite dimensional manifolds and this work by Zhang seems to be a very good starting point.

Regarding the other paper 'Unifying Divergence Minimization and Statistical Inference via Convex Duality', it is a nice piece of work trying to put in a common framework a variety of commonly used algorithms. This is surely a useful endeavour and a good way to clarify things.

These are two good examples of papers trying to provide a clear and solid theoretical framework for further study of learning problems and algorithms.

Hi Daniel,

I guess you are right on the points you make:
- ML is not that young, but I think, that unlike other (younger fields), it has yet to build its own foundations.
- It is surely not a requirement that everyone agrees on the definitions and vocabulary, diversity is a sign of health and activity. But I am just afraid that many people are building on fragile grounds and that much time is wasted working on irrelevant problems or already solved ones, just because things are not more organized and clarified.
- Machine Learning is too far off from applications: I cannot agree more! However, I strongly believe that some abstraction (and thus some theory) is needed. Of course, this should not be the starting point, and this is probably the mistake that has been made. But if nobody cares about abstracting things, people just reinvent the same things or just make epsilon changes to existing algorithms and nothing is gained in the long run.

Olivier, I basically agree with all the points you make. One problem in my view, is that "Machine Learning" may be in itself too vast a field in order to hope to be able to find a reasonable unifying set of concepts and goals withoug being reductive. Consider such diverse topics as classification, detection, reinforcement learning, image analysis, and game theory which all overlap with ML. However, because the task is difficult does not mean one should not try it; at the very least unifying some subparts of ML can help tremendously so that, for example, a "fundamental" result of one subcommunity can be ported more easily to another.

I also wanted to comment on your previous and related entry, although it is more specifically focused (whether LT is diconnected from practice). As a typical "guy who proves bounds", I have become increasingly irritated recently that theoretical results have, in fact, so little impact on practice. How do we justify our existence then? Come to think of it, in what other scientific field do we have such a gap between theory and practice?

The NIPS workshop "(ab)use of bounds" from 2 years ago was more or less centred on this issue and it is not clear what progress has been made since. John (Langford) drew in introduction a half-joke diagram with two axes respectively representing "fear" and "respect" that engineers felt for various theoretical results. Information theory, theoretical physics and signal processing where pretty high on both axes, while LT was close to the origin...

As you point out, one reason for this lack of respect might be the absence of a strong foundation that would at the same time be indispensable for anyone working in the field and relatively easily taught. Let's consider, say, the fundamental law of dymanics for mechanics, the Fourier transform for signal processing, Shannon's theorems for information theory... What do we have in LT? VC theory? But no practitioner really cares about it. It seems that all that a practitioner needs to get going is a informal understanding of what is under-, over-fitting, and regularization. Do we have something better to offer?

Following up on my comment above, I'm collecting together a list of papers on information geometry and machine learning. Details and opportunity to suggest additions here (http://www.dcorfield.pwp.blueyonder.co.uk/2006/07/information-geometry-and-machine.html).

I'm probably not experienced enough to make a very informed comment on this topic. But in my personal opinion, since most of our problems are close to computer science and are usually evaluated using measures derived from computer science theory, it would be a good idea to first extend the notions of computer science to learning theory. For example, in theoretical computer science, the terms "problem" "solution" and "performance of a solution" have well-defined meanings. This is not exactly the case for learning theory. The recent work on reductions is perhaps a step in this direction. Nevertheless, much remains to be done.

I do not care about the 'true' age of machine learning, but I am pretty sure that there is still a lot to do.

I copied your text completely to
http://miningdrugs.blogspot.com/2006/09/machine-learning-lacking-standards.html
because this is a slightly different audience, and if you would know the literature in cheminformatics you would not see any problems in machine learning.

I can tell you that the bad quality, the lacking data, the 'term invention' for old algorithms, and so on is a huge problem. Once, an editor was requesting major revisions for a cheminformatics journal publication, because 'the term cross-validation is not know to the main audience'. Well, then they have to learn it!!!

Joerg

Olivier,

I read your excellent post as "making machine learning more rigourous", and, while there is a lot of room for improvement, I think we have done a fairly good job here. While we still have a lot to learn from staticticians, Machine learning folks have been leading the fight for more systematic experiments in computer vision and computational linguistics.

One just had to read Vapnik's introduction about Popper to realize how obsessed machine learning has been with being the ultimate realization of the "scientific method". And the fact that we do not agree on the method underlies the central discussions in our field. But is our obsession focused in the right direction?

Patrick

A great man is always willing to be little—R. W. Emerson
It is not enough to be industrious, so are the ants. What are you industrious for?— H. D. Thoreau

We’re conditioned to think that our lives revolve around great moments. But great moments often catch us unaware - beautifully wrapped in what others may consider a small one.

Wow Adam Smith, of John Maynard Keynes, and of Ronald Coase they are really economic geniuses..Nice article..I liked reading it..

Rightly said, machine learning has not gone so further. It's still in the early age and there more things that we can do to make to systematic and scientific.

The broad goal of machine learning is to automate this type of process, so that computer-automated predictions can make a task more efficient, accurate, or cost-effective than it would be using only human decision making.People should understand these things rather than getting confused.

there more things that we can do to make to systematic and scientific.

Never leave that until tomorrow, which you can do today.

I'm probably not experienced enough to make a very informed comment on this topic. But in my personal opinion, since most of our problems are close to computer science and are usually evaluated using measures derived from computer science theory, it would be a good idea to first extend the notions of computer science to learning theory. For example, in theoretical computer science, the terms "problem" "solution" and "performance of a solution" have well-defined meanings. This is not exactly the case for learning theory. The recent work on reductions is perhaps a step in this direction. Nevertheless, much remains to be done.

I won't accept such sort of family though it is quite comfortable. I prefer the family with mom which is not rich but ample in my heart

But if not, you are out of luck.

Im positive about it. I guess im going to finish it.

Hola,
Ha hecho un trabajo muy bueno. Hay muchas personas en busca de eso ahora van a encontrar suficientes fuentes por tus consejos.
espera para obtener más consejos acerca de que

I am not aware of ML but I find it interesting after reading your post.Points which we are lacking need to be overcome so that we can enhance our knowledge and skills.All the information of the post shows your effort.Thanks for working so hard for us.Keep doing it

Machine learning does have a long way to go still, but it will get there eventually. Anytime we deal with something so advanced it takes time to reap the benefits.

The comments to this entry are closed.