The title of this post may seem awkward as most people working on ML would consider themselves as scientists.
However, ML is still a young field, and as most scientific fields in their youth, the methods and practices are still being defined and formalized. Indeed, scientific fields usually start unorganized and, as they mature, begin to have proper definitions for their goals, proper vocabulary and proper methods for verifying their results
My feeling is that Machine Learning is still at an early age of its development. Many things are still lacking and I will try to list some of them:
- Foundations
- Agreed-upon vocabulary: there are many different fields that concurred to the infancy of the domain, they all have different ways of calling the same object, for example: function, model, hypothesis, concept... It would be nice to speak the same language.
- Statement and definition of the goals: what is the object under study?
- Statement of main problems: providing a list of problems in a clear and formalized way (this has already started with the open problems sessions at the COLT conference, but there is yet to be a consensus about which ones are more important for advancing the field).
- Experimental methods
- Common language for specifying algorithms and experiments so that they can be reproduced. Indeed, too often, the papers describe results that no one can reproduce because some specific ad-hoc tuning of the parameters was used. I recently found this article which describes an attempt to provide a language for automating experiments.
- Define and agree upon a set of measures or criteria that are relevant to assess the success of an algorithm (there could be many, depending on the field of application, the idea would be to use as many as possible rather than focusing only on a handful)
- Datasets: go beyond the UCI database, share the data, define a common standard for this data or provide tools to convert from one format into another.
- An idea could be to run evaluations more or less like a challenge (as the ones that have recently been proposed, for example by the Pascal network), but with a way to share even the code of the algorithm: you could send your code to a server and the server runs it on many databases, and measures many different criteria, generates a report and adds the entry to a database, thus creating a big source of data for meta-learning.
- The goal is to say which algorithm is better for which problems, but not in general, and especially to avoid the dataset selection issue.
- Knowledge management
- Write more introductory papers, or textbooks.
- Collect and maintain relevant bibliographic references.
- Revisit what has been done or discovered so far with a fresh look: avoid repeating always the same things without trying to understand them or putting them under a new light (e.g. "SVM are great because they are well-founded", "Boosting is maximizing the margin", "Fisher's discriminant is valid only when the classes are distributed as Gaussians", "What if the data is not iid", "Bayes rule is optimal")
Of course, this may take some years and a lot of effort, but hopefully, as more money is poured into Machine Learning research and applications, this will happen...
As for foundations, something tells me we ought to be trying to understand 'Nonparametric Information Geometry' (http://www-personal.umich.edu/~junz/zhang%20aism.pdf ) and interpreting 'Unifying Divergence Minimization and Statistical Inference via Convex Duality' (http://ttic.uchicago.edu/~altun/pubs/AltSmo-COLT06.pdf ) in its terms.
Posted by: David Corfield | June 21, 2006 at 02:04 PM
Doesn't machine learning goes back to the fifties? I don't see what is young about from a computer science perspective.
Reference:
Samuel, A.L., Some studies in machine learning using the game of checkers, IBM. J, 1959.
Also, you have to be realistic as to what can be agreed upon and what cannot be. People have been doing Fourier analysis for literally two centuries and even the basic definitions will vary, sometimes not even in trivial ways.
You could argue that once you agree on all the terms, there is nothing left do discuss and a field dies. ;-)
Anyhow, I think that Machine Learning, just like AI in general, is too far off from applications for its own goods. Abstracting problems is nice, mathematicians have been doing it for centuries... and see where it lead them.
Well, I better not get started with this topic.
Posted by: Daniel Lemire | June 21, 2006 at 07:54 PM
Thanks for the pointers, David!
Regarding the nonparameteric information geometry, I think this is quite relevant indeed: in my opinion, information geometry is a very nice way to study the properties of statistical models and it has provided deep insights in various domains of statistics, however it has been so far restricted to finite dimensional manifolds (i.e. parametric models). I have been looking for extensions of this framework to infinite dimensional manifolds and this work by Zhang seems to be a very good starting point.
Regarding the other paper 'Unifying Divergence Minimization and Statistical Inference via Convex Duality', it is a nice piece of work trying to put in a common framework a variety of commonly used algorithms. This is surely a useful endeavour and a good way to clarify things.
These are two good examples of papers trying to provide a clear and solid theoretical framework for further study of learning problems and algorithms.
Posted by: Olivier Bousquet | June 25, 2006 at 11:21 PM
Hi Daniel,
I guess you are right on the points you make:
- ML is not that young, but I think, that unlike other (younger fields), it has yet to build its own foundations.
- It is surely not a requirement that everyone agrees on the definitions and vocabulary, diversity is a sign of health and activity. But I am just afraid that many people are building on fragile grounds and that much time is wasted working on irrelevant problems or already solved ones, just because things are not more organized and clarified.
- Machine Learning is too far off from applications: I cannot agree more! However, I strongly believe that some abstraction (and thus some theory) is needed. Of course, this should not be the starting point, and this is probably the mistake that has been made. But if nobody cares about abstracting things, people just reinvent the same things or just make epsilon changes to existing algorithms and nothing is gained in the long run.
Posted by: Olivier Bousquet | June 25, 2006 at 11:35 PM
Olivier, I basically agree with all the points you make. One problem in my view, is that "Machine Learning" may be in itself too vast a field in order to hope to be able to find a reasonable unifying set of concepts and goals withoug being reductive. Consider such diverse topics as classification, detection, reinforcement learning, image analysis, and game theory which all overlap with ML. However, because the task is difficult does not mean one should not try it; at the very least unifying some subparts of ML can help tremendously so that, for example, a "fundamental" result of one subcommunity can be ported more easily to another.
I also wanted to comment on your previous and related entry, although it is more specifically focused (whether LT is diconnected from practice). As a typical "guy who proves bounds", I have become increasingly irritated recently that theoretical results have, in fact, so little impact on practice. How do we justify our existence then? Come to think of it, in what other scientific field do we have such a gap between theory and practice?
The NIPS workshop "(ab)use of bounds" from 2 years ago was more or less centred on this issue and it is not clear what progress has been made since. John (Langford) drew in introduction a half-joke diagram with two axes respectively representing "fear" and "respect" that engineers felt for various theoretical results. Information theory, theoretical physics and signal processing where pretty high on both axes, while LT was close to the origin...
As you point out, one reason for this lack of respect might be the absence of a strong foundation that would at the same time be indispensable for anyone working in the field and relatively easily taught. Let's consider, say, the fundamental law of dymanics for mechanics, the Fourier transform for signal processing, Shannon's theorems for information theory... What do we have in LT? VC theory? But no practitioner really cares about it. It seems that all that a practitioner needs to get going is a informal understanding of what is under-, over-fitting, and regularization. Do we have something better to offer?
Posted by: G. Blanchard | July 05, 2006 at 12:43 PM
Following up on my comment above, I'm collecting together a list of papers on information geometry and machine learning. Details and opportunity to suggest additions here (http://www.dcorfield.pwp.blueyonder.co.uk/2006/07/information-geometry-and-machine.html).
Posted by: David Corfield | July 26, 2006 at 02:28 PM
I meant of course
http://www.dcorfield.pwp.blueyonder.co.uk/2006/07/information-geometry-and-machine.html
Posted by: David Corfield | July 26, 2006 at 02:31 PM
I'm probably not experienced enough to make a very informed comment on this topic. But in my personal opinion, since most of our problems are close to computer science and are usually evaluated using measures derived from computer science theory, it would be a good idea to first extend the notions of computer science to learning theory. For example, in theoretical computer science, the terms "problem" "solution" and "performance of a solution" have well-defined meanings. This is not exactly the case for learning theory. The recent work on reductions is perhaps a step in this direction. Nevertheless, much remains to be done.
Posted by: Alekh Agarwal | August 31, 2006 at 07:49 AM
I do not care about the 'true' age of machine learning, but I am pretty sure that there is still a lot to do.
I copied your text completely to
http://miningdrugs.blogspot.com/2006/09/machine-learning-lacking-standards.html
because this is a slightly different audience, and if you would know the literature in cheminformatics you would not see any problems in machine learning.
I can tell you that the bad quality, the lacking data, the 'term invention' for old algorithms, and so on is a huge problem. Once, an editor was requesting major revisions for a cheminformatics journal publication, because 'the term cross-validation is not know to the main audience'. Well, then they have to learn it!!!
Joerg
Posted by: Joerg Kurt Wegner | September 15, 2006 at 01:58 AM
Olivier,
I read your excellent post as "making machine learning more rigourous", and, while there is a lot of room for improvement, I think we have done a fairly good job here. While we still have a lot to learn from staticticians, Machine learning folks have been leading the fight for more systematic experiments in computer vision and computational linguistics.
One just had to read Vapnik's introduction about Popper to realize how obsessed machine learning has been with being the ultimate realization of the "scientific method". And the fact that we do not agree on the method underlies the central discussions in our field. But is our obsession focused in the right direction?
Patrick
Posted by: Patrick Haffner | October 02, 2006 at 07:13 PM
A great man is always willing to be little—R. W. Emerson
It is not enough to be industrious, so are the ants. What are you industrious for?— H. D. Thoreau
Posted by: lacoste shirts | July 08, 2010 at 03:10 AM
had fun reading all your thoughts...thank you!
Posted by: Nursing top | July 12, 2010 at 07:12 PM
We’re conditioned to think that our lives revolve around great moments. But great moments often catch us unaware - beautifully wrapped in what others may consider a small one.
Posted by: coach sale | July 16, 2010 at 08:33 AM
Wow Adam Smith, of John Maynard Keynes, and of Ronald Coase they are really economic geniuses..Nice article..I liked reading it..
Posted by: cna certification test | July 16, 2010 at 01:51 PM
Rightly said, machine learning has not gone so further. It's still in the early age and there more things that we can do to make to systematic and scientific.
Posted by: How to become a CNA | September 15, 2010 at 06:05 AM
The broad goal of machine learning is to automate this type of process, so that computer-automated predictions can make a task more efficient, accurate, or cost-effective than it would be using only human decision making.People should understand these things rather than getting confused.
Posted by: Dog Food | October 05, 2010 at 05:30 PM
there more things that we can do to make to systematic and scientific.
Posted by: coco chanel | November 02, 2010 at 07:56 AM
Never leave that until tomorrow, which you can do today.
Posted by: cheap jordans | November 04, 2010 at 03:52 AM
I'm probably not experienced enough to make a very informed comment on this topic. But in my personal opinion, since most of our problems are close to computer science and are usually evaluated using measures derived from computer science theory, it would be a good idea to first extend the notions of computer science to learning theory. For example, in theoretical computer science, the terms "problem" "solution" and "performance of a solution" have well-defined meanings. This is not exactly the case for learning theory. The recent work on reductions is perhaps a step in this direction. Nevertheless, much remains to be done.
Posted by: Alex | January 18, 2011 at 07:05 AM
I won't accept such sort of family though it is quite comfortable. I prefer the family with mom which is not rich but ample in my heart
Posted by: Chanel Handbags | February 18, 2011 at 04:16 AM
But if not, you are out of luck.
Posted by: Louis Vuitton Handbags | February 24, 2011 at 02:09 AM
Im positive about it. I guess im going to finish it.
Posted by: vigilon security | March 11, 2011 at 10:54 AM
Hola,
Ha hecho un trabajo muy bueno. Hay muchas personas en busca de eso ahora van a encontrar suficientes fuentes por tus consejos.
espera para obtener más consejos acerca de que
Posted by: Careprost | March 16, 2011 at 07:16 AM
I am not aware of ML but I find it interesting after reading your post.Points which we are lacking need to be overcome so that we can enhance our knowledge and skills.All the information of the post shows your effort.Thanks for working so hard for us.Keep doing it
Posted by: software testing webinar | March 18, 2011 at 10:10 AM
Machine learning does have a long way to go still, but it will get there eventually. Anytime we deal with something so advanced it takes time to reap the benefits.
Posted by: how to become a cna | March 22, 2011 at 06:38 PM