Learning theory and Physics

Learning theory is about the process of induction, that is the process of building theories or models from observations.
Most of what physicits do is actually induction about natural phenomena. So one may wonder whether there can be some relationship between Physics and Learning Theory.
One could argue that Physics is using induction in a particular setting, while Learning Theory is studying induction in general, so they cannot really be compared.
But here are some surprising connections:

1. In quantum physics Bell's inequality provides a test for the existence of "hidden variables" that explain entanglement. This inequality is based on a statistical reasoning.
2. Some physicists study the connection between Bayes formula for updating probabilities and the collapse of the wave function when measurements of a quantum system are made (see e.g. the work of Christopher Fuchs). There are even "Bayesian" and "non-Bayesian" physicists, just like the Machine Learning people !
3. Even further, Lucien Hardy tries to rethink the way physics theories are built up. His starting point is that the work of any physicist is to accumulate and correlate data. He thus develops physics theories as theories for how data should be handled ! (see e.g. http://arxiv.org/PS_cache/gr-qc/pdf/0509/0509120.pdf)

Comments

[Matthias]

Hi Olivier,

It's nice to follow your blog. It reminds me of our long discussions...
Physics and learning theory, as someone coming the physics community and now working in learning theory the connection of these two fields is a very interesting question.
In principle I think you are right when you are saying physics and learning theory are similar, since physics basically tries to infer natural laws from measurements. However I see two main differences.

First a more direct comparison. In physics we want to infer a law, that is a differential equation, and not only a function. However one could argue that we would like to learn the solution (a function) to the differential equation for all possible initial conditions. The problem I see here is that there are probably a lot of
initial conditions (so to say the differential equation is a very good compression algorithm). A second difference is that measurements are usually not
stochastic but deterministic.
The second difference is the level of abstract thinking needed to infer physical laws (which is done by humans). As an example take Newtons gravitational law. To infer that an apple falling from a tree follows the same law as the earth going around the sun requires a very high level of abstract thinking. I don't see how you can build this into a traditional statistical learning framework.

Nevertheless this should not prevent research into this direction. I remember
that there exists a book like "Physics from Fisher information". But the book
seems not to be appreciated very much, see the review of R. F. Streater on his homepage.

[Olivier Bousquet]

Hi Matthias,

Nice to hear from you and thanks for the comment.

You are right about the differences you mention, but my feeling is that there should be a way to place learning theory at a higher level (possibly forgetting the statistical aspect). What I mean is that we could think of learning theory as a theory of how to build theories from observations (not just functions).
It is clear that we are far from being able to account for the thinking that led to Newton's gravitational law, but it would be nice to set this as some sort of long-term goal of the development of the theory.