Overfitting and psychology

I was thinking about writing a post in order to explain overfitting in simple terms.
The way I wanted to start was something like:
"Overfitting is the phenomenon of building a model that agrees well with the observed data but has no predictive ability (it does not agree with unseen or future data)"

This is probably fine, but somewhat formal, and sometimes a simple example is better than a long explanation. The example I needed was given to me by Bernhard Schölkopf who noticed a connection between a psychological disease called "obsessive compulsive disorder" and the overfitting phenomenon:

Someone who is O.C. will for instance take on and off his trousers five times in a row, since he once did it and it causes something positive to happen (or he thinks it helped avoid a catastrophe).
A lot of children have this, in a mild form (e.g. they do not want to step on the edge of the tiles in the floor, etc.). So maybe it is actually part of our inference engine, trying to learn decisions/actions. If this is true, then O.C. is nothing but an error in the inference enginge. Maybe the wrong capacity, leading to overfitting.

Comments

[Christian Walder]

Interesting. Similar perhaps is superstitious behaviour - attributing random events (noise) in a causal way. I heard (thanks jez) of an experiment in which a group of chickens were fed at random intervals. After some time, the chickens behaved strangely, with various bizarre dances - but in fact they were just repeating what they had by chance done simultaneously to being fed in the past. I don’t know if this really occurs (they say chickens have a short memory) but the idea makes some sense.

[Michael Schröder]

Dear Olivier, dear Christian,
I saw similar behaviour in rats that had been trained in skinner boxes. The animals had learned by try and error to perform a sequence of certain actions in order to obtain a reward. (Example: activation of a number of handles in a defined order.)

Even though the reward was given deterministically some rats had their eyes half-closed or held their tails in strange positions during that procedure.

[buy viagra]

Congrats. This article is very good.

[guanacaste costa rica]

You would think people would wake-up after that one, but no -- they keep looking for that pot of gold that will send them to Hawaii for the rest of their lives.

[Asics shoes]

A man is not old as long as he is seeking something. A man is not old until regrets take the place of dreams.

[Generic Viagra]

In my view -- and I know this is controversial -- a much larger proportion of the US population will have to be employed in growing the food we eat. There are many ways of arranging this, some more fair than others, and I hope the better angels of our nature steer us in the direction of fairness and justice.

[viagra online]

This is going to be a great deal of fun, which is one of the things I love about both shows - the sense of playfulness with the genre. Seems like they're going to make good use of Fargo's history of working with AIs with personality issues, a nifty choice to hang a plot around. Looking forward to it!

[Start playing]

Greets
i want to thank the blogger very much not only for this post but also for his all previous efforts. I found your site to be very interesting. I will be coming back to site for more information.
Laters

[cialis super active]

i am agree with the previous comment.

[aöf]

Congrats. This article is very good.

[Boopsie Woopsie]

Dude, you're catching up to animal learning psychology of the 1960s.

http://en.wikipedia.org/wiki/B._F._Skinner#Superstition_in_the_pigeon

[Air Jordan]

A sound mind in a sound body is a short but full description of a happy state in this world. Do you agree?

[spokane physician]

My thoughts are similar when i also observed it.