Thomas Bayes

photo from Wikipedia

This is possibly a portrait of Thomas Bayes, the creator of Bayes' Theorem. He was an English non-conformist Presbyterian minister and statistician. But you don't need me to regurgitate his Wikipedia entry. He never actually published the equation for which he is now the man of the moment-- it was published after his death.

He was elected a Fellow of the Royal Society in 1742, some think because of his 1736 work (published anonymously) "An Introduction to the Doctrine of Fluxions, and a Defence of the Mathematicians Against the Objections of the Author of The Analyst. I gotta say, I love that title. "Fluxions" was the name for Newton's calculus, and the "Author of The Analyst" was Bishop George Berkeley, who apparently had issues with it.

I'm not familiar with the religious or political climates of Europe in the 18th century, but in general I like seeing words like "non-conformist".

I would have loved to have been able to say, "Yeah I aced my Fluxions midterm." I wonder why that didn't stick. Maybe a branding thing from the publisher: "Hey Newt, I know you like Fluxions, but I think Calculus has legs."

Bayes' Theorem is basically a method of correcting our assumptions with data. I'll illustrate with a typical example next.

Stats 101

It was spring or summer term. I was sitting in a classroom in a building adjacent to a grassy quad. Bikini-clad co-eds outside the window sunning themselves in the days before sun was considered unhealthy.

It was the first day of my first statistics class, and the professor opened with this story problem:
"A family moves in next door. You know they have 2 kids. You can see one of them playing in the yard-- it's a girl. What are the odds that the other is a boy?"

The answer wasn't 50-50. It also wasn't a trick question. Nothing about slightly more women in the population, or including potential effects of nuclear radiation on sexual identity.

The answer went like this:
With 2 kids, you have 4 possibilities.

Older	Younger
Boy	Boy
Boy	Girl
Girl	Boy
Girl	Girl

We see a girl, so the first option "boy-boy" is impossible. Out of the 3 remaining options, 2 out of 3 have a boy as the other child, so the answer is 2/3 or 67% chance that the other child is a boy.

Straightforward? Yep.

Clear? Absolutely.

Was I confident that I would be able to come to that conclusion by myself? Not at all. The fear I felt is likely the reason I still remember that story problem.

The trick is that when we see the girl playing in the yard we DON'T know if she is the older or younger child.

This leads to another interesting feature:

When we DON'T know if the girl is the older or younger child, we are 67% certain that the other child is a boy.

If we are told that, for example, the girl in the yard is the older child, then we are back to 50% certain that the other child is a boy. In this case, having MORE information makes us LESS certain.

Machine Learning vs. Statistics

Machine Learning uses statistical methods. It is rooted fairly firmly in maths. However, there seems to be somewhat of a rivalry between Statisticians and Machine Learning people.

Statisticians are mathematicians. Purists. They are concerned with things like model confirmation and population sampling. New approaches are expected to be accompanied by a proof.

Machine Learning people seem to be pragmatists. Their main concern is, "Are you making better predictions?" If the answer is "yes", then they're interested and open to new techniques.

Intro

I've been looking into machine learning for several months now, and I have the distinct impression that, even though I'm seeing machine learning being used damn-near everywhere, what I've seen is still the very tippy-tippy-top of a humongous can-eat-the-Titanic-iceberg-for-a-light-snack iceberg. This blog will be a place where I can put my latest learnings and musings on the subject.