12-18-2013, 07:07 PM
It's time to play contrarian -- the eye test actually is useful in certain circumstances, generally in life. I'd prefer pure computer/statistics to the "eye test" as applied by most in college football -- which generally seems to be defined as "motivated reasoning to back up whatever result I've previously determined is correct -- but it's not the optimal approach.
What is the optimal approach? Before answering the question, let's define what I mean by eye test. I think it's the idea that, "perception using human senses, intuition and cognition is superior than machine- or statistically-based approaches." Well, oftentimes, a combination of computer/statistical approaches plus expertise (that is, the eye test) will prove superior to a mere computer. A computer is better at chess than a human, but a well-trained human and a computer is better than both. Similarly, if you're traveling to the Rose Bowl by plane, most of your trip will be on autopilot -- that is, by computer -- but the human pilot will go ahead and land the plan for you. If you get injured and get a scan, odds are your radiologist will be doing some computer-aided diagnosis.
Humans, of course, are biased -- predictably so. The question often is whether you're able to detect under what circumstances you're just letting your "fast" thinking substitute for correct thinking -- that is, whether you're meta-rational or not. Most people (Dunning-Kruger Effect) are not, which is sad, as they'd be well-served by letting the computer take control. However some aren't and those philosopher-kings should be ruling in tandem with computers.
The problem is that computers often take population-based approaches and sand over the quirks inherent in any piece of data. In general, that's fine; many of the things we think of as errors are actually just motivated reasoning, meant to convince us that This Time Is Different. But oftentimes there are flaws in the model. For example, looking back to the last time we saw Stanford take the field, I'm convinced part of the reason computer models were so bullish on the Sun Devils was their twenty-point victory over SC. The computers believe SC is one of the best teams in the country; it's not hard to disagree about a team with Lee, Agholor, Williams, Allen, etc. But what the computers didn't know is that K. was fired directly after that game, Orgeron took over, and (unfortunately for us) SC began to play much better. Someone who appreciates the flaws in the model can therefore correct a bit for its error.
(Of course, sometimes both models and humans suffer from the same errors, i.e. the "black swan" events. There's no real cure for this.)
Anyway, I guess the real solution here is to be smarter, or at least be smarter about your own faults.
What is the optimal approach? Before answering the question, let's define what I mean by eye test. I think it's the idea that, "perception using human senses, intuition and cognition is superior than machine- or statistically-based approaches." Well, oftentimes, a combination of computer/statistical approaches plus expertise (that is, the eye test) will prove superior to a mere computer. A computer is better at chess than a human, but a well-trained human and a computer is better than both. Similarly, if you're traveling to the Rose Bowl by plane, most of your trip will be on autopilot -- that is, by computer -- but the human pilot will go ahead and land the plan for you. If you get injured and get a scan, odds are your radiologist will be doing some computer-aided diagnosis.
Humans, of course, are biased -- predictably so. The question often is whether you're able to detect under what circumstances you're just letting your "fast" thinking substitute for correct thinking -- that is, whether you're meta-rational or not. Most people (Dunning-Kruger Effect) are not, which is sad, as they'd be well-served by letting the computer take control. However some aren't and those philosopher-kings should be ruling in tandem with computers.
The problem is that computers often take population-based approaches and sand over the quirks inherent in any piece of data. In general, that's fine; many of the things we think of as errors are actually just motivated reasoning, meant to convince us that This Time Is Different. But oftentimes there are flaws in the model. For example, looking back to the last time we saw Stanford take the field, I'm convinced part of the reason computer models were so bullish on the Sun Devils was their twenty-point victory over SC. The computers believe SC is one of the best teams in the country; it's not hard to disagree about a team with Lee, Agholor, Williams, Allen, etc. But what the computers didn't know is that K. was fired directly after that game, Orgeron took over, and (unfortunately for us) SC began to play much better. Someone who appreciates the flaws in the model can therefore correct a bit for its error.
(Of course, sometimes both models and humans suffer from the same errors, i.e. the "black swan" events. There's no real cure for this.)
Anyway, I guess the real solution here is to be smarter, or at least be smarter about your own faults.