This article has been inspired by two separate sources.
The key substance is clearly an exposition of some of the views of the late and great Richard Feynman. If you do not know anything about Feynman, then you owe it to yourself to find out. Surely You're Joking, Mr. Feynman is a good start - order it from Amazon today.
Another article that contributed to some of these thoughts is Tversky & Kahneman (1974) - "Judgment under uncertainty; heuristics and biases", that I found fascinating.
The main thrust of Feynmans paper "This Unscientific Age" published as part of "The Meaning Of It All", is that, despite the progress and impact of science on our lives, in many ways we cannot say that we live "in a scientific age".
Feynman makes many interesting points, from the lack of appreciation of the beauty of natural laws, to the willing acceptance of mystical claptrap.
It is interesting to consider his arguments alongside the Tversky & Kahneman paper, in that they demonstrate how poor humans are at making rational judgements.
Feynman identifies the handling of uncertainty as a key element in a scientific approach.
A common misconception is that science deals with hard knowledge. At one level this is correct since it is rigorous testing and analysis of real events that lead to scientific theories. But a theory is always just that - a theory. The theory is the embodiment of scientific knowledge. The payoff is the ability to use this knowledge to predict future behaviours. Different levels of confidence may be placed in any theory, but there is always a degree of uncertainty. At the simplest level no measurement can be absolute, so any theory can only be tested to the accuracy of possible measurements at the time.
But this is not considered a bad thing by scientists. They are on a voyage of discovery. Observe an event, produce a theory to explain the event, design experiments to test predictions, test the theory, revise the theory... etc.
Theories are never proven, simply awaiting to be disproved, discarded and replaced.
The Tversky & Kahnman paper focusses on the abilities of humans to make rational judgement in situations of uncertainty. If we use intuition and gut instinct to guide our actions it can be demonstrated that we have very poor abilities.
The point of all this is that we should rarely trust our instincts when it comes to making decisions. As humans, it seems that we are not naturally very smart.
This is well learned when it comes to designing systems of any type. We need to continually ask ourseleves if we have really understood things correctly, because it is most likely that we have made fundamental mistakes.
Game show setting. There are 3 doors, behind one of which is a prize. Monty Hall, the host, asks you to pick a door, any door. You pick door A (say). Monty opens door B (say) and shows voila there is nothing behind door B. Gives you the choice of either sticking with your original choice of door A, or switching to door C.
Should you switch?
If you think that you now have a 50-50 chance, think again.
Intuition is a dangerous thing!