Rethinking the Paradigm Paradigm

A few years ago, I wrote a magazine article on the mathematics of the stock market. The assignment required that I spend considerable time interviewing the experts and studying the various theories, which is to say, whether stocks engage in a random walk of unpredictable fluctuations or whether their movements can be predetermined. If the latter is true, then the market is indeed a game that can be beaten by the better players, not just the lucky ones. I concluded, as do most experts and virtually all experimental studies on the subject, that for 99.99 percent or so of traders, buying a stock is a proposition, over the short term, at least, no more predictable than a coin toss, and losing money is as likely a result as making it. The remaining infinitesimal fraction comprises those pros who have spent fortunes on computing systems that will sift through vast amounts of data and find the exceedingly subtle patterns in the ebb and flow of stocks. They are also the ones who have the financial wherewithal to profit from those patterns before they vanish.

The “new economy” arrived, however, shortly after the publication of my article. I watched my friends and relatives, none of whom had shown particular signs of genius, cash in on the explosive growth of high-tech, Internet and dot-com stocks. New rules were in effect, I was told, and money could be made risk-free and hand-over-fist. After 18 months of passive-aggressive skepticism, I decided that perhaps I was wrong and they were right, and I bought a few tech stocks. The market then crashed, as the new economy revealed itself to be the old economy in the emperor’s new clothes, and it took the bulk of my investment with it.

The moral of this story, as I see it, is that despite anything the philosopher of science Thomas Kuhn might have said to the contrary, new paradigms are extraordinarily enticing. In science, they are the breakthrough theories or remarkable discoveries, the revolutions-Kuhn’s “conceptual world views”-that take a moribund field, mired in a swamp of conflicting data, and move it en masse to a new and fertile intellectual realm. In the business of technology, they are, in effect, the physical or financial manifestations of our prayers being answered: the next ubiquitous operating system, the latest revisionist approach to curing cancer or selling widgets online. They are, in their most trivial manifestations, “the new new thing,” to quote the author Michael Lewis. They are also usually too good to be true-but we will get to that later.

As Kuhn saw it, and several generations of scientists, historians and journalists have told it since, new paradigms are accepted slowly, if not over the dead bodies of those who grew up with the old ones. Kuhn documented one great scientist after another, from Copernicus to Darwin to James Clerk Maxwell, who struggled relentlessly against the resistance of mediocre minds and later was vindicated. It was the German physicist Max Planck who set down the definitive words on the subject: “a new scientific truth,” Planck wrote, “does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.”

The salient question, however, is why the existence of “opponents”? And the answer, noted by Kuhn, although often ignored since, is surprisingly simple: a potential new paradigm or a remarkable breakthrough has opponents primarily because the data supporting it are not persuasive. Albert Einstein, for instance, may have refused to accept that God plays dice with the universe-the essence of quantum mechanics, which holds that the universe at its heart is a probabilistic and uncertain place-simply because the data supporting the existence of God’s alleged gambling habit were then ambiguous. After a sufficient hypothesis and test, the data supporting the quantum mechanics revolution became compelling, and even Einstein was convinced (although perhaps not happy about it).

And that’s the point: while scientists and technocrats will exhibit their fair share of inertial pettiness, the rejection of a potential new paradigm by the relevant experts is inevitably due to that institutionalized skepticism without which science no longer functions as a means to finding reliable knowledge. There are, after all, an infinite number of spectacular but erroneous breakthroughs for every one that stands the test of time. With these overwhelming odds, skepticism serves as the immune system for science, protecting the well-tested body of reliable knowledge from chronic infection by pathological phenomena that may play well in the media but can’t be reproduced in the laboratory. This skepticism is even more important in confronting the latest alleged technological wizardry, because lives are often on the line, not just investments. As the physicist Richard Feynman put it, speaking in particular of the demise of the space shuttle Challenger, “reality must take precedence over public relations, for nature cannot be fooled.”

Such is not the case with mankind in general, however, scientists or otherwise. Good scientists are trained to be skeptical so as not to be deluded or, more particularly, as Feynman said, not to delude themselves. Indeed, in both science and technology, it is not embarrassing to be skeptical of a brilliant new paradigm or a remarkable breakthrough that turns out years later to be right. It’s simply the nature of the job.

This leads us back to Kuhn, because virtually every paradigm-busting breakthrough, whether scientific or technological, is launched from a position of evidentiary weakness. The brilliant minds, the progenitors of new technological or scientific paradigms, are the ones who can extract the truth while it is still mired in that swamp of conflicting data. So it is that a new paradigm is likely to attract opponents: it is proposed while the supporting evidence is still ambiguous. Most experts will then oppose it for the right reasons (the evidence ain’t convincing) rather than the wrong ones (close-minded adherence to the old world view)-despite anything Kuhn, the press or the beleaguered scientists and inventors might say to the contrary.

The history of science and technology is actually rife with new paradigms that were so compelling they were accepted with little opposition. The emergence of molecular biology is one example, as the British biologist Lewis Wolpert has noted. “The evidence from the structure of DNA and other key discoveries was so persuasive that almost everyone-certainly the young-got caught up in the excitement of what is clearly a new age for biology,” he writes. Or take Claude Shannon’s theory of information, backbone of today’s digital revolution. When Shannon published his theory in 1948, it was so compelling, recounts author M. Mitchell Waldrop (see “Claude Shannon: Reluctant Father of the Digital Age,” TR July/August 2001), that it “exploded with the force of a bomb. Around MIT, the reaction was, ‘Brilliant! Why didn’t I think of that?’” Simply put, the excitement of a legitimate new discovery is a stronger motivating force in science than petty self-interest.

In this era when a potential new paradigm can be worth the wealth of OPEC, when an entire field of science or technology can emerge virtually overnight on the basis of a single scientific publication, the challenge lies in differentiating the valid paradigmatic breakthrough from the compelling fantasy. Both, after all, will garner supporters and copious press, because both promise wondrous things. The press, however, will be worthless at rendering sober judgment; reporters will back the new-paradigm angle because that’s the better story. They will allude to the abundance of believers as proof that the new paradigm is correct. If there happens to be an abundance of skeptics, the reporters will cite them as evidence that Kuhn was right and that the experts are petty and lacking in vision, rather than as evidence that the new new thing is worthy of skepticism.

The reality is that vigorous skepticism aimed at a potential new paradigm means one of two things and usually both: first, that the spectacular breakthrough or the wondrous paradigm is indeed too good to be true, and second, that the reasons to be skeptical are very good ones. If the evidence supporting the new paradigm reaches a high enough pitch, then the skepticism will fade. (Although, as the new economy suggests, even if it does, we should keep our heads.) The longer that skepticism lasts, the more likely it is that the new paradigm is delusional and will eventually evaporate in the harsh light of reality. Should you buy into it despite the skepticism, then, as one of my old engineering professors liked to say, you pays your money and you takes your chances.