Randomness: Why the Future Is So Hard To Predict

Which is why, when you tell someone, in jest, just how random they are, you are far closer to the truth than you realize. JL

Shane Parrish reports in Farnam Street:

The course of history was determined by a small chance event which had no seeming importance when it happened. This is why predicting the future, even with immense computing power, is impossible. The effects of randomness, with small inputs (be)ing disproportionate and massive, makes prediction very difficult. That’s why we must appreciate the role of randomness. “Each of us human beings is the product of a long sequence of accidents, any of which could have turned out differently.”

What parts of reality are the product of an accident? The physicist Murray Gell-Mann thought the answer was “just about everything.” And to Gell-Mann, understanding this idea was the the key to understanding how complex systems work.

Gell-Mann believed two things caused what we see in the world:

1. A set of fundamental laws
2. Random “accidents” — the little blips that could have gone either way, and had they, would have produced a very different kind of world.

Gell-Mann pulled the second part from Francis Crick, co-discoverer of the human genetic code, who argued that the code itself may well have been an “accident” of physical history rather than a uniquely necessary arrangement.
These accidents become “frozen” in time, and have a great effect on all subsequent developments; complex life itself is an example of something that did happen a certain way but probably could have happened other ways — we know this from looking at the physics.
This idea of fundamental laws plus accidents, and the non-linear second order effects they produce, became the science of complexity and chaos theory. Gell-Mann discussed the fascinating idea further in a 1996 essay on Edge:

Each of us human beings, for example, is the product of an enormously long sequence of accidents, any of which could have turned out differently. Think of the fluctuations that produced our galaxy, the accidents that led to the formation of the solar system, including the condensation of dust and gas that produced Earth, the accidents that helped to determine the particular way that life began to evolve on Earth, and the accidents that contributed to the evolution of particular species with particular characteristics, including the special features of the human species. Each of us individuals has genes that result from a long sequence of accidental mutations and chance matings, as well as natural selection.
Now, most single accidents make very little difference to the future, but others may have widespread ramifications, many diverse consequences all traceable to one chance event that could have turned out differently. Those we call frozen accidents.

These “frozen accidents” occur at every nested level of the world: As Gell-Mann points out, they are an outcome in physics (the physical laws we observe may be accidents of history); in biology (our genetic code is largely a byproduct of “advantageous accidents” as discussed by Crick); and in human history, as we’ll discuss. In other words, the phenomenon hits all three buckets of knowledge.
Gell-Mann gives a great example of how this plays out on the human scale:

For instance, Henry VIII became king of England because his older brother Arthur died. From the accident of that death flowed all the coins, all the charters, all the other records, all the history books mentioning Henry VIII; all the different events of his reign, including the manner of separation of the Church of England from the Roman Catholic Church; and of course the whole succession of subsequent monarchs of England and of Great Britain, to say nothing of the antics of Charles and Diana. The accumulation of frozen accidents is what gives the world its effective complexity.

The most important idea here is that the frozen accidents of history have a nonlinear effect on everything that comes after. The complexity we see comes from simple rules and many, many “bounces” that could have gone in any direction. Once they go a certain way, there is no return.
This principle is illustrated wonderfully in the book The Origin of Wealth by Eric Beinhocker. The first example comes from 19th century history:

In the late 1800s, “Buffalo Bill” Cody created a show called Buffalo Bill’s Wild West Show, which toured the United States, putting on exhibitions of gun fighting, horsemanship, and other cowboy skills. One of the show’s most popular acts was a woman named Phoebe Moses, nicknamed Annie Oakley. Annie was reputed to have been able to shoot the head off of a running quail by age twelve, and in Buffalo Bill’s show, she put on a demonstration of marksmanship that included shooting flames off candles, and corks out of bottles. For her grand finale, Annie would announce that she would shoot the end off a lit cigarette held in a man’s mouth, and ask for a brave volunteer from the audience. Since no one was ever courageous enough to come forward, Annie hid her husband, Frank, in the audience. He would “volunteer,” and they would complete the trick together. In 1980, when the Wild West Show was touring Europe, a young crown prince (and later, kaiser), Wilhelm, was in the audience. When the grand finale came, much to Annie’s surprise, the macho crown prince stood up and volunteered. The future German kaiser strode into the ring, placed the cigarette in his mouth, and stood ready. Annie, who had been up late the night before in the local beer garden, was unnerved by this unexpected development. She lined the cigarette up in her sights, squeezed…and hit it right on the target.
Many people have speculated that if at that moment, there had been a slight tremor in Annie’s hand, then World War I might never have happened. If World War I had not happened, 8.5 million soldiers and 13 million civilian lives would have been saved. Furthermore, if Annie’s hand had trembled and World War I had not happened, Hitler would not have risen from the ashes of a defeated Germany, and Lenin would not have overthrown a demoralized Russian government. The entire course of twentieth-century history might have been changed by the merest quiver of a hand at a critical moment. Yet, at the time, there was no way anyone could have known the momentous nature of the event.

This isn’t to say that other big events, many bad, would not have precipitated in the 20th century. Almost certainly there would have been wars and upheavals.
But the actual course of history was in some part determined by small chance event which had no seeming importance when it happened. The impact of Wilhelm being alive rather than dead was totally non-linear. (A small non-event had a massively disproportionate effect on what happened later.)
This is why predicting the future, even with immense computing power, is an impossible task. The chaotic effects of randomness, with small inputs having disproportionate and massive effects, makes prediction a very difficult task. That’s why we must appreciate the role of randomness in the world and seek to protect against it.
Another great illustration from The Origin of Wealth is a famous story in the world of technology:

[In 1980] IBM approached a small company with forty employees in Bellevue, Washington. The company, called Microsoft, was run by a Harvard dropout named bill Gates and his friend Paul Allen. IBM wanted to talk to the small company about creating a version of the programming language BASIC for the new PC. At their meeting, IBM asked Gates for his advice on what operating systems (OS) the new machine should run. Gates suggested that IBM talk to Gary Kildall of Digital Research, whose CP/M operating system had become the standard in the hobbyist world of microcomputers. But Kildall was suspicious of the blue suits from IBM and when IBM tried to meet him, he went hot-air ballooning, leaving his wife and lawyer to talk to the bewildered executives, along with instructions not to sign even a confidentiality agreement. The frustrated IBM executives returned to Gates and asked if he would be interested in the OS project. Despite never having written an OS, Gates said yes. He then turned around and license a product appropriately named Quick and Dirty Operating System, or Q-DOS, from a small company called Seattle Computer Products for $50,000, modified it, and then relicensed it to IBM as PC-DOS. As IBM and Microsoft were going through the final language for the agreement, Gates asked for a small change. He wanted to retain the rights to sell his DOS on non-IBM machines in a version called MS-DOS. Gates was giving the company a good price, and IBM was more interested in PC hardware than software sales, so it agreed. The contract was signed on August 12, 1981. The rest, as they say, is history. Today, Microsoft is a company worth $270 billion while IBM is worth $140 billion.

At any point in that story, business history could have gone a much different way: Kildall could have avoided hot-air ballooning, IBM could have refused Gates’ offer, Microsoft could have not gotten the license for QDOS. Yet this little episode resulted in massive wealth for Gates and a long period of trouble for IBM.
Predicting the outcomes of a complex system must clear a pretty major hurdle: The prediction must be robust to non-linear “accidents” with a chain of unforeseen causation. In some situations this is doable: We can confidently rule out that Microsoft will not go broke in the next 12 months; the chain of events needed to take it under quickly is so low as to be negligible, no matter how you compute it. (Even IBM made it through the above scenario, although not unscathed.)
But as history rolls on and more “accidents” accumulate year by year, a “Fog of the Future” rolls in to obscure our view. In order to operate in such a world, we must learn that predicting is inferior to building systems that don’t require prediction, as Mother Nature does. And if we must predict, must confine our predictions to areas with few variables that lie in our circle of competence, and understand the consequences if we’re wrong.
If this topic is interesting to you, try exploring the rest of the Origin of Wealth, which discusses complexity in the economic realm in great (but readable) detail; also check out the rest of Murray Gell-Mann’s essay on Edge. Gell-Mann also wrote a book on the topic called The Quark and the Jaguar which is worth checking out. The best writer on randomness and robustness in the face of an uncertain future, is of course Nassim Taleb, whom we have written about many times.