The Optimism Pessimism Continuum

What about change in general?

The lesson here is clear: We can’t sustain our plans and projects if we’re disconnected from our passionate and distorted perceptions about their future. So even if a pill should come into existence that would allow you to disconnect from the ups and downs of your feelings about the future, you would be foolish to take it. Contending with our conflicting waves of optimism and pessimism is an inescapable part of planning for the future. As we cannot do without them, we’d do well to try to harness their benefits and minimize the damage they do.

Half of this case is easy to make: Convincing human beings that they need optimism has never been difficult. A conviction that one can beat unfavorable odds seems to be baked into the human psyche. After all, worldwide, lotteries collect the equivalent of $275 billion annually, even though any rational person would describe them (as one rational person did) as “a tax on people who don’t understand probability.”

Perhaps we are inclined to ignore the odds against us because we, like all living creatures, have already beaten tremendous odds. We did it just by existing. In a universe with plenty of barren rocks, we live on a planet capable of supporting life, which actually has living creatures on it, some few of which are pretty intelligent—something 4 billion years of evolution never managed before us. All very unlikely success stories. And what is true about our species is also true for any individual human being. At the most literal level, the odds are always against any particular egg being fertilized and growing to become a healthy newborn. And the odds of any particular sperm cell reaching its goal are infinitesimally low. That you exist to cherish a certain mad optimism is a sort of proof that mad optimism can be justified.

Some medical research suggests that optimism accords with our biology. People who score high on standard psychological measures of optimism tended to have healthier diets, have lower Body Mass Index scores, lower blood pressure, blood sugar and cholesterol, according to a recent study in the journal Health Behavior & Policy Review. Optimists are also known in the epidemiological literature for getting fewer colds and suffering less from flu. When they get cancer, they tend to have better outcomes. As friends, neighbors and colleagues, they are supposedly more pleasant and productive—“happiness magnets,” as the former Olympian Mary Lou Retton puts it.

Yet a closer look at the science makes it clear that optimists have—surprise, surprise—overstated their case. Pessimists (if they thought it would do any good, which they might not) can offer their own studies to counter each point about the merits of the rosy view.

Health benefits? A very long-term Stanford University study of 1,528 people over eight decades was able to track the entire lives of people whose relative optimism and pessimism had been assessed when they were children. It found that the children who scored highest in optimism were the soonest to die as adults. Similarly, a study in the journal Psychology and Aging, reported last year in The Wall Street Journal, found that people older than 65 who expected to be less satisfied with their lives in five years were more likely to be alive and healthy years later, compared to those who were more optimistic. Better work and better friendships? Not necessarily, when the going gets tough. When events fail to confirm their outlook, optimists cope with the stress of adversity less well than pessimists, according to a host of studies. One likely reason: Pessimists aren’t easily disappointed.

Even if there is no clear verdict about which outlook is healthier, there is still a widespread prejudice that optimism makes for better friends, neighbors and co-workers. Are there any reasons you might prefer engaging with gloomy Gusses rather than people of cheerier outlook?

As it happens, there are. One is rooted in the fact that pessimism isn’t merely the absence of optimism—it is its own mode of thinking and feeling about the future. One that offers some strengths that optimists lack.

By dint of their uncertainty that they can prevail over future challenges, pessimists tend to worry. As they game out potential failures, they can be more likely to imagine solutions. In other words, while optimists on a team will expect solutions to arise if needed, generating those solutions may require a few pessimists who actually thought in detail about how to handle trouble. According to Adam Perkins, a psychologist at King’s College, London, the pessimistic tendency to imagine threats is a natural spur to thoughts about how to meet those threats—in other words, to creativity.

Other psychological research aligns with his theory of a link between angst and performance. Experiments conducted on students—some as young as 6 and 7, some in college—continually find that those who have been lulled into a happy mood score lower on tests and games than do those who have been made sad or angry just before the exercise.

What about change in general?

These results suggest that overconfidence arising from optimism leads people to skip the effort they need to anticipate and solve future problems. But the problems of optimism may run deeper than lack of interest in details. When they do settle down to predict how the future will treat them, optimists often fail to engage reality—even when they have experience to guide them.

This is because our innate tendency to prefer rose-colored glasses causes us to think our goals can be met more easily than they can be in reality. In fact, the reliable tendency of estimates to be too rosy has a name, coined by the Nobel Prize winning psychologist Daniel Kahneman and his late colleague Amos Tversky: the “planning fallacy.” This is the almost unavoidable tendency of organizations to underestimate the amount of time and money required to complete a project, due to an optimistic take on potential future problems and the organizations’ own capacities.

For example, Kahneman has noted, among rail projects undertaken worldwide between 1969 and 1998, estimates of the number of passengers who would use the system were on average 106 percent too high, while cost overruns ran an average of 45 percent. As Kahneman notes in his summary of his work on human bias and error, “Thinking Fast and Slow,” these data are striking because they cover three decades. They show that two decades of experience in railroad projects since the late 1960s did not improve the forecasting of projects started in the 1990s. The planning fallacy is as much a component of our personal lives as it is of our work. As Kahneman recounts, in 2002, a survey of American homeowners who had remodeled their kitchens found that, on average, they had expected the job to cost $18,658; in fact, they ended up paying an average of $38,769.

There are two types of errors in thought involved in these pratfalls of optimism. One is giving too much weight to social factors: We human beings are highly alert to the way others see us, and in convincing onlookers that we are skilled and efficient, we convince ourselves all too easily.

And often organizational politics adds its weight as well. As an acquaintance told me recently, discussing meetings about his construction work for a big urban government, when confronted by the boss’s grand but impossible vision, direct reports faced a choice: They could say “it cannot be done,” and earn a lot of ill will from their superior. Or they could say “yes, indeed!” and later report that they failed—which earns them a pat on the back for at least having tried. As the project planning wore on, he said, everyone involved learned that the second option was the only safe choice.

Aside from pressure from others to be too optimistic, there is a second source of error in optimistic assessments. It is, simply, making plans using the wrong frame of reference. To estimate future costs in money and time, your project should be compared objectively to others like it. Instead, people tend to compare it to the last one they personally worked on, or to those that seem to them to be similar. If that comparison yields an unacceptably gloomy projection, they then look for reasons why, to use a familiar jingle beloved of optimists, “This time is different.”

The cure for all this is simple: Request—and heed!—someone who has no incentive to misjudge the future. Planning needs an outside observer who has no stake in being admired, liked or rewarded, and has no cherished notion about why this project is special and no knowledge of your team’s last success story. The cold eye of the outsider, using the correct reference class to make comparisons, is much more likely to be accurate than the “inside view.”

This is easy to say, but social pressure and habits of thought incline us to resist or suppress the “outside view” of our plan. In a project Kahneman himself worked on, one team member with a great deal of experience had projected that the work would take seven to 10 years. But before Kahneman had prompted him to bring his knowledge to bear, the very same man had estimated that the work would take around two. In this case, a team member needed help from others to see his own “outside view” clearly.

What about change in general?

How, then, can you leverage the benefits of each outlook on life without succumbing to its ill effects? It’s a tricky challenge in management, both for yourself and your team.

Consider the experience of Keith Blount, creator of Scrivener, a writing app beloved of novelists, bloggers and scholars for its writer-friendly features (full disclosure: this article was created on the app). Begun as a personal project about 10 years ago, Scrivener launched as a commercial product in 2006, and Mac and then Windows versions of the app did well for six years. But by 2012, with the world going mobile, Blount decided it was time to create an iOS version.

He knew this would be a big project — it involved not only translating the app into a new operating system but also creating bomb-proof syncing for very complex files. He knew that the Windows version of the app had taken two and a half years to perfect, even though developers had told him it would take six months. (“Whenever you speak to a developer they say, ‘I can do that in six months, easy,’ ” he says wryly.) He optimistically announced that the iOS version was in development, and that it would be ready by the end of the year.

Then came the unpredictable events that dog every endeavor. Some were directly related to the work: Bomb-proof syncing of the app’s complex files, which users would need to switch between computers and mobile devices, proved much more difficult than anyone expected. Others were the sorts of curveballs that pessimists are sure will come and optimists think they’ll handle if and when necessary. In this case, the iOS developer hired for the project developed serious health problems, but didn’t want to admit defeat.

In retrospect, Blount reflects, he should have separated from the developer and found someone else. But optimism—intellectual, emotional and social—held him back. “The person wants to do it,” he says. “You want to give them that chance because you’ve got a personal relationship with that person. And they know the project really well at that point, so you’re kind of hoping they can get past these problems.”

The iOS app had to be delayed. Adding to the stress, some potential users complained—some to tech support staff that should have been spending their time on users’ issues with the Mac and Windows versions of the app.

A second developer came on to the project, then “hit a brick wall,” Blount recalls, and, again, was kept on too long. Much as we look to leaders to optimistically inspire us, he has concluded that there are other times when leadership requires insisting on pessimism. “That’s part of being the big bad boss,” he says. “Being the person who says, ‘This isn’t going as well.’ It’s hard to be that person when sometimes you need to be.”

Now, three years later, Blount is on his third developer and finally testing a product. However, he says, the company’s stance is now “we’re not going to make guesses about when it will be released, it’s just going to be finished when it is. We’re on that swing of the pendulum now.”

Of course, he reflects, he could have simply announced nothing and kept the iOS project a secret. However, in the software universe, “if you do that people think you’re not working on anything at all. People can be equally frustrated by that approach.”

The dance of optimism and pessimism, both in one’s own head and organization and in what one tells the public, is not easy to maintain. Both outlooks have their uses, and either, applied wrongly can sink a project. It is, as Blount says, “a really difficult balance.”