INAUGURAL ARTICLE by a Recently Elected Academy Member:Why postmortems fail

Most high-profile disasters are followed by demands for an investigation into what went wrong. Even before they start, calls for finding the missed warning signs and an explanation for why people did not “connect the dots” will be common. Unfortunately, however, the same combination of political pressures and the failure to adopt good social science methods that contributed to the initial failure usually lead to postmortems that are badly flawed. The high stakes mean that powerful actors will have strong incentives to see that certain conclusions are—and are not—drawn. Most postmortems also are marred by strong psychological biases, especially the assumption that incorrect inferences must have been the product of wrong ways of thinking, premature cognitive closure, the naive use of hindsight, and the neglect of the comparative method. Given this experience, I predict that the forthcoming inquiries into the January 6, 2021, storming of the US Capitol and the abrupt end to the Afghan government will stumble in many ways.

In the wake of high-profile disasters like the terrorist attacks on September 11, 2001, the destruction of the Challenger space shuttle, or the discovery that Iraq did not have active programs to produce weapons of mass destruction (WMD) in the years before the 2003 invasion, there are demands for an investigation into what went wrong with intelligence and policy making. Even before they start, calls for finding the missed warning signs and an explanation for why people did not “connect the dots” will be common, along with the expectation that a good inquiry will lead to changes that will make us much safer. Unfortunately, however, the same combination of political pressures and the failure to adopt good social science methods that contributed to the initial failure usually produces postmortems that are badly flawed, even if they produce some good information. This leads me to predict that the inquiries into the January 6, 2021, storming of the US Capitol and the abrupt end to the Afghan government in August 2021 will stumble in many ways. [Exceptions to this otherwise dreary pattern—studies that are better done—are generally produced by researchers who have had more social science education, are highly skilled, or who do the task well after the events, creating more room for perspective (1–3). There is also some evidence that organizations conducting routine postmortems, such as in the investigation of transportation accidents and medical mishaps, do it better (4–8).]I will look most closely at the American postmortems conducted over the past decade that examined major foreign policy failures. Because they were salient to the polity and generously funded, it is reasonable to expect they would have been done as well as possible. I have omitted only the congressional reports on the attack on the American diplomatic outpost at Benghazi on September 11 to 12, 2012, and the Mueller report on whether the Trump campaign conspired with Russia during the 2016 election and subsequently sought to obstruct the investigation. The former were so driven by the politics of attacking or defending Hillary Clinton, who was simultaneously the Secretary of State at the time and the Democratic candidate for president in 2016, that to take them as serious attempts at unraveling what happened would be a strain. The Mueller report was largely a fact-finding and legal document and so did not have the same purpose of understanding the events and the causal relationships at play. I believe that I have avoided the trap, discussed below, of only looking at cases that are likely to support my argument.There is no simple recipe for a successful postmortem, but there are roadmaps if not checklists that can help us judge the ones that have been done. To start with, humility in a double sense is in order. Not only is it likely that the case under consideration will be a difficult one, which means that the correct judgments were not likely to have been obvious at the time, but even later conclusions are likely to be disputable. A good postmortem then recognizes the ambiguities of the case, many of which may remain even after the best retrospective analysis.In this, it is important to separate judgments about why incorrect conclusions were reached from evaluations of the thinking and procedures that were involved. People can be right for the wrong reasons, and, even more troubling, can be wrong for the right reasons.In the analysis of why the contemporary judgments were reached and the causes of the errors that are believed to have been involved, standard social science methodology points to the value of the comparative method in trying to see whether the factors believed to have marred the effort were also present when the outcome was better. Related, it is not enough for postmortems to locate bits of evidence that are consistent with the explanations that they are providing; they must also try to see whether this evidence is inconsistent with alternative views. Common sense embodies cognitive shortcuts that need to be disciplined to avoid jumping to conclusions and seeing the evidence as pleasingly clear and consistent with favored views.It is of course easy—or at least easier—to be wise after the fact, something a good postmortem has to recognize and internalize. It is not only unfair to the contemporary actors but an impediment to good retrospective understanding to seize on evidence that now seems to be crucial or interpretations that we now think to be correct without taking the next step of analyzing whether and why they should have been seen as privileged at the time.The very fact that a disastrous failure occurred despite the existence of individuals and organizations designed to provide warning and take appropriate action indicates that the meaning of the course of events is not obvious, and this widens the space for the political and psychological biases that I will discuss below. Sometimes there may be bits of vital information that were not gathered, were overlooked, or blatantly misinterpreted, or there may have been outright incompetence, but most organizations are better than that. This means that in many instances, the case under examination will be an exception to many of our generalizations about how the world works (9, 10). To take a case I studied for the CIA, the Iranian revolution of 1979 is an exception to what political scientists and policy makers believe, which is that leaders who enjoy the support of their security forces will not be overthrown by mass movements. Indeed, the universal assumption behind postmortems that analysts and decision makers should have gotten right may not be correct. As I will discuss later, we live in a probabilistic universe and sometimes what happens is very unlikely, in which case it is far from clear that the decision makers should have acted differently. Such a conclusion is almost always psychologically and politically unacceptable if not unimaginable, however, and is almost never considered, let alone asserted as correct. The fact that the explanation for the events is not obvious also means that postmortems that are conducted without more attention to good methodology than was true of the analysis that produced the policy and the underpinning beliefs are likely to fall into traps similar to those that played a role in the failure itself.The area of my own expertise, international politics, presents three additional reasons why it is difficult both for contemporary actors to judge their environments and for postmortems to do better. Although these issues are not unique to international politics, they arise with great frequency there. First, it is often hard to understand why others are behaving as they are, especially when we are dealing with actors (individuals, organizations, or governments) who live in very different cultural and perceptual worlds from us. One of the other postmortems that I did for the US intelligence community (IC) was the failure to recognize that Saddam Hussein did not have active WMD programs in 2002 (11). The belief that he did rested in part on the fact that he had expelled the United Nations weapons inspectors at great cost to his regime. We now know that, contrary to what was believed not only by the United States, but by almost all countries, the reason behind Saddam’s decision to do so was not that he was hiding his programs. In retrospect and with access to interviews and an extensive documentary record, it is generally believed that Saddam felt he had to pretend to have WMD in order to deter Iran (12, 13). Even this explanation has been disputed (14, 15), however, which underscores the point that contemporary observers face very difficult problems when it comes to understanding the behavior of others, problems that may not be readily resolved even later with much more and better evidence. Iraq may be a particularly difficult case, but it is telling that more than 100 y later and with access to all existing records (some have been destroyed) (16), historians still debate the key issues related to the origins of World War I, especially the motives and intentions of Germany and Russia. In fact these debates largely mirror the ones that occurred among policy makers in 1914. It is very hard to get inside others’ heads.The second problem is that many situations that lead to postmortems involve competition and conflict. In the classical account (17, 18), the actors are in strategic interaction in that each tries to anticipate how others will act, knowing that others are doing likewise. This poses great intellectual difficulties for the actors and is one reason that policies can fail so badly that postmortems follow. These interactions also pose difficulties for the postmortems themselves. In some cases, leaders behave contrary to the theory and act as though they were playing a game against nature rather than against a protagonist in strategic interaction (19). When this is not the case, tracing how the actor expected others to behave is often difficult because decision makers rarely gratify historians by spelling out their thinking. Additionally, antagonists in conflict often have good reason to engage in concealment and deception (20–23). Of course actors understand this and try to penetrate these screens. But success is not guaranteed and so errors are common and can result in disastrous policy failures. Furthermore, the knowledge that concealment and deception are possible can lead the actor to discount accurate information. This was the case in the American judgment that Iraq had WMD programs at the start of the 21st century. Intelligence analysts knew that Iraq had been trained by the Soviet Union in elaborate concealment and deception techniques and they plausibly—but incorrectly—believed that this explained why they were not seeing more signs of these programs (24). In retrospect these puzzles are easier to unravel, but that does not mean they are always easy to solve and deception can pose challenges to postmortems.A third problem both for contemporary decision makers and for those conducting postmortems is that when mass behavior plays an important role, events can be subject to rapid feedback that is difficult to predict. Revolutions, for example, can be unthinkable until they become inevitable, to borrow the subtitle of a perceptive book about the overthrow of the Shah of Iran (25). That is, in a situation in which a large portion of the population opposes a dictatorial regime backed by security forces many people will join mass protests only when they come to believe that these protests will be so large that the chance of success is high and that of being killed by participating is low. And the bigger the protests one day, the greater the chances of even larger ones the next day because of the information that has been revealed. Related dynamics were at work with the disintegration of the Afghan security forces in August 2021. But the tipping points (26–28) involved are hard to foresee at the time and only somewhat less difficult to tease out in retrospect.The difficulties of the task of conducting adequate postmortems make it easier for biases to play a large role. In prominent cases the political needs and preferences of powerful groups and individuals come in, sometimes blatantly. When President Lyndon Johnson established the Warren Commission to analyze the assassination of his predecessor, he made it clear to Chief Justice Earl Warren and other members that any hint that the USSR or Cuba were involved would increase the chance of nuclear war. In parallel, he did not object when Allen Dulles, Director of the CIA and member of the commission, withheld information on the plots to assassinate Fidel Castro and on Lee Harvey Oswald’s contacts with Cuba since knowledge of them would point to the possibility that the Cuban leader was involved (29). When the space shuttle Challenger exploded a few minutes into its takeoff, President Ronald Reagan similarly appointed a national commission chaired by former Secretary of State William Rogers to get to the bottom of what happened. But Rogers understood that the program was essential to American prestige and the heated competition with the Soviet Union and so while the commission looked at the technical problems that caused the disaster, it did not probe deeply into the organizational and cultural characteristics of NASA that predisposed it to overlook potentially deadly problems. It also shied away from acknowledging that the complex advanced technology incorporated into the program made it essentially experimental and that even with reforms another accident was likely (30). It took a superb study by organizational sociologists to elucidate these issues (30, 31), and in an example of the impact of organizational politics, NASA ignored this situation until it suffered another disaster with the shuttle Columbia.Politics can also limit the scope of the inquiry. When the bipartisan 9/11 Commission decided that its report would be unanimous this had the effect if not the purpose of preventing a close examination of the policies of the George W. Bush administration. The public record was quite clear: President Bush and his colleagues believed that the main threat to the United States came from other powerful states, most obviously China and Russia, and terrorism was not only a secondary concern, but could only be significant if it was supported by a strong state. It is then not surprising that al Qaeda received little high-level attention in the first 9 mo of the administration. While incorrect, I do not believe this approach was unreasonable, a product of blind ideology, or the result of the rejection of everything the previous administration had done. But it was an important part of the story, and one that could not be recounted if the report was to be endorsed by all its members.Sometimes the political bias is more subtle, as in the Senate report on the program of Rendition, Detention, and Interrogation (RDI) involving secret prisons and torture that the Bush administration adopted in the wake of the terrorist attacks of 9/11. According to the executive summary (the only part of the report that is declassified), the Democratic majority of the Senate Select Committee on Intelligence (SSCI) concluded that congressional leaders (including the chair of the SSCI) were kept in the dark about the program and that it did not produce information that was necessary to further the counterterrorism effort, including tracking down Osama bin Laden’s hiding place (32). These conclusions are very convenient: the SSCI and Congress do not deserve any blame and because torture is ineffective the United States can abjure it without paying any price. If the report had found torture to be effective, even on some occasions, it would have made the committee, the government, and the American public face a trade-off between safety and morality, and obviously this would have been politically and psychologically painful. It was much more comfortable to say that no such choice was necessary.As a political scientist, not only am I not shocked by this behavior, but I also believe that it is somewhat justifiable. Keeping tensions with the Soviet Union and Cuba under control in the wake of Kennedy’s assassination was an admirable goal, and Reagan’s desire to protect the space program could also be seen as in the national interest. The SSCI was more narrowly political, but it is not completely illegitimate for political parties to seek advantage. What is crucial in this context, however, is that the politicization of the postmortems limits their ability to explain the events under consideration. This is not to say that many of their conclusions are necessarily incorrect. Although Johnson suspected otherwise, Oswald probably did act alone (33); cold weather did cause the shuttle’s O rings to become rigid and unable to provide a protective seal. But it is unlikely that the relevant congressional leaders were not informed about the RDI program. More importantly, the claim that even had torture not been used the evidence could and should have yielded the correct conclusions, while impossible to disprove, was not supported by valid analytical methods, as I will discuss below.Both politics and psychology erect barriers to the consideration of the argument that a conclusion that in retrospect is revealed to have been disastrously wrong may have been appropriate at the time. Given the ambiguity and incompleteness of the information that is likely to be available, the most plausible inferences may turn out to be incorrect (34). Being wrong does not necessarily mean that a person or organization did something wrong, but any postmortem that reached this conclusion would surely be scorned as a whitewash. This may be the best interpretation of the intelligence findings on Iraq leading up to the Second Gulf War, however. Although the IC’s judgment that Iraq had active WMD programs in 2002 was marred by excessive confidence and several methodological errors, including the analysts lack of awareness that their inferences were guided at least as much by Saddam’s otherwise inexplicable expulsion of international inspectors as by the bits of secret information that they cited as reasons for their conclusions (35), it was more plausible than the alternative explanations that now appear to be correct. It is telling that the official postmortems and the journalistic accounts do not reject this argument, they do not even consider it. The idea that a disastrously wrong conclusion was merited is deeply disturbing and not only conflicts with our desire to hold people and organizations accountable, but clashes with the “just world” intuition that even though evil and error are common, at bottom people tend to get what they deserve (36–38) and our sense that well-designed organizations with excellent staffs should be able to understand the world.To put this another way, people tend to be guided by the outcomes when judging the appropriateness of the procedures and thinking behind analysis and policies. If the conclusions are later revealed to be correct or if the policies succeed, there will be a strong presumption that everything was done right. While intuitively this makes sense, and it would be surprising if there were no correlation between process and outcome, the correlation is not likely to be 1.0. Toward the end of the deliberations on whether the mysterious person that the United States located in a hideout in Abottabad really was Osama bin Laden, Michael Morell, Deputy Director of the CIA, said that the case for this was much weaker than that for the claim that Saddam had active WMD programs in 2002 (39). Although this assessment is disputable and by necessity subjective, Morell was very experienced in these matters and his argument is at minimum plausible.The quality of postmortems is impeded by the almost universal tendency to ignore the fact that people may be right for the wrong reasons and wrong for the right ones. For example, the SSCI praised those elements of the IC who dissented from the majority opinion that Iraq had active WMD programs without investigating the quality of their evidence or analysis (40). It was assumed rather than demonstrated that the skeptics looked more carefully, reasoned more clearly, or used better methods than did those who reached less accurate conclusions. To turn this around, this and many other postmortems failed to use standard social science comparative methods to probe causation but instead criticized those who were later shown to be wrong for being sloppy or using a faulty approach without looking at whether those who were later shown to be right followed the same procedures (41).Both the analysis being examined and the subsequent postmortems are prone to neglect the comparative method in another way as well. They usually look at whether a bit of evidence is consistent with the favored explanation without taking the next step of asking whether it is also consistent with alternative explanations. Just as CIA analysts seized on the fact that trucks of a type previously associated with chemical weapons were being used at suspicious sites and did not ask themselves whether an Iraq without active chemical production would find other uses for the trucks (42), so the SSCI report did not consider that the errors they attributed to bad tradecraft could also be explained by political pressures (which is not to say that the latter explanation is in fact correct), and those who argued for the importance of these pressures did not look at the other areas of intelligence, especially on the links between Saddam and al Qaeda, to see whether the same pressures had the expected effect (in fact, they did not). Here factors were judged to be causal without looking at other cases in which the factors were present to see if the outcomes were the same.A related set of failings, and ones that are central, include the hindsight fallacy, cherry picking of evidence, confirmation bias, and ignoring the costs of false positives. These are epitomized by what I noted at the start—in the aftermath of a disaster, people ask how the decision makers and analysts could have been so wrong, why the dots remained unconnected, and why warning signs were missed or discounted. Hindsight bias is very strong (43, 44); when we know how an episode turned out we see it as predictable (and indeed often believe that we did predict it) if not inevitable. Just as we assimilate new information to our preexisting beliefs (45, 46), once we know the outcome, when we go back over the information or look at old information for the first time (as is often the case for those conducting postmortems), there is a very strong propensity to see that it unequivocally points to the outcome that occurred. In a form of confirmation bias and premature cognitive closure that comes with our expectations about what we are likely to see, knowledge of how events played out skews our perceptions of how informative the data were at the time. The problematic way of thinking compounds because once we think we know the answer, we search for information that fits with it and interpret ambiguous evidence as supportive.This is not to suggest that good postmortems should ignore the light shed by the outcome and what in retrospect appears to be the correct view. Without this, after all, later analysts would have no inherent advantages over contemporary ones. But authors of postmortems must struggle against the natural tendency to weigh the evidence by what we now know—or believe—to be correct. Given all the information that flows into the organization, it is usually easy to pick out reports, anecdotes, and data that pointed in the right direction. But to do this is to engage in cherry picking unless we can explain why at the time these indicators should have been highlighted and interpreted as we now do and why the evidence relied on at the time should have been ignored or seen differently. As Roberta Wohlstetter pointed out in her path-breaking study of why the United States was taken by surprise by the Japanese attack on Pearl Harbor (47), most of the information is noise (if not, in competitive situations, deception), and the problem is to identify the signals. To use the language that became popular after 9/11, to say that analysts failed to connect the dots is almost always misleading because there are innumerable dots that could be connected in many ways. For example, official and journalistic analyses of 9/11 stress the mishandling of information about the suspicious behavior of some people attending flight instruction schools. True, but if the attack had been by explosive-laden trucks I suspect that we would be pointing to suspicious behavior at trucking schools.Fighting against the strong attraction of knowing how the dots were in fact connected is a difficult task, and one that must be recognized at the start of a postmortem lest it make debilitating errors. A prime example is the SSCI majority report on the RDI program mentioned above. The grounds for the conclusion that torture was ineffective were that the correct conclusions could have been drawn from the mass of information obtained from other sources, especially interrogations that used more benign techniques. The problem is not that this claim is necessarily incorrect, but that it commits the hindsight fallacy and cherry picks. Knowing the right answer, one can find strong clues to it in the enormous store of data that was available to the analysts. But this information could have yielded multiple inferences and pictures; to have established its conclusion the postmortem would have to show why what we now believe to be correct was more plausible and better supported than the many alternatives (48, 49). Such an argument will always be difficult and subject to dispute, but like most retrospective analyses, the SSCI report did not even recognize that this was necessary.The frequently asked question of what warning signs were missed points to the related problem of the failure to recognize the potential importance of false positives. This comes up with special urgency after mass shootings and often yields a familiar and plausible list of indicators: mental health problems, withdrawal from social interactions, expressed hostility, telling others that something dramatic will happen soon. But even if we find that these signs universally preceded mass shootings, this would not be enough to tell us that bystanders and authorities who saw them should have stepped in: looking only at cases of mass shootings (what is known as searching on the dependent variable) makes it impossible to determine the extent to which these supposed indicators differentiate cases in which people go on shooting sprees from those in which they do not. Acting on these indicators could then lead to large numbers of false positives. Economists have the saying that the stock market predicted seven of the last three recessions; those who do postmortems should take heed.Put differently, the “warning signs” may be necessary but not sufficient conditions for the behavior. Their significance depends on how common they are in the general population, and this we can tell only by looking at the behavior of people who do not commit these terrible crimes. The same point applies to looking for predictors of other noxious behavior, such as terrorism, violent radicalization, and domestic abuse, to mention just several that receive a great deal of attention (or of unusually good behavior like bravery or great altruism).Examining people who do not behave in these ways or instances that do not lead to the undesired outcomes would be labor intensive and can raise issues of civil liberties. But this is not always the case, and was not in one well-known instance (although this was not a postmortem, but predecisional analysis). Before the doomed launch of the Challenger, because the engineers knew that the O rings that were supposed to keep the flames from escaping through the boosters’ joints might be a problem in cold weather, they provided the higher authorities with a slide showing the correlation between the air temperature and the extent of the damage to the rings in previous launches. Because this showed some but not an overwhelming negative correlation it was not seen as decisive. The slide omitted data for launches in which there was no damage to the rings at all, however. This showed that partial burn-throughs occurred only when the temperature fell below 53°, and had these negative cases been displayed the role of cold would have been more apparent.With these past cases as guides, we can expect that absent self-conscious efforts to counter the biases discussed above, the attempt to understand the failure of the authorities to anticipate and control events of January 6, 2021, will be suboptimal. That is not to say that these postmortems will be without value. There are things to be learned about why communication channels were not clearer, what the barriers to cooperation among the diverse law enforcement organizations were, why there was so little contingency planning, and why the decision to deploy the National Guard was delayed. The analysis of the lack of forewarning, however, is likely to fit the pattern of hindsight bias, cherry picking, and the neglect of comparisons. The media has said that an alarming report from an FBI field station was not passed on and that insufficient attention was paid to the “chatter” on social media (50–53). It is almost certain that a more thorough investigation will turn up additional bits of information that in retrospect pointed to the violence that ensued. If the analysis stops here, however, the obvious conclusion that these indicators should have been heeded will be incorrect. Better methodology points to the next steps of looking at some of the multiple cases in which large-scale violence did not occur or was easily contained in order to ascertain whether the reports preceding January 6 were markedly different. We should also look for indications and reports that the demonstration would be peaceful.It is not surprising that the fall of Kabul has led to widespread calls for an inquiry into what went wrong. In all probability, however, these postmortems are also likely to be deficient. They will be highly politicized and subject to the hindsight fallacy and related methodological shortcomings. Because the stakes are so high and involve so many different entities and actors, political pressures will be generated not only by the Democrats and Republicans, but also by different parts of the government, especially the military and the civilian intelligence community.Politics will be involved in how the postmortem is framed. Democrats will want to focus on the intelligence while Republicans will seek a broader scope to include if not concentrate on the decisions that were made. Disputes about the time period to be examined are also likely. Republicans will want to limit the study to the consequences of President Biden’s April 14, 2021, announcement that all troops would be withdrawn by September 11; Democrats will want to start the clock earlier, with the Trump administration’s February 2020 agreement with the Taliban to withdraw by May 1, 2021. More specifically, Democrats will want to look for evidence that the Trump agreement led to secret arrangements between the Taliban and local authorities that laid the foundations for the latter’s defections in the summer.Because the issues are so salient and emotion-laden, there will be pressure to have the investigations be entirely disinterested, which means that the people conducting them should not have had deep experience with any of the organizations involved. This impulse is reasonable, but comes at a price because reading and interpreting intelligence reports requires a familiarity with their forms and norms (which differ from one organization to another), and outsiders are at a disadvantage here. The distinction between strategic and tactical warning is easily missed by those new to the subject, as is the need to determine whether and how assessments are contingent (i.e., dependent on certain events occurring or policies being adopted). An understanding of how policy makers (known as “consumers”) are likely to interpret intelligence is also important and needs to be factored in. In the case of Afghanistan, as in other military engagements, military intelligence is prone to paint a relatively optimistic picture of the progress that is being made. Experienced consumers understand this and can apply an appropriate discount factor. A further complication is that as the pressure for a full American withdrawal increased, the military’s incentives changed, and pessimism about the prospects for the Afghan army in the absence of American support came to the fore. If a postmortem takes these estimates at face value and legitimately faults them for their organizational bias, it would be important to also probe how the estimates were interpreted.Here as in other cases, the overarching threats to a valuable postmortem are hindsight bias and cherry picking, which will be especially strong because we now more clearly see the weaknesses of the government and the positive feedback that led Taliban victories to multiply. If the study tries to uncover when various consumers and intelligence analysts and organizations concluded that a quick Taliban victory was likely, it will have to confront not only the obvious problem that people have incentives to exaggerate their prescience, but that memories about exactly when certain conclusions were arrived at are especially unreliable when events are coming thick and fast, as was true of the analysts charting the unrest that led to the fall of the Shah.^* In retrospect, the more accurate assessments will stand out and there will be strong impulses to claim that at the time they had more support than the alternatives and so should have been believed.None of this is to say that a well-designed postmortem of either Afghanistan or the events of January 6 would conclude that few errors were made or that the information was analyzed appropriately. But it would provide a more solid grounding for any conclusions, attributions of blame, and proposals for change if blame is the ending rather than the starting point, hindsight and confirmation biases are held in check, and the ambiguity of the evidence and frequent need for painful value trade-offs are recognized. Postmortems are hard and fallible, but the country can ill afford to continue mounting ones that have unnecessary flaws.