H-Diplo Essay 374
Essay Series on Learning the Scholar’s Craft: Reflections of Historians and International Relations Scholars
5 October 2021
Panning for Gold
Series Editor: Diane Labrosse | Production Editor: George Fujii
Many of my colleagues have contributed essays revisiting their graduate school days, full of commendations to friends and collaborators. I could do that too—and, in fact, my friends include many of the very authors of these essays—but I thought it more useful to spend this time on tools and methods. As I sit to write this, Athan Theoharis, a friend, just passed away after encountering crazy complications from a semi-routine medical procedure. That reminds me of how ephemeral we are. Also, so many of these essays concern academic careers, which is not where we all end up. As I think back on my doctoral cohort (admittedly, in international relations, or IR), not one of us ended up in academia. Some went to the State Department or other government agencies. One taught in private school. I guess he came closest to academe, but ended up at the Central Intelligence Agency (CIA). Others went to oil companies or beltway bandits. I am a working historian. My experience shows that it is possible to train in this discipline and branch out far afield.
First, a few words on fashion. There are always fashions, both in subject and in methodology. National security was in fashion when I was in grad school, and diplomatic history well established. My history department rebelled against that to the degree that it gave me the impression that history ended in 1945 (or even 1914). If students wanted to study the contemporary era that meant political science. In that field “bureaucratic politics” was in fashion. I proposed a dissertation on the origins of the Cold War. One of my advisers challenged me, “Tell me, what would be the political science contribution here?” Of course, we were debating the origins of the Cold War almost every day at that time. The objection seemed nonsensical. So I went off and, instead of writing some detailed dissertation proposal, outlined five different possible topics, including applying bureaucratic politics to the Peacemaker ICBM basing decision, or to CIA estimates of Soviet strategic forces. I wanted my adviser to agree on “political science value” before I went to the trouble of the full proposal. When I presented the outlines at office hours, my other adviser popped into the room, a mischievous grin on his face. “You MIRVed him!” he exclaimed. The adviser let me pick any one I wanted, so I chose the CIA. That paper became my first book, The Soviet Estimate.
This example is not selected at random. At the time there was a huge debate going on in national security circles—almost of the dimensions of the “Missile Gap” of the 1950s—as to whether the CIA routinely underestimated the size of Soviet nuclear forces. Strategic analyst Albert Wohlstetter had published a series of papers (overly influential in my view) that made the charges in the then new journals Foreign Policy and Strategic Review. Here’s the point for a scholar: start out with a question that merits an answer. It is true that the Soviets built 1,518 ICBM silos, more than predicted in the CIA estimates. But did that matter? On the issue of technological development the estimates were exactly right in predicting when the Russians might field a MIRVed missile. On Soviet missile defenses, bomber forces and much else too, the estimates were on the money.
Another question. Authors need to find the data to fuel their inquiries. I knew where to put my fingers on the data set Wohlstetter had used—the Pentagon had recently declassified a set of “classified posture statements” from the years when Robert McNamara had headed it. For older years a few documents were starting to become available. Then we had the tool of the Freedom of Information Act (FOIA) and other declassification procedures that could be applied to documents in the archives. Opening up the records, like making use of someone’s private papers, offers new perspectives on whole histories. This became a feature of my research endeavor, to the extent that there were subjects I selected for books not just for questions that needed answers, but for records which needed to be opened.
A case in point would be my study of America’s near-intervention in Vietnam at the time of Dien Bien Phu, originally published as The Sky Would Fall (1983), but which I have more recently restored, expanded and revised as an e-book titled Operation Vulture: America’s Dien Bien Phu. When I looked at the records there were huge gaps representing stuff that remained top secret. What was in the Foreign Relations of the United States was not complete. Then I noticed that many of President Dwight D. Eisenhower’s diplomatic documents of the period had been sequestered in an inner file set called “Project Clean Up.” I made it my business to apply for the declassification of everything related to this crisis. A lesson of this project is that time and patience are required. I began this study in 1977 and published Sky Would Fall with many of the important documents yet to be released, and my own requests still languishing. Many of them were only declassified in the 1990s. It was worth it. The documents turn this story on its head. The original spin had been that Eisenhower labored to keep the U.S. out of war in Indochina in 1954. The true story is that he labored to make intervention possible but was stymied at every turn. In 1983 I argued that premise on limited evidence. By 2014 the evidence was authoritative. The takeaway for researchers should be: always look at the file labelled “miscellaneous,” and always work to perfect the evidence.
Every project opens the door to expanding our universe of source material. Think of it that way. Whether the material is papers never seen or underutilized, piles of unopened files, or stacks of documents awaiting declassification, opportunity is there. Projects can be configured that way, much as was Operation Vulture. It’s like panning for gold. In 1991 I published a history of the National Security Council (NSC) staff called Keepers of the Keys. (6) That afforded the possibility of lining up the records of presidents from Harry Truman to the first President Bush and framing presidents’ use of these NSC staffs as policy tools. Plenty of documents were declassified along the way. David Rothkopf may have gotten more word of mouth for his version of NSC history, but I don’t think his work did as much to widen the field for research. Panning for gold, both in the form of neglected or fresh sources and declassified documents, is what matters. William Colby and the CIA (originally, The Lost Crusader in 2003) showed that it was possible to write intelligible biography of a denizen of the secret world. Randall B. Woods, also author of a Colby biography, admits he could not have written his book without the benefit of Lost Crusader. The Ghosts of Langley began as an overview CIA history but morphed into a device for understanding why agency covert operations kept spiraling out of control. Whether one’s field is gender studies, ethnic studies, or diplomacy, the well-aimed project has better chances.
There are fashions in research approaches. One current one is the multinational history, in which the scholar uses archives in different lands to tell parts of the story. This is a great addition to scholarly practice—and it has advantages in overcoming secrecy strictures in different countries—but I would argue that the sense there are different sides to a story is equally if not more important. My first “book,” if you could call it that, came in the sixth grade. I filled one of those green-and-white speckled composition books with a treatise on the Russo-Japanese war. Perhaps it was that America did not figure in the story, but I learned that sides matter. At that time popular history was full of World War II. I was struck with how much of it—particularly histories of the Pacific war—focused on the U.S. (or the ‘friendly’ side). That got me involved in researching the other sides of those events. In the 1970s, starting with revelations of how the Allies were reading German codes, the secret history of the war began to unspool. In the next decade the National Security Agency and other stewards of that material yielded up original sources on codebreaking. For the Pacific war a little bit of this had been in the public domain since the investigations of the Pearl Harbor disaster back in 1945-46, but in the 1980s the trickle became a tsunami. For me, this material illuminated the other side in unprecedented detail. Raw access to assorted nations’ archives proved much less important than the actual content of the source material. It also pointed to a new need in history—conventional accounts of battles and campaigns need to be rewritten to integrate the role of and insights from intelligence. I have so far written four books (Combined Fleet Decoded, 1995; Normandy Crucible, 2011; Islands of Destiny, 2012; and Storm Over Leyte, 2016) to contribute to this endeavor.
The Vietnam War was going on as I went through school. At one point I thought I would be part of it, later that I would write its history. Many concerns came together there. From an early date I accumulated material on both Vietnamese sides of the war, and from the late 1970s I began using declassification regulations to open the records of the war. For Vietnam, the Pentagon Papers early on offered a window into the inner workings of U.S. decision-making, and for a long time historians relied upon them for their works on that conflict. But the Papers were limited by the way they were compiled—to avoid White House scrutiny and President Lyndon Johnson’s anger, analysts deliberately avoided any effort to access presidential records. Histories based exclusively on Pentagon Papers material were therefore limited, especially because four volumes of the Papers that centered on peace feelers had been left out when Dan Ellsberg originally leaked them.
Much agency stuff was omitted simply because government generated so much more paper than could be taken into account. Plus the Pentagon Papers’ focus on decision-making left out or touched only lightly on a host of other Vietnam issues. The notorious “Phoenix” program, for example, does not appear in the Pentagon Papers. Yet the Johnson and Nixon Libraries contain literally tens of thousands of pages of records associated with Projects “ICEX,” “Take Off,” and the “High Value Detainee Rewards Program,” that were the very stuff of Phoenix. Then there were subjects like Dien Bien Phu that were passed over very quickly in the Pentagon Papers but really represented much deeper episodes in the American experience. There were, and remain today, plenty of records still to be opened on the Vietnam War. Bulk declassification programs have been largely ineffectual because agencies are reluctant to relinquish their secrets. Even with the Pentagon Papers. The U.S. government made a great show of “fully” declassifying the full Papers to mark the fortieth anniversary of the leak (in 2011), but punted and kept eleven words classified. At this writing, ten years later, that text is still secret.
Shaking loose research material proceeds in tandem with charting the history. My Vietnam book after the one on Operation Vulture was called Valley of Decision: The Siege of Khe Sanh. This I wrote together with Ray W. Stubbe, who had been chaplain of the Marine unit that constituted the core of the defenses at Khe Sanh, where perhaps the greatest battle of the war was fought. Ray sought to erect a verbal memorial to his brave comrades, but his immense collection of notes and details needed coherence, context, and the kind of top-level perspective I had been researching. Khe Sanh was important as an entryway to South Vietnam from the Ho Chi Minh Trail—and that idea of blocking off the South from the North Vietnamese framed one of America’s key strategies. In 2018 the Office of the Director of National Intelligence made another of those showy displays of supposed transparency, with a fiftieth anniversary release of masses of material surrounding the Tet Offensive, including Khe Sanh. I commented on this collection for the National Security Archive’s online blog. The holes in the documents were disturbing. There were a few that were newly whole, but looking at their Khe Sanh material I could see that some of the documents were the exact same versions, with the same declassification dates and markings, as Ray and I had gotten released for Valley of Decision.
Next was The Hidden History of the Vietnam War. That work aimed at counteracting various claims as to how the U.S. would have/should have “won” the war, showing how these “perfect strategies” did not work. From the strategic perspective, victory in Vietnam required a triangulation of existing (not idealized) military and intelligence methods, South Vietnam’s capacity to absorb and utilize them (hence the importance of Vietnamese material), against an evolving threat (the Vietnamese again, this time from the other side), within the framework of rising American domestic opposition. Victory was an illusion. I returned to that theme in my 2009 book Vietnam: The History of an Unwinnable War, 1945-1975, where I spelled out and developed in detail themes from Hidden History. There had been a data problem in Vietnam, the most studied conflict the U.S. had yet waged. Saigon politics still held sway, Hanoi’s determination overcame American ingenuity, and President Richard Nixon’s secret plan to end the war was to win it—which could by now be shown from documents. In my earlier NSC history Keepers of the Keys I had also selected U.S. Vietnam policy as one thread to follow, including being among the first studies to focus on Nixon’s war strategy.
Vietnam was a major focus in my summary study US Special Forces: What Everyone Needs to Know. Following this muse across the changing technology of books, I turned Operation Vulture into an e-book, and also published in 2015 A Streetcar Named Pleiku: Vietnam 1965, A Turning Point. As compendium books gained popularity, for this subject I contributed In-Country: Remembering the Vietnam War, a collection excerpted from memoirs by veterans, in 2012.
Meanwhile I revisited the idea of blocking off South Vietnam in The Blood Road: The Ho Chi Minh Trail and the Vietnam War. That study inverted the lens, using The Trail as a microscope through which to re-examine the history of the conflict. I went looking for material on the Vietnamese side. That, it turned out, existed in massive amounts in U.S. records of questioning Vietnamese prisoners—intelligence officers had a standard menu of questions asked of enemy personnel, so all of them related their stories of growing up in North Vietnam, joining or being drafted into the army, being trained and sent to South Vietnam, and so on. Combined with captured documents, North Vietnamese and Liberation Front wartime publications, and postwar memoirs and recollections, not to mention U.S. intelligence reporting, the story of The Trail could be told in considerable detail.
Another way to open up sources is to illuminate subjects with conferences that shine light on them. I’ve participated in many, many conferences, but there are two dealing with Vietnam that merit special mention. One was the “Missed Opportunities” meeting that took place in Hanoi in June 1997, where mixed delegations of scholars and former government officials from both the United States and North Vietnam met to discuss the war. General questions were discussed in advance, readings and sets of documents were assembled as references, and scholars prompted the former government officials by offering what were called “provocations.” The Thomas J. Watson Jr. Institute of Brown University and the Institute of International Relations of the Ministry of Foreign Affairs of the Socialist Republic of Vietnam were the sponsors. Robert S. McNamara led the American delegation, of which I was a member. I’d compiled the document briefing book from the American side. Brown University scholars James Blight and janet M. Lang designed the format. Vietnamese participants were restricted by their formal regulations but we did develop significant findings, including that there had been no second attack at the Gulf of Tonkin incident in 1964, that North Vietnamese officials believed Hanoi had missed an opportunity to cooperate with a neutralized South Vietnam, and that the Pleiku shelling of February 1965 (then considered a terrorist “spectacular”), which set off the U.S. bombing of North Vietnam, had been a local enterprise, not an operation ordered from Hanoi.
Speaking of the Pentagon Papers, the role of that leak as stimulant for certain Nixon administration actions, their value as historical resource, and the paucity of knowledge regarding origin and elaboration represented a lacuna in our knowledge. The Vietnam Veterans of America (VVA), a veteran’s association to whose magazine I’ve contributed dozens of historical articles—including one about the Hanoi conference—came to me with the idea of a conference on the Papers. We presented that event in Washington in June 2001, a few months before 9/11. The meeting ranged widely. We had panels on the creation of the study, with four analysts who had worked on it; on publishing the Papers, with journalists who had worked on them for both the New York Times and Washington Post, and others who had written about them; on the court case, with legal scholars plus lawyers who had represented both newspapers. Ellsberg gave the keynote address, and Senator Mike Gravell the lunchtime talk. I edited a proceeding of this conference with VVA’s Margaret Pratt Porter, adding extensive commentary, including an analysis of how Nixon had decided to try and suppress the study, plus extensive treatment of flaws in the government’s specific claims to secrecy. We also sought out and transcribed Nixon’s telephone records regarding the affair. VVA legal counsel Michael J. Gaffney wrote an essay putting it in the wider context of government secrecy. This strong package was published as Inside the Pentagon Papers.
Nixon’s telephone records bring up another subject worthy of a word. A source developing most recently has been audio tapes, where various figures on the historical stage have left evidence embedded in tape recordings. This is particularly significant for the United States, where several presidents used taping systems to record telephone calls, meetings, and so forth. To furnish scholars a whiff of what was becoming available, in 2003 I published White House Tapes: Eavesdropping on the Presidents, which was unusual in that I not only commissioned enhancements of the audiotapes but transcripts of the conversations, and covered presidents from Harry Truman to Gerald R. Ford. The actual recordings and the transcripts appeared as parts of the package.
Through all of this I remained a child of the Cold War, and there too, was gold to be panned. During research for my dissertation I filed FOIA requests for some of the national intelligence estimates (NIEs) from the 1950s, the era of the notorious “Missile Gap.” Those documents were released almost without censorship and with blinding rapidity. Perhaps that experience gave me confidence to undertake the long struggle against secrecy that ensued. In any case, I continued to study and write on Soviet-American relations, the nuclear balance, and arms control. Those issues formed another foci for Keepers of the Keys. There were numerous articles for the Bulletin of the Atomic Scientists, a couple for Scientific American, papers on Cold War decisionmakers, Soviet ballistic missile defense, NIEs, the “Team B” report, intelligence as a driver for foreign policy, and other subjects, appearing in scholarly compilations or in such sources as the 2002 edition of the Encyclopedia of American Foreign Relations. I joined the debate over the Reagan administration’s endeavors with my book How the Cold War Ended. There I added value by including a selection of key documents, making a suitable reader for those seeking primary sources. In addition, there was a methodological component, wherein I dissected the 1980s as a “research project” and examined what we could learn from applying assorted types of historical analysis.
Intelligence is a wider subject than the NIEs. From an American perspective it has migrated over the years from covert operations through domestic surveillance to international terrorism. The CIA covert operations document sets I am gathering for the National Security Archive will elevate the study of U.S. intelligence to a new level. Here much gold has appeared. The segments of the CIA collection already or about to be published by the ProQuest Corporation include nearly 8,000 documents. Among them, for example, in 2007 the CIA declassified the infamous “Family Jewels,” a compendium of documents assembled for a CIA director worried about accusations of agency participation in illegal domestic activities. The Archive circulated the documents themselves, but I included them in the CIA collection and wrote The Family Jewels: The CIA, Secrecy and Presidential Power. On terrorism, going back to the days of 9/11, I collected the most important documents then current in America Confronts Terrorism.
Of course, narrative is the bread-and-butter for a historian and there I have investigated agency covert operations. In the 1980s, at the height of Iran-Contra, I published President’s Secret Wars (republished in 1996). I amplified that treatment in my book Safe for Democracy. Bridging the era of classic covert operations and that of black prisons and counterterrorism I then wrote The Ghosts of Langley (2017). The ghosts theme is pursued in a work that for the first time will tell the full story of the CIA’s Vietnam War. That should be the next to appear.
There is a whole other story I could tell that revolves around gaming. I published a boardgame even before graduate school. I’ve continued to employ history quite directly in designing strategy games, including some very well-known ones, ever since. But the gaming is about understanding and simulating real world events and physical processes. It’s a tale for another day. Enough now. In the meantime, never stop panning for gold!
John Prados heads the National Security Archive’s Intelligence Documentation Project and its Vietnam Project, assists on other subjects, and is a Senior Fellow on national security affairs, including foreign affairs, intelligence, and military matters. He holds a Ph.D. in Political Science (International Relations) from Columbia University and has authored many books, most recently The Ghosts of Langley: Into the CIA’s Heart of Darkness (The New Press, 2017).
 Harrison Smith, “Athan Theoharis, historian who exposed FBI misconduct, dies at 84” (Obituary), Washington Post, 12 July 2021, https://www.washingtonpost.com/local/obituaries/athan-theoharis-dead/2021/07/12/69c35bdc-e31a-11eb-a41e-c8442c213fa8_story.html.
 John Prados, The Soviet Estimate: U.S. Intelligence and Soviet Strategic Forces, 2nd ed. (Princeton: Princeton University Press, 1986 ).
 Albert Wohlstetter, “Optimal Ways to Confuse Ourselves,” Foreign Policy no. 20, Fall 1975; Wohlstetter, “Legends of the Strategic Arms Race, United States Strategic Institute, USSI Report 75-1, 1975.
 Prados, The Sky Would Fall: The Secret U.S. Bombing Mission to Vietnam (New York: The Dial Press, 1983). Revised as Operation Vulture (New York: iBooks, 2002). Restored as Operation Vulture: America’s Dien Bien Phu (New York: Don Congdon Associates, 2014).
 Prados, A History of the National Security Council from Truman to Bush (New York: William Morrow, 1991).
 David Rothkopf, Running the World: The Inside Story of the National Security Council and the Architects of American Power (New York: Public Affairs Press, 2005).
 Prados, The Lost Crusader: The Secret World of CIA Director William Colby (New York: Oxford University Press, 2003). Republished as William Colby and the CIA: The Secret Wars of a Controversial Spymaster (Lawrence: University Press of Kansas, 2009).
 Randall J. Woods, William Egan Colby and the CIA (New York: Basic Books, 2013).
 Prados, The Ghosts of Langley: Into the CIA’s Heart of Darkness (New York: The New Press, 2017).
 Prados, Combined Fleet Decoded: The Secret History of U.S. Intelligence and the Japanese Navy in World War II (New York: Random House, 1995, republished U.S. Naval Institute, 2003); Prados, Normandy Crucible: The Decisive Battle that Shaped World War II in Europe (New York: Penguin/Caliber, 2011); Prados, Islands of Destiny: The Solomons Campaign and the Eclipse of the Japanese Empire in World War II. (New York: Penguin/Caliber, 2012); Prados, Storm Over Leyte: The Philippine Invasion and the Destruction of the Japanese Navy (New York: Penguin, 2016).
 Prados and Ray W. Stubbe, Valley of Decision: The Siege of Khe Sanh (Annapolis: U.S. Naval Institute, 2004 [1991, 1993]).
 Unredacted at nsarchiv.gwu.edu, search for Prados articles.
 Prados, The Hidden History of the Vietnam War (Chicago: Ivan R. Dee, 1995).
 Prados, Vietnam: The History of an Unwinnable War, 1945-1975 (Lawrence: University Press of Kansas, 2009).
 Prados, US Special Forces: What Everyone Needs to Know (New York: Oxford University Press, 2015).
 Prados, A Streetcar Named Pleiku: Vietnam 1965, A Turning Point (ebook, Now and Then Reader, 2015).
 Prados, In-Country: Remembering the Vietnam War (Chicago: Ivan R. Dee, 2012).
 Prados, The Blood Road: The Ho Chi Minh Trail and the Vietnam War (New York: John Wiley, 1998).
 Conference, “Missed Opportunities: Revisiting the Decisions of the Vietnam War, 1961-1968,” Thomas J. Watson Institute for International Relations, Brown University, and Institute for International Relations, Foreign Ministry, State of Vietnam, June 1997.
 Prados and Margaret Pratt-Porter, eds., Inside the Pentagon Papers (Lawrence: University Press of Kansas, 2004).
 John Prados, ed., White House Tapes: Eavesdropping on the President (New York: The New Press, 2005).
 A few citations would include Prados, “Certainties, Doubts and Imponderables: Levels of Analysis in the Military Balance,” Intelligence & National Security 26:6 (December 2011); Prados, “The Strategic Defense Initiative: Between Strategy, Diplomacy and U.S. Intelligence Estimates,” in Leopoldo Nuti, ed., The Crisis of Detente in Europe: From Helsinki to Gorbachev, 1975-1985 (London: Routledge, 2009); Prados, “The National Intelligence Estimates, 1976-1988: Successes and Failures,” in Malcolm Muir, ed., From Detente to the Soviet Collapse: The Cold War from 1976 to 1991 (Lexington: Virginia Military Institute, 2006); Prados, “Team B: The Trillion Dollar Experiment,” The Bulletin of the Atomic Scientists, 49:3 (April 1993); Prados, “European Nuclear Forces,” with Joel S. Wit and Michael A. Zagurek, Scientific American 255:2 (August 1986).
 Prados, How the Cold War Ended: Debating and Doing History (Washington, D.C.: Potomac Books [now University of Nebraska], 2011).
 Cf. Digital National Security Archive and/or ProQuest. CIA Set IV is currently in production and Set V in preparation. Set I covered the period from President Carter to Obama, Set II the Year of Intelligence (the 1975 investigations); Set III the presidents from John F. Kennedy to Gerald R. Ford. Set IV is Dwight D. Eisenhower’s era with some overlap with Kennedy.
 Prados, The Family Jewels: The CIA, Secrecy and Presidential Power (Austin: University of Texas Press, 2014).
 Prados, ed., America Confronts Terrorism (Chicago: Ivan R. Dee Publisher, 2002).
 Prados, Presidents’ Secret Wars (Chicago: Ivan R. Dee, 1995 [1986, 1988]).
 Prados, Safe for Democracy: The Secret Wars of the CIA (Chicago: Ivan R. Dee, 2006).