CommunityThe Dissenter

Organizations Tracking Drone Strike Deaths Provide Flawed or Incomplete Data

Graph from the Human Rights Clinic study on drone strike deaths

Students and staff at the Human Rights Clinic at the Columbia Law School have released a report examining tracking organizations’ estimates on drone strikes by the United States. The clinic’s systematic review found reports often rely on unnamed Pakistani government officials or unnamed witnesses. The organizations were all “hampered methodologically and practically.” They “consistently underestimated the potential number of civilians killed in Pakistan during the year 2011.”

The report concluded the US government should release estimates of drone casualties. Media and tracking organizations need to acknowledge the reporting being used for data has its limits. It also recommended media avoid using “militant” unless they are quoting a government official. Use other “specific identifiers where possible” because the term is “ambiguous and controversial”

Tracking organizations the clinic focused upon were the Bureau of Investigative Journalism, the Long War Journal and the New America Foundation. This is because they are “the most influential of such organizations” and “their work has in many instances catalyzed debate about the effectiveness and humanitarian cost of strikes.”

Recounting the Number of Reported Casualties

The clinic’s report found somewhere between 72 and 155 civilians were killed in 2011 in Pakistan. Fifty-two of the reported deaths identified the civilians killed by name, “a relatively strong indicator of reliability.”

…By comparison, New America Foundation’s count is just 3 to 9 “civilians” killed during this period; Long War Journal’s count is 30 civilians killed. In percentage terms, and based on their and our minimum numbers, we counted 2300 percent more “civilian” casualties than the New America Foundation, and 140 percent more “civilian” casualties than New America’s “civilian” and “unknown” casualty counts combined. We counted 140 percent more minimum “civilian” casualties than the Long War Journal…

The Bureau’s count was much closer to the clinic’s recount. They had counted 68 to 157 civilians killed, “only 5.9 percent more minimum civilian casualties.”

After the recount, the clinic concluded the Long War Journal and New America Foundation had a lower number of reported deaths overall and civilian deaths. Only the Bureau had “consistently purported to actively track civilian casualties—as opposed to focusing on providing an estimate of the overall number of individuals killed.”

The clinic arrived at these different estimates after examining assumptions and the “accuracy of information about drone strike deaths” the tracking organizations had used. They looked at the media reports referenced and any independently counted casualties. They looked at 2011 in particular to show, through a “strike-by-strike comparison,” the discrepancies in how the tracking organizations count casualties.

What makes the discrepancies particularly problematic is how “news analysis and political commentary” have frequently cited the Long War Journal and New America Foundation. The clinic wrote:

…Exclusive or heavy reliance on the casualty counts of these two organizations is not appropriate because of the significant methodological flaws we identify. While we do not agree with the Bureau’s analysis of media sources in all cases, it appears to have a more methodologically sound count of civilian casualties, commensurate with its special focus on that issue…

Not only did the clinic find the Bureau’s methodology to be more sound but the Bureau was also the only organization to “respond directly to [the clinic] about each strike count where [the clinic] reached a different result.”

The students and staff contacted all three organizations. “Bill Roggio of the Long War Journal spoke with [the clinic] extensively about the organization’s methodology,” however, although the New America Foundation altered their data after “detailed strike counts” were provided to them by the clinic, they “declined to comment.”

The Reliance on Unnamed Pakistani Government Officials

One of the more noteworthy sections of the report is an examination of media coverage that depended on “unnamed Pakistani government sources” when reporting “militants” were killed.

…We do not know who the unnamed Pakistani officials are, although observers believe they are Pakistani Army officials. What definition these officials use to categorize a person as a militant or civilian is unknown. Nor do we know how the Pakistani Army confirms such deaths or the quality of information it is able to rely on, given the limited accessibility of some of the tribal regions to even the Army. Critics of the Pakistani military argue that it does not conduct on-the-ground investigations before issuing “on the condition of anonymity” announcements that the dead are militants; these critics contend that it is plausible the Army has a political interest in categorizing as many of those killed in drone strikes as “militants”…

The report noted this issue was further compounded by the fact that the media does not typically seek to provide “additional identifying information about the dead that would enable the reader—or tracking organizations—to reach their own judgment on the matter.”

What the clinic found when putting together their report was that 330 to 575 militants had been reported dead. One hundred to two hundred and nineteen of casualties identified as militants came from “anonymous officials.” Additionally, “In the case of 36 to 71 of those dead, absolutely no further identifying information [was] provided beyond the claim that they were militants or alleged militants.” And, “For the remaining 64 to 148 dead, some identifying information” was provided, like “an alleged connection to a militant” but that was all.

The Ambiguity of Terms Used to Describe Drone Deaths

Particularly insightful is a section of the report that details the problem with using terms like “militant” or even “civilian” to describe those killed by drones. The report determined, “There is no standard definition that media sources use to categorize a person as a militant or a civilian, nor a standardized measure by which the media sources weigh and corroborate their information.”

The report described what those involved in the review looked at the ambiguity of the terms and examined “the hidden bias of this categorization.”

…They are not defined by the US government, though US officials use them; the terms sound vaguely legal, although they only loosely track legal and scholarly debates about who may be lawfully targeted. Without a universally accepted or standardized definition for these terms, categorization of “militant” and “civilian” deaths is biased by the definition of the individuals to whom media reports cite for identifying the dead. Whether these primary sources are unnamed US and Pakistani officials, or unnamed local villagers and witnesses, identification of those killed as “militants” or “civilians” is likely driven by political interests, and colored by the perspective and experiences of the source. As these terms appear in media reports and the tracking organization studies, they might be better understood as moral categories of who should and should not be killed. They are, to that extent, inherently limited and biased…

Although it might seem clear to denote “named and well-known militant group leaders” as “militants” and “young children and women” as “civilians,” like media typically do, this does not reflect how the government might interpret the terms. “Under some circumstances,” the report argued, “civilians may lawfully be targeted” because the “circumstances under which civilians lose protection under humanitarian law and become subject to direct attack is a matter of hotly contested debate among lawyers and scholars in the US and internationally.”

Government officials might say there few or no civilian deaths have been caused by drone strikes because they think civilians “suspected of some affiliation of providing some material support to militant groups” and individuals providing supplies to a local militant group can be targeted, even if the supplies provided were food or medicine and provided “under duress.” People meeting with “militants” may be legitimate targets for making contact, whether they were engaged in reconciliation or peacemaking efforts.

The Limits of Media Reporting

The report found coverage of drones strikes often come from the “same few journalists and news outlets providing the same materials to multiple wire agencies and national or international press.” They come from “limited on-the-ground investigations,” unless a high-level militant leader has been killed or a “high number of overall and reportedly civilian casualties” have occurred. And the content of coverage is typically no more than a location where the strike happened, what the “alleged or apparent target” appeared to be (like a “compound” or vehicle), how many people were reportedly killed and some kind of claim from an official that “militants” were some of the casualties if not all of them.

There is the issue of “stringers,” which is described in the report:

…While the public and policymakers may view international media reports as especially credible, these outlets—including the New York Times, the BBC and wire services such as Reuters and the Associated Press—are generally unable to access the tribal areas where drone strikes are occurring, making them reliant on local journalists or “stringers” for their reporting. International media sometimes report the number and identity of those killed based on the reports of several local stringers. However, the stringers themselves may be unable to go to the area where a drone strike actually occurred due to security issues, making their reports substantially dependent on the word of a handful of local officials…

Stringers able to reach area where drone strikes occurred might conduct their own reporting and, if only one stringer was able to confirm there were civilian casualties, civilian deaths will not appear in news reports on the strike. However, if three or four stringers find civilians were killed, then civilian deaths will be reported.

It is also possible that “unnamed witnesses” or “local villagers” give flawed statements as well. “Local militant groups” (provided they aren’t killed in “double tap” strikes) might move bodies of those killed so they can be buried or their identities can be kept hidden or secret. Witnesses might see what really happened and fear retaliation. Local government officials, covert agents or militant groups could, hypothetically, influence witnesses. (However, there are no known reports that this has happened.)


All of the above led the clinic to determine the tracking organizations are problematic because, “in the absence of other information,” such as official counts from the US government, estimates rife with flaws have been treated as “actual body counts.”

Neither of the tracking organizations has claimed their data is definitive, but what they have put together has been discussed as if it is “credible, corroborated and firsthand research.” To the extent that the data is “assimilated into fact,” it threatens to “become what everybody knows about the US covert drone strike program.”

Furthermore, as the report astutely stated:

…The estimates provide a dangerous assurance: the human toll is something we have identified, debated and considered. If we know who and how many people we have killed, calls to examine and deliberate on the drone program—and calls to end it—lose their urgency. We may come to falsely believe that covert drone strikes are an “open secret” when, in fact, the US government continues to resist disclosure of basic and important information about the drone strikes program. Moreover, where the tracking organizations’ estimates significantly undercount the number of civilians killed by drone strikes, they may distort our perceptions and provide false justification to policymakers who want to expand drone strikes to new locations, and against new groups…

This is incredibly critical. The data has been cited when making arguments that drones are not as bad as the Taliban or the Pakistan army because the Taliban and Pakistan army have caused much more death. The data has been used to try and stifle criticism. (Which is not to say, if the data was more accurate and there were a few hundred more civilian casualties counted, the debate would shift and it would be proven drones are worse then the Taliban or Pakistan army. All violence caused by the Taliban, Pakistan army or drone warfare creates horror.)

The report further proves the exact costs are unknown. Civilian casualty counts are consistently lower than the number actually killed. The government and the media have contributed to an uninformed public understanding of drone attacks.

There is an ever-expanding amount of available material, which shows the state-sponsored violence being carried out must be subject to much more scrutiny by policymakers and the public. Responding to efforts to ignite debate with more indifference, propaganda or secrecy ensures more atrocities occur.

Previous post

Why Has the Director of Minnesota's Chief Anti-Marriage-Equality Group Suddenly Gone Silent on That Issue?

Next post

Sunday Talking Heads: October 14, 2012

Kevin Gosztola

Kevin Gosztola

Kevin Gosztola is managing editor of Shadowproof Press. He also produces and co-hosts the weekly podcast, "Unauthorized Disclosure."