Unscientific Methods?

Also In This Issue Data Bomb · Counting Corpses · Hidden Patterns Related Resources On National Journal.com National Journal: "Iraq's Slippery Polls" (12/01/07)

By Neil Munro, National Journal
© National Journal Group Inc.
Friday, Jan. 4, 2008

The 2006 Johns Hopkins University study [PDF] on war deaths in Iraq includes eight pages of numbers, tables, and explanations, plus a short, placid paragraph declaring, "This study received ethical approval from the Committee on Human Research at Johns Hopkins Bloomberg School of Public Health."

Skeptics say that the controversial study, published in The Lancet, is built on unreliable data, perhaps even faked responses. In examining the scientific controversy, National Journal also reviewed a variety of statements and documents suggesting that the study's researchers may have sidestepped standard ethical practices designed to safeguard respondents from reckless surveyors—and perhaps even skirted federal law.

Any door-to-door survey in a war zone presents ethical problems for researchers because the surveyors may be put in danger and because political factions have an incentive to skew the results. In Iraq, for example, Al Qaeda, some Iraqi Sunni insurgents, and some Shiite militias want the United States out of Iraq because they say that American intervention is costing Iraqi lives. Each group has the incentive and the means to pressure locals and surveyors to exaggerate the death toll from the U.S. overthrow of Saddam Hussein. The Hopkins authors recognized some of these dangers, and they took steps to reduce the risks to surveyors and respondents who participated in the study.

It's unclear, however, if the steps they took complied with relevant U.S. ethical norms and regulations governing scientific research.

The federal government drafted patient protection regulations in the 1970s and finalized them in 1991, responding to public outrage over a series of medical experiments that harmed prisoners and poor African-Americans. Like many other universities, Johns Hopkins agreed to apply those rules to all of its "human subject" projects.

The regulations are crucial because anyone can use them to lodge a complaint against a university with the federal Office for Human Research Protections, which can bar individual researchers or entire universities from conducting further experiments that involve people, and can cut off federal funding.

In 2001, the government temporarily froze all of Hopkins's federal funding—about $992 million -- because of complaints about the death of a volunteer in a medical trial. The university subsequently agreed to tighten oversight of its research. Hopkins now receives more federal funding -- almost $1.3 billion in 2004 -- than any other university. The OHRP, part of the Health and Human Services Department, says that no one has filed a complaint about the Hopkins study of Iraqi mortality.

The designated principal investigator in the study was Gilbert Burnham, a professor of international health at Johns Hopkins Bloomberg School of Public Health. He worked under the supervision of the university's Institutional Review Board, whose legal duty is to verify that proposed projects comply with the protection rules. Panels typically have five members; their sole role is to protect the subjects involved in a study, not to protect the quality of the science or the welfare of the researchers. Hopkins declined to release the IRB's review or to name the reviewers for the Iraq study.

Burnham said that the IRB approved the survey plan with some departures from routine practice, as allowed by law. For example, the panel said that the survey teams had to get only verbal consent, rather than signed consent, from Iraqi respondents, Burnham said, because of the fear that militias might capture the signatures from the surveyors and then attack the respondents.

Burnham briefly discussed the IRB review with National Journal in November, but he subsequently declined to comment once questions were raised about Hopkins's compliance with the standard rules and practices. Michael Klag, who is the school's dean and Burnham's boss, also declined to discuss compliance issues. Instead, the university e-mailed a statement saying, "The Johns Hopkins Bloomberg School of Public Health is satisfied that Dr. Burnham and his research team conducted their studies in an ethical manner and in compliance with the Bloomberg School's policies and procedures."

Anonymous Surveys
A central question is whether the Iraq survey was conducted anonymously. Under section 46.101b(2) of the regulations, if surveyors record the names of respondents, federal law requires that risks to respondents be minimized. If a survey does not collect names, the IRB can exempt it from the federal rules, although not from commonly held ethical duties. (The regulations can be found here.) Johns Hopkins officials declined to say if they deemed the study to be anonymous, or to release the answer form that surveyors used to record respondents' answers during the survey.

Burnham and his colleagues have frequently said that their Iraqi surveyors did not record names. But the Iraqi researcher who directed the survey may have used an answer form that had spaces to record names, with or without Hopkins's approval. In May 2007, the chief of the Iraqi survey team, Riyadh Lafta, gave a copy of what he said was his form to Ali Mohamed, a United Nations official who tracks deaths in Iraq, Mohamed told National Journal. On the form that Mohamed provided to NJ, the top line has spaces to record the location of the survey and the "name of householder." The form also has spaces to record the names of infants and deceased people.

The form is a Microsoft Word document that includes a "properties" feature. The feature reveals that the file was created by "Dr. Riyadh" in December 2005 but that it had been modified to an unknown extent by an unknown person most recently in mid-November 2007. Lafta did not reply to several e-mail queries.

If a survey team collected names after being exempted under section b(2), "they would be carrying out what is arguably covered research without IRB approval," Ivor Pritchard, the acting director at the OHRP, said in an interview about surveys and the federal regulations. However, he added, the regulations also say that a survey can be exempted from the rules even if it records names, providing that the IRB finds that the survey is not "damaging to the subjects' financial standing, employability, or reputation."

Such a finding would be a stretch for Hopkins because Burnham also said that the IRB determined that the Iraqi respondents faced risks for participating in the survey, and thus classified the respondents as "vulnerable." OHRP regulations require extra safeguards for "vulnerable" populations.

Consent
Even if the Hopkins's board declared the survey exempt from federal regulations, evidence suggests that the survey ignored normal practices intended to protect respondents. "The regulations are a floor and not a ceiling," Prichard said. "Institutions are free to impose additional [ethical] requirements" on exempted surveys.

The federal regulations [PDF], and customary medical practice, for example, say that surveyors must inform respondents in their own language of the risks and benefits of the study, and that respondents usually must give their consent before they are asked questions. But although the researchers submitted an English-language consent form to the IRB, the review panel did not check the teams' Arab and Kurdish translations, Burnham said. The consent document "was a basic thing," he said, and "they did not require us to get approval for the back translation," because such approval is needed only when the IRB believes "risks to participants … are substantially above what they regularly experience."

Yet in a 2006 article "The Human Cost of the War in Iraq" [PDF], published by the Massachusetts Institute of Technology, the study's authors wrote that the Iraqi respondents did fear the survey. Once they arrived in a neighborhood, "the teams faced suspicion initially, especially at the first house," the article said. "Lengthy explanations of the purpose of the survey -- and that it would help the Iraqi people -- were necessary to allay fears."

Modifying Procedures
Hopkins's guidelines say that substantial modification of survey procedures without approval by the university's IRB can be deemed a serious violation. "All intentional noncompliance is considered to be serious," the guidelines state.

The survey "was carried out as we designed it," Burnham told National Journal.

He also said, however, that the survey teams worked to reduce risks to themselves by asking neighborhood children to spread the news of their arrival. "That’s actually what happened; that wasn't part of the study design," he said. The survey teams also wore white doctor's coats and conducted all interviews on the doorsteps of respondents' houses, Burnham said, not in the privacy of their homes. Lafta, the survey team's Iraqi leader, acknowledged contact with local militias. "The militias," he said, "are unpredictable, [but] they are very smooth when they know that we are from ‘their side,' " according to the authors' "Human Cost" article.

Minimizing Risk
The federal law states that the IRB can approve a non-exempt study only if risks to the respondents are "minimized."

But the survey teams' interaction with local militias, the doorstep interviews, and the use of children ensured that respondents' participation in the study was public and thus potentially exposed the respondents to pressure from local armed groups. Doorsteps are "not a private location for the conduct of a survey that may have sensitive information" such as whether members of the household had died violently, said Moira Keane, director of Research Subjects' Protection Programs at the University of Minnesota.

In response to this critique, Burnham said that the IRB had not barred the surveyors from conducting doorstep interviews. Also, the use of children "probably does not to add to the risk" of a privacy violation, he said.

"There may be some studies exempt from the regulations where there are identifiable risks to subjects," Pritchard said. "Hopefully, ethical researchers will ask themselves to take steps beyond the regulatory requirements to protect those subjects."

Sign up now to get four free issues of National Journal or two free weeks of CongressDaily.