Wednesday, October 11, 2006

New Report on Deaths in Iraq

A new study is out purporting to show that over 600,000 Iraqis have died as a consequence of the U.S-lead war in Iraq. John Podhoretz of National Review asked me to compile into a set of questions some of the issues I raised about the study with him, and I do so below.

You can read the study here.

I should note that, while this isn't my first time around a random sample, a survey instrument, or a log-linear regression model, I do not study mortality rates and would defer to the expertise of scholars in the field (i.e. medical research) who have experience with these models. However, I do have some questions from a semi-lay perspective.

1. The survey reports a pre-invasion crude mortality rate of 5.5 deaths per year out of 1000 persons in the population (based on [edit: 82, not 2] 2 pre-invasion deaths reported in their sample). Is this consistent with death rates in countries similar in development to Iraq?

2. A one year period prior to the invasion was chosen for the pre-invasion period comparison baseline in the study. Why was this interval chosen? Note that this is a period with a coalition enforced no-fly zone in effect. Would a different interval have produced a different mortality rate? For example, what about one that included post-Gulf War reprisals? This is very significant. The pre-invasion mortality rate is integral to calculating the # of deaths attributable to the war in Iraq. Higher pre-invasion mortality rate, fewer deaths attributed to the war. And note, they get their estimate of the pre-invasion mortality rate *from* their sample. So a household is asked about deaths from each of the periods. They may be underestimating the deaths in the earliest period simply because it was so long ago.

3. Why are the confidence intervals around their estimates so large? It sends up a pretty big red flag. While sensitivity checks were done according to the authors, I wonder whether outlier clusters might be driving the inflated death totals. Their model assumes that the "variation in mortality rates across clusters is proportional to the average mortality rate." Perhaps non-constant error variance in their data may account for the inflated confidence intervals they estimate?

4. I think there is a fundamental question as to whether this baseline-comparison model is appropriate for assessing the impact of the war on the number of deaths in Iraq. It is very sensitive to the baseline selected as well as the variance in deaths across clusters. While the authors acknowledge several possible sources of systematic bias in their sample (i.e. the possibility of over-sampling higher-mortality clusters as a result of population migration), I don't think they appreciate just how much of an effect that could have on their estimates. Their method does not and cannot account for the essential non-randomness of warfare. Coalition forces do not drop bombs randomly. Insurgents do not plant IED's or car-bombs randomly. As such their sample may be severely biased by the fact that individuals living in Iraq make rational decisions to move away from hot-spots when able to do so. Less people in the household means a higher mortality rate.

5. Why was no effort made to distinguish combatants from non-combatants? I understand that there is concern for risk to the interviewers (as stated in the article), but this is pretty important. The authors argue that this is the "this the deadliest international conflict of the 21st century" and that they have urged investigation into the "excess" mortality rate in Iraq. But what if a large percentage of that 'excess' is, in fact, coalition forces killing insurgents? From a policy perspective, there doesn't seem to be much logic in treating the death of an innocent child from an IED the same as the death of the terrorist planting the IED at the hands of coalition forces...yet that is exactly what the study does. Note the study acknowledges that "Across Iraq, deaths and injuries from violent causes were concentrated in adolescent to middle age men." The exact population from which combatants are drawn.

UPDATE: IUD vs. IED spelling error corrected. Thanks to Craig for noting. The universe of possible jokes as a consequence of that error I leave to the imagination of the reader. ;)

13 Comments:

Blogger Craig said...

While it can certainly be argued that innocent children are killed with IUDs, you perhaps meant IED here.

Good analysis, though.

Warmly,

2:28 PM  
Blogger Donald said...

Woops. Thanks for the correction! D.GOOCH

3:13 PM  
Blogger chew_2 said...

If you read the 2004 Lancet study using the same methodology, I recall there was a discussion that Jordan and Syria had a death rate around 5.+ per thousand. The 2004 study also yielded a baseline death rate of 5.+ per thousand.

Regarding separating out combatants. Not worth it. Too dangerous and you couldn't trust the responses anyway. Plus, a comabatant could have been police, security forces or a sectarian militia or home guard, not just an "insurgent".

4:50 PM  
Blogger d said...

Note the study acknowledges that "Across Iraq, deaths and injuries from violent causes were concentrated in adolescent to middle age men." The exact population from which combatants are drawn.

"If it's dead, it's VC."

1:22 AM  
Blogger Donald said...

And it may be too dangerous. But that means that you don't argue that your study, which has no effort to distinguish the two, should be a clarion call for attention to an 'international' crisis in Iraq as a consequence of 'excess' deaths. Killing insurgents is a big part of the military mission in Iraq. You don't make policy proscriptions based on a study that fails to distinguish when the policy succeeds (i.e. kills insurgents) and when the policy fails (i.e. protect innocent Iraqi citizens).

9:34 AM  
Blogger tempjcp said...

Above you wrote: “(based on 2 pre-invasion deaths reported in their sample).” I believe you meant 82 deaths.

Your typo did shock me into looking at the study though, and their methodology is seriously flawed. The most significant flaw is revealed on page 2: “...or those that refused to participate were passed over until 40 households had been interviewed in all locations.”

Since they state they relied on word of mouth to inform people of the purpose of the survey before their arrival, that decision means the sample was no longer random but self selected by the participants, and the study useless.

I wasted no time reading the study further.



-jcp-

2:05 PM  
Blogger Donald said...

Sorry, that's right. 2 violent deaths, 80 non-violent deaths in their sample.

2:08 PM  
Blogger steve said...

A few comments;
tempjcp writes; "Since they state they relied on word of mouth to inform people of the purpose of the survey before their arrival, that decision means the sample was no longer random but self selected by the participants, and the study useless."
I agree that the locals needed to be apprised of the investigators arrival. The identity of the teams were announced, not vital statistics. This was done for security purposes, not to introduce bias. Refusing to answer questions is not haphazard and the investigators did keep to the N=40 for each cluster. They did acquire death certificates for a vast majority of those that were asked for.
The OP states in 1) that the one year period prior to the invasion is questioned. What year would he choose? It would need to be recent. It could not be a year in whichunusually large numbers of the population were dying, such as the first Gulf war or those terrible years fighting Iran. During some of the non-war years there were a large number of Kurds dying. I'd say that the actual year that was needed was fairly limited. He further states that the pre-invasion mortality rate is integral to calculating the # of deaths in Iraq. Yes, but a baseline is needed, one in which there is a 'normal' rate of deaths and this is compared to war years.
OP's 3) Confidence levels are published at 95%. This is the gold standard of modern surveys. This means that they are 95% positive that using a standard deviation of the population that a given percentage of the population density curve lies within this range. That is why for published surveys, you see a tag that says, 'this survey is 5 +/-, or 3 +/-'. This is done with that figured out for you.

6:57 PM  
Blogger Donald said...

The problem isn't that the publish 95% confidence intervals. The problem is that their 95% confidence intervals are very wide. Not quite as bad as with the first study...but still very large.

7:01 PM  
Blogger tempjcp said...

Steve,

You are obfuscating. Either the participants knew the purpose or they did not. If they knew the purpose it was not a random study.

The paper states that “By confining the survey to a cluster of houses close to one another it was felt that the benign purpose of the survey would spread quickly by word of mouth among households...”

Once the participants know what is going to be asked they have control over the results and the survey is useless.

8:39 PM  
Blogger ed said...

"...5.5 deaths per year out of 1000 persons....Is this consistent with death rates in countries similar in development to Iraq?"

According to the "CIA Factbook," yes. That is about the same death rate as Iran or Egypt, and much higher than Jordan, Oman, or Libya (for example).

Most developed countries have much higher mortality rates, but I expect that is because they have much older populations. I don't know how infant mortality is factored into these numbers.

https://www.cia.gov/cia/publications/factbook/rankorder/2066rank.html

3:06 AM  
Blogger Donald said...

But if you go by the CIA Factbook, that *remains* the mortality rate for Iraq.

11:42 AM  
Blogger Ryder said...

Seems to me that there are at least two elements that are suspect in all of this:

1 - The use and issuance if death certificates. If death certificates are used in *any* payment or benifits systems in Iraq, then the counterfiet death certificate market would be massive, as would reports of deaths.

2 - Crime. General lawlessness may have increased dramatically in Iraq, especially since Iraqi security forces are now the lead agencies in maintaining domestic peace. Assaults in simple crimes, resulting in violent death (which is more a measure of Iraq culture and society than an attribute of our military actions in Iraq) would be measured here.


I could see either of these as being major roadblocks in getting an accurate assessment of the results of our military involvement in the region.
-Ryder-

6:42 PM  

Post a Comment

<< Home