I was so close to asking Tom to summon the Ghost of D-squared in response to IBC's criticism. Still, the widespread assumption that IBC is accurate is certainly wrong. In a country where people are afraid to leave the house for being shot, the number of confirmed deaths is bound to be below the actual number of deaths. Bodies rotting in the streets and all. Would like IBC to do a similar survey, but it still would be like discussing how horrible war is. Whanking, in other words.

Anyway, loved this quote from Logic Times: [War opponents] argue that chaos in Iraq represents failed policy.

Indeed they do.

Aside from IBC only counting reported deaths, a certain undercount, I thought the other advantage the Lancet study would have would be that it counted all extra deaths, not just people being shot or blown up.

I'm curious as to how death certificates work. If an entire family gets blown up, would someone from the government still find them and issue the certificates? Are all insurgents claimed by their families, even the foreign ones and the ones buried under rubble they bring down upon themselves? What about the people who go missing and don't come back? How much of this is reported and/or compiled?

I'm not in a position to really judge the accuracy of the numbers reported here, but I would expect a study compiled in this way to have a far higher total than the IBC one. And also that if (and this is a big if) the sampling is done correctly that it would more accurately reflect reality.

I don't share Matthew's scepticism about how so many deaths could have escaped media attention, though he makes good points generally - in a civil war situation where journalists are getting killed in greater numbers than in any other conflict ever, it seems unlikely that they would fearlessly run off to document every death.

IBC makes very good points and addresses a question I had about whether the 92% death-certificate-backed figures matched up to figures from certificate-issuing authorities. But they over-emphasize the Coalition's role in causing these deaths (perhaps reflecting Lancet's own emphasis on coalition agency) - while it may not be credible for the coalition to have killed this many Iraqis this year, all the bomb blasts and sectarian killings might have done the job, no?

Dear all,

I do not quite understand why none of the previous commenters, while rightly pointing out to the IBC's very likely under-registering (it is, after all, just a news aggregator) of deaths in Iraq, has answered its critique of the Lancet Study.

The IBC guys themselves agree that their numbers are most likely less than that of the actual casualties.

However, the Lancet Study has so many gaping holes in it - pointed out by both Matthew supra, the IBC, Mike Dunford (http://scienceblogs.com/authority/), and others - that its failure to provide USEFUL data has been demonstrated.

Iraq is very well covered by media attention - foreign and (Iraqi) domestic. I should know - I work in that field. Add to that a political situation where there are plenty of political entities who would LOVE to see high casualty numbers ... and there is simply no way that a situation where the actuall number is TEN TIMES as large as the official one will go unnoticed, unreported, undiscussed, unspun, unused, un-screamed-off-the-roofs-&-tv-stations.

Car bombs and other explosions are PARTICULARLY well-recorded by the various authorities, but also by the national, provincial, and local media.

Now, given the fact that there are areas in Iraq that are more or less free of violence - Kurdistan, vast parts of the South - that would mean that almost all of those 600,000 deaths have occured in Baghdad, Sunni-stan, and Karbala/Najaf/Basra. A fifth of those would've had to have happened in the capital, home to 20% of Iraq's population.

Trust me - SOMEone would've noticed if 120,000 people had been killed in Baghdad over the past 3 years. And you & I & everyone else would've read or heard about it.

I have no idea why the Lancet Study people fucked up so badly. I'm no statistician and thus cannot say anything to/about their methodology. But I've worked in Iraqi media, with Iraqis on a daily basis for almost 2 years now & I CAN say that those numbers are simply wrong.

And it pisses me off. It pisses me off for the simple reason that, now that the Lancet Study will be pulled apart and shown to be flawed in ways that were EASILY avoidable (hello, why didn't anybody use their BRAIN when they collated the data & arrived at, chiching!!!, an unbelievably high number???), it will be dismissed and laughed at by the pro-war factionistas. A truly well-done study that still would've arrived at, say, 100K dead or even 200K, with a sound method and - preferably - having it cross-checked by Iraqi scientists on the ground in Iraq would've been actually useful.

This one ... I wish they'd never done it. It brings more harm than good.

In the end, of course the numbers should not be the yardstick for the evaluation of the war & its aftermath. The situation should be discussed in terms of quality not quantity, or else we'll get into ludicrous comparisons of bodycounts.

--Matthias S. Klein
www.niqash.org

Read the report properly, Aqoul! The methodology stands up.
Just because the figures are shocking does not make them false. Iraq Casualty Count only tallies deaths reported by the media. This is extremely unscientific, passive and fails to count deaths outside the gaze of journalists.

regards
Simon

I basically agree with the criticisms of the Lancet study, but not with this:

[Iraq Body Count's] 40-some-odd thousand reported to date [is] probably low, but probably includes a large proportion of non-civilian insurgents...

From what I gather from the IBC methodology, it probably does NOT include a large proportion of non-civilian insurgents, and that is the main reason why it's a low count. They (claim to) only count deaths that have been explicitly reported as civilian by two independent and credible media sources. So, even if both the BBC and the Washington Post gives good evidence that "40 people were killed in widespread violence in Baghdad today", that registers exactly zero deaths with the IBC, since they are not explicitly marked as civilians by BOTH sources.

Sure, a couple of thousand militiamen could have been mixed into that, but if they are serious about how they do their counting (I don't know), then it doesn't affect the final numbers much. Also, the IBC only counts violent deaths (as opposed to Lancet-type studies, who include epidemics etc), and, importantly, it has no way of following up on people injured in attacks, who die unreported deaths in the hospital or at home a few days later.

Presently the IBC counter stands at about 44-49,000, but I would argue this must be a gross underestimate of total war-related deaths (combatant & noncombatant, violent & non-violent), even if the Lancet study is probably off the mark.

MSK makes my points a bit better.

IBC doesn't claim to make a full count, just the most verifiable. Personally, I would guess the toll to be almost double their figures, espcially in the last several months. They themselves don't claim to be the final word.

But if a study says that half of the United States die of HIV in the last 50 seconds, there needs to be a second look. (Yes, that's exaggeration but people need to get a grip on numbers.)

One needs extraordinary delivery of firepower or an extensive series of mass executions DAILY over 3-4 years.

To illustrate again, this is the same proportionately as the the entire death toll of the Lebanese civil war period of 15 years including one major Israeli invasion, two shorter ones, two Syrian serious intervention and three phases of sectarian bloodletting without real limits, including the great massacres of Sabra, Shatila, and other places.

Don't think so.

Now, with the civil war raging at a higher order of magnitude one is only now getting a rate that comes (if one doubles the reported figures) close to generating these casualties in three years.

Casualties like that also yield mass relocations, refugee crises, secondary injuries.

3 dead Iraqis for every single US soldier in Iraq?

If this study gains conventional wisdom status it will make efforts to limit civilian casualties or accentuate their horror impossible. The first studies finding of two thirds of violent deaths in one cluster in Falluja alone -- and then later extrapolating 100,000 violent deaths is just a clear go back to drawing-board warning.

This is from peer-reviewed medical literature, the place where wine is good one week but kills you the second. And second-hand smoke will give you cancer if you just look at it. (Yes, I do believe second-hand smoke is harmful, and I am exaggerating, but once politicized, facts can run wild, just take a look at revisions in HIV and AIDS deaths in Africa).

Dear Simon,

the numbers aren't false because they are "shocking" but because the methodology seems to have gaps and because a number of more than 600,000 killed would not have been undiscovered until now.

I have not claimed that only journalists can count dead people - there are many NGOs in Iraq as well as government institutions that work in all the regions. Many, if not most, of them are patently anti-occupation. NONE of them corroborates the numbers provided by the Lancet Study.

Please engage Matthews, the IBC's, and my arguments against the Lancet Study.

I would really like the commentators to talk about the Lancet Study - yay or nay - and not about the IBC's own problems. IBC, btw, does NOT claim to provide complete, authoritative data - CONTRARY to the Lancet Study.

--Matthias
www.niqash.org

I found the best criticism of Lancet to be the car-bomb argument. Lancet suggests that 9 out of 10 car bombs go unreported, but IBC receives, on average, 6 unique reports on each one they tally. Which makes no sense at all.

I found the best criticism of Lancet to be the car-bomb argument. Lancet suggests that 9 out of 10 car bombs go unreported, but IBC receives, on average, 6 unique reports on each one they tally. Which makes no sense at all.

Indeed. The whole thing is questionable and far from any final word.

And we haven't even gotten into reporting bias.

Claims of coalition destruction or death get compensation sometimes. That can inflate reports and "documentation". I haven't looked close enough to see if this is accounted for, or not relevant, but it is one of those things that should be red flag fluttering in the background.

And as MSK notes, the figures are not shocking, they are ridiculous, and inconsistent with prevailing long-term reality.

IBC references another report/survey from early 2004 done by UN development program. Not allowing for biases and confidence-intervals, they got somewhere around 20,000 dead in the first year or so. Extrapolate, some years were quieter, some had fallujahs and that one they used had shock and awe and invasion.

One problem is the conspiracy of silence -- no one has an incentive to be factual -- the antiwarriors like to say war is just hyperhumongous atrocious bad (and "Bush and Co" especially who will stop at nothing), while the empire-warriors want to say -- see, who can avoid civilian casualties? Israel was so nice and careful and sweet, see; and those bombs now hitting Teheran in 2007, a mere 10,000 dead, a drop in bucket compared to Iraq!

The IBC release nicely points out several strange implications of the Lancet study, such as the inexplicable number of wounded that would clog up the health system (even more). This article perhaps gets at some of the the cores problem with the methodology that produced those results.

Thanks for this, when I get a chance I may update and link. Makes sense on the surface.

I would really like the commentators to talk about the Lancet Study - yay or nay - and not about the IBC's own problems.

Well, in that case: nay, but thanks for trying.

Apparently these methods of research ("clustering" or whatever it's properly called) have been very successful elsewhere, such as in Rwanda and the Congo. I remember reading somewhere credible that a team using identical methods produced the figure of 800,000 dead for Rwanda's genocide, which has later been confirmed by counting and naming victims (perhaps I'm wrong here, I don't know much about Rwanda, but that's what it said). So it's probably not the methods per se that are flawed -- they're the best available -- but the selection. I assume it's harder to investigate an extremely lethal war zone such as Iraq through remote controlled local staff (which will obviously have their own limitations of movement and reach), than it is to sift through the Rwandan countryside post facto.

As for my private, uneducated guess, I wouldn't be at all surprised if the full total numbers of war dead (violent/nonviolent and civilian/combatant) are now around 250-300,000, but that's still far below the Lancet's lowest likely outcome.

One of the points in the link in my last comment is that although clustering may be a good method, the author's opinion was that not nearly enough cluster points were used in the study.

Added the Wall Street Journal commentary. The criticism there sounds like it might be reasonable, despite the biases of the source and environment.

Also withdrew my Rwandan genocide comparison due to an extra 0 I omitted. Added an Iran-Iraq war comparison.

Moore:
"With so few cluster points, it is highly unlikely the Johns Hopkins survey is representative of the population in Iraq."

That's just wrong. So long as the cluster points are unbiased, they should be a reasonable representative of the Iraqi population. Too few points would make the confidence interval larger, but it would not change the result.

"However, there is a definitive method of establishing if it is. Recording the gender, age, education and other demographic characteristics of the respondents allows a researcher to compare his survey results to a known demographic instrument, such as a census."

Sure. If this information was available the confidence interval would be smaller. But how would it change the result?

Naturally, before writing the above I should have checked on the fine folks at Stats.org. You don't have to agree with the results, but cluster sampling isn't voodoo. I realize nobody on this board probably likes Deltoid, but Tim Lambert has the best roundup I can find of actual statisticians discussing the study.

Anon

I like deltoid, but I'm too busy and brain-burned to engage in the required spirited debate & near-plagiarism it would take to get into this fight.

In the meantime, ghost of D-squared invoked

Too few points would make the confidence interval larger, but it would not change the result.

That actually sounds wrong too. (And yes, I know an article at Stats.org says this.) Even two unbiased sets of cluster points will give a different result, and a difference confidence interval. Now, I mean this in practice, not in a theoretical proof of the accuracy of the cluster point method, or an algebraic representation of its result and error. True, the difference may be small, if both sets are truly "unbiased", ie. properly represent the distribution of population or other factors. But they'll be different. Would you expect two valid and proper statistical studies to produce identical results? Of course not. So the questions remain, if the number of cluster points is small enough, is it not inevitable that it becomes unbiased, and did it reach that point?

But wait, there's more trouble: there may be inherent problems with the choices and methods used in the Lancet study that fall outside the usual clustering method. In this article, Seixon, who apparently isn't very impressed with Tim Lambert, points out some serious problems with the 2004 Lancet study, where the claim was 100,000 deaths. It's a very interesting article, worth reading, although he sums it up nicely in a recent comment on Deltoid:

Great, now if only [she] could explain the scientific basis for the 2004's study doing the following:

1. Using "cluster clumping". Give me ONE example of another study using this methodology. ONE.

2. What's the scientific basis for pairing up all but five of the governorates in Iraq?

3. What's the scientific basis for the pairings at all? Where's the evidence of the pairs being similar? There is none!

I fear for science if this sort of thing was overlooked by the scientists supposedly endorsing the 2004 study's methodology.

The pairing is discussed in the article, but, briefly, the Lancet authors paired up provinces in order to cut down on surveying, by allowing one of the provinces to represent both in each pairing. Unfortunately, the provinces paired in many cases had wildly differing media-reported body counts, making it very unlikely that they were at all similar with regards to actual death rates. It also meant large sections of southern and northern Iraq were not surveyed at all. (Map link from the article.) The Lancet report even admitted that these techniques were used to cut down travel in Iraq, and that they don't know how to quantify the error produced by this. In other words, the confidence interval doesn't represent a possibly large source of error.

Seixon writes about the 2006 report here.

Now, I don't know too well the political biases of these critics and defenders of the Lancet study. Most of these people I'm reading for the first time, so feel free to chime in with characterizations. The critics have so far pointed out some serious problems with the study, very clearly and specifically. I haven't found any defense that specifically refutes these points, or provides more compelling ones that render them moot. The Stats.org articles mostly defend cluster sampling and statistical analysis in general, which no one has a problem with. I didn't start with a preformed opinion on the Lancet study, and I didn't reject it when I first heard it (Iraq is doing badly no matter what study you look at), but recent reading is making me doubt it.

I think the basis of dismissal here is thin. For example, compare that map link above to this map of Iraq by population density. They didn't sample Muthanna, but nobody lives there, so nobody dies there either.

Regarding the pairing, what makes you think the media-reported figures for - say - Anbar have any validity? You can't go there to report it and neither can you go there to sample it, because you might get killed, but this is evidence of a low death rate?

And the rest is really just argument from incredulity. But surely all these people couldn't have died and nobody noticed! Well, quite a lot of us have noticed there's a war on, and secondly, how can you be so sure we haven't just noticed?

I have to say I am finding this entire debate uninteresting. A large number of civilians have died in Iraq, I believe few rational people would contest that. It is also well-nigh impossible to do proper survey work in a civil war.

So, all figures are likely to boil down to guestimates. So a pox on both critics and the study.

I fail to see the real importance of the debate, when it comes right down to it. Future looking subjects are rather more interesting, such as what the civil war is going to evolve towards.

Alex: A few pairings are very odd, especially Dehuk and Ninawa. Ninawa has four times the population of Dehuk, but 500 times the death rate according to the IBC. Whether or not those are complete, it's the only verifiable information we really have, so it's odd that the study would pair them together, and count Ninawa's death rate as counting for both. And yes, who's to say whether the IBC numbers are representative, but they're all we really have. The study made choices that fly in the face of that information apparently without any other justification.

But I won't harp on about this, I just happened to do a lot of reading on it last night. I agree that in relation to Iraq's real problems, this type of discussion doesn't amount to much. To me personally it's a question of whether there was any irresponsible science being practiced, even if this may not be the most appropriate forum for that. And it may be impossible to prove either way (the accuracy, and the soundness of the techniques used).

"I fail to see the real importance of the debate, when it comes right down to it."

In terms of what we all know to be true, an expanding civil war, the numbers dont mean much.

But if someone says Exchange X has a P/E of 40 and another says it's 600, it matters, though both are ridiculously high.

Precise numbers are the basis for analysis of current events and prediction of future ones. Casualties say alot about policies undertakena nd military power (and $$$) expended.

Further, it distorts objective measurement in the direction of the whole growing mythology that civilian casualties are promiscuously inevitable, and therefore it all makes no difference.

Now with civil war more and more full blown, real time casualty figures will be hard, but until the big mosque bombing the level of violence was more reasonably watchable, and the post big mosque bombing period (after Feb 2006) is not very much covered in the study so it was done in a time where (compared to these past months) relatively clear data are available.

Also, on a financial level, if some country, international body, or private firm were to take on reparation costs or medical expenditures (eg USA pays comp for coalition damages some time), etc. such tasks at some point, they need realistic numbers to budget and solicit by. A factor of 10-20 is a big difference in real money too.

It matters alot and is also an illustration of how credulous and lazy conventional wisdom can be, even on the war skeptic side.

Perhaps a more concrete illustration -- sometime back in 04 was it, the US announced a sinle incident in which about 50 insurgents were killed.

Realizing that was an anomaly, I originally feared it was a massacre misreported or covered up as a battle. I had called it wrong, it was not a massacre, it was a falsehood.

But I was correct to note that that report really needed a second really close look, because the sheer numbers alone illustrated something EXTRAORDINARY happened -- 50 dead IS HIGH for a single day in a single place.

It turned out that the best close-up accounts, hospital wintesses weapons bodies showed there were maybe three dead.

What do we learn by having a good handle on what is high and what is not -- whether or not a policy is working. The fact is, such an event was clear objective numerical indication that the US military did NOT, contrary to its reports, have the upper hand or control over the situation.

Nor incidentally were its troops just shooting up villages and kiling Iraqis willynilly, which would also be implied by the Lancet figures.

In that incident, the reporters back then all noted the same thing - that about 50 dead in an engagement was something special and many took special care to race out there (at a time when it was easier) to independently confirm the information and failed to do so.

The number made all the difference in illustrating the whole nature and effectiveness and style of US policy in Iraq.

If people just accept high figures without realizing something is wrong with them and acting on it, they start to accept false pictures of the situation.

Indeed, if the US/coalition had killed as many as the Lancet alleges while suffering just 2000 soldiers dead, it would probably indicate the unvarnished SUCCESS of US policy to crush the insurgency, a higher rate of actual proven dead in Fallujah (perhaps less) caused the insurgents -- and even the entire people for a while -- to abandon the city, and it has ceased to be a center of resistance. Imagine the whole country.

Put another way

-- small number of dead = tragedy
-- large number of dead = crime/horror/shock
-- humongous number of dead = victory

In the cruel calculus or war, overestimating the dead, as in Vietnam, is worse than a crime, it's a blunder. It give wrong indications as to real measures of success and failure.

If tomorrow the US came out swinging and really DID kill that many Iraqis over the country, the US would be called many many things bad by many more people, but it would, in a real local sense, have undeniably won.

(Which is why war is so appealing and so evil at the same time.)

Re: death certificates and point number 5 of IBC's rebuttal. My understanding is that these days you can simply report the death (usually with some confirmation) to a local police station and they will issue a death certificate. There is no need to go through the government and its ministries to receive it, so it should hardly be surprising that they do not have a record of said certificates.

Re: point about lack of hospital patients. Perhaps doesn't account for the entire gap, but it has been well documented that Shiite death squads often prey on hospitals killing every Sunni in sight. In fact several Iraqis were quoted as saying they would prefer to chance death on the streets then go to a hospital where they believe it a certainty.

This WaPO blog post has some pretty interesting commentary on the issue, from other experts who have made casualty surveys in Iraq.

I have to say, I pretty much agree with Lounsbury's Oct 19 sentiments. This hasn't stopped me from getting involved in a wankathon about the reliability of this study over on my main BB. I personally feel the study is off by a factor of 2-3, but 'feeling' has diddle squat to do with evaluating a quantitative study. Looking at the various critiques of the report I am starting to suspect it may be right and I might be wrong.
Pertinent to this is Mike Dunford's reconsideration of the study, which concludes:

The more I read the Lancet paper, and the more I examine my own concerns with that article, the more reasonable their conclusions appear. I still think that their number is likely to be on the high side, but I don't think it's going to be high by as much as I would have liked to believe. In fact, I think that the actual number is quite likely to fall within their stated 95% confidence interval. It's painful to think that we have so much blood on our hands as a nation, but the concerns that I still have with the paper's methodology are almost certainly not severe enough to account for even half of their estimate.