“If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming you know nothing about the person's symptoms or signs?”
This assumes that the test had a sensitivity of 100%.
One of the bloggers surveyed his residents and got a similar rate of wrong answers.
The answer, as revealed in the paper, is 1.96%. The authors accepted any “ballpark” answer (2% or less) as correct for survey reporting purposes. I got the right answer but cheated a little by consulting this resource.
So what's the real problem here? I think it's a little over the top to say we're dumb as rocks about Bayesian statics, but the blog author is correct, in my opinion, in his assertion that, as a profession, our overall foundation in EBM is poor. (I would digress for a second to add that it goes way beyond our inability to do the math; or misunderstanding of EBM is pervasive on many levels).
I think most of us understand Bayesian principals qualitatively. We know, for example, not to rely on the D dimer assay as a rule out for VTE in a high risk population. That's Bayesian thinking. But the math is not something we do everyday. The challenge question set a trap for the survey respondents by applying a test with good inherent characteristics (low false positive rate) to a low disease prevalence population. Unless you really stop and think you're tempted to jump to an inappropriately high probability of disease.