(04-22-2020, 08:50 PM)BostonCard Wrote: Also, it is possible that the concerns about selection bias and the test specificity are overblown, and, in fact, the numbers that are quoted in the article are broadly right. For the record, I don't think so, but it is possible.
Take the concern about sensitivity. Yes, based on the confidence interval of the sensitivity it is possible that all the positives are false positives. But based on the point estimate of the sensitivity, that is not the case. So, bottom line, I think the confidence interval that the paper cites is too narrow, but their point estimate is unaffected. I believe that will need to be fixed during peer review and when this paper comes out, I'd say it is likely that it will feature broader confidence intervals that propagate the sensitivity uncertainty better.
The concern about selection bias is legitimate, but they acknowledge it in the paper. Unless you can come up with a way to measure and adjust for it, the best they can do is what they have done; list it as a limitation. You might speculate on its directionality, but there is no way to be sure of its magnitude.
Yesterday, I posted a link to an article about hydroxychloroquine that has also been making the rounds on news outlets. Unfortunately, it showed that hydroxychloroquine doesn't do anything for patients with COVID-19. It was a retrospective study, and suffers from the same bias (patients with more severe disease are more likely to be treated, a phenomenon known as confounding by indication). I noted the bias when I pointed the paper and said I'd wait until higher quality papers would come out to make up my mind for certain. However, I also pointed out that it is unlikely that HCQ is a "game changer" because we would have seen it. So, even though there is likely bias in the paper, I wouldn't demand its retraction.
Same here. I will wait until true random sampling (or saturation testing) is performed before I make up my mind for certain on the prevalence and IFR. But the study still moves the needle, and it is reasonable to think that the prevalence might be higher and the CFR might be lower than previously reported.
BC
Much of my job involves designing sampling schemes and analyzing the resulting data. The whole point of these sampling regimes is to be able to make inferences about the population. If the sampling is biased--unrepresentative of the population--the resulting estimates are essentially useless. Without accessory information about the nature of the bias, you have zero basis for estimating population means.
For this study, because there was self-selection in the study participants (we have evidence for this from facebook and nextdoor threads), we have no way of knowing how representative the 3000+ people sampled were of the county. But with only 50 positives, it would not take many 'excess' positives (beyond what you would get if you sampled everyone or did a truly random sample) to make the resulting estimate wrong.
With that as a given, we simply have no idea how unrepresentative the sample was . Was there one excess positive? Five? Twenty? No one knows. And no one can know with this dataset. You could apply a correction using accessory information from the study participants like history of symptoms, exposure to ill people, and desire to be tested, and you could correct using the same information gathered at random in the population. But because that was not done, we're back to an unrepresentative sample and no ability to make inferences about the population.
Like I said earlier, this isn't just a mild methodological dispute. It's a fundamental error. I don't ascribe any bad intentions to the authors of the paper. Maybe none of them have ever analyzed a sample survey. Or maybe they just didn't consider biased responses a possibility. But this error makes all of their conclusions invalid. This study should not move the needle any more than my making up fake data and writing a paper with an IFR of 3% should. We need to be guided by good data and good analyses. Period.