Communities across the globe are trying to understand what percent of their population has been exposed to COVID-19 by searching random samples of residents for antibodies against the virus.
Los Angeles County, the most populous county in the U.S., tested 863 people in an antibody study. New York tested 3,000. A Boston study looked at just 200 people passing a certain street corner. But one sparsely populated county in the Rocky Mountains has done a more thorough job than any of the others. It tested about 70% of its population -- almost 5,500 people. And the results are now in.
San Miguel County in Colorado teamed up with a biotech company (founded by a couple of its residents) to offer a COVID-19 antibody test to all residents who wanted one. It took about a month to get results back, but they show that about 30 people came back positive.
“Only 0.5% of our population tested positive for the antibodies,” said Susan Lilly, a spokesperson with the county. Even if you add in 80 more people -- those who were “borderline,” meaning there’s a chance they were in the early stages of producing antibodies -- that’s still less than 2%.
“In a lot of ways it’s very good news because we haven’t had the health impact or devastation that so many people and so many areas have,” said Lilly. “On the other hand, it can be considered bad news if you say, ‘Well, gee, we haven't been hit. Yet.’”
This fits with the picture emerging as regions search their residents’ blood for antibodies. Antibody surveys in other places have also found evidence of low prevalence, with exceptions like New York City (21% positive) and Heinsberg, Germany (14% positive).
Public health professionals were crossing their fingers that antibody studies would reveal lots of people who’d already been infected and could return to regular life with their newfound immune armor. But, as The Atlantic wrote last month, “if the first results from antibody surveys, also known as ‘serosurveys,’ in the U.S. are anything to judge by, simply not enough people are immune. Too many Americans are still vulnerable to COVID-19 infection for these tests to be the ‘game changer’ that many were hoping for.”
“You would have hoped for 45% or even 60% positive,” Mark Perkins, a diagnostics expert at the World Health Organization, told Science Magazine in late April. “That would mean that there is lots of silent transmission, and a lot of immunity in the population. It now looks like, sadly, that’s not true. Even the high numbers are relatively small.”
Dr. Ajay Sethi, an infectious disease epidemiologist at the University of Wisconsin, Madison, says the San Miguel County numbers are, “in a lot of ways, expected” due to results from other places. But what’s different about San Miguel County’s antibody testing is its scale.
For example, in Santa Clara County, California, researchers were criticized for their methods of recruiting volunteers and extrapolating their findings to the overall population, which some said led to an overestimate of the percent infected.
“There's always a question of how representative the sample that was studied is of the entire population," Sethi said. "Sampling matters, especially in the context of shelter-in-place policies, where so many people are not even leaving their homes in order to be at a supermarket to get tested or being selected for testing. So you have to, you know, take those results with a grain of salt. The San Miguel County study, though not very many people came back positive, what was interesting about that was two-thirds of the county was tested. And that's a very large sample. I think that's pretty representative.”
Scientists say there are a lot of caveats with antibody testing -- it’s still unclear to what extent antibodies against COVID-19 mean immunity against COVID-19, and if so, for how long. And antibody test results can be difficult to interpret, with the possibility for both false positives and false negatives. A lot of that has to do with statistics, rather than with the performance of a specific test.
"It is kind of a strange thing," Dr. H. Gilbert Welch, who studies issues with tests and screening at Brigham and Women's Hospital, told NPR. "An antibody test is much more likely to be wrong in a population with very little COVID exposure."
And, unfortunately, it means that even a bigger, more representative sample doesn’t guarantee better results.
Say the same test was used on two towns of equal size, the only difference being that one had a worse outbreak than the other. As a calculator created by NPR shows, in the town where 20% of people had recovered from COVID-19, each person who got the test would only have a 4% chance of being told they had antibodies against the virus, when in fact they didn’t. In the town where only 1% had recovered from COVID-19, the chance of false positives would rise to 52%. (ProPublica has some good visualizations illustrating this issue).
A lot of this has to do with two somewhat slippery measures of a test’s reliability: sensitivity and specificity.
Ideally, the perfect antibody test would identify all positives (also known as “100% specific”) and wouldn’t misidentify any negatives (also known as “100% sensitive”). COVAXX, the company that did the San Miguel County antibody tests, claims its test has such performance. An FAQ page from March from its parent company, United Biomedical, said the sensitivity finding was based on trials in which the test successfully identified COVID-19 antibodies in the blood of an undisclosed number of patients who were confirmed to have COVID-19 by other tests. It said the specificity finding was based on trials in which the test successfully came back “negative” after testing 900 blood samples collected from patients before the COVID-19 outbreak started. (That’s well above the FDA recommendation to test at least 75 negative samples.)
But Dr. Lee Riley, a professor of epidemiology and infectious diseases at University of California, Berkeley School of Public Health, said such perfect performance is so far mythical. “There’s no such antibody test. It just doesn’t exist,” he said.
COVAXX, the company that makes the test, said it was unable to comment on performance, following recently updated guidance from the Food and Drug Administration, and said the company has submitted documentation to the FDA for review.
Assuming 100% specificity is impossible, Riley said, say the test is actually 99% specific, meaning that 1% of those tested will be false positives. Of the 5,500 people tested in San Miguel, 55 people should come back as false positives. The total number of positives, Riley explained, should be 55 plus the number of actual positives, so the fact that San Miguel County came back with just 29 positives is a red flag. “I would not rely on the results of that test,” he said.
That’s not to say this test is any different from others. As the New York Times reported, lots of tests likely boast better performance than in reality. A group of scientists checked 14 different antibody tests (which did not include the one used in San Miguel County). They found that only one test was 100% specific, meaning all the people it identified as positive were, indeed, positive. Two others were 99% specific, meaning only 1% of people tested were given a falsely positive result. But the three tests with high specificity didn’t have high sensitivity, at best missing 10% of the people who were, in fact, positive. Initially the FDA wasn’t inspecting COVID-19 antibody tests the way it normally would in non-pandemic times, but after enough concerns arose about their validity, this week the agency started requiring companies to submit data about their tests to the agency for review.
If all of this seems confusing, you are not alone. As STAT pointed out, two things are clear about antibody studies so far: “The first: Whether the study was conducted in California or Denmark, in the Netherlands or Germany, most have shown the virus has not yet infected a big portion of the screened population … The second thing: The world still has no idea how to interpret the import of any of these studies.”
However, as Riley pointed out, despite those drawbacks, antibody testing in San Miguel County could still prove useful to gauge whether the disease is spreading or not. “If they repeat the same study and they find the prevalence hasn’t changed, that’s a good sign that the virus is not spreading,” he said.
A Florida county is doing just that, randomly testing hundreds of people each week for antibodies, which showed that over at least two weeks in April, the infection rate remained at 6%. According to a county press release, “the fact that this number held steady implies that our physical distancing is working and that together, we are ‘flattening the curve.’”
In Colorado, San Miguel County is just coming out of a stay-at-home order. It plans to offer the antibody test a second time at some point in the future.
This story was produced by the Mountain West News Bureau, a collaboration between Wyoming Public Media, Boise State Public Radio in Idaho, KUNR in Nevada, the O'Connor Center for the Rocky Mountain West in Montana, KUNC in Colorado, KUNM in New Mexico, with support from affiliate stations across the region. Funding for the Mountain West News Bureau is provided in part by the Corporation for Public Broadcasting.
Do you have questions about COVID-19? How has this crisis affected you? Our reporters would love to hear from you. You can submit your question or share your story here.