The complications have complications…


It’s really hard to determine which policies control COVID efficiently

Places don’t just implement one policy, and erratic testing complicates everything.

Nobody wants to go back under heavy social restrictions. But the surging case numbers are causing many countries to put in place targeted lockdowns and other limits to try to get the pandemic back under control—a move that has sparked a backlash in a lot of places. So, it seems like it’s worth asking what the optimal combination of restrictions might be. How do you get the most pandemic control for the least restrictive social environment?


That’s precisely what an international team of researchers attempted to find out, as described in a paper published today. And, while the researchers come up with some potential answers, their paper ends up with an additional message: this is a really hard question to answer. So, to an extent, many countries are going to have to act with imperfect information and hope for the best.

How do you answer that?


In an ideal world, we’d have some sense of the impact of each possible social restriction: closing restaurants, starting contact tracing, shutting schools, and so on. Given that information, we could look at the rate of infections and the virus’s trajectory, then figure out the smallest possible set of restriction that could cause the infection rate to drop. But the real world is very far from this idealized situation at the moment, which is what motivated the researchers to try to provide a bit more certainty regarding the effectiveness of different restrictions.


But the real world also makes that a difficult question to answer. After all, nobody adopted just a single restriction; generally, a suite of limits were put in place in most countries, with additional restrictions added gradually. In addition, different cultures might find it more difficult to comply with limits on bars and restaurants than they would with having a mask-wearing mandate—different limits will see different levels of compliance.


To try to get around this, the researchers gathered large collections of public health policies put in place during the initial wave of infections in March and April. They then analyzed all of them using four different methods and looked for the ones that were consistently associated with better outcomes in terms of the total number of infections.


This turned out to be a lot of information. The researchers’ initial analysis was a list of over 6,000 different policy changes implemented in 79 different countries, states, or provinces. But they then went on to use two additional sources of data on restrictions to validate the initial analysis, adding a total of 42,000 additional interventions.


The correlations between these interventions and infection rates were then tested using four different mathematical approaches. (These were a case-control matching algorithm, two different types of regression analysis, and transformer modeling.) Approaches that produced significant results in three or four of these tests were considered likely to be effective.


Not so fast


Unfortunately, one of the limitations of this analysis became obvious pretty early. During the first wave of the pandemic, testing capacity was generally limited, most notably in the United States. As such, the established number of cases often correlated more with testing capacity than the presence of the virus. Not surprisingly, the algorithms typically picked out policies like “enhancing testing capacity” and “increased surveillance” as having a negative effect on controlling the pandemic. The reality is that, in most countries, these policies probably led to increased identification of cases.


That sort of limitation aside, there was a high degree of correlation between policies flagged as effective by different algorithms, which should provide some reason for confidence in other results. Some of these are exactly the things people don’t like: national lockdowns and stay-at-home orders. Banning small gatherings came out as the top intervention. Another top intervention, increasing the supply of personal protective equipment, isn’t really a surprise. Same with the fact that educating the public about the virus and pandemic seemed to have a positive effect in three of the four analyses.


Other things were somewhat surprising. Border restrictions turned out to be quite effective, although that could be skewed by a number of island nations, like Taiwan and New Zealand, that saw great success in controlling the virus’s entry. Another consensus choice was school closure. Some previous studies had suggested spread in schools was minimal, while others had supported the measure, so the evidence here has been ambiguous.


When the research team turned to the larger database of policy interventions, however, there wasn’t complete agreement with the initial analysis. Of the seven that had seen all four analyses flag them as effective, only three saw some level of agreement when the same four analyses were repeated on the larger database. But again, social restrictions like ending small or large gatherings and closing offices came out looking good, as did border closures. So, while a specific policy might not have replicated, closely related policies often did, which might suggest that we’d get clearer results by lumping similar restrictions into a single group.


A further indication of this may be the fact that national lockdowns were only rated as having inconsistent effectiveness. The researchers suspect this was caused by lockdowns only coming on the heels of a variety of earlier restrictions, and so lockdowns didn’t add as much as they might if implemented in isolation.


One policy that specifically scored poorly across all data sources was widespread disinfection of the environment. It’s not clear why that was the case. Another policy that didn’t seem to do much was social-distancing measures specifically targeted toward public transit.


Overall messages


One other thing that the research team checked is what they call the “entropy” of the results. That is, if a policy is universally effective wherever it’s tried, it will have a low entropy. From this, we can infer that cultural differences were unlikely to have had a huge impact on whether the policy was successful. It turns out that some of the more effective measures, like social distancing and travel restrictions, had high entropy, meaning that they worked in some countries but not as well in others. One only has to look at the different levels of compliance with restrictions within the United States to understand how this could happen.


Still, the local culture seemed to influence how effective the policy was, not whether it was effective at all. Various forms of social distancing and restrictions on gatherings, businesses, and restaurants consistently scored well in most analyses.


Also telling was the fact that better communications about the motivations for policies often had just as strong an effect as implementing the policy itself. If you explain to people why you might need to close restaurants, then chances are good that a lot of people will stop going to restaurants whether the rules are in place or not.


Still, the information is too complicated for the ideal goal we’d like to see: identifying the minimal level of restrictions that can keep the pandemic in check. As hospitals fill to capacity, we’re probably going to have to institute severe social restrictions just to be certain we don’t allow the health care system to get overloaded. One thing the study does provide, however, is another piece of evidence that those restrictions will do the job.


Nature Human Behavior, 2020. DOI: 10.1038/s41562-020-01009-0 (About DOIs).