'Flawed' findings on lockdowns fed province's decision to end mandates

The New Brunswick Department of Health's March 2022 decision to end mask mandates and other COVID-19 measures was made after officials circulated an academic paper widely criticized as "fundamentally flawed."

The paper, by three researchers tied to libertarian think-tanks, concluded that lockdowns did not significantly reduce the death rate during the first wave of COVID-19 in early 2020.

But the study, a so-called "meta-analysis" that synthesizes dozens of individual studies, is full of problems, according to other researchers.

"The study has some methodological issues which make the conclusion just not reliable. So there's a high risk of potential bias," says Adrian Lison, a doctoral student and infectious disease researcher at ETH Zurich.

"And as the study is not yet peer-reviewed, we think it does not really add reliable new insights to the body of knowledge, so we recommend to not use the quite strong conclusions in this review in policy."

The paper was circulating among New Brunswick Department of Health officials in early February. It has not been published in an academic journal.

Lockdowns "have had little to no public health effects" but enormous economic costs, it says. They are "ill-founded and should be rejected as a pandemic policy instrument."

Health Department spokesperson Adam Bowie said officials review data "from many different studies" as well as from epidemiologists in the province when making decisions.

"A recommendation from Public Health to introduce or remove any health mandate would never be based on any one study or report," he said.

The meta-analysis "would have only been one source of information under consideration," he said, though he didn't identify any other studies they used.

Kathleen Gadd of the pro-mandate citizens' group Protect our Province, which obtained the study through a right-to-information request, says it was the only so-called academic research the province handed over.

"The fact that they didn't supply any other papers says a lot," she said.

"If other studies and peer-reviewed papers and evidence were used to inform the lifting of the mandatory order, this would have been the time to supply the public with that information."

The Department of Health did not respond to a request for an interview on what role the paper played in the mandate decision.

Use of broad term 'lockdown' is 'dangerous,' researcher says

Among the flaws identified by critics is the paper's use of the general term "lockdown" for a range of policies from stay-at-home orders to mask mandates.

That makes measuring the impact of individual restrictions difficult, says Lison, who co-authored a rebuttal posted to the Social Science Research Network website.

He says it's "dangerous in a systematic review to use such a broad term, because if you make conclusions in the end, there's a high risk that people will misinterpret your findings."

The paper also measures death rates without looking at other impacts, such as reduced transmission and a reduced impact of hospitalizations on the heath care system.

Lison says transmission is a better measure of lockdown impacts because it's most directly affected by restrictions, whereas hospitalization and death happens after a delay.

Health official called paper's conclusions 'interesting' 

The province lifted its mandatory COVID-19 order on March 14, ending mask mandates, proof-of-vaccination requirements and limits on gatherings.

The 60-page paper, "A Literature Review and Meta-Analysis of the Effects of Lockdowns on Covid-19 Mortality," was the only research study included in the 200 pages turned over to the PoP group when it asked for documents relating to the decision.

In a Feb. 4, 2022 email, Nina van der Pluijm, the director of well-being, legislation and standards at the department, sent the study to several colleagues, remarking, "Interesting info re: lock-down effectiveness."

She noted its conclusion that the average lockdown had reduced mortality by only 0.2 percent.

Gadd, a former health sciences librarian, says "quite a number" of people with expertise at universities and health authorities could have supplied officials with better studies.

"There's no sign that information professionals, such as health science librarians, were engaged in these quite significant decisions that were made regarding the health of New Brunswickers," she said.

"There's a large amount of peer-reviewed, high-quality evidence that has been completely ignored by those in power regarding how to manage the pandemic."

Paper criticized as 'unbalanced' for excluding other studies

The meta-analysis, published in January, synthesizes the findings of 34 separate studies. The authors began with hundreds of studies but screened out most of them.

"Unfortunately this review has only focused on a very small subset of analysis … and has excluded a large part of the overall body evidence, the overall epidemiological research, so it is also rather unbalanced," Lison said.

Dr. Seth Flaxman, a computer science professor at the University of Oxford, called the meta-analysis "fundamentally flawed" for excluding studies rooted in epidemiology.

Researchers at Imperial College in London said in June 2020 that lockdowns in the first wave of the pandemic probably averted 3.1 million deaths in 11 European countries.

Another study estimated that lockdowns in six countries, including the U.S. and China, prevented or delayed 530 million cases of COVID-19.

"It is just problematic to exclude a majority of the research that is available, and it risks your review becoming very unbalanced and not representative," Lison said.

"There are still of course some uncertainty and a lot of open questions, but there's already substantial evidence on the effectiveness of interventions overall."

Authors tied to libertarian think-tanks

The authors are not epidemiologists or public health experts, and PoP has highlighted their connections to libertarian think-tanks opposed to most government interventions in society.

"The authors don't represent the fields of study that are applicable to epidemiology, public health, disease transmission, those kinds of fields that we should be looking to for information around our pandemic response," Gadd said.

Co-author Jonas Herby, an economist at the Centre for Political Studies in Copenhagen, Denmark, is a contributor to the American Institute for Economic Research, which bills itself as promoting "pure freedom and private governance," with government "sharply confined."

Another co-author, Steve Hanke, is a fellow at the right-wing Cato Institute who has referred to lockdowns and mandates as "cracking the fascist whip."

In an interview, Herby acknowledged the conservative leanings of the authors and said that shaped the question they wanted to pose in the paper. But their research method was sound, he added.

He said the meta-analysis focused on deaths because there were not many studies available on hospitalizations.

On criticism that the authors used the broad term "lockdown" for a range of different restrictions, Herby said that's why the reduction in mortality of 0.2 per cent is based on an "average" lockdown.

"I think it's bad science to say your science is bad just because you have a different political view than I have."

Revised version shows larger reduction in deaths

The paper's three authors published a revised version in May.

It recalculates the reduction in death caused by lockdowns as 3.2 percent – still lower than other studies, but 16 times higher than the 0.2 percent estimate in the first version.

Mask mandates had the largest effect, reducing COVID-19 mortality by 18.7 percent, it says.

The authors say what matters  is not particular figures but the fact the number of lives saved by lockdowns "are much smaller and far removed" from what epidemiologists, politicians and the media promised.

Most of the "curve flattening" of COVID deaths early in the pandemic was due to voluntary distancing, not lockdown requirements, Herby says.

People changed their behaviour of their own free will in response to a threat, he says.

"The question is can lockdowns enhance this behaviour change? I think they can, but the effect is very limited."

The new version includes a statement that it "does not imply that lockdowns do not work.

"It simply indicates that the most lenient lockdowns had virtually the same effect on mortality as stricter lockdowns. Since no country did nothing, we cannot reject the thesis that some NPI would be required, e.g., to spur voluntary behavioral changes."

Lison said he found the authors' defence of the meta-analysis "largely not convincing" and said the revised version did not change his view that the paper should not be used by governments as policy advice.

"Most problems of the first version remain," he said.