Anxiety remains high after multiple code greys at Queensway Carleton Hospital

A doctor at the Queensway Carleton Hospital says frontline staff are still bearing "the brunt of responsibility" for a major IT failure that remains unresolved more than four months after it knocked out phones, computers and medical equipment for nearly 20 hours within the facility.

Since the "catastrophic" code grey incident last September, which hammered the west Ottawa hospital's emergency department, the same system has experienced at least nine more outages of varying severity, including five that were triggered by maintenance staff as "planned network downtimes" — far more than the usual once per month, according to the doctor.

"Our trust in the hospital has rapidly abated," said the physician. "It's evaporated."

CBC News has agreed not to identify the hospital employee or their gender due to concerns over professional retaliation and reputational damage for speaking out.

"The emergency department ... is extremely chaotic," said the doctor. "If you happened to be scheduled for one of these shifts then you sort of go in with a heavy heart, a worry of what's going to happen."

Documents obtained by CBC News show the latest outages — both planned and unplanned –– ranged from 10 minutes in duration to 11 hours.

"We did not communicate publicly as we were able to provide service to the vast majority of our patients and spoke directly to the small number of patients that it impacted," a hospital spokesperson wrote in a statement in response to CBC's questions about why some of those incidents were not announced.

Attempt to revert to primary system failed

To date, the hospital has not identified exactly what caused the original code grey (short for "critical infrastructure failure") on Sept. 9, which caught medical staff flat-footed, limiting their ability to access patient records, perform diagnostic scans and communicate with each other.

The code grey was cleared after the hospital switched over to a backup, secondary system.

Staff conducted multiple planned downtimes in the weeks following in what appears to be an effort to replicate the problem and identify the cause, according to internal emails.

However, an attempt to "switch back" to the hospital's primary network failed on Sept. 28.

"Unfortunately the updated network core would not establish a connection with the existing infrastructure during the downtime window, even with support of our onsite external experts," wrote Tim Pemberton, the hospital's vice-president of information management and clinical support, in an internal email.

Pemberton then offered a medical analogy to explain the issue.

"Sometimes when you are in surgery and have opened up the patient, you discover something that makes it unwise to continue the surgery that day," he wrote. "Despite extensive preparations, it was unwise to keep going given the potential impact to clinical users."

After nearly five hours of downtime, hospital management made the decision to "revert to the previous state."

Back-to-back code greys

The network was later fully restored in mid-October, but downtimes continued to happen, according to documents.

"Downtimes, whether planned or not, are extremely disruptive," said the doctor who spoke confidentially to CBC.

"They add to the anxiety of going into work a shift."

As an example, the doctor pointed to two recent back-to-back code greys, which happened on Jan. 15 and 16 of this year.

The first code grey, which began at night and lasted 11 hours into the morning, was called when one of the two CT scanners available to the emergency department stopped working.

The second incident came a few hours after the first code grey, when the system used by patients to call nurses was also "not functioning," according to an internal memo.

The second code grey lasted six hours.

"We have a deep-seated and perpetual fear that we're going to misdiagnose, that we're going to send someone home and that person will die because we missed something," said the doctor.

"And so when there's a code grey, it overlays a level of chaos into an already chaotic system, and that makes that fear a whole lot worse."

The doctor said when an outage knocks out a major diagnostic tool like CT scanning, it can lead to delayed and inaccurate diagnoses — and potentially harmful treatments.

"If you do not have the diagnostic tools that are standard in 2023, then you are put in a position where you're not practicing medicine up to the standard of that year," they said.

"The hospital doesn't experience the moral distress that frontline staff do."

According to the hospital's statement to CBC, three patient appointments had to be rebooked as a result.

"We transitioned our cases over to the second machine temporarily while the other was being serviced. The second machine was temporarily not able to deliver contrast solutions. Fortunately we were able to provide service to most of our patients without a problem."

The hospital characterized the second code grey on Jan. 16 as the result of a "five-minute network outage."

"It was resolved immediately," read the statement to CBC. "However there were several users who reported difficulties with the handheld devices that connect to the nurse call bell system."

The spokesperson said the problem was resolved once staff reset the device.

"The overall system itself was operational during the entire time," the statement read.

Doctors frustrated over lack of response

As previously reported by CBC, several doctors expressed anger and frustration in the hours following the Sept. 9 code grey, when the Queensway Carleton Hospital kept its emergency department open through the night during a severe outage — against the advice of some.

"Once again I shake my head at the fact that our collective voice in our hospital isn't valued," one ER physician wrote in an email at the time.

In November, the hospital told CBC in an email statement: "When clinicians sound alarm bells, we understand. … We want to fix the system."

But the unnamed doctor said the system is still not fixed.

"The hospital has a habit of not listening to us," said the doctor, "Management simply is not listening and that's why things are not getting better and that's why things continue to fail."