In November 2016, the Montreal-based International Civil Aviation Organization (ICAO) was hit by the most serious cyberattack in its history, and internal documents obtained by CBC suggest key members of the team that should have prevented the attack tried to cover up how badly it was mishandled.
As the United Nations body that sets standards for civil aviation around the world, ICAO is the gateway to everyone in the aviation industry, so an uncontained cyberattack left not just ICAO vulnerable, but made sitting ducks of its partners worldwide.
The documents obtained by CBC suggest the hacker was most likely a member of Emissary Panda, a sophisticated and stealthy espionage group with ties to the Chinese government.
At ICAO, investigators found a network full of holes, with security vulnerabilities that should have been flagged years earlier.
José Fernandez, a cybersecurity expert and professor at Polytechnique Montréal, said what happened at ICAO is akin to leaving your car unlocked and allowing a criminal to use the vehicle to commit a crime.
"If a large organization like ICAO leaves its infrastructure unprotected, or not well protected, it is allowing criminals or, in this case, cyberspies to use that infrastructure to spy on other people."
The documents show that the breach was discovered by an outside agency, and what should have been a race to contain it was mired in delays, obstruction and negligence. The documents suggest that four members of ICAO's information and communications technology (ICT) department tried to hide evidence of their own incompetence, and their absentee supervisor allowed that to happen.
Despite the gravity of the attack, and the confusion of the ICT team's response to it, confidential sources have told CBC that ICAO Secretary General Fang Liu shelved internal recommendations to investigate the four ICT team members and their boss, James Wan, ICAO's deputy director of information management and general administration.
All five still work at ICAO.
Classic 'watering hole' attack
The documents obtained by CBC, which are assessment reports that include emails and an "information security incidents summary," show that a cyberintelligence analyst working for an independent agency known as the Aviation Information Sharing and Analysis Center first flagged the cyberattack on Nov. 22, 2016.
That analyst, Adam Weidmann, contacted ICAO's information security officer, informing that officer that a hacker had control of two of ICAO's servers and was using them to spread malware to foreign government websites.
The type of attacker they were dealing with posed "a significant threat to the aviation industry," Weidmann said.
Since ICAO's role is to set standards for civil aviation rather than keep planes in the air, the hacker was not likely scheming to disrupt flights or airlines, said Fernandez.
But for the purposes of cyberespionage, "ICAO would be a natural choice," Fernandez said. "They would have been a one-stop shop for hacking everybody else in the aerospace industry."
This attack had all the hallmarks of a classic "watering hole" attack, in which hackers find a website that their targets frequent and infect it with malware in order to gain access to those targets.
Within 30 minutes of the hack on ICAO, at least one of the UN agency's 192 member states, Turkey, had been compromised.
It turned out the attackers had set up a chain of watering holes, which included ICAO's online store for aviation publications, as well as the Turkish treasury board's website.
Anyone visiting either site had the potential of becoming infected.
Widespread privacy breach
Alarmed, ICAO's information security officer gave the ICT team until noon on Nov. 23, the day after the discovery of the hack, to get the infected servers offline, and contacted a UN-affiliated IT agency in New York to tell them what had happened.
"Timing is of the essence," said Ali Arasteh, a cybersecurity consultant at FireEye, which investigates attacks of this nature. "You need to line up all of your organizational resources to abruptly remove the attackers from the net."
The documents obtained by CBC suggest that wasn't the case at ICAO. Its ICT team dismissed the expertise of the New York-based UN analysts, handing over data that was not useable and late, and in some cases, not bothering to answer emails for days.
On Dec. 5, ICAO's information security officer, who was co-ordinating the recovery response with investigators, finally sought and obtained the go-ahead to fly in one of the UN analysts for four days. But even when face to face with the ICT team, the documents show it took three days of repeated requests before the analyst was granted access to the data logs and to the infected servers.
At first, the ICAO attack was thought to be limited to "one severe incident" on two of the organization's most sensitive servers. But on Dec. 7, the analyst brought in from New York discovered it was more widespread.
ICAO's webmail server, domain administrator and system administrator accounts were all believed to have been compromised, giving the cyberspy access to past and current passwords of more than 2,000 ICAO users, which would allow the spy to read, send or delete the email of any of those users.
It also meant the hacker could access personnel records of past and current employees, medical records of those who had used ICAO's health clinic, financial transaction records and the personal information of anyone who had visited the ICAO building or registered on an ICAO website.
Encrypted files go home
Upon the discovery of the more extensive breach, the documents show, ICAO's information security officer asked that the infected webmail server be decrypted, so that people who may have had their privacy invaded could be identified and advised that their personal information was at risk.
Wan, the ICT team's boss, rejected that request outright. However, a couple of days later, one of the ICT team did just that, taking an encrypted file home to try to decrypt it.
"He ought to have known that through his actions, he recklessly compromised the security of confidential data," read one of the documents obtained by CBC.
The same day, the New York-based UN IT analysts were struggling to decrypt the file. They were told by the ICAO ICT team that if they didn't succeed in doing so by day's end, they were to delete the file.
However, the New York team did succeed in decrypting it, and what they found further alarmed them.
The file tied the superuser account of one of the ICT team members, the systems infrastructure associate, to the attack.
That could mean that a hacker remotely accessed that superuser account, or it could mean that the superuser himself, the infrastructure associate, was party to the cyberattack: the analysts had no way of knowing which it was.
Despite the suspicions raised about that superuser, he was given the job of validating the New York analysts' forensic work. That ICT superuser disputed the analysts' findings, concluding their detection of malware was a "false positive" — in other words, that no malware was to be found.
Based on that report, Wan reported to ICAO's secretary general that the entire cybersecurity incident was a minor one, overblown by the New York analysts.
The documents indicate that ICAO's information security officer asked for an independent review of the false positive findings. That request was declined.
ICT team escorted from ICAO
The documents obtained by CBC show that by this time, Wan's superior, Vincent Smith, appeared to lose faith in the ICT team.
On Dec. 20, the documents show, Smith lodged a formal complaint with ICAO, alleging that the four ICT team members had "acted with intent to disguise the source, nature and impact of a breach of the ICAO network."
The following day, all four were escorted from ICAO headquarters and placed on paid administrative leave pending further investigation.
ICAO's malware problems still weren't resolved.
On Jan. 4, 2017, more than six weeks after the discovery of the breach, a representative of ICAO's Nordic delegation notified ICAO that someone she didn't know had used her account to send an email, making it look as though it came from her.
The documents show James Wan, deputy director of information management and general administration, told her that was a "common threat in today's digital world," advising her to permanently delete the suspicious email, without investigating further.
Wan, meanwhile, was himself under scrutiny.
By the new year, an independent cybersecurity firm, SecureWorks, was brought in to ICAO to carry out its own forensic analysis of the attack.
Investigator David Peck complained on Jan. 17 that Wan had "engaged in a pattern of obstruction, deception, insubordination, and incompetence in his handling of the ongoing cybersecurity response."
Peck concluded the security issues dated back at least three years.
Crucially, the malware used in the cyberattack had been identified by ICAO's anti-virus software 12 months earlier, but the network was never disinfected — even though one of the ICT team's most basic responsibilities is to identify viruses and get rid of them.
Peck blamed Wan, who, according to the documents, was advising other managers that "ICAO systems were clean and safe," even while SecureWorks was reporting that there was no guarantee the hacker couldn't compromise "the new systems using the same vulnerabilities."
David Peck did not respond to requests for comment from CBC News.
In the critical weeks right after the attack, Wan took leave or simply stayed home on three different occasions. He made ICAO's information security officer accountable for the forensic analysis in his absence, but didn't grant that officer the authority to act.
Sure enough, during one of Wan's absences, ICAO received a malware alert about a password-stealing virus on a server.
A flurry of urgent emails requesting Wan's approval to isolate the server went unanswered for three and a half days.
On Jan. 12, when he was asked in an email to approve a long-term action plan to improve ICAO cybersecurity that had been developed by the New York IT analysts and SecureWorks, Wan never replied.
Several followup emails over the span of a week, in which the SecureWorks investigator told Wan that ICAO "still remains at high risk for another issue" and that "what could happen could be far worse than before," were never answered.
Wan went on emergency leave, and the plan was never approved.
CBC News contacted the four ICT team members, James Wan and Fang Liu for comment. None responded to repeated requests for an interview.
ICAO's chief of communications, Anthony Philbin, did provide a statement.
"Decisions made by ICAO regarding the 2016 incident you've referenced were based on forensic evidence provided by two independent expert bodies," Philbin said.
"I'm sure you'll understand that it wouldn't be prudent for me to discuss more specific details with media on matters relating to ICAO security measures, cyber or otherwise."
Philbin offered reassurance that "ICAO maintains no type of financial or other private information which could possibly pose risks to individual Canadians."
Later Wednesday, after this article first appeared, Philbin sent a second statement, saying the gravity of the malware found on ICAO's servers in 2016 "has been greatly exaggerated in the CBC account."
"We're not aware of any serious cybersecurity ramifications for external partners which resulted from this incident, and as a standards-setting body, with no operational role or mandate in aviation, the inference that our data security could pose risks to the combined aviation and aerospace sectors, or the general public, is grossly inaccurate," Philbin said.
He also said that since the 2016 event, "ICAO has made robust improvements to its cybersecurity posture and approaches to mitigate further incidents."
Federal Transportation Minister Marc Garneau said Wednesday that he would be speaking to ICAO's secretary general to find out more about the incident.
"Obviously, we, as a member of ICAO, will want to make sure that any information that we share with them is protected," Garneau said.
No heads rolled
The full extent of the cyberattack and what the attacker was truly after is not revealed in the documents obtained by CBC.
A full list of organizations that may have been compromised was also never uncovered, because a file in the server containing that information mysteriously disappeared.
The four ICT members marched out of ICAO on Dec. 21 were back at their jobs six weeks later, on Feb. 6, 2017.
A confidential source told CBC there was pressure from higher up the United Nations chain to return them to work, where they are to this day.
Polytechnique Montréal's Fernandez said ICAO should have done better. He said it has a responsibility to protect its own data and that of its direct partners, as well as the general public's confidential information.
If you have a tip about this story, send us an email or call our confidential tipline: 514-597-5155.