Sandworm Read online

  * * *

  ■

  Several days after Henrik Jensen had watched all the screens around him go dark in Maersk’s headquarters, he was at home in his Copenhagen apartment, enjoying a brunch of poached eggs, toast, and marmalade. Since he’d walked out of the office the Tuesday before, Jensen hadn’t heard a word from any of his superiors. Then his phone rang.

  When he answered, he found himself on a conference call with three Maersk staffers. He was needed, they said, at Maersk’s office in Maidenhead, England, a town west of London where the conglomerate’s IT overlords, Maersk Group Infrastructure Services, were based. They told him to drop everything and go there. Immediately.

  Two hours later, Jensen was on a plane to London, then in a car to an eight-story glass-and-brick building in central Maidenhead. When he arrived, he found that the fourth and fifth floors of the building had been converted into a 24/7 emergency operations center. Its singular purpose: to rebuild Maersk’s global network in the wake of its NotPetya meltdown.

  Some Maersk staffers, Jensen learned, had been in the recovery center since Tuesday, when NotPetya first struck. Some had been sleeping in the office, under their desks or in corners of conference rooms. Others seemed to be arriving every minute from other parts of the world, luggage in hand. Maersk had booked practically every hotel room within tens of miles, every bed-and-breakfast, every spare room above a pub. Staffers were subsisting on snacks that someone had piled up in the office kitchen after a trip to a nearby Sainsbury’s grocery store.

  The Maidenhead recovery center was being managed by the consultancy Deloitte. Maersk had essentially handed the U.K. firm a blank check to make its NotPetya problem go away, and at any given time as many as two hundred Deloitte staffers were stationed in the Maidenhead office, alongside up to four hundred Maersk personnel. All computer equipment used by Maersk from before NotPetya’s outbreak had been confiscated, for fear that it might infect new systems, and signs were posted threatening disciplinary action against anyone who used it. Instead, staffers had gone into every available electronics store in Maidenhead and bought up piles of new laptops and prepaid Wi-Fi hot spots. Jensen, like hundreds of other Maersk IT staffers, was given one of those fresh laptops and told to do his job. “It was very much just ‘Find your corner, get to work, do whatever needs to be done,’ ” he said.

  Early in the operation, the IT staffers rebuilding Maersk’s network came to a sickening realization. They had located backups of almost all of Maersk’s individual servers, dating from between three and seven days prior to NotPetya’s onset. But no one could find a backup for one crucial layer of the company’s network: its domain controllers, the servers that function as a detailed map of Maersk’s systems and set the basic rules that determine which users are allowed access to which machines.

  Maersk’s 150 or so domain controllers were programmed to sync their data with one another so that, in theory, any of them could function as a backup for all the others. But that decentralized backup strategy hadn’t accounted for one scenario: where every domain controller is wiped simultaneously. “If we can’t recover our domain controllers,” a Maersk IT staffer remembered thinking, “we can’t recover anything.”

  After a frantic search that entailed calling hundreds of IT admins in data centers around the world, Maersk’s desperate administrators finally found one lone surviving domain controller in a remote office—in Ghana. At some point before NotPetya struck, a blackout had knocked the Ghanaian machine off-line, and the computer remained disconnected from the network. It thus contained the singular known copy of the company’s domain controller data left untouched by the malware—all thanks to a power outage. “There were a lot of joyous whoops in the office when we found it,” a Maersk administrator remembers.

  When the tense engineers in Maidenhead set up a connection to the Ghana office, however, they found its bandwidth was so thin that it would take days to transmit the several-hundred-gigabyte domain controller backup to the U.K. Their next idea: put a Ghanaian staffer on the next plane to London. But none of the West African office’s employees had a British visa.

  So the Maidenhead operation arranged for a kind of relay race: One staffer from the Ghana office flew to Nigeria to meet another Maersk employee in the airport to hand off the very precious hard drive. That staffer then boarded the six-and-a-half-hour flight to Heathrow, carrying the keystone of Maersk’s entire recovery process.

  With that rescue operation completed, the Maidenhead office could begin bringing Maersk’s core services back online. After the first days, Maersk’s port operations had regained the ability to read the ships’ inventory files, so operators were no longer blind to the contents of the hulking 18,000-container vessels arriving in their harbors. But several days would pass after the initial outage before Maersk started taking orders through Maerskline.com for new shipments, and it would be more than a week before terminals around the world started functioning with any degree of normalcy.

  In the meantime, Maersk staffers worked with whatever tools were still available to them. They taped paper documents to shipping containers at APM ports and took orders via personal Gmail accounts, WhatsApp, and Excel spreadsheets. “I can tell you it’s a fairly bizarre experience to find yourself booking five hundred shipping containers via WhatsApp, but that’s what we did,” one Maersk customer said.

  About two weeks after the attack, Maersk’s network had finally reached a point where the company could begin reissuing personal computers to the majority of staff. Back at the Copenhagen headquarters, a cafeteria in the basement of the building was turned into a reinstallation assembly line. Computers were lined up twenty at a time on dining tables as help desk staff walked down the rows, inserting USB drives they’d copied by the dozens, clicking through prompts for hours.

  A few days after his return from Maidenhead, Henrik Jensen found his laptop in an alphabetized pile of hundreds, its hard drive wiped, a clean copy of Windows installed. Everything that he and every other Maersk employee had stored locally on their machines, from notes to contacts to family photos, was gone.

  * Fernández is not his real name. Like Henrik Jensen, this source asked that I refer to him using a pseudonym.

  27

  THE COST

  Five months after Maersk had recovered from its NotPetya attack, the company’s chair, Jim Hagemann Snabe, sat onstage at the World Economic Forum meeting in Davos, Switzerland, and lauded the “heroic effort” that went into Maersk’s IT rescue operation. From June 27, when he was first awakened by a 4:00 a.m. phone call in California, ahead of a planned appearance at a Stanford conference, he said, it took just ten days for the company to rebuild its entire network of four thousand servers and forty-five thousand PCs. (Full recovery had taken far longer: Some staffers at the Maidenhead operation continued to work day and night for close to two months to rebuild Maersk’s software setup.) “We overcame the problem with human resilience,” Snabe told the crowd.

  Since then, Snabe went on, Maersk has worked not only to improve its cybersecurity but also to make it a “competitive advantage.” Indeed, in the wake of NotPetya, IT staffers told me that practically every security feature they’ve asked for has been almost immediately approved. Multifactor authentication has been rolled out across the company, along with a long-delayed upgrade to Windows 10.

  Snabe, however, didn’t say much about the company’s security posture pre-NotPetya. Maersk security staffers told me that some of the corporation’s servers were, until the attack, still running Windows 2000—an operating system so old Microsoft no longer supported it. In 2016, one group of IT executives had pushed for a preemptive security redesign of Maersk’s entire global network. They called attention to Maersk’s less-than-perfect software patching, outdated operating systems, and above all insufficient network segmentation. That last vulnerability in particular, they warned, could allow malware with access to one part o
f the network to spread wildly beyond its initial foothold, exactly as NotPetya would the next year.

  The security revamp was green-lighted and budgeted. But its success was never made a so-called key performance indicator for Maersk’s most senior IT overseers, so implementing it wouldn’t contribute to their bonuses. They never carried the security makeover forward.

  Few firms have paid more dearly for dragging their feet on security. In his Davos talk, Snabe claimed that the company suffered only a 20 percent reduction in total shipping volume during its NotPetya outage, thanks to its quick efforts and manual workarounds. But aside from the company’s lost business and downtime, as well as the cost of rebuilding an entire network, Maersk reimbursed many of its customers for the expense of rerouting or storing their marooned cargo. One Maersk customer described receiving a seven-figure check from the company to cover the cost of sending his cargo via last-minute chartered jet. “They paid me a cool million with no more than a two-minute discussion,” he said.

  All told, Snabe estimated in his Davos comments, NotPetya cost Maersk between $250 million and $300 million. Most of the staffers I spoke with privately suspected the company’s accountants had lowballed the figure.

  Regardless, those numbers only start to describe the magnitude of NotPetya’s damage. Logistics companies whose livelihoods depend on Maersk-owned terminals weren’t all treated as well during the outage as Maersk’s customers, for instance. Jeffrey Bader, president of a Port Newark–based trucking group, the Association of Bi-State Motor Carriers, estimates that the unreimbursed cost for trucking companies and truckers alone was in the tens of millions. “It was a nightmare,” Bader said. “We lost a lot of money, and we’re angry.”

  The wider cost of Maersk’s disruption to the global supply chain as a whole—which depends on just-in-time delivery of products and manufacturing components—is far harder to measure. And, of course, Maersk was only one victim. Only when you start to multiply Maersk’s story—imagining the same paralysis, the same serial crises, the same grueling recovery—playing out across dozens of other NotPetya victims and countless other industries does the true scale of Russia’s cyberwar crime begin to come into focus.

  Merck, the $200 billion, New Jersey–based pharmaceutical giant, was hit early on the morning of NotPetya’s judgment day. It lost fifteen thousand Windows computers in ninety seconds, according to one of the company’s IT staffers, before administrators managed to shut down its entire network. Merck maintained a backup data center for exactly this sort of crisis, but the staffer told me it was a “hot site,” connected to Merck’s network to enable faster recovery, rather than a “cold site,” which would have been disconnected for greater security. That meant that it, too, was wiped out in NotPetya’s tsunami. “We didn’t have a great plan for what we’d do if both sites get infected at the same time, and that’s exactly what happened,” the IT staffer told me. “Something that would just take down all of our Windows systems—we hadn’t imagined something of that scale.”

  Just as NotPetya shut down Maersk’s port terminals worldwide, it immediately rippled out to Merck’s physical processes, too, paralyzing its drug research and shutting down a significant swath of its pharmaceutical manufacturing. “Without computers these days you can’t do anything,” one Merck scientist lamented to The Washington Post. In its financial report to shareholders a few months later, the company would reveal that it had been forced to borrow a quarter-billion dollars’ worth of its own vaccine for cancer-causing human papillomavirus from the federal Centers for Disease Control and Prevention. Two congressmen would write in a letter to the Department of Health and Human Services that the effects of NotPetya on Merck “raise questions about how the nation is prepared to address a significant disruption to critical medical supplies.”

  Eight months after the attack, Merck told shareholders it had totaled its losses due to the malware to a staggering $870 million. FedEx, whose European subsidiary, TNT Express, was crippled in the attack and required months to recover some data, took a $400 million blow. The French construction giant Saint-Gobain lost around the same amount. Reckitt Benckiser, the British manufacturer, lost $129 million, and Mondelēz, the food producer, took a $188 million hit. Untold numbers of victims without public shareholders counted their losses in secret.

  In total, the result was more than $10 billion in damages, according to a White House assessment confirmed to me by the former homeland security adviser Tom Bossert, who at the time of the attack was President Trump’s most senior cybersecurity-focused official. Bossert emphasized, in fact, that this eleven-figure number represents a floor for their estimate, not a ceiling; it might well have been much higher. “While there was no loss of life, it was the equivalent of using a nuclear bomb to achieve a small tactical victory,” Bossert said. “That’s a degree of recklessness we can’t tolerate on the world stage.”

  To get a sense of what that $10 billion in damages means on the spectrum of cyberattacks, consider that when a nightmarish but more typical ransomware attack paralyzed the city government of Atlanta in March 2018, it cost an estimated $17 million. In other words, less than a fifth of a percent of NotPetya’s price. Even WannaCry, at the time an unprecedented internet catastrophe, was believed to have cost around $4 billion by most estimates. Nothing since has come close.

  “This was a very significant wake-up call,” as Maersk’s chairman, Snabe, had said at his Davos panel. Then he’d added, with a Scandinavian touch of understatement, “You could say, a very expensive one.”

  * * *

  ■

  But not all of NotPetya’s costs could be measured in dollars. Another of its collateral victims was a little-known company called Nuance, focused on speech-recognition software. Nuance’s code was used in the first version of the iPhone’s Siri, for instance, and the voice command system in Ford cars. By 2017, however, much of Nuance’s business came from a vast array of institutions that relied on its technology in matters of life and death: hospitals.

  As it had for so many other massive multinationals, NotPetya sprang out from Nuance’s Ukraine office to instantly paralyze the company’s digital systems across its seventy locations, from India to Korea to its headquarters in Burlington, Massachusetts. And just as at Maersk, desperate IT administrators would struggle for weeks to recover thousands of PCs and servers encrypted by the worm. “It was trench warfare,” one former Nuance staffer who participated in the rescue effort told me. “The office was in a state of triage. People were working 24/7. Every empty conference room had beds in it.”

  Ultimately, Nuance would report a loss of $92 million from NotPetya, just a fraction of the damage to firms like Merck and FedEx. But Nuance’s transcription service for electronic medical records, aided by the company’s team of human transcriptionists, was used by hundreds of hospitals and thousands of clinics around the world. And that’s where the real toll of its outage would be felt.

  On the morning of the attack, Jacki Monson was sitting in a conference room in an office park in Roseville, California, a suburb of Sacramento. Monson served as the chief privacy and information security officer for Sutter Health, a network of more than twenty-four hospitals and clinics from Utah to Hawaii. Early that morning, she’d received a jarring message from Merck’s chief information security officer about the company’s crippling NotPetya infection, via a mailing list for the Health Care Industry Cybersecurity Task Force, a group created by the Obama administration to examine cybersecurity risks to medical organizations. By 9:00 a.m. Pacific time, Monson was on a tense conference call with health-care security executives around the world, all hoping to somehow avoid NotPetya’s ballooning effects.

  Half an hour into that meeting, Monson received another call from Sutter’s head of health-care information management systems. Sutter hospitals still didn’t seem to be infected with NotPetya, Monson was relieved to hear. Instead, they were facing a less obvious pro
blem: For the last hour, Nuance’s systems had been down and, with them, the ability of every doctor at every Sutter hospital to dictate changes into patients’ medical records.

  Monson quickly began to see the seriousness of that bottleneck. All across Sutter’s hospitals, doctors were reading changes into Nuance’s transcription service—in some cases, hours of audio at a time—and now none of those changes would show up in patients’ files. People scheduled to go into surgery that morning might not have the final approvals they needed to be cleared for their operations. Others, like transplant recipients whose doctors constantly monitor and adjust their drugs, might miss crucial changes in treatment.

  Sutter’s emergency response team soon began racing to sort through thousands of patients’ records at dozens of hospitals, trying to identify which ones might face serious consequences from their Nuance choke point. Meanwhile, Monson and her IT colleagues were desperately searching for an alternative system that would allow their hospitals’ doctors to keep making changes to health records at their normal pace. Though Nuance’s human-aided dictation services were off-line, its fully automated software, installed on Sutter’s own systems, was still working. But that software was error-prone and struggled with accents. The hospitals’ own transcriptionists were overwhelmed. It would take Sutter two weeks to switch to one of Nuance’s competitors. And within just twenty-four hours, Sutter was facing a backlog of 1.4 million changes to patients’ records, every one of which might have a real impact on a human being’s health.

  On the other side of the country, another hospital network was grappling with NotPetya more directly. Heritage Valley Health System, a small two-hospital network in Pennsylvania, had itself been infected by the worm. According to one of the IT staffers at those hospitals who spoke to me, its administrators had been logged in to a Nuance server at the time of the company’s infection, allowing the worm to spread directly into the hospitals’ own systems. Before 8:00 a.m. eastern time, it had corrupted two thousand computers and hundreds of servers.