Endpoint Detection & Response (EDR)
,
Incident & Breach Response
,
Next-Generation Technologies & Secure Development
Microsoft’s Tool Requires Physical Access, a ‘Time-Consuming and Laborious Task’
Microsoft’s statement discounting Friday’s massive CrowdStrike-linked outage by saying it affected “less than one percent” of all Windows machines is a transparent attempt at corporate spin for an incident that still reverberates through the global economy. Recovery is harder than a blog post.
See Also: 5 Requirements for Modern DLP
The 8.5 million systems disrupted in a software update to CrowdStrike’s Falcon platform that went wrong didn’t affect a random sample of Windows machines. The incident affected machines used by larger, more well-funded organizations for sensitive functions that require the sort of protection that CrowdStrike promises – as demonstrated by disruptions in critical sectors such as healthcare, banking and finance, transportation and public safety.
“Those 8.5 million devices were in large organizations whose threat and risk profile meant they could afford to invest in CrowdStrike’s solution,” said Brian Honan, head of Dublin-based cybersecurity consultancy BH Consulting.
Three days after the faulty update went out, cleanup remains “ongoing,” British cybersecurity expert Kevin Beaumont said in a Monday post to social platform Mastodon.
Affected systems are stuck in an endless cycle of crashing out to a Windows blue screen of death, rebooting and loading the faulty CrowdStrike file and then crashing all over again. Because of CrowdStrike’s direct access to the operating system kernel, Microsoft didn’t have built-in defense. Remediating affected systems requires eliminating the file. Microsoft has released< a tool that can help, or those affected can do wiping and restoring (see: CrowdStrike, Microsoft Outage Uncovers Big Resiliency Issues).
“I know plenty of orgs less than a third through remediation,” Beaumont said.
Among airlines, Delta continues to be particularly hard-hit, leading to disruptions at the corporate level and for customers. By midmorning U.S. time Monday, the airline had already canceled 700 or 19% of the day’s flights, making for more than 5,000 flight cancellations since Friday, according to flight tracking site FlightAware. The cancellations and widespread delays have left many travelers stranded, and Delta has given no timeline for when it expects to restore normal operations, reported Reuters.
Microsoft on Sunday released a tool it developed with CrowdStrike that gives administrators the ability to remove the faulty update from affected systems, albeit with caveats.
Any system locked down using full-disk encryption via Microsoft BitLocker or a similar product – a regulatory requirement in some industries – first has to be unlocked by administrators entering the recovery key.
Some organizations store BitLocker keys centrally, but many organizations only do this for critical servers, rather than every last desktop or laptop. As a result, they may need to involve end users in the recovery process.
Another caveat: The Microsoft utility runs from a bootable USB drive, so using the utility requires physical access to the system. “Your life will be driving from site to site, applying the fix and moving on. Forget about sleeping in your own bed, or eating a meal that wasn’t bought from a fast food restaurant,” said IT personality Ed Zitron in an encomium to system administrators.
Needing to get hands-on with nonworking device will add to the recovery timeline, especially “for organizations that have many workers employed remotely or in hybrid working scenarios,” Honan said. These staff “will need to come to a local or central office to have their device recovered, or an IT person will need to go to where the staff member and their device is.”
CrowdStrike has promised to provide more options to get affected Windows clients, servers and Hyper-V virtual machines back up and running. On Sunday, it reported testing “a new technique to accelerate impacted system remediation,” saying it’s been “making progress by the minute” but offering no further details.
On Monday, “BradW,” a moderator of the CrowdStrike subreddit, previewed “a cloud remediation that has been showing major success,” saying more information will be forthcoming, and customers will be given the ability to opt in.
CrowdStrike’s leadership has continued to apologize for the disruption caused and said it’s been working nonstop to help. “Thousands of our team members have been working 24/7 to get our customer systems fully restored,” Shawn Henry, an FBI veteran who’s now CrowdStrike’s CSO, said in a Sunday post to LinkedIn.