CrowdStrike Confirms Faulty Software Update for Falcon Sensor, Details Workaround
Banks, airlines, major media firms and others are experiencing business disruptions due to a mass, global IT outage tied to Windows PCs.
See Also: 5 Requirements for Modern DLP
Security and IT experts report that the outage appears to be due to a faulty software update released by Austin, Texas-based cybersecurity firm CrowdStrike, which leaves Windows systems displaying the dreaded “blue screen of death.”
CrowdStrike said it’s aware of the outages. A prerecorded messaging on CrowdStrike’s telephone support line said that “CrowdStrike is aware of reports of crashes on Windows” that appear to be “related to the Falcon sensor,” reported Sky News.
Microsoft said the outages appeared to begin around 6pm U.S. Eastern Time on Thursday and that it’s taking “mitigation actions.”
The resulting disruptions have reportedly led to boarding delays at multiple airports in the U.K., EU and beyond, with Hong Kong International Airport being left “in chaos” as staff reverted to using manual procedures for checking in passengers, reported South China Morning Post.
Sky News is also experiencing disruptions, including not being able to “to broadcast live TV this morning” in Britain or Australia, said David Rhodes, its executive chairman, in a post to social media platform X. “We are working hard to restore all services.”
Not all versions of CrowdStrike Falcon are affected. “It is our understanding that any business running versions 7.15 and 7.16 are affected by the outage, but 7.17 seems to be OK,” said Ajay Unni, CEO of Australian cybersecurity service firm StickmanCyber.
Multiple IT administrators report receiving this workaround from CrowdStrike’s support team:
- Boot Windows into Safe Mode or the Windows Recovery Environment;
- Navigate to the
C:/Windows/System32/drivers/CrowdStrikedirectory; - Locate the file matching
C-00000291*sysand delete it; - Boot the host normally.
While that might be fine in theory, the workaround would be difficult to implement at scale since it can’t be automated, said British cybersecurity expert Kevin Beaumont in a post to Mastodon. Also, what affect deleting the file might have – for example, if it would compromise the ability of the Falcon sensor to detect or block malicious code – remains unclear.
“If anybody is wondering the impact of the Crowdstrike thing – it’s really bad. Machines don’t boot,” he said. “Basically Crowdstrike will be in very hot water.”
“IT security tools are all designed to ensure that companies can continue to operate in the worst-case scenario of a data breach, so to be the root cause of a global IT outage is an unmitigated disaster,” StickmanCyber’s Unni said.
Multiple IT support teams have reported implementing their incident response plans to deal with the outages.
IT administrators have been taking to message boards seeking advice on workarounds as well as the potential efficacy of any update CrowdStrike might push.
“I have 40% of the Windows Servers and 70% of client computers stuck in boot loop (totaling over 1,000 endpoints), one posted to the CrowdStrike subreddit. ” I don’t think CrowdStrike can fix it, right? Whatever new agent they push out won’t be received by those endpoints coz they haven’t even finished booting.”
“Here in the Philippines, specifically in my employer, it is like Thanos snapped his fingers,” another posted. “Half of the entire organization are down due to BSOD loop. Started at 2pm and is still ongoing. What a Friday.”
Many report having to immediately update large numbers of PCs, in some cases by in-person teams.
“I’m planning a weekend trip to 15 sites with all the IT staff to bring systems up one by one. Hilarious,” one Australian IT administrator who oversees about 200 Windows PCs, said in a post to Mastodon, noting that every system uses BitLocker whole-disk encryption plus a local administrator password solution.
“We’re 100% bitlockered and LAPS’ed, so I have to wake every machine by hand to delete the file, AFAIK,” the admin said. “Happy to accept advice on a better way.”
Others also reported needing to get IT hands on keyboards to deal with affected systems. “All of our work computers use bitlocker for certain government contract requirements (consulting). So no employees can do the official workaround on their own since they won’t have the bit locker recovery key,” another IT administrator posted to the r/sysadmin subreddit. “So there goes the weekend I guess.”
Stay tuned for updates on this developing story.