97% of Disrupted Endpoints Restored

Cybersecurity endpoint giant CrowdStrike says almost all of the computers disrupted by a faulty software update on July 19 have now been fixed.

“Over 97% of Windows sensors are back online as of July 25,” CrowdStrike CEO George Kurtz said in a post to LinkedIn. That assessment is based on a “week-over-week comparison” from before the faulty software update was distributed.

“This progress is thanks to the tireless efforts of our customers, partners and the dedication of our team at CrowdStrike,” Kurtz said. “We understand our work is not yet complete, and we remain committed to restoring every impacted system.”

Last Friday’s global IT outage sent technology professionals scrambling, and many worked nonstop over the weekend to access affected systems and attempt to restore them.

Disruptions resulting from the faulty “channel file” update affected banks, stock exchanges, doctors’ offices and hospitals, airlines and more. Multiple airlines experienced disruptions, and Delta in particular faced days of flight cancellations and delays, leaving many travelers stranded.

The company said the flaw involved a relatively new threat detection feature that uses configuration data that “maps to specific behaviors for the sensor to observe, detect or prevent.” While the firm tested the integrity of those configuration files, which previously performed as expected, one of the files distributed July 19 “passed validation despite containing problematic content data,” CrowdStrike said (see: CrowdStrike Says Code-Testing Bugs Failed to Prevent Outage).

Microsoft on Saturday said about 8.5 million Windows hosts – comprising PCs, servers and virtual machines – crashed and got stuck in an endless reboot loop as a result of the faulty software update.

The incident disrupted systems at one-quarter of the 500 most profitable, publicly traded companies, estimated cloud outage risk modeler and underwriting agency Parametrix Solutions, which calculated that those Fortune 500 companies will collectively see $5.4 billion in direct losses. That doesn’t include Microsoft, which Parametrix said suffered “very significant intangible losses” due to the incident, making its losses tough to predict (see: CrowdStrike Outage Losses Will Hit Healthcare, Banking Hard).

Microsoft has tasked “over 5,000 support engineers working 24×7 to help bring critical services back online,” said John Cable, vice president of Windows Servicing and Delivery, in a Thursday blog post.

CrowdStrike has promised to overhaul its quality assurance and testing processes and make other improvements, including not pushing updates to all endpoints at once, but rather in a staged fashion to detect any problems. The company has also pledged to release a full “root cause analysis” into the incident once it wraps its investigation.

Hearings and Investigations Loom

Kurtz is being called on to testify before Congress, and at least one lawmaker has called on the U.S. Cyber Safety Review Board to probe the incident. Authorized in 2021 by President Joe Biden, the public-private CSRB’s remit is “to review and assess significant cyber incidents and make concrete recommendations that would drive improvements within the private and public sectors.”

Attackers have already been attempting to turn the outage to their advantage in a variety of ways, including through phishing attacks that spread malicious files purporting to be recovery tools. On Thursday, CrowdStrike said one of the latest attack attempts detected involves phishing attacks designed to deliver “an inauthentic CrowdStrike Crash Reporter installer via a website impersonating a German entity.”

Free Coffee – Or Not

Many analysts and cybersecurity experts have lauded the firm’s transparent and forthright response to the incident, saying it will likely minimize customer and prospect attrition.

Even so, company is facing backlash after it emailed $10 Uber Eats voucher codes to staff and business partners, as first reported by TechCrunch. “To express our gratitude, your next cup of coffee or late night snack is on us!” CrowdStrike wrote in the emails, directing recipients to use the included voucher code.

Multiple individuals took to social media to report that the codes didn’t work. CrowdStrike has apologized, saying that it “did not send gift cards to customers or clients,” a spokesperson said. “We did send these to our teammates and partners who have been helping customers through this situation. Uber flagged it as fraud because of high usage rates.”

Some have questioned the move altogether. “The gesture of a cup of coffee or Uber Eats credit as an apology doesn’t seem to make up for the tens of thousands lost in man hours and customer trust due to the July 19 incident,” said one individual who says he works for a business partner and received one of the vouchers.