CrowdStrike, the cybersecurity firm that broke the internet, has released a comprehensive post-incident review (PIR) detailing the cause of last week’s widespread Windows crash that affected 8.5 million machines and outlining steps to prevent future occurrences. The company is so repentant they are sending certain users $10 Uber Eats gift cards as an apology.
The Verge reports that in a detailed post-incident review, CrowdStrike has shed light on the recent incident that caused millions of Windows machines to crash due to a buggy update in its Falcon software. The cybersecurity company, whose software is widely used by businesses globally for malware protection and security breach management, has identified the root cause and proposed a series of measures to prevent similar incidents in the future.
The issue stemmed from a content configuration update released on Friday, which was intended to “gather telemetry on possible novel threat techniques.” This update, part of CrowdStrike’s regular update process, unexpectedly triggered Windows crashes across numerous systems. The company’s update system consists of two types: Sensor Content, which directly updates the Falcon sensor operating at the kernel level in Windows, and Rapid Response Content, which modifies how the sensor detects malware. The problematic update was a mere 40KB Rapid Response Content file.
CrowdStrike’s investigation revealed that a bug in their Content Validator system failed to properly validate one of two Template Instances (another term for Rapid Response Content) released last week. This oversight allowed problematic content data to pass through the validation process unchecked. The company admitted that while they perform extensive automated and manual testing on Sensor Content and Template Types, the same level of scrutiny was not applied to Rapid Response Content.
The cascading effect of this oversight led to the sensor loading the problematic Rapid Response Content into its Content Interpreter, triggering an out-of-bounds memory exception. Unable to handle this unexpected exception, the Windows operating system crashed, resulting in the dreaded Blue Screen of Death (BSOD) for affected users.
In response to this incident, CrowdStrike has outlined a comprehensive strategy to enhance its testing and deployment procedures. The company plans to implement more rigorous testing for Rapid Response Content, including local developer testing, content update and rollback testing, stress testing, fuzzing, and fault injection. Additionally, they will introduce stability testing and content interface testing specifically for Rapid Response Content.
Improvements to the cloud-based Content Validator are also in the works. CrowdStrike stated, “A new check is in process to guard against this type of problematic content from being deployed in the future.” This enhancement aims to provide an additional layer of security in the content validation process.
On the driver side, CrowdStrike has committed to enhancing the existing error handling capabilities in the Content Interpreter, which is an integral part of the Falcon sensor. This improvement should help mitigate potential issues arising from problematic content in the future.
Perhaps one of the most significant changes in CrowdStrike’s approach will be the implementation of a staggered deployment system for Rapid Response Content. This new method will ensure that updates are gradually rolled out to larger portions of the install base, rather than pushing updates simultaneously to all systems. This approach, which has been recommended by security experts, should provide an additional safeguard against widespread issues caused by potentially problematic updates. Members of the security community expressed shock that the giant company already didn’t use this approach to releasing updates.
How a massive publicly traded company is only going to just now start doing canary deployments is so far beyond my comprehension… pic.twitter.com/IAjhueM5Eu
— Kelly Vaughn (@kvlly) July 24, 2024
CrowdStrike has responded to the issue by offering $10 Uber Eats gift cards to affected customers. In an email, CrowdStrike recognized “the additional work that the July 19 incident has caused,” and added that they send their “heartfelt thanks and apologies for the inconvenience.”
The email ends with the company saying: “To express our gratitude, your next cup of coffee or late night snack is on us!”
Read more at the Verge here.
Lucas Nolan is a reporter for Breitbart News covering issues of free speech and online censorship.
