On the cloud side CrowdStrike manages its own system that performs validation checks on content before it’s released to prevent an incident like Friday from happening. CrowdStrike released two Rapid Response Content updates last week, or what it also calls Template Instances. “Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data,” says CrowdStrike.
While CrowdStrike preforms both automated and manual testing on Sensor Content and Template Types, it doesn’t appear to do as much thorough testing on the Rapid Response Content that was delivered on Friday. A March deployment of new Template Types provided “trust in the checks performed in the Content Validator,” so CrowdStrike appears to have assumed the Rapid Response Content rollout wouldn’t cause issues.
This assumption led to the sensor loading the problematic Rapid Response Content into its Content Interpreter and triggering an out-of-bounds memory exception. “This unexpected exception could not be gracefully handled, resulting in a Windows operating system crash (BSOD),” explains CrowdStrike.
To prevent this from happening again, CrowdStrike is promising to improve its Rapid Response Content testing by using local developer testing, content update and rollback testing, alongside stress testing, fuzzing, and fault injection. CrowdStrike will also perform stability testing and content interface testing on Rapid Response Content.
CrowdStrike is also updating its cloud-based Content Validator to better check over Rapid Response Content releases. “A new check is in process to guard against this type of problematic content from being deployed in the future,” says CrowdStrike.
On the driver side, CrowdStrike will “enhance existing error handling in the Content Interpreter,” which is part of the Falcon sensor. CrowdStrike will also implement a staggered deployment of Rapid Response Content, ensuring that updates are gradually deployed to larger portions of its install base instead of an immediate push to all systems. Both the driver improvements and staggered deployments have been recommended by security experts in recent days.