Preventing future outages: insights from our CTO

A faulty software update from CrowdStrike caused widespread disruptions across Microsoft Windows systems globally in July 2024. The failed update led to crashes in approximately 8.5 million devices, affecting businesses and services worldwide and leading to billions of dollars in financial losses. 

Business leaders have since been asking themselves the same thing: How can I protect my organization from such outages in the future?

What set this incident apart was that it was not a cyberattack, but a configuration error in CrowdStrike’s security software, stemming from a flawed configuration change during a CrowdStrike software update. Although designed to enhance security against malicious activities, this update introduced an accidental logic error that had severe consequences. Programs in Ring 0, like CrowdStrike, have full control over a device’s CPU. Their failure can incapacitate the entire system. Recovery, therefore, required manual intervention on each affected machine. In some cases, it took several days to get systems back up and running again.

The CrowdStrike outage underscores the potential vulnerabilities in the global digital infrastructure and the need to build a security-friendly culture across organizations. In this interview-style blog post, our CTO, Shreerang Gondegaonkar, shares his insights on optimizing security across organizations and building a culture where security is no longer an afterthought.

What are the preventive steps that businesses should take to minimize the damage of a similar outage?

Shreerang: Firstly, I would recommend establishing rigorous testing protocols. This means setting up multiple testing environments, including staging, sandboxing, and ones that mimic the production environment as closely as possible. Both automated and manual testing are essential to catch potential issues before we deploy anything live.

Next, I’d suggest adopting a staggered rollout approach. Start by deploying updates to a small group of users – this is what we call canary deployments – so we can monitor their impact before going for a full-scale rollout. Keeping an eye on performance and gathering user feedback is crucial before expanding the deployment.

Another critical step is ensuring you have redundancy or failover systems in place. This means having robust backup solutions and regularly testing our recovery procedures to ensure they work. Implementing redundant systems can help maintain operations if the primary systems fail. It’s also vital to have a well-developed incident response plan. Regular updates and drills will prepare your team to respond swiftly and efficiently to any disruptions that might occur.

On the security front, implementing robust access controls is a must. This involves using multi-factor authentication (MFA) and strict access control policies to limit who can access your sensitive data and systems.

Lastly, businesses should conduct regular security audits and penetration testing. Periodic assessments of our security infrastructure will help us identify vulnerabilities and address them proactively.

How can businesses build a security-friendly culture?

Building a security-friendly culture starts with leadership commitment. Leaders need to set the tone by prioritizing security and showing their dedication through their actions and decisions. Integrating security practices into daily operations is vital – they should be embedded into workflows and project management processes across the organization.

Employee training and awareness initiatives are also essential. I recommend having regular security training sessions that cover security best practices and new securing threats. It’s important to make these sessions engaging by using interactive methods like workshops, simulations, and newsletters. At Openprovider, we have yearly training sessions for our employees on the principles of ISO 27001.

Having clear policies, procedures, and reporting mechanisms about security in place is also crucial. These policies should be easily accessible and understood by everyone. It’s also important to create safe and anonymous channels for reporting any suspicious activities.

Promoting responsibility and recognition plays a big role, too. Businesses should empower employees to take personal responsibility for security and reward those who demonstrate strong security practices and a proactive approach to security.

Finally, fostering open communication and collaboration is key. Regular meetings to discuss security issues and updates are important, as well as having feedback mechanisms in place. Encouraging cross-department collaboration and identifying security champions will strengthen your security culture even further.

What products and services do you recommend to minimize the risk and impact of future outages?

There are various solutions and tools available. Firstly, firewalls and intrusion detection/prevention systems (IDPS) are crucial. These devices or software solutions monitor and control network traffic based on security rules you’ve set, helping to keep your network secure from unauthorized access and potential threats.

Next, endpoint protection solutions are essential. These software tools are designed to detect and respond to threats on individual devices like laptops, desktops, and mobile phones, ensuring each endpoint is secure.

Backup and recovery solutions are another must-have. These systems regularly back up your data and provide ways to restore it in case of data loss or corruption, helping you quickly recover and continue operations.

Network monitoring tools are also important. They continuously monitor the network for any slow or failing components and alert network administrators to any issues, allowing you to address problems before they escalate. Meanwhile, security information and event management (SIEM) solutions provide real-time analysis of security alerts generated by applications and network hardware, helping businesses stay on top of any security threats as they arise.

Lastly, I would recommend investing in DDoS protection solutions, such as Premium Anycast DNS. These services filter out malicious traffic from distributed denial of service attacks, ensuring that only legitimate traffic reaches our network and keeps our services running smoothly.

Protect your customers with Openprovider

The 2024 CrowdStrike incident was a stark reminder of the importance of a proactive and comprehensive approach to resiliency and cybersecurity. By implementing rigorous testing protocols, fostering a security-conscious culture, and investing in robust solutions that align with recognized standards and frameworks, organizations can enhance their resilience and ensure the continuity of their operations in the face of outages and other potential cyber threats.

Your customers expect you to keep their websites and data secure and their operations running. Start protecting your customers’ websites and email servers with security products from Openprovider – the perfect add-on to a domain name or hosting package. From SSL certificates and Premium DNS to SpamExperts’ intelligent spam filters, our security products are a high-value yet cost-effective solution that will add a new dimension to your portfolio. And, while you’re at it – Openprovider Members save more on all security products. Click here to find out how you can protect your customers and save more at the same time.