July 19 marked one year since what has been described as “the largest IT outage in history” when cybersecurity company CrowdStrike updated its software that led to roughly 8.5 million systems crashing and refusing to properly restart.
A number of broadcasters were significantly impacted by the outage, including the U.K.’s Sky News and Sky Sports News. Sky News went off-air for parts of the morning on July 19, 2024, before returning with a much-changed backdrop, no graphics, autocue or packages.
In the United States, station group E.W. Scripps Co. told TV Tech that a strategy put in place over the past 10 years was able to quickly muster technical resources and identify and correct issues when they arose.
To mark one year since the outage, TV Tech and sister brand TVBEurope asked key media figures if they believe the industry has taken sufficient steps since then to prevent similar disruptions in the future.
Tim Claman, chief technology officer, Avid
One of the big learnings from that outage was that security strategies need to be more proactive and adaptive, rather than just responding to incidents after the fact. Since the breach, we have seen wider adoption of strategies that combine prevention and resilience methods.
Customers are bolstering traditional detection methods (like antivirus, EDR) with technologies that more dynamically identify potential threats, including AI-based monitoring, zero-trust models, dynamic policy management and more rigorous update testing, to help prevent and minimize impact from security incidents like we saw last year. While a lot of the technologies and approaches (CNAPP, IAM, EDR, Endpoint Isolation, Zero Trust, etc.) are not new, they are now being used more intelligently in combination with operational best practices to reduce risk.
Beyond the technological improvements and the evolution of best practices, we have seen a shift in mindset. The outage was a wake-up call for our customers, as well as for the vendor community. Media enterprises realize that they need to own their own destiny on security, rather than trusting security vendors, so they are more proactively managing their security strategies and practices. As a technology vendor, Avid is receiving more detailed and thorough security questions and requirements in RFPs, a reflection of our customers’ more intensive approach to security strategy.
Rowan de Pomerai, CEO, DPP
I think it’s probably fair to say that no, not enough focus has been given. There’s been a huge amount of business transformation and disruption, and AI has taken so much of the technical limelight, so it’s been easy to lose focus on concerns like security.
But perhaps AI could be the solution as much as the problem: contributors to the “DPP 2025 Predictions” said that “security concerns will go beyond human scale” as the capability to keep up with ever-changing threat vectors will become so complex that AI and other automation tools become essential.
Our “State of Media Technology Security” report also exposed a gulf between customers and vendors when it came to their assessment of the security of modern media technology tools. Clearly, there is more collaboration needed as we move forward.
Neil Maycock, TVBEurope contributor and business adviser
The CrowdStrike incident really highlighted the ever-evolving threats in cybersecurity, the fact that major operations were so severely impacted is a testament to that. These were companies who take security very seriously and will have had extensive security protocols in place, and yet an unforeseen scenario had devastating consequences. Therefore, to answer the question, have sufficient steps been taken to prevent a repeat, I think the answer is, of course industry will have protected itself from another CrowdStrike, but the real question, is it ever possible to anticipate all possible scenarios?
Cybersecurity is a classic risk-management exercise, where risk must be balanced against cost, both financial and operational impact. If we lock our systems down to the extent that it compromises the ability of an organization to operate, then the cost may outweigh the potential risks. Quantifying the risk and cost impact is a massive challenge, and one that is getting harder all the time. We frequently talk about the impact of AI in the media industry, but across all sectors AI is now being leveraged to implement ever more sophisticated cyberattacks. For example, AI’s ability to impersonate key personnel or create very specific personalized phishing attacks is creating new challenges.
Ensuring companies are adequately protected is an unenviable challenge.
John Naylor, VP of product security and Ross Research Labs, Ross Video
The root cause of the disruption was a supply-chain attack, which was enabled by a flaw in CrowdStrike’s content validator that enabled malicious content to be included in its global, automated software update. Subsequently, CrowdStrike has improved the rigor with which they validate the validator and changed their deployment process to be more gradual so that similar problems can be spotted before they’re globally deployed.
The impact of the CrowdStrike flaws was so severe for Windows users in particular because it, like other endpoint-protection software, has privileged access to the Windows kernel. Last month, Microsoft announced a preview of its endpoint protection APIs that enable them to execute in user space, which dramatically reduces the harm they can do, either maliciously or by accident.
So, in a narrow sense, yes, “the industry” has taken steps to prevent products from endpoint protection vendors creating similar future disruptions. According to the most recent Verizon Data Breach Investigations Report, supply-chain attacks are up 68% year on year, which is why they now list them in a new category of breach types. This makes future disruptions with a root cause somewhere in the supply chain almost inevitable, so ensure your incident response, command, control and communication plans are ready and tested!
Karl Paulsen, TV Tech contributor and retired CTO
I suspect this won’t be the last time we see something like this happen when there are so many moving parts in a single system. With Windows 10 support evaporating, much of the base software for users will be shifted and everybody and their dog will be wondering how to address a significant change in their hardware and software (including third-party support).
Impacts on multi-cloud will need to be managed, and it is still unclear what “AI” is going to do to everyone’s architectures. Errors are still likely to occur and an overload of systems management is a potential liability.
Dan Pisarski, chief technology officer, LiveU
The notorious CrowdStrike outage in 2024 taught an important lesson about shared vulnerabilities: you could have an entire, physically diverse set of servers and workstations—even in a building designed to withstand disasters (what if a meteor falls on the building, after all?)—but if every one of those servers runs CrowdStrike, you’re still at risk. This event has put real pressure on media organisations to rethink what disaster recovery looks like, especially when it comes to intentionally building diversity into Disaster Recovery (DR) plans.
Is your primary production on-prem? Then your DR should be in the cloud. Is your playout path based on a local fibre run? Then your DR plan should use wireless. The goal is simple: if something happens that wipes out an entire class of options (as the CrowdStrike incident did to Windows PCs and servers), you need a diverse DR plan that can avoid the fate that takes down not just one component, but an entire class of your mission-critical systems.
Cloud-based production platforms provide a diverse form of backup to hardware-based production solutions running Windows. If there is another day when “all Windows PCs don’t work in the world”, then you have a backup plan. The cloud solution does not require integration with Windows, and while it is not an exact match 1:1 of features of a hardware-based solution, that matters less when it is becomes the primary solution in a disaster recovery scenario.
This kind of built-in diversity applies at both small and large scales. At the small scale, for example, your bonded-cellular active/active wireless transmissions should use multiple carriers—not just for performance, but also “just in case” one carrier experiences a localized or nationwide outage (it’s happened!). On a much larger scale, diversity might mean having an elastically scalable cloud production service ready to go, even if you “usually” rely on on-prem production.
