As I type this, parts of the Pacific Northwest are recovering from a power outage cascading across multiple towns. The cause? A contractor with a piece of heavy equipment severed a buried copper power line. The contractor is very sorry (and poorer), and we all now understand how secure we are against bulk power outages — digital or otherwise.
Digital technology is new for the power grid. Whereas in the computer security world, we focus on things such as system integrity or confidentiality as our primary goal, those are far from the top driver for the power grid folks. Here, the focus is system availability, where typical system uptimes are measured in decades. No one calls the power company to report that the grid is running smoothly, but have an outage and a flood of complaints pours in within seconds. This dynamic drives the lack of appetite for potentially vulnerable digital systems that could affect uptime.
It makes a certain kind of sense. After all, what if your computer was designed before the Internet existed, had to run for decades, cost millions, arrived on a train car, and required a crane to install? Would you upgrade when a new app came out because some guy in IT thought doing so “might” be a good idea? Not likely.
What about the personnel running the grid: Should they be anxious to install remote management software they don’t totally understand because it “might” be better in some way? Again, not likely.
The amazing part is that the grid actually works, and for very long periods of time. But enter a new threat: foreign (or domestic) actors bent on crippling commerce, the ability to run hospitals, and provide transportation; and now you can understand the temptation to meddle digitally with the power grid, and the need to defend it all. And digital attacks are on the rise, as we recently investigated.
When we observe the progression of attacks against critical infrastructure, they start with large-scale reconnaissance, where would-be attackers assess the attack surface and build dossiers of weaknesses. While there may be some specific attacks against high-value targets, think of it largely as weapons stockpiling based on gathered intelligence.
In the few actual attacks seen to date, the hackers’ next step has been to attempt some low-level attacks to judge the readiness of the adversary to detect and respond to an attack and the response time. After that, the more sophisticated attacks ramp up.
However, because potential attackers have their own goals and targets in mind, there’s no such thing as a one-size-fits-all attack. But the security goal from the defenders’ mindset is the same — to protect what matters.
I recently interviewed a security staff member working in the power sector, and he related a close call in which attackers almost succeeded in crippling a large power transformer supplying a major tech metropolitan area. The attack: taking out a critical bottleneck, unfortunately located right next to a major freeway — providing easy access, anonymity, and ease of egress.
The attack didn’t succeed, but not for reasons you might expect. The attackers damaged a link from the transformer to the bulk transmission lines but didn’t use quite enough force. The company’s response was to replace parts and get the system back up and running, not necessarily to assess what other potentially crippling attack vectors might exist or to perform a comprehensive post-mortem investigation. If the attacker been more successful, it might have taken a month to replace some of the more specialized parts, had they failed.
Recently, at a summit on Capitol Hill, I spoke during a collaborative event for private, public, legislative, and military personnel to discuss the way forward. While no single piece of that puzzle is a silver bullet, direction and budget from the Department of Energy, the National Institute of Standards and Technology, and others, along with industry technology can help.
Initiatives aimed at information sharing among electrical grid players are a positive step forward but are still hampered by barriers created by security clearance requirements. Also, participants need safe harbor initiatives to encourage sharing without fear of retribution. Technology solutions, however, such as supply chain integrity testing and multifactor authentication, are slowly moving forward.
Still, underlying it all is a people problem. The most senior folks (nearing retirement) — the ones with the experience to keep the power grid running — are reluctant to embrace digital security. After all, they’re not going to get raises if they learn this new-fangled digital security thing (since they’re at or near the top of the pay scale anyway), and they stand a chance of being punished for potential missteps.
Until digital natives who also have mastered the art of keeping the grid humming can begin to view the problems through a security lens, we will continue to see low-level hacks against important systems.
This is why the scammers don’t even need elite technologists and zero-day exploits when they can gain access through ancient operating systems and operators who don’t feel all that comfortable with technology.
Meanwhile, some grid equipment still runs Windows NT, where no security patches are even available. These systems have little or no authentication and run on horribly insecure protocols like Modbus. But the incentives to upgrade a $5 million generator to increase communication security are low.
As I finish typing this, the media is reporting an outage in Louisiana caused by a squirrel chewing through some electrical equipment, leaving thousands without power. While the squirrel wasn’t part of an international cadre of elite hackers, the result was similar — the lights went out. And in the end, that’s the part that everyone cares about, whether caused by rodents of unusual skill level or rogue hackers from across the globe.
We have a lot of work to do.
Cameron Camp is a researcher for global security provider ESET, and has played a critical role in growing the ESET North America Research Lab. Cameron has been building critical technology infrastructures for more than 20 years, beginning as an assembly language programmer in … View Full Bio