Hacker Breached Florida Treatment Plant To Poison The Water Supply

• Managing critical infrastructure security comes with several challenges. It entails massive environments that can’t experience downtime, and safety is often prioritized over security. For example, vulnerability scanning and remediation on OT devices often only occur once or twice a year. As a result, we are potentially leaving the back door wide open for nefarious attackers to our critical infrastructure.

• Now that OT environments are no longer insulated from internet-based risks, threat actors have seized on the opportunity: IBM reported a 2,000% increase in cybersecurity incidents targeting OT in 2019. In fact, the number of advisories published by the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT), a U.S. government entity and foremost authority on OT, has dramatically increased by 30% to 224 advisories.

• Security leaders of local governments and city municipalities must consider a new approach to vulnerability detection and management. There needs to be a paradigm shift in mindset. Organizations must move beyond detection and response and adopt more proactive and preventative security strategies for critical infrastructure.

In 2020 we saw a dramatic increase in Nation State actors attempting attacks on critical infrastructure like power and utility companies. The number of warnings, and specifically where they originate, insinuate that the level of activity has been elevated. Moreover, we are now witnessing these Nation State actors attempting to gain a foothold into utilities in order to build proactive attack capabilities - and they are trying to manipulate them with deadly consequences.

The change is partly due to the fact that a few hackers who have gained these attack capabilities are also more inclined to be aggressive - with Iran being the number one proponent. In Israel, Iranian state actors attempted, without success, to attack Israeli water utilities last year. While this isn’t the first effort to manipulate US water supplies, this new attack in Florida is the first time we have seen an attempt with lethal consequences. This is in contrast to the spate of ransomware attacks like those currently victimizing Florida hospitals, which points to a different trend where criminal attackers aim to profiteer. 

This event reinforces the increasing need to authenticate not only users but the devices and machine identities that are authorized to connect to an organization's network. If your only line of protection is user authentication, it will be compromised. It's not necessarily about who connects to the system, but what that user can access once they're inside. If the network could have authenticated the validity of the device connecting to the network, the connection would've failed because hackers rarely have possession of authorized devices. This and other cases of highjacked user credentials can be limited or mitigated if devices are issued strong, crypto-derived, unique credentials like a digital certificate. In this case, it looks like the network had trust in the user credential but not in the validity of the device itself. Unfortunately, this kind of scenario is what can happen when zero trust is your end state, not your beginning point.

The incident at the Oldsmar, Florida water treatment plant is a reminder that our nation’s critical infrastructure is continually at risk; not only from nation-state attackers but also from malicious actors with unknown motives and goals. Our dependency on critical infrastructure – power grids, utilities, water supplies, communications, financial services, emergency services, etc. – on a daily basis emphasizes the need to ensure the systems are defended against any adversary. Proactive security measures are crucial to safeguard critical infrastructure systems when perimeter defenses have been compromised or circumvented. We have to get back to the basics – re-evaluate and rebuild security protections from the ground up.

The thing we need to understand is that you don’t have to be a highly skilled attacker to be able to successfully breach a system like this. Although alarms would’ve been triggered before any dangerous water reached anyone’s taps, this plant was very lucky that the worker noticed his mouse moving and was able to address it quickly. Water plants are not known for their security resources, and between budget cuts and COVID keeping people working remotely, they’re even more vulnerable. It’s becoming easier and easier to access systems like these by people who have hardly any experience at all.

The area this happened it has a high population of children, and it’s disturbing to think someone would attempt to do harm like this.

The news that a hacker infiltrated a water treatment facility in Florida and changed a configuration setting to increase the volume of a dangerous chemical (lye) has rightly been greeted with concern by the media and cybersecurity community. The cyber threat to critical infrastructure has been increasing steadily as hackers, whether nation-state actors, criminal enterprises, or lone individuals better understand how to exploit operational technology (OT) in addition to IT systems. While much of the coverage of the cyber risk to critical infrastructure to date has focused on the age of many industrial control systems and the fact that they were not designed and deployed with security in mind, in this case, the attack vector appears to have been the increased level of remote access enabled by the Florida county.

In the rush to support remote operations during the global pandemic, there are very likely many organizations who have increased remote access to industrial engineering workstations and operator consoles. Fortunately, in this case, there was a vigilant operator who noticed the 111x increase in the chemical (from 100ppm to 11,100ppm) and was able to take quick corrective action to return the configuration setting to its prior level. While industrial espionage remains a significant threat (not all cyber attacks are focused on disruption), the worst fears of many in the OT cybersecurity community were realized in this episode; namely, changing a configuration setting to harm the community served by the facility. It is a poignant reminder that the best foundation for effective OT cybersecurity is a detailed and broad asset inventory that includes relationships and dependencies among OT systems and a baseline of configuration settings. With this in place, risk assessment is far more informed, enabling organizations to more effectively assign and limit remote access at both the system and account levels. Indeed, the combination of an up-to-date asset inventory and risk-based remote access management policies is more critical now than ever before, as it enables both reduced risk as well as faster recovery in the event of an unauthorized change.

Yesterday's hack of the Oldsmar, Florida water treatment plant again highlights the importance of maintaining critical infrastructure with a virtual air-gap (being off the network) from remote access. These systems should not be reachable by unauthorized attackers because of the sophistication of modern penetration tools and the complexity of these systems to make them completely free of vulnerabilities. Traditional firewalls and other remote access or VPN solutions are proving inadequate against these threats. We need to block any unauthorized access from ever reaching these critical and life maintaining systems while still allowing authorized, fully identified users remote access through secure tunnels using military-grade encryption. We have many such water districts using our solution for just these types of scenarios. 

One of the best ways to run a company network is to constantly think like a hacker. Connecting systems to the internet that have the potential to cause critical changes with relative ease is asking for trouble. Luckily, they had redundancies in place that would have made a fatal outcome unlikely.

However, whenever anything is connected to the internet there is a level of vulnerability, especially if remote tools such as Teamviewer are set up. Segregating networks for maximum security is vital; if their network could be controlled externally by anyone then it offered up the chance to be controlled nefariously.

Thankfully the potentially lethal actions were spotted whilst in progress, but this highlights that humans still look for the easiest path of resistance and will connect remote tools for ease of use, sparing the thought of them being misused. Teamviewer and other remote tools have greats uses, however, if there is the potential for users to change sodium hydroxide levels, which would end up in people’s homes, then it really should be reconsidered.

Since last year, Mandiant Threat Intelligence has observed an increase in cyber incidents by novice hackers seeking to access and learn about remotely accessible industrial systems. Many of the victims appear to have been selected arbitrarily, such as small critical infrastructure asset owners and operators who serve small populations. Through remote interaction with these systems, actors have engaged in limited-impact operations but none of these cases has resulted in damage to people or infrastructure. Fortunately, industrial processes are often designed and monitored by professional engineers who incorporate safety mechanisms to prevent unexpected modifications. We believe that the increasing interest in industrial control systems by actors of this nature is the result of the increased availability of tools and resources that reduce the barrier to learn about and interact with these systems.

While the incident does not appear to be particularly complex, it highlights the need to strengthen the cybersecurity capabilities across the water and wastewater industry similarly to other critical infrastructure sectors.

The attack against Oldsmar's water supply is precisely the kind of assault on critical national infrastructure (CNI) that cybersecurity experts have been fearing for years. It is frightening to think what might have happened if it was not for the vigilance of one of the plant's operators. 

COVID-19 has already placed enormous strain on UK infrastructure. As the government and NHS wrestle with the pandemic, it's hard to imagine how the country could cope at this time if there was any major disruption to the UK's supply of electricity or water. Nonetheless, key facilities worldwide are constantly being probed for weaknesses, and there are still significant concerns about the readiness of CNI to weather increasingly sophisticated cyber-attacks, with many facilities believed to run on out-of-date and vulnerable IT systems. The incident in Florida will go down as yet another near miss, but it is clear that CNI will remain a key target for hackers - inaction can no longer be tolerated.

In today's extremely volatile cyber landscape and faced with a surging threat of nation state actors, the UK government has rightly placed the resilience of CNI at the heart of its National Cyber Security Strategy in 2021. Thwarting cyber-attacks against key utilities and services has never been more critical and the severe consequences of failing to do so are only exacerbated by the unprecedented events of the past year. Organisations responsible for the security of our CNI need to ensure that a layered approach to cybersecurity is in place, focusing on installing the best and most up-to-date software and technology possible, supplemented by investment in both people and process. Only then will we have the right combination of safeguards in place to ensure that our critical infrastructure, key services, and health and safety, is not solely reliant on the watchfulness of the man or woman on duty at the time of an attack.

Critical infrastructure, such as water treatment plants, need to be treated as such. Normally, critical systems, such as this water treatment system, do not allow remote access. Risk is the impact if something bad happens times the likelihood of it happening. In this case, the impact (poisoning, possible death) to the population using the water from this facility is quite severe. The overall risk is normally manageable though because controls, such as disallowing remote access, are put in place to make the likelihood of something bad happening very unlikely. The challenge we are facing with these types of scenarios is that most organizations do not understand cybersecurity risk. In fact, convenience is often the primary driver for decisions with cybersecurity a mere afterthought.

The cyberattack against the water supply in Oldsmar, Florida, last week should come as a wakeup call. Cybersecurity professionals have been talking about infrastructure vulnerabilities for years, detailing the potential for attacks like this, and this is a near perfect example of what we have been warning about.  Though this attack was not successful, there is little doubt a skilled attacker could execute a similar infrastructure attack with more destructive results.  Organizations tasked with operating and protecting critical public infrastructure must assume the worst and take more serious measures to protect their environments.

With so much emphasis recently placed on hacks for the health care and financial services industry, an infrastructure hack such as this tends to hit much closer to home as it regards our physical safety.

As this is the case, it is critical to consistently review and monitor such critical administrative accounts that control such systems.  Alarms and logs for critical infrastructure systems should be reviewed and attended to constantly, and if such a hack or changes in set tolerances were to occur, a root cause analysis is imperative to mitigate such an event from happening in the future.

With the U.S. Secret Service and FBI involved in trying to determine the cyber culprits poisoning the Pinellas County, Florida water supply, this is another reminder that cyber threats against critical infrastructure networks are real. For nearly one year since the beginning of COVID-19 pandemic, threat actors have carried numerous acts of war against research companies, hospitals and other first responders. These attacks are brazen, shocking and downright maniacal. While this attack wasn’t against Florida’s two largest counties, Miami-Dade or Broward County, any attempt to poison a water supply should raise the eyebrows of local and state officials.

What's surprising about the manipulation of chemical levels in Florida’s water supply is the bad actors tipped their hands without first doing proofs of concept or stockpiling attacks for later use. What we don’t know if any successful attacks have taken place over the past few months and possibly not reported.

It is premature to infer what the motive of the attackers were and who they are. The actor at this point could be script kiddies, terrorists, criminal ransom, nation-state of any other actor. The correct response should be due process: investigate, understand, learn, improve, follow the investigation and data and constantly get better. Acts of War are determined by the State and among states. If the U.S. can point to a culprit and says it is, then that's what matters. The details thus far are scant but we will all be listening to the postmortem and hope the current administration provides a deeper response and holds the adversaries responsible for this act responsible. To be clear, the investigation is what matters. Where is leads, who it involves and how we interpret that are all to be determined.

A similar attack was reported by Verizon in 2016. Back then it was a water filtration plant in Syria, during the civil war. 

The underlying security issue is one of SCADA vulnerabilities. Supervisory Control and Data Acquisition networks are relied upon to manage critical infrastructure across the globe but they are predominantly reliant upon older, legacy systems which were not designed to be integrated or connected to the internet. Pre-digital design was based on ‘air gapping’ the critical components but it has become more and more obvious to malicious actors that those gaps present unprotected points of entry for malicious software. 

Nation State Security Services are aware of these vulnerabilities and I would expect the authorities involved to provide a solution to the citizens of Florida currently affected by this incident.