Oldsmar, Fla., averted a potential catastrophe in February when someone breached the computer system at the Bruce T. Haddock Water Treatment Plant and changed the levels of sodium hydroxide from 100 parts per minute to 11,100 ppm.

An operator monitoring the system happened to see the cursor moving across the screen and corrected the levels.

“That’s called getting lucky,” Kevin Morley, manager of federal relations for American Water Works Association, said.

During a press conference organized by the sheriff and city officials following the event, Pinellas County Sheriff Bob Gualtieri said it was an “unlawful intrusion” to “part of the nation’s critical infrastructure.”

The perpetrator actually made two attempts. The first was at 8 a.m. Feb. 5, but it was very brief, and the operator thought maybe supervisors were accessing the system through remote access. At 1:30 p.m., the system was again breached, but this time the perpetrator changed the amount of sodium hydroxide — a caustic ingredient in drain cleaners — to “significant and dangerous levels,” according to the sheriff.

The sheriff and city officials — Mayor Eric Seidel and City Manager Al Braithwaite — stressed residents were never in danger because even if the cyber attack had not been detected, there are other controls in place that would’ve set off alarms before the increased ingredient could have entered the drinking water system. They also noted it takes 24-36 hours to hit the water system. At the time of the press conference on Feb. 8, Sheriff Gualtieri said they didn’t know whether the threat came from inside or outside the country.

The FBI is still investigating the incident, so when called, Oldsmar’s assistant city manager said, “We’re not engaging in any conversations on that topic at this time.”

During the press conference, Braithwaite responded to a reporter’s question by stating, “We anticipated this day coming — we talked about it and studied it.”

However, Morley said the Oldsmar system had no firewall and a weak password, so “it didn’t require a lot of sophistication to hack it.”

In a press report days after the incident, an FBI investigator was cited as stating the cyber actors likely accessed the system by exploiting cybersecurity weaknesses, including poor password security and an outdated Windows 7 operating system, and they likely used a shared software, Team Vision, to gain unauthorized access to the system.

Morley said this is not an isolated incident and shared a man was just indicted in Kansas for breaching a water utility. In that case, it was a terminated, disgruntled employee who shut down the water system.

A press release dated March 31 stated that Wyatt A. Travnichek, 22, of Ellsworth, Kan., was indicted with one count of tampering with a public water system. On or about March 27, 2019, Travnichek knowingly accessed the Ellsworth County Rural District’s protected computer system without authorization. It is alleged Travnichek performed activities that shut down the processes at the facility, which affected the facility’s cleaning and disinfection procedures, with the intention of harming the Ellsworth Rural Water District #1, known as Post Rock Rural Water District.

A special agent in charge of the U.S. Environmental Protection Agency’s Criminal Investigation Division in Kansas said the indictment sends a clear message that individuals who intentionally violate these laws will be vigorously prosecuted. Upon conviction, the alleged crimes carry the following penalties: tampering with a public water system — up to 20 years in federal prison and a fine up to $250,000 — and reckless damage to a protected computer during unauthorized access — up to five years in federal prison and up to a $250,000 fine.

When asked how common these instances are, Morley replied, “It’s the number one threat to critical infrastructure and not exclusive to water.”

Prioritizing and protecting cybersecurity

Daniel Kapellman Zafra, manager of analysis, Mandiant Threat Intelligence at FireEye, a cybersecurity firm, spoke to the threats utilities face.

He said all organizations are vulnerable to cyberattacks. “However, the water and wastewater sector is generally less mature in cybersecurity than other sectors. This is partly caused or exacerbated by the local nature of most operations: mostly small, municipality-owned utilities that rely on limited resources to support complex cybersecurity programs. Additionally, there is currently a lack of regulation and resources developed specifically to guide and enforce implementation of security controls in this sector.”

Zafra added, “Some of the challenges water utilities face — similar to other critical infrastructure industries — include slowly maturing cybersecurity infrastructure, outdated hardware/software, unauthenticated protocols and lack of security resources and knowledge. This is especially important in water utilities, given their relevance for other industries and social well-being.

“More than large-scale cyber physical attacks, water utilities face an immediate threat resulting from financially motivated criminals and low-sophistication actors that leverage commodity-type attacks or target internet-exposed assets (such as the case of the recent hack on a Florida water utility).”

To protect against these types of attacks, Zafra said, “Water utilities should focus on implementing best security practices such as avoiding exposure of critical assets to internet, establishing redundancy mechanisms for critical assets, employing strict access control policies and raising security awareness among employees. They should also place emphasis on understanding the safety processes in place and how these could be (or not) bypassed by attackers via cyber means.”

He added, “Further investment in regulation and guidance for supporting water utilities to mature their security programs would also be beneficial.”

Even if a utility is confident it has a working security system in place, Zafra said, “Ideally any organization should invest at least in regular monitoring of traffic from non-trusted networks. In the case of organizations such as utilities, this traffic volume could be limited by implementing a robust segmentation that limits access to and from production networks. Organizations that reach a high maturity level and have access to the resources may also be able to hunt for malicious behaviors based on threat intelligence.”

Morley pointed out in an article, “Priority on Cybersecurity,” which he wrote for DC Beat in March 2019: “Drinking water and wastewater systems not only manage sensitive personal data, they also operate process control systems that are essential to day-to-day operations. A cybersecurity breach in the water sector could result in serious harm to public health and safety, as well as other damages from service interruptions, lost data, compromised systems, litigation and repair costs and reputational harm. In fact, government intelligence has confirmed that drinking water and wastewater systems have been directly targeted by nation states, as part of multi-stage intrusion campaigns, and by individual criminal actors and other groups seeking to harm the United States or obtain illicit proceeds.”

He wrote those attack campaigns used various tactics, including spear-phishing emails from a compromised legitimate email, watering hole domains, credential gathering, open-source and network reconnaissance, host-based exploitations and industrial control system infrastructure targeting.

But AWWA has developed tools and training to help. “Since 2014, we developed a set of guidance and assessment tools for utilities to examine vulnerability in line with the National Institute of Standards and Technologies (NIST) cybersecurity framework,” Morley said, adding these were developed along with the Obama administration under executive order 13636.

He said the impetus for the tools goes back to 2008 when officials developed a roadmap for security processes and discovered there was “a whole lot of different standards but no consistent template to apply those standards.”

So in 2013, at the same time as the Obama executive order, AWWA put in guidance for water utilities. Morley said the organization collaborated with the Department of Homeland Security, EPA and NIST “to ensure the end product aligned with the federal family.”

“We had the first sector-specific approach to applying controls for cybersecurity framework,” he said.

According to Morley, AWWA recognized it couldn’t expect someone in small-town Mid-America to become a cyber expert, so the association developed the tools in a way that its 52,000 diverse community water systems would find relevant.

“We said let’s take this from the perspective of the utilities — how do these controls apply to the guidance they use?” he said.

The guidance tool asks questions about how utilities use technology, such as whether they allow employees to “bring your own device” or not? He said there’s a set of 22 questions that utility managers either answer that they do or they don’t, and if they don’t know, they’ll need to find out.

“So there’s some self-discovery there, and that’s really important,” Morley said.

The tool also allows them to prioritize controls, with priority one being a must-have. “They found it to be very useful in the decision-making process,” Morley said.

As the utility answers questions, one feature is a status check of 100 controls, and the utility gets a scorecard of its level of progress. The scorecard is something that can be taken to management to spur action. For instance, if there are 20 priority one controls but the utility is only implementing 10, managers can see the other 10 controls need working on ASAP.

Morley stressed all these safeguards don’t mean a utility won’t be breached, “but it makes it harder to do so. This is risk management, not risk elimination.”

AWWA also has a set of resources for small systems and developed training as well through a USDA grant. “It’s been very successful in helping smaller utilities move along in getting things done.”

Risk and resilience assessment

Morley shared a reminder that communities are under a statutory obligation under the America’s Water Infrastructure Act of 2018 to provide a risk and resilience assessment, including cybersecurity, and also to have an emergency response plan, which requires an action plan. Larger systems serving over 100,000 had their deadline last year. Systems serving between 50,000 to 100,000 had their first deadline in December 2020, and the next one to certify the emergency response plan with the EPA is due June 30. Smaller systems, which compose the largest group — utilities serving 3,300-50,000 — have a deadline of June 30 for its risk and resilience assessment and Dec. 30 for the emergency response plan.

“Getting cybersecurity right is not an easy issue — threats are persistent and mutable,” Morley said, adding, “The scary thing is, from a cybersecurity perspective, the parameter isn’t at the gate — the criminal element can be anywhere in the world.”

But he said, “The priority one controls are foundational speed bumps mitigating a lot of the large number of threat attempts we see.

“If the Kansas utility had disabled the employee’s credentials when he was fired and if Oldsmar had a firewall and a strong password — in these cases, there were no real speed bumps to accessing the systems.”

Utilities “should have a basic working knowledge of how their system is moving information through the system. Once they know that, it’s easier to assess the risk.”

“Cyber risk is a serious threat, and it is critical that organizations in the water sector make cybersecurity a top priority,” Morley said, adding, “Use the tools.”