Welcome to the next installment of Security Sessions, a regular feature focused on security-related issues, policies and procedures. Over the past few years I have worked with both IT folks and I&C folks to bridge the “gap” that exists in the expertise, and quite often the terminology, of the two communities. Sometimes it seems like the two groups don’t even speak the same language. And to a large degree, I guess that is actually true. Like it or not, I&C personnel need to learn about and understand the ‘lingo’ of IT so that they can have meaningful conversations. It’s kind of like when my wife tries to tell me about a problem with her laptop PC (or the car.) She and I don’t use the same terms and when we seem to be using the same terms it often turns out that we mean totally different things. A common basis for understanding and communications are essential for collaboration and collaboration with IT is essential for establishing and maintaining adequate cyber security for our industrial automation systems – Tim.
William T. (Tim) Shaw
PhD, CISSP
By now most of us are aware of how industrial automation systems have evolved and transmogrified into things that look frighteningly like IT business systems. They are built out of PCs and “servers” and switches and routers. They run Windows and Linux and Oracle and Apache. We have centralized RADIUS and Active Directory authentication and “policy” servers. We accessorize them with firewalls, and “security appliances” and NAC appliances and IDS/IPS systems. They use Ethernet and TCP/IP and UDP and EtherNet/ IP and OPC communications. We implement black listing and white listing and grey listing and…”lions and tigers and bears, oh my!” I could keep on going, using lots more techno-terms, but the point of all of this is that we toss these terms and words around while most I&C people are never given the additional education they really need to truly understand what these things mean, how they work and how they play together. And that is dangerous when, for the most part, it is still the I&C staff that is expected to keep these systems up and running.
One of the things I do in my copious spare time – other than writing these articles – is teaching various courses for the International Society for Automation (ISA). Although some courses I teach are still on basic I&C and computer-based control and automation topics, more and more courses are oriented towards what I like to call “IT survival basics for the I&C staff”. It is interesting to take a group of highly competent automation/I&C personnel and guide them into the wondrous world of “how this stuff really works”. Just showing them how much hidden functionality is supported in a basic Ethernet switch can be an eye-opening event.
Most of my latest students were not aware of the fact that a network of interconnected Ethernet switches will intercommunicate and coordinate and automatically respond to network changes/failures (using “rapid spanning tree” messages) and that this functionality is pretty-much standard in most switches. Imagine their surprise when we disconnected an Ethernet cable and watched the flurry of message traffic initiated by that simple act.
Likewise, they didn’t know that the essential configuration settings in that same switch – which are necessary for the switch to function – could be manipulated using SNMP communications and freely available software tools, from across the network. Or, that the only thing preventing such manipulation was a set of built-in passwords (called “community strings”) that had universally-known default factory settings (i.e., the words ‘public’ and ‘private’) until specifically changed. They also did not know that a special type of connection (a “trunk port”) must be used between interconnected switches in order to make VLANs work – and most did not know what a VLAN was or even why it was useful. To most of them a switch was just a box with several RJ-45 jacks into which you can plug in Ethernet cables so that computers can talk to each other.
My point is that these are basic “survival” issues for today’s I&C engineer since switched Ethernet LANs form the basic backbone of all modern DCS and PLC automation systems. Today, in the same way that an I&C engineer needs to be knowledgeable about PID loop tuning, instrument calibration and sequential function charts, that same engineer needs to have a firm grasp of basic IT concepts and technologies. Those same students, when initially asked if they knew the difference between a hub and a switch (prior to being taken through the course training material) basically could not explain what made them any different. When asked what they thought was meant by a “managed” switch, they figured that it had to do with a manufacturer’s support contract or the fact that IT was responsible for the switch, which I suppose is somewhat correct, but that’s really not the primary issue of concern as regards cyber security.
Now, at this point you might be thinking, “but the IT folks are responsible for all of that stuff, so we don’t need to know about it.” That has been the general consensus to this point in time. But, as I mentioned earlier, in many plants support of the automation systems falls on the shoulders of the I&C staff – at least partially because they fought long and hard to keep the IT people from touching those systems. But there may not be IT support staff available on a 24/7 basis, so unless a problem happens during normal working hours it will be up to the I&C people to take the proper corrective actions. Not understanding about trunk ports and VLANs could result in an engineer mis-wiring a replacement switch, or not making necessary VLAN settings, and having communications blocked between system components.
For example, I was recently at a plant where the I&C engineers described a strange and transient problem with their network. They were making Ethernet wiring changes to add a new switch in another plant area when, for no reason, they could understand, the communications all stopped for a few seconds, and then recovered. They had been able to recreate the event, but could find no hardware reason to explain how changes made in one plant area could cause the whole network to be impacted. When I asked about Spanning Tree they told me that they didn’t use that vendor. (Maybe now you catch my drift about needing basic IT skills?)
Sure, you can have a lot of carefully written and tested procedures that could take that same engineer through the steps needed to replace a failed network component without their having to understand the steps. Or, you could rely on the backup/redundant system elements to keep you operating until the IT support staff shows up. But I personally feel it is best that the I&C personnel receive essential IT training.
I am of the same opinion as it relates to having I&C people understand the concepts of cyber security and all of the fancy technical controls and countermeasures that IT will sprinkle around a plant to make it secure from cyber attack. One of the challenges for I&C personnel – and even for some IT folks – is the mind-numbing array of meaningless pseudo-technical marketing gobbledygook that is used to “explain” and “differentiate” various kinds of techno-toys. Believe me, I’ve read and re-read those so-called technical brochures on high-tech products for equipment such as intrusion detection and prevention systems and walked away feeling that I had no idea what they did, how (or if) they worked, or how they could (and could not) be applied. Although I wasn’t entirely sure what they meant by a ‘platform’ from reading their literature, I sure knew what kind of ‘per-platform’ licensing fees they wanted to be paid!
In their rush to bring forth their techno-panaceas and capture the market, while cyber security is still the hot topic of the day, some companies are doing a disservice by letting their sales and marketing staff explain their products to the masses. From my viewpoint at least, they are doing a poor job.
In some of my ISA courses we have reviewed marketing literature for cyber security products, and it is always interesting to see the class reaction. For example, I’ve had students tell me that if we had not discussed what a firewall did and how they generally work, they would never have come away understanding it from reading the sales literature we reviewed. In my opinion, any I&C engineer that can’t come to a course and is left to figure it out on their own by reading vendor product literature has a great challenge ahead of them. One would think that by attending one of the endless parades of on-line product webinars you could gain the requisite knowledge and understanding. In my role as a contributing editor, I’ve signed up and participated in quite a few during the past year.
And while we’re talking about professional associations, you may be interested in a new ISA certification activity that was launched as an offshoot of the activities of the ISA-99 industrial cyber security committee and working groups. (I should note for transparency purposes that I participate as an observer on that committee). They have established a set of manufacturer/ vendor requirements and testing through which a product can be certified.
Quoting from the ISA website: “The ISASecure designation is earned by industrial control suppliers whose products demonstrate adherence to an industry consensus on cyber security specifications for security characteristics and supplier development practices.” To begin with, they seem to be focusing on instrumentation and controllers with their Embedded Device Security Assurance (EDSA) certification, which can be obtained in one of three (3) levels of increasing security assurance. But in order to obtain certification for a specific product, the manufacturer must submit to a Functional Security Assessment (FSA), a Software Development Security Assessment (SDSA) and to Communication Robustness Testing (CRT).
Although this is a fledgling program, and there will undoubtedly be issues and challenges, it is a good starting point in addressing the challenges of supply chain protection. (See my last column for a discussion of supply chain issues.) Of course we will have to see how rigorous they are in their software development security assessment process, particularly as regards embedded firmware that comes already installed in basic building block integrated circuits. If that software isn’t being included – and I don’t know that it isn’t – then the SDSA process is seriously flawed. But that will be the subject matter for a future column… Tim
About the Author
Dr. Shaw is a Certified Information Systems Security Professional (CISSP) and has been active in industrial automation for more than 30 years. He is the author of Computer Control of BATCH Processes and CYBERSECURITY for SCADA Systems. Shaw is a prolific writer of papers and articles on a wide range of technical topics and has also contributed to several other books. He is currently Principal & Senior Consultant for Cyber SECurity Consulting, a consultancy practice focused on industrial automation security and technologies. Inquiries, comments or questions regarding the contents of this column and/or other security-related topics can be emailed to Tim@electricenergyonline.com.
