SCADA Control Room Management

B

Blunier, Mark

Unfortunately, you are comparing apples to oranges. In your Linux distribution, with several different versions of a specific application, take an FTP server for instance, if a couple of distributions have a couple of different ftp server packages that have a vulnerability, you're counting that as 4 bugs, even though only one FTP server is installed. In other cases, such as in web servers, a vulnerability you'd count against Linux would not count against windows, as the windows OS does not include a web server. If I were to release MBWS (marks buggy web server) for both Linux and windows, would it be fair to bash Windows for vulnerabilities in the windows version of MBWS? That's what you're doing to Linux.

I don't see how you can even suggest that they are similar. When Linux gets a vulnerability identified, it gets fixed. When Windows gets a vulnerability, its 'fire walling' and virus protection software gets updated, and then later if we're lucky, the software that is vulnerable gets fixed.

As far as the original thread goes, it doesn't matter what the title is of the person is that is managing the systems. The person should be competent in technology required for the work. Unfortunately, both IT and Maintenance have plenty of people that aren't competent in their traditional areas, that to expect them to be competent in both is unreasonable. But there are a few exceptional people that could do both competently.

Mark
 
How about thwacking yourselves? End-users have enormous influence with DCS vendors. If you don't think so, go sit in on a Honeywell User Group meeting, where the user group itself designs and commissions products, and has over 25 engineers working directly for them.

The same is true for Yokogawa, Invensys, Emerson, and so forth. End users have stroke, money, and feet to vote with. And they use them.

The fact is, Michael, it is easier to blame Microsoft or your least favorite DCS vendor for poor security practices on the part of end users and system integrators. Microsoft and the DCS vendors have their problems, it is true, but you're acting like there is no responsibility to be accountable on the part of the end user. That's not right.

Sure, it would be nice to have the feature you describe...have you asked a DCS vendor to supply it? If you haven't, then shame on you.

The debunking is coming unraveled. It was easy to thrash MS in the general computer press until OS X started exhibiting similar vulnerabilities, and xNIX distributions have always practiced your security by obscurity.

I completely agree with you that anybody who thinks security by obscurity is going to save them is foolish, and likely to do serious damage or even kill people from neglect of following some very simple rules.

We are all going to have to re-think what we do as far as security is concerned.

Walt Boyes Editor in Chief Control magazine www.controlglobal.com blog:Sound OFF!! http://waltboyes.livejournal.com
_________________

Putman Media Inc. 555 W. Pierce Rd. Suite 301 Itasca, IL 60143 630-467-1301 x368 [email protected]
 
N

Nathan Boeger

Walt,

Microsoft does not currently produce bad software - they have a large enough economy of scale and development team to produce software of a quality where it's rare for a user to find significant bugs that affect productivity - the same is true of many large open source projects.

Industrial software, on the other hand, is written by much smaller companies for a much smaller industry that has orders of magnitude less volume. Developers are then pressured to release the latest technologies AND support legacy systems - a tall order. Take any HMI system from 10 years ago and try to get it working on new machines - good luck. My point is that it's commonplace, it's industry standard as "normal", to tolerate lots of version updates and hotfixes to be necessary to keep up with the times. Software that randomly crashes without a patch or has tech support telling you to reinstall Windows to fix your problem - This can seldom be said of commercial software. For example, when SP2 came out for XP, most major vendors issued instructions to NOT INSTALL IT until they figured out how to deal with it (in fairness Microsoft screwed them over with the default Windows Firewall settings). Similar problems are commonplace with DCOM settings, problems when installing different combinations of software (RSView and MS Word), etc, etc. If you think that the current generation of industrial software programs are up to par with commercial software - and they should be more stable given the criticality of their function - I would argue that your experience is out of touch with reality in most of the industrial world - talk to industrial integrators or manufacturing managers about the stability of their PC based control systems.

I think that you assumed a lot more than I meant because I used the term "shoddy". By that I meant that industrial software is seriously sub-par to where it should be, specifically with respect to normal computer workstation maintenance from IT - a heated topic that has caused many controls guys in this post to question the competency of IT departments in general. My point was that IT still possesses the expertise to manage the system - they just need to know what pitfalls to avoid and be keenly aware that when dealing with control systems, a running process is #1, security and updates come at a distant second.

My other point is that industrial systems should utilize developed (commercial or otherwise) software when applicable. For example, Microsoft SQL Server is a solid platform - great for commercial or industrial use. Why were most industrial software companies logging data to flat files or their own proprietary formats for so many years until relatively recently? Who knows, but the trend to move to a more stable platform makes too much sense. And IT would love to support that SQL database. Why do so many industrial software companies insist on writing their own pieces and plugins for their packages when they do a second rate job compared to existing technologies?

----
Nathan Boeger
Inductive Automation
"Design Simplicity Cures Engineered Complexity"
 
N

Nathan Boeger

Dennis,

I certainly agree with you on this one. I've seen it happen far too many times that IT person innocently tries to patch an industrial production machine without having any idea of what could ensue. Like you said, cooperation's really the way to straighten out this sort of thing. Once IT understands the systems and priorities, they tend a knowledgeable resource for this arena.

----
Nathan Boeger
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
N

Nathan Boeger

First, I'd like to say that Michael Griffin's 14DEC 06 post is dead on.

Walt, I have to give it to you on the pizazz and elocutionary flair. Unfortunately on this last post I'll have to take exception to certain statements of yours.

"xNIX distributions have always practiced your security by obscurity" - Where did this come from? Please elaborate. Unix communities tend to provide source code for their security implementations, relying on algorithms that will stand up to an attacker who knows the system, the POLAR OPPOSITE of security by obscurity. Much details of the Windows kernel and security implementation are closely guarded trade secrets at Microsoft - providing a level of security by obfuscation. Maybe their code's solid, who knows? Ask someone that work for them.

Windows 9x and before were designed as single process applications without security in mind. Windows NT was designed to be a secure system, with many provisions that weren't actually implemented. Over time, Microsoft's been doing better and better. Vista should be pretty solid at the core. My interpretation is that Michael and Mark's qualms relate to the way the Microsoft deals with its problems - they choose to play the PR game and sweep it under the rug, or best case hotfix, where xNIX communities tend to release the problem and fix. Since the source code is available, the fix is open to pubic scrutiny.

> How about thwacking yourselves? End-users have enormous influence with DCS vendors. <

Should we blame ourselves? As integrators, engineers, and opinion leading writers, probably. As end users? Give me a break. End users understand computer security implementation details in even more limited scope than you. That's why they pay the big bucks to have the "experts" implement it. How can they be expected to correctly influence vendors in this regard?

> I completely agree with you that anybody who thinks security by obscurity is going to save them is foolish, and likely to do serious damage or even kill people from neglect of following some very simple rules.
>
> We are all going to have to re-think what we do as far as security is concerned. <

You're absolutely right here. SCADA security is currently almost non-existent from a real computer security standpoint. Obfuscation won't save a system from even a novice attacker. I think that the responsibility should fall on vendors, integrators, and finally, end users.

----
Nathan Boeger
Inductive Automation
"Design Simplicity Cures Engineered Complexity"
 
N

Nathan Boeger

Dennis,

That answer is convenient for small systems, but what happens when you grow to a large distributed system? Is the HMI guy going to learn how to program Cisco routers? What about when your customer gets into MES/ERP systems as all the major vendors are beginning to promote? Is the PLC programmer going to become an Oracle expert?

IT people should already be PC experts, more so than controls engineers. Of course they need to learn about the software they're supporting before you unleash them on your SCADA system! Why not work with them?

----
Nathan Boeger
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
M

Michael Batchelor

OK, here I am ranting again.

Touche' on this point Walt, but flavor of OS aside, how do we get the organizational structure of companies changed to get rid of the war between IT and Maintenance?

That point is a hundred time more important than what OS the stuff runs on.

MB
 
I have taken to bringing a lawn chair and a magazine or two with me wherever I go. The emphasis on security has been detrimental to the automation business. The measure of a control system's value is correctness and availabiliy, not security.

Security is important, but so is making widgets. An unsecure control system has the potential of failing, lowering both its MTBF and availablity. But an overly secure control system has the potential of being time consuming to repair, raising its MTTR and lowering its availabilty sharply. (Just in case, the definition of availability is the MTBF divided by the sum of the MTBF and the MTTR.)

The IT people I have dealt with, by and large, seem to be highly competent WRT MTBF, but are oblivious WRT control system MTTRs. A meeting is required for anything and everything, there's always someone "who needs to be there" that is on vacation or too busy to attend. Thus, the lawn chair and magazines.

My recommendation, for now, is to regard enterprise IT departments as a valuable service provider only, much the same way we look at the phone company or an ISP.
 
M

Michael Griffin

In reply to Walt Boyes - Not everyone can attend user group meetings. It also helps to discuss problems in public ("thwacking vendors") to build a consensus amongst users as to whether there really is a problem, and as to what the possible solutions may be. If software vendors are not reading this and other mailing lists, they are missing out on important information about their market. No doubt the user group meetings help in getting more detailed feedback, but these will always have the disadvantage of representing a narrow selection of opinion.

As to whether "the debunking is coming unraveled", I don't want to turn this into a debate on the security of different operating systems. This would simply be duplicating detailed discussions that have taken place elsewhere in the general computer press.

Please note though that what was being "debunked" was the explanation of *why* there are more security problems with MS-Windows than with other operating systems. I was responding to your comment on "if you look at the size of the distribution of MS apps vs the size of the xNIX apps, if there were as many Linux boxes as there are Windows boxes, we'd all be complaining about how shoddy Linux is, or how virus and trojan vulnerable OS X is". The premise of your own argument is that MS-Windows has more security problems, and your explanation for why this is so is an old argument which I said has been debunked. If you would like more details, the following article explains it rather well.

http://www.theregister.co.uk/security/security_report_windows_vs_linux/

I would also point out that Microsoft is not the only software vendor with a poor security record. Oracle has an equally poor reputation in this regards. This seems to be a common failing among very large software vendors who dominate their respective markets. Competition seems to be the most reliable means of getting companies to respond to problems, so companies that don't feel that competition very strongly tend to avoid taking difficult measures to correct those problems.

The relevance of the above to SCADA and MMI systems is that unless customers consider security to be a priority and make it a criteria in their purchasing decisions, the vendors have little incentive to devote resources to security instead of on developing new features. Many vendors have been counting on the fact that SCADA systems are relatively obscure and feel that this affords a degree of protection ("security by obscurity").

We discussed SCADA security earlier this year under the topics

HMI: COMM: An interesting article on a SCADA security vulnerability.

HMI: SCADA Security Developments

These discussions were based on articles in the computer security press about vulnerabilities in SCADA systems and how new legislation in some countries is requiring plant owners to address this problem for "critical infrastructure".

I will repeat one of my own conclusions from that discussion, as I think it bears further consideration. The SCADA security studies pointed out that the security of the SCADA application itself is only one element in overall security, and in many cases, only a minor part of that security. The security of the operating system, database server software, and other third party applications are equally or even more critical.

An operating system which is in a configuration used for typical desktop applications will not meet the security requirements laid out in the studies. If it is possible to configure a system to meet these requirements it would require more knowledge than all but a very few SCADA integrators or plant operators possess.

I think that SCADA vendors have a role to fill here; and no, it doesn't involve standing around and getting "thwacked". While every SCADA application is unique, the software components and configurations are common to most. The problem is that for most SCADA systems the application integrator is expected to purchase various third party components separately and to put them together himself (hopefully doing it securely). The plant operator is then expected to maintain that system (while supposedly keeping track of all applicable security threats). The SCADA vendor doesn't take responsibility for anything that doesn't come on the CD they supply.

An alternative approach would be for the SCADA vendor to supply a customer with a CD that contains *all* the software they are likely to need for most SCADA applications, and to support *all* of that software (via a service contract). That would include having it install by default in a secure configuration, and supplying all updates from a single source. They would supply and take responsibility for not just the actual SCADA software, but also the operating system, the database, and any other software that a typical system would require (web servers, programming languages, etc.).

If that sounds impractical, it is worth pointing out that there are companies that do precisely that for critical business systems. They do not create most the software, but they package it together and support it for paying customers. That is in fact the actual business of most Linux distributors. The operating system itself is only a small part of what they deliver and support.

I don't see why SCADA vendors couldn't pursue a similar strategy, whereby they package their proprietary SCADA software with all the third party components necessary and then take responsibility for the system as a whole. If some customers needed to go outside the supported configuration they would still be free to do so, but they would be on their own as far as supporting those changes are concerned.

This doesn't mean using a proprietary OS on proprietary hardware. The customer would use common PC hardware purchased from any qualified vendor, the OS would be a readily available one, as would the other required software. The SCADA vendor would strip out software components which are not needed for SCADA applications, and configure the remainder in a reliable and secure manner. They would then supply additional software to manage installation, configuration, and downloading updates. They would operate a software repository that would provide all updates from a single source and notify their customers of any problems.

This would be a value added business for the SCADA vendors that would let integrators and plant operators concentrate on their application, while letting the SCADA software vendors take care of the complete software application platform. There are I believe some specialist SCADA vendors who already do this, but I don't know why this shouldn't be the rule for most SCADA software rather than the exception.

Notice I haven't just "thwacked" the SCADA vendors, nor have I thrown my hands up in the air or pretended that there is no problem. I think the above is a viable solution for both vendors and customers. Unless that is, nobody is really all that concerned about security. I think that is in fact the case today, but that could change in future. The SCADA vendors who have put themselves in a position where they could take advantage of that change when it comes may be the ones who survive while the rest go out of business.
 
M

Michael Batchelor

OK, I'm still on an organizational rant here. I don't want to turn this into a debate about security differences between two, ten or fifty operating systems either.

I *DO* want to turn this into a discussion about how to effect the organizational changes necessary to solve the problem of the continuous struggle between the IT staff and the maintenance/production staff.

Even if I had a magic wand and could change every computer inside Organization "X" into a computer running OS distribution "Q" I'm still going to have the struggle between the competing needs of the Control Room HMI and the receptionist's email "terminal" and the Corporate mainframe in the glass house.

What can we do to address the organizational issue? I think this is far more important than the OS.

Michael Batchelor
www.ind-info.com
 
M

Michael Griffin

In reply to Michael Batchelor - If the control network used RS-485, we wouldn't even consider the possibility that IT should take care of it. Just because it happens to use ethernet shouldn't change any conclusions we may have about who should look after it. The answer would seem to be to have separate IT and control room networks and let each department look after their own.

That doesn't mean that people in different departments shouldn't help each other out when they are in difficulty. It does mean however that when you buy some fancy equipment you should make sure that your own people can look after it. Moving the responsibility for a combined network from IT to the Maintenance department doesn't change the problem, it just changes who is complaining about it.

The real problem should come when you want to exchange data between the two systems. For things like having an MRP/ERP system get data from production equipment it's probably a good idea to have an interface "box" that intermediates between them rather than having the MRP/ERP system reach directly into individual machines. For things like allowing test equipment to back up test data to the IT file servers, it's a good idea to set up a special working relationship for those specific cases. In these types of applications though, you have limited and well defined interfaces that both sides can come to an agreement on how to deal with.
 
Michael

I work for a consulting engineering outfit which would never allow a supplier or integrator to define the security of a system. The specification defines the requirements and the contractor bids to complete the job WRT the specs --security is defined in the specs. We would be a fool not to include security in those specs. The security runs the whole gambit of software, hazardous areas, employee safety and public safety all of which are in the realm of consulting engineers and we try to protect these areas.

Dennis
 
N

Nathan Boeger

In response to Michael Griffin's post:
I appreciate the general insight that you provided on the above post (16DEC06 9:27AM). Your points about the relevance and issues relating to SCADA security are dead on.

I wanted to address your "Alternative approach" for SCADA vendors - supplying and supporting all the necessary software like Linux distributors. I think that's a good idea in terms of leaving the responsibility in the hands of a group that should be able to handle it (integrators tend to be too small and specialize in controls and end users tend to lack the expertise). Inductive Automation goes part of the way toward what you described. Because end users often have such different standards and considerations, we support many databases (MySQL, MS SQL Server, Postgres, Oracle, DB2, etc, etc). Similarly we'll work with pretty much any OPC server to talk to any kind of PLC. This makes securely locking up and bundling a single package impractical. However, we do support integrators on these installations (a free service for integrators) and provide direction in terms of implementation/security. We recommend that they work with their IT department to figure out their best implementation and secure it. We also support the sensible use of 3rd party applications or consultants.

----
Nathan Boeger
Microsoft Certified Systems Engineer
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
N

Nathan Boeger

Kirk,
I have had similar experiences with some IT departments. More recently I have been recommending that integrators approach IT prior to implementing a project and include them on the relevant details. You will find that the details can be established beforehand, preventing all the meeting where they may have been trying to stall on you. I hear such experiences as "they just set up a computer for me, everything installed, patches, automatic backups and all, and it was ready to put the control software on"

I hate to be a nay-sayer, but SCADA security is often taken way too lightly. Think pre-911 airports. I remember a vendor at a gun show bragging that it was nothing to get a stun gun through airport metal detectors. How many incidents do you think it'll take before SCADA security will matter? Think ammonia systems, water treatment plants, power plants, etc. Protection from a knowledgeable assailant is practically non-existent.

This should be the responsibility of all - vendors, integrators, IT departments, and end users. Designing a secure system doesn't have to equate to a long mean time to recovery.

----
Nathan Boeger
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
M

Michael Batchelor

In a different life I worked in IT, and I had far more RS-232 links to worry about that I've seen in most production facilities. OK, so RS-232 isn't RS-485, but they're truly cousins. An had one of the non-IT guys wanted to monkey with it, he would have been stopped.

I see your point, but I think it's deeper than that.
 
M

Michael Griffin

In reply to Dennis - There is a group called the "SCADA and Control Systems Procurement Project" who are a group of large users of SCADA systems. They are busy writing a set of standard contract language intended to be inserted into SCADA purchasing contracts to cover security issues. I have mentioned them before, so before replying to your message I thought I would have a look at their current draft. I found the following contract language which I had not noticed before:

"Post contract award, the vendor shall provide notification of a known vulnerability affecting vendor supplied or required OS, application, and third party software within a negotiated period of time after public disclosure. The vendor must apply, test, and validate the appropriate updates and/or workarounds on a baseline reference system before distribution. These steps could be a subset of FAT testing. Mitigation of these vulnerabilities should occur within a negotiated period of time." (section 2.6.2 - draft version 1.5).

Note they intend to have the *vendor* be responsible for the "supplied or required OS, application, and third party software". If a vendor's software requires a third party OS, this group (and anyone else who follows their recommendations) are going to require the SCADA vendor to take responsibility for the security of the OS (and other software) and that responsibility will continue after the sale and installation.

It's nice that you include software security in your specs, but do you take responsibility for the software for the life of the plant? Do you continuously analyse the security threats and test and provide the application, operating system, and other third party security patches? Most consulting engineers are on to the next project after the plant is up and running.

What I was referring to was a software vendor business model similar in intent to what the quoted text above is stating. That is, a business relationship that doesn't end after the initial sale.

For this to be feasible, I think the vendor should provide all the software. This doesn't mean the vendor has to *write* all the software (other than the SCADA package). These days operating systems and databases are commodities so it would be foolish for them to write their own. I think though it all needs to go through the vendor's hands so they have some control over what they are supporting and can strip out anything that isn't necessary for their customers.

It will be interesting to see how SCADA vendors respond to this requirement and how well their solutions work.
 
Michael

The answer is -- sit down and talk to the management and IT people.

You may be surprised where it will lead. They all have the same interest (the company) at heart.

Dennis
 
B
One of the things that has never failed to amaze me is that an organization may have numerous procedures and policies regarding the need to analyse any proposed changes to operating plant to the nth degree, but can totally ignore the impact of anything that does not involve cutting metal.

The need to make any modifications at all to SCADA related hardware can have a major impact on production. Even in a well-structured system with all control done at the field end and the SCADA performing only supervisory actions, loss of a server will ultimately affect performance - if it doesn't why have the SCADA in the first place? I'm sure many of us have been in situations where even adding the latest service pack upgrade has impacted on operational software.

(Incidentally, about 20 years ago, I was responsible for the instrumentation of a small methanol plant - including the "Process Computer" which was the nearest we got to a SCADA system. There was also a "business" computer system which was independently managed, and was eventually PC-ised. When the organization got around to replacing typewriters with word-processing software on desktop PC's, there was a major bust-up between the head of the typing pool and the IT manager over control of these machines. But because the plant computer was a mainframe there were no issues at that end of the plant. The IT manager also managed to seriously p%#$ off the drawing office guys and most of the engineers by disestablishing their very functional and home-brewed drawing data base system and insisting on writing his own - 5 years later it was just about up to the same level of functionality as the original.)

This issue is one of organisational management but all involved need to be very aware that any change to anything connected to the plant can have major and unforeseen effects on production and safety. This needs to be appreciated by all involved, from the CEO to the HR group to purchasing to the cleaning service. ANY change must undergo a formal review and sign-off process involving any affected discipline.
 
C
That's pretty simplistic. And a quote from the party line. Considering that the source is available for most of the *nix variants, it's several orders of magnitude easier to find (and fix) vulnerabilities as there are many orders of magnitude more eyeballs on the code. Think for a moment how many more vulnerabilities would be found if the Windows source were available for all to see. And consider for a moment how many fewer vulnerabilities exist after all that scrutiny. And then there is the "exploitability" factor. Those that are left in say, Linux, are much less open to exploit than those in Windows. The relative number of boxes is a far smaller factor as is evidenced by the number of _successful_ exploits, economic damage, business disruption or practically any other metric. But, yes, I will easily find more typos in the book I have read, than the one that remains closed. No matter how many copies of each I have.

Regards
cww
 
Top