SCADA Control Room Management

C

Curt Wuollet

In answer to the last question, on a private network there are few non-obvious vulnerabilities. Open that network to the World and while vulnerabilities can be limited, it will need management. Automation folks tend towards set it and forget it. But it is several orders of magnitude less likely to be compromised than simply plopping another Windows box on the net.

Regards
cww
 
C

Curt Wuollet

Hi Brian What you describe sounds a lot like general office use and the problems there most certainly are different than setting up a particular application on a system where the purpose is to run that application. Like a SCADA system. Once you solve the problems for that, they should stay fixed.

I have done office settings as well and they are full of headaches, with either OS. But switching to Linux has impediments like hardware that only works on Windows, people who will not accept any change in how they do things and people who want non-business related software on "their" machine. The best way to deal with these is to solve them before you begin and get management on board so you can say NO until the dust settles a bit. One of the biggest problems I had was a saboteur who worked hard to wreck things because MS Golf wouldn't run anymore. It will all work out if people want it to work.

Regards
cww
 
C

Curt Wuollet

Exactly!
Any number of schemes could be used, the only trick is to get any two vendors to use one. I don't think OLE was chosen because of it's merits, but because it was all that was provided.

Regards
cww
 
D

Davis Gentry

Since I first logged onto the internet in 1983 (through a Commodore 64 talking to Compuserve) I have had a virus or worm only a couple of times - and not once in the last decade on any PC for which I am/was responsible. My kids corrupted their PC with adware a year ago, and I took away some of the privileges for their login (write access to the registry mainly) and we haven't had that problem since then. The rest of the computers on my home network repulsed attempts from the adware to load. I've done this by consistently following a few simple rules which I have also followed on all installs for machines in the field, of which 80+% have been MS of some flavor:

1) Keep antivirus up to date - manually, as I don't like auto update 2) Turn off unused services 3) Even with the AV running don't do something stupid like run the .exe or .gif with the latest porn/joke/whatever on it.

Keeping AV up to date can sometimes be simplified by bringing the IT guys on board. I am in dozens of plants each year, and I've seen everything from warfare between IT and Engineering to a nicely symbiotic relationship. If you can maintain a good relationship with IT your life can be much easier. Bring them in to look at your stuff, show them why they can't just delete files from a production machine to save space (yes, I've had that happen), basically try to make them part of your team. If they are at all reasonable (maybe yes, maybe no) and you can show that you have some expertise in their area but that you are not trying to exclude them from an area where they do have some responsibility (does your data go out onto their network? If the answer is yes then they do have responsibility) then you can often work together. If not, wait for them to screw up and go over their heads.

It is also an oversimplification to suggest that IT guys know only MS OSs. I would say that just as many IT guys know other PC OSs as do engineers, maybe more. After all, many of the IT types are running web servers which are often Apache.

I agree that MS has a bad habit of turning things on which should be left off. If you do updates manually (or if you coopt your IT guys into doing it for you) then you can handle this. Or run Embedded XP and load only those things that you want.

Michael Griffen has some great points with regards to having a SCADA/HMI package which by default sets things up to a secure status. It has been a while since I used any package other than Visual Studio or Labview to build an HMI, but a good while back I used Wonderware and it defaulted to a secure setup at least from the standpoint of password access, including across the network. What is the status of the many packages now available? Do ANY of them handle AV and internet security? Would certainly be nice if they did.

Davis Gentry
 
in reply to Michael Griffin's observations
> Note that this isn't an "MS versus Linux" argument.

I differ. Mr. Wuollet's pointed comment on tithing to MS reflects religious struggle going on here. The High Priests of MS are making lists of who is clean and who is unclean.

> The problem which is arising is that the 64 bit version of the new version of MS-Windows ("Vista") will require drivers to be cryptographically signed by Microsoft before the operating system will load them. To get the drivers signed by Microsoft will require that the driver author is listed as an "approved vendor" by Microsoft. Microsoft has stated that they will only approve a limited number of vendors, as they need to keep the vendor list small enough to be manageable. >

It will be interesting how this situation develops, given that XP will load unsigned drivers after the requisite warning notification (Repent, the end of world is at hand!).

If, in fact, unsigned drivers will not load or function in Vista, then MS moves outside the orthodoxy of "open architecture" that IBM brought to the PC world 25+ years ago. If an O/S excommunicates unsigned hardware drivers, then hardware standards are only meaningful for the formally ordained, because hardware without a loadable driver is a non-entity, merely an unclean leper to be banished from communion with those holy (big & rich) enough to be confirmed into the faith.

Could Linux be walking the road to Jericho, the Samaritan ready to aid the beaten man, one of "these 3 who was neighbor to him who fell among the thieves"?

Daniel
 
Michael,

I would love to see my life modified by this document, even though it is a draft, (I don't think it will happen in my lifetime -- to many preliminary legal battles to go thought to set precedence). The constant reminder of inaccurate sight supervision ( Suncor law suit for example) will keep my mind busy for years lest I fall into the same battle. My goal is not to follow Suncor's engineers path but to perform due diligence -- This spec helps but it is not a complete solution (all is still in flux). Maybe it will be refined with time and experience.

Dennis
 
B

Brian E Boothe

I've been following this thread for a while now, and it seems that a lot of IT Organizations have to many Tightly Wound, Flat 1 dimensional People Running the show, in not just the Control Industry. I can say I know for a fact they have no idea about control issues or Automation software. I'm basically the opposite of that Scenario you see in use in most IT organizations and Offices. IT and Control Should merge into 1 Application. And if the Administrator wants to check his/her internal Company email And Check production on the floor he/she can do it all on 1 machine in his office, thru a Spam filter gateway/router Combination w/ Logins, all on a local intranet with segmented IP addresses.

IT Guys, stop being so anal about everything, by the book isn't always the way to follow.
 
> It is not true, in my experience, that vendors produce either shoddy programs or support them badly. Nor does Microsoft produce bad software. <

No, Microsoft produces fine software for their market. The problem is that market is not the industrial controls market. Windows is fine for the average home or office user, plenty of assists, everything nicely packaged, "easy" to run, lots of multimedia support, etc. But that is not what we need for controls. We need small, tight, tailorable systems that have only what is needed and nothing more. Not bloatware.

> Before everybody from the "Church of Kill Bill" leaps on me again, let me point out that CERT shows a large number of vulnerabilities in every distro of xNIX from OS X to Linux. If you look at the size of the distribution of MS apps vs the size of the xNIX apps, if there were as many Linux boxes as there are Windows boxes, we'd all be complaining about how shoddy Linux is, or how virus and trojan vulnerable OS X is... <

Yep, remember, the Linux kernel was written by one guy, a college student at the time. It is of questionable quality even now. The problem is an intractable one of testing coverage. Even Microsoft with all the talent and tools at their disposal can't get it right.

> The issue, as Joe Weiss from Kema and I were talking about on the phone last night, is not really one of coding. It is about policies and procedures, auditing and training. <

Policies, procedures, auditing and training attempt to make up for the fact that the software is not 100% tested. Limit how it is used and check up on everybody and maybe you can get away with sloppy code.

> I read somewhere that on the order of 40% of all control systems still have "password" as the root administrator password years after delivery, and the "guest" signon is still enabled in more than half.
>
> You can't legitimately thwack vendors for that. <

Yeah, you can. Those items plus many others are configurable from within the product. The problem is that most vendors would rather use ignorance than responsible design decisions because it is cheaper in the short run. A good design would mandate the user enter a "strong" password, enforce mandatory password changes, and not have a guest account at all. Systems were being designed and delivered back in the 1960s that were secure, because they were designed to be secure and to take into account the inherent insecurity of people. Most of this work by the U.S. government was categorized into the "Red Book" (I believe) that listed the various security levels and what it would take to be certified to a level. Granted, it took specially designed systems (hw and sw) to reach the upper levels, but it could and was done. The modern computer industry is too in love with flashy toys and simpleton marketing departments to focus on the meat and potatoe issues.

Mark my words, it's going to take a cyberattack by some foreign government that takes out a refinery or other large chemical facility with a large loss of the surrounding population, then watch the government crack down. I can see where inherently dangerous facilities will be outright banned in this country because the populace won't want the risk of having them around. Heck, how long has it been since we've had a new refinery, not to mention a nuclear power plant? Better than 30 years now.
 
You and I are going to have to disagree about whether the end-users are part of the "Stop me before I kill again" school of security management.

Yes, vendors could mandate strong security. None of them do, AND THE END USERS AREN'T ASKING THEM TO.

I guarantee you that if a major end-user asked any of the two dozen or so people that make HMI or optimization software to include strong encryption
and security out of the box, they'd do it, hands down, no questions asked.

But security costs money, just like whisky, and talk remains inexpensive.

If you missed my nightmare, I happen to think you're right. It is going to take a cyberattack that brings down a refinery or bulk chem plant to get people to put money where their mouths are.

And then, you have to ask yourself if the money is going to be well spent, or if it will just be another instance of "security kabuki" like airline
passenger security is.

Walt Boyes
www.waltboyes.com
[email protected]
630-639-7090
 
M

Michael Griffin

In reply to Walt Boyes: If you look back to some of the earlier posts on this precise topic, you will see there was mention of the "SCADA and Control Systems Procurement Project" (http://www.msisac.org/scada/). This is an organisation representing end users who *are* asking for better security, and are coming up with specifications to try to get vendors to provide it. The present technical and business models of most vendors are however not able to provide it at the present time.
 
B
Hi Walt,

Most organisations seem to rely excessively on passwords as the front line of protection. This is about the only one that is available as a general rule. The problem with the "password" model of security is that it is extremely difficult for Joe (or Jo) User to comply with all the requirements: Make it non-obvious and don't use personal information Don't write it down Use a strange mixture of lower and upper-case letters, numbers and characters. Don't use the same password for different systems Change it every month, etc.

If you are dealing with more than one or two isolated systems you would need to have an exceptional memory to do this - and if you have to log on to a system once a month or so then forget it. A notebook PC used by the on-call tech to access a PLC has to have a password known to the whole maintenance team. I have seen a research paper (just had a quick look with Google but can't re-find it) that made the point that banks seem to do very well with 4-digit PINs - why can't software vendors do something the same?

The other issue is that software system vendors are trying to use internet access as a selling point, but this opens up a whole lot of cans of worms as regards security. This has always been a problem of remote monitoring systems but things have changed a bit since we used to phone up the telemetry number of the local waterworks to find out how much the reservoir level fell during the half-time break in the big football match. Surely the first point that needs to be checked in any attempt to deliberately open up a system for remote dial-up style access is its security?

The parts of the user's organisation that make these decisions need to take action - but the IT industry in generally seems to be blaming the guys and gals at the consoles for not following the rules. As instrument people, one of our first concerns should be to avoid any complications to operators that might hinder their access to information, and the password system as currently implemented is one of those complications. In just about every other activity we are trying to reduce stresses on operators so they can focus on the ob of running the plant. Cheers,

Bruce
 
M

Michael Griffin

In reply to Rich Wargo: There is a good deal of academic research being put into "micro-kernal operating systems". Some of that is focused on producing an operating system kernal that is provably correct. That is, the design can be subjected to formal proofs which can determine if the design will meet the objectives. A micro-kernal is not synonymous with "provably correct" however; most micro-kernals are not written with this in mind.

Despite their academic popularity, micro-kernals have been much less popular in commercial applications. The only reasonably successful one which comes to mind is QNX (Mac OSX was based on the Mach kernal, but had a huge monolithic chunk grafted onto it so it's not a micro-kernal). Micro-kernals have a number of practical disadvantages (particularly high system call overhead) which have limited their use so far despite their theoretical appeal.

As far as more conventional operating systems are concerned, some of the problem with relying on in house testing is that you only end up testing the problems you expect to have. There is really no substitute for getting feed back from as many real users as possible. The more people you have using a piece of code before it is formally released, the more likely it is that any problems with it will turn up before it is used in serious applications.

None the less, I agree with your point about "small, tight, tailorable systems" for industrial controls applications. What I think this requires is a modular (but not necessarily micro-kernal) operating system based on widely used components. Most modern operating systems would in fact be capable of being readily adapted to provide this. The exception may be Microsoft Windows (since everything in it is integrated with everything else), but as you said in your original message, they have a different vision of where their market lies.
 
I guess the relationships between IT and Industrial Software Engineers will improve with time, when the rolls are more clearly defined, this will help.

If IT management quit their 'holier than thou' 'I’m the king of the domain’ attitude toward everything, would also help to improving this relationship. But I don’t see that happening soon, because to what I’ve seen, that’s a prerequisite to getting into IT in the first place.

Dennis
 
N

Nathan Boeger

Walt,

Great points! SCADA security is certainly something that hasn't been taken seriously on a wide scale. And you're right, vendors don't do much about it, end users don't ask, integrators go with the flow, and the government doesn't seem to be doing anything effective. I certainly hope that this gets sorted out before rather than after we have to regret it.

To add to your point about end user complacency, I've experienced end users FIGHTING security and standard IT technologies because of the natural trade off between security and usability. While strong security certainly doesn't have to get in the way of usability, many end users have experienced downtime due to security implementations - a big problem for them. I've spoken with many end users who would rather just have no security in place - for simplicity. It's a scary thought to me, but I certainly see where they're coming from.

----
Nathan Boeger
http://www.inductiveautomation.com
Total SCADA Freedom
 
Hi Dennis,

Please read past the first line... ; )

I'm a technical recruiter with an award-winning firm that recently was assigned to look for information about SCADA and RTU positions. On my own I have been seeking out any information I can find for some clients on mine. Any help or pointers would be greatly appreciated.

I’ve done some reading and the articles that this site and others have discussed seem very interesting. I’ve been an IT and Engineering recruiter for 10 years and have seen lines blurred in many roles regarding IT. Your posting sheds a little light on the mentality of the shops that I will be dealing with.

Please feel free to respond directly to my email if you can offer any help, msole @ sbcglobal. net.

-Marcus
 
W
With the utmost of respect, Michael, what you're saying is not really true.

It doesn't take a village, or a committee, or a project to get the suppliers to provide what you say the end-users want.

It takes a grand total of five specific people to do it. These five people command enough purchasing power that all of the first and second tier process automation vendors would fall over their own feet to install whatever security those end users wanted.

They don't seem to be asking.

Walt

Walt Boyes
Editor in Chief
Control magazine
www.controlglobal.com
blog:Sound OFF!! http://waltboyes.livejournal.com
_________________

Putman Media Inc.
555 W. Pierce Rd. Suite 301
Itasca, IL 60143
630-467-1301 x368
[email protected]
 
M

Michael Griffin

In reply to Walt Boyes: While it is quite possible that I am wrong about whether there is any customer demand for better SCADA security, I can point to the following information.

From the web site which I referenced (http://www.msisac.org/scada/): "The SCADA Procurement Project, established in March 2006, is a joint effort among public and private sectors focused on development of common procurement language that can be used by everyone. The goal is for federal, state and local asset owners and regulators to come together using these procurement requirements and to maximize the collective buying power to help ensure that security is integrated into SCADA systems."

The "leadership council" of this organisation is composed of the following members: NYS Public Service Commission, New York Power Authority, ConEdison, NY Independent Systems Operator, Central Hudson Gas & Electric, Municipal Electric Utilities Assoc. (MEUA), National Fuel Gas Dist. Corp, KeySpan Energy, Entergy, PSEG Fossil, LLC, NY Water Service Corp., National Grid, United Water NY, Rochester Gas & Electric, Dynegy, NYS Electric & Gas, Reliant Resources, Constellation Energy, Energy Association of NYS, Independent Power Producers of NY.

The particular organisation I have referenced above is in the US, but there are similar efforts in Canada and other countries who are cooperating with each other, as is evident by the list of members in the working groups.

The list of companies and organisations which are "Workgroup Coordinators" is much too long to be reproduced here in full (see the web site for more information), but I will list a few examples: Australia Department of Communications, Information Technology and the Arts, Bechtel, BP, Chevron, ConEdison Energy, Inc., DuPont, Electric Power Research Institute, Exxon Mobil, NYS Office of Cyber Security & Critical Infrastructure Coordination, Royal Canadian Mounted Police/Critical Infrastructure Intelligence Section, Shell, Siemens Power Transmission & Distribution, Inc., Suez Energy NA, Swedish Emergency Management Agency/Swedish Defense Research Agency, United Kingdom National Infrastructure Security Coordination Center, US Army, US Department of Homeland Security.

While I admit that I am not as familiar with the SCADA industry as you are, I would be interested in just what these people are all up to if nobody wants better security. When they say they wish to "maximise the collective buying power to help ensure that security is integrated into SCADA systems", I more or less took them at their word.
 
B

Bruce Durdle

M

Michael Griffin

In reply to Bruce Durdle: I had a look at these SCADA security documents from Emerson, and there is not as much to them as may appear at first sight. Much of it is just general IT security recommendations for PCs in general, or for configuring MS-Windows. It doesn't however tell you how to actually do any of the recommended general actions. It also requires you to source, test, and install third party security products.

In short, if you know how to make use of their advice, you probably don't need their advice. I would consider these to be some nice short documents that tell you that you may have a problem in terms that relate to a SCADA system. They do not however actually provide you with a solution.

I would agree that "at least one major vendor recognises that there is a problem". They however still appear to feel that it is up to the end user to provide a solution.
 
J
Dennis,

I think you are right on the mark. But to take the issue a step further, I am convinced the heart of the turf war problems between Control Systems & IT is not technical, but personal. Control Systems professionals typically hold (traditional - accredited) engineering degrees related to their field. Often these degrees are advanced and highly specialized. IT professionals typically hold 2-year technology-based "degrees." This is not to say that IT professionals are less qualified for their positions or have inferior knowledge of their field, but there does seem to be a jealous tendency toward engineering & controls professionals. Envy & jealousy are the root causes of the strife between IT and any other technically oriented profession.
 
Top