SCADA Control Room Management

C
Hi Michael, Nathan, etal.
This would indeed be a major improvement and it is doable. The very large number of Linux distributions available demonstrates that a company could "own" the Linux their package runs on and have far better control over the lifecycle of the product. Upgrades could be coordinated, and no one, not even a predatory monopoly, could simply declare all your hard work obsolete. This could be done with very limited resources as you can start with a good distribution and simply maintain it as long as you want. All necessary upgrades would be provided by the community and could simply be reviewed for applicability. Many "old" releases are well supported because there are still large numbers of people using them. The work needed simply to ensure their product continues to run well would be far less than what MS churn requires and if they write reasonably portable code there would be minimum disruption when they want to rev the whole package because unlike with MS, they could know exactly what changes are happening in the base distro as they happen and plan as they go along.

The cost savings would have to be large and the number of crunch events would be close to zero with good management. This is in comparison with developing on Microsoft's rather erratic schedule with bumps and panics every so often even after the release and first rounds of fixes. Automation people have shown they greatly prefer not to mess with that which is not broken and this would make it practical and practicable to accommodate them.

As for not having Windows on their automation machines, I think people would deal far better with that than all the consequences of having Windows on their automation machines. The OS drops into the background (or should) and the operations become prime.

Regards
cww
 
N

Nathan Boeger

Curt,
Thanks for the clarification. I agree completely with what it would take to be viable for the customer. Unfortunately, I've yet to see a SCADA vendor take this approach. To make matters worse, selling new ideas to end users seems really difficult when it comes to controls. Ironic for an industry that's been historically defined by innovation. Perhaps open source projects will pick up the pace in terms of viability. They could fill in some gaps here.

----
Nathan Boeger
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
N

Nathan Boeger

CWW,
Accidently responded to Michael Griffin's response thinking it was yours. Oh well.

On Dec 31, 2006 12:51 pm, Curt Wuollet wrote:
> That's been a really rocky road for automation customers. Most, if not all own unsupported systems of some type. Don't you? And, there is room for other support options. Being wholly dependent on a single shrinkwrap vendor is a fairly large liability in itself. One can read from this fora that that isn't necessarily working that great. With all the mergers, acquisitions and consolidation there are large numbers of orphans under the current model. Add to that many packages that are simply obsoleted with no recourse, and having stuff that any qualified programmer can work on begins to appear much more attractive long term. What the commercial vendors see as a viable option often hinges upon the transfer of large amounts of money from your pocket to theirs and very few other choices. Single sourcing is risky business, why do business types see that for everything but software? >

Absolutely! I've integrated many a project with unsupported legacy systems. It often pisses me off and the customer. In my experience, I've had (slightly) less trouble with old HMI applications that old custom programs. I'm thinking of a certain distributed Delphi app that was ahead of its time, and a Linux C based program where the original author passed away a few years ago. Going open source or even with the big guys should tend to minimize your risk, but you're right - these software companies have been going crazy with buyouts recently! It's tough, I'm not sure what to tell end users. Our industry has a poor track record - and this madness ends up costing manufacturers lots of money. I work for a small industrial software company. We strive to be as open and standards based as possible for commercial software. We're also not interested in a buyout. But I'm sure many companies have been there before.

cww: > Or a Tivo? :^)

Lol, I was going to say Tivo. I never did figure out how to program my VCR ;)

on the Linux vs Windows support - I think that a major problem is a lack of people giving Linux a legitimate shot. Usually when I mention Linux to manufacturers their eyes roll as they're reminded of some ancient project gone arye. Even the industrial IT departments that I've dealt with are often against Linux desktop support - even when they're supporting Linux servers. Any ideas why that might be? It's an uphill battle!

----
Nathan Boeger
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
Michael,

I would be interested to here more about this.

Contact me at phair1 @ rogers. com

Do you have any web sites?

Dennis
 
M

Michael Batchelor

Well, here I am back again, trying to change the subject.

I think we should all support Ght56dfN OS because it's going to be open by the time I get around to writing it, but in the mean time I want to get back to the problem of the "war" between IT and Production.

Operating System support aside, how do we effect the organizational change that alleviates the tension, and sometime open warfare, that exists because IT's needs conflict with Production's needs.

I've thrown my proposed solution into the ring several times. What do others suggest, besides trying to hide under a rock and hope the IT guys don't find us. That's not a solution.

Michael
--
Michael R. Batchelor
www.ind-info.com

GUERRILLA MAINTENANCE [TM] PLC Training
5 Day Hands on PLC Boot Camp for Allen Bradley
PLC-5, SLC-500, and ControlLogix

If you aren't satisfied, don't pay for it. Guaranteed. Period.

[email protected]
 
M

Michael Griffin

In reply to Curt Wuollet: If someone wanted to use a Linux OS in their "SCADA appliance", probably the best strategy would be to base the OS part of it off a major well established "full featured" distribution. You could track that distribution, and just have a different standard package list (i.e. strip out the stuff that is not required, and add in your own).

It is important to remember that "stability" is not the same thing as "stasis". New releases are good if they fix bugs in the old ones or provide features that people are waiting for. What people are going to want to know is that the new release is going to continue to work on their hardware. The only way to really know that is to try it out and the more people trying it out the better.

The SCADA appliance market is likely to be small compared to the general computing market. If you track another much larger distribution, you can take advantage of their larger user base in testing the software you have in common with them (which would be most of it).

What would differentiate the SCADA appliance distribution from the one it is based off would be the SCADA software itself (of course), the different standard package list, and the system management package (which would be SCADA oriented).
 
M

Michael Griffin

In reply to Nathan Boeger: I am in the middle of a PC based automation project. The customer for the software was offered an OS choice of either MS-Windows or Linux. They chose MS-Windows and I'm not inclined to argue with them if that's what they want. The software was developed and tested on Linux and deployed on MS-Windows (it runs on either), so the project has given me a fairly direct side by side comparison of what it is like to manage MS-Windows versus Linux in the same industrial application. It was a lot easier to set up a computer with Linux for the application than it was using MS-Windows.

For most manufacturing people though, asking them what OS they want is like asking them whether they want an Intel CPU or an AMD CPU. They don't know the difference and they don't want to know the difference. Most IT departments are going to oppose any change in their daily routine the same way they opposed introducing PCs in the days when "IT" meant mainframes. IT departments tend to prefer vendors who take their department heads to "workshops" in sunny climates and they roll their eyes up at people who think that saving other department's money matters to them.

If you want to sell something different to people, sell them an advantage not a name. If you try to sell them a name they will stick with the name they already know.

If my project time line had allowed it, I would have asked the customer if they wanted to have a high reliability PC with no hard drives and no fans. If they said yes, I would have shown them the hardware and the software to do it. The software would have included using a Linux OS, because that's a low cost off the shelf solution for this sort of application. Reduced downtime is something that people can understand and it's the sort of question they want to be asked about.
 
C

Curt Wuollet

Hi Nathan

The point that almost everybody misses is that you don't have to really convince the users, <dons flameproof union suit> they are used to running whatever comes in the box. Seriously. Automation folks already run nearly everything with non-standard, single sourced, binaries that have no commonality to speak of. So most of automation could be running on Linux and no one, well very few, would ever know. What runs on your SLC, your S7-400, your converter box, your intelligent sensor, etc.? These could all run Linux under the covers, and some of recent vintage probably do. After all, Wind River, whose products grace lots of embedded gear, work with Linux now. SixNet produces an RTU and friends that run on Linux. Your phone may well run Linux.

The only places where people would notice is in tools, SCADA, HMI, and the various comms and IPC schemes designed to ensure that every automation project include a tithe to the church of Gates and a Windows box as it's least reliable component.

None of those functions really depend on Windows, that is, there is nothing there that can't be done with any other graphical OS. One would need a replacement for OPC and a few other Windows toll booths, but I doubt anyone would shed any tears since most are not supported anymore by MS. Replacements built for automation purposes would probably be simpler, easier to use, and more reliable . Vista is going to be a much larger PITA than switching to Linux would be and may even dump some of these old favorites.

The big problem here and the "Skunk on the table" is that this would be very doable in a cooperating world, but may well be impossible with the degree of consensus and cooperation our vendors exhibit. This Skunk prevents anything from being standardized or even modernized to a large extent, but would be particularly thwarting to shedding the MS lock-in. MS is the only thing they have ever agreed upon and it's only because they were forced to by monopoly omnipresence.

As it's New Years Day, I will make a prediction that some company will make a 2007 move. Probably a EU company where the legislators and regulators and everyone else aren't totally owned by Microsoft and the current Anti-Trust trials would discourage Microsoft from retaliating. Witness what happened when MA tried to mandate Open Document Format.

Regards

cww
 
N

Nathan Boeger

This thread is out of control - I love it! It was obvious that Tony was opening up a hornet's nest, but I had no idea that we'd get this much good information. End users and integrators deserve to know what software's available - especially the open source/free stuff. And "everybody" knows that this SCADA/IT convergence is happening.

This is a bit of a tangent, but it really pisses me off when vendors spread ideas claiming that "industrial software is somehow, magically, different" - that it won't perform on any platform other than theirs. Or "industrial data" is different than any other type of data. So THAT'S why I have to spend so much more money on a custom, obfuscated (enhanced, really...) version of MS SQL Server instead of running free, MySQL or PostgreSQL databases! I wish I could find a link to an article that I read that claimed that open source software wasn't viable, secure, and get this, stable enough to control real world processes. I challenge you to ask any end user/integrator about their experience with stability in typical industrial software... especially from the era this article was written.

The example that I do remember is this: Google "web based hmi". You will get one of the worst nay-saying articles that I've ever seen. Not only does it patently misinform, but it reinforces the pessimistic notion that progress won't meet needs or even be possible - the entrenched vendors clearly have all your problems solved. Granted, it was written Sept '02, but it's the first search result in Google! Unacceptable! Users deserve far better data - like the information that posters to this thread have contributed.

----
Nathan Boeger
"Design Simplicity Cures Engineered Complexity"
Inductive Automation - total SCADA freedom
 
D

Davis Gentry

Michael -

Thanks for the clarification. I did misunderstand
your point.

> As to what operating system a vendor may choose to
base this sort of business
> model on is another issue altogether. Simply having
> an integrator install a
> different operating system (what ever that may be)
> while otherwise doing
> everything else the same as before doesn't really
> address the issue. <

I agree fully.

> this is not the customer looking for problems and
hiring
> someone to fix them; this is someone doing this for
the customer without
> prompting on a continuous basis. <

Very good idea.

> I don't see this as an issue that integrators can
> properly address. Rather, I
> see this as an issue for SCADA and MMI package
> vendors, the solutions for which integrators then apply. <

I'm not sure that I agree with this. I see security risks in two major areas - local access (i.e. operations, maintenance, engineering, and IT with direct access to the computer), and remote access, usually across a corporate network. Many local access problems can be limited by setup of the computer - password access levels being the major necessary step - if the operators have no root and/or registry privileges (regardless of OS). Having no keyboard and limiting file input capabilities also limits possible problems. Limiting or excluding internet access from the operator end also blocks a number of vulnerabilities. Another good tool is the use of a comprehensive security tool such as Norton, or Panda, or whatever your preferred package is. This is an area where coordination with local IT can be useful - they can work with production/engineering to keep the package up to date (and no - I do not advocate auto update).

The maintenance and engineering people have greater access, which also leads to greater vulnerability. I'm not sure what, if anything, can be done to completely remove all vulnerability. If you have the access you can cause damage. Anyone know of any way around this? If so I would love to get details on that.

Remote access vulnerabilities are easier to stop. Cut off all unnecessary services on the computer and greatly limit what anyone can do from outside. Pass production data through directories which are read only from outside. If you have to open a space for write access (recipe information for example) then limit the types of files it will accept, and keep a process running that auto deletes anything else as it is written.

Again - security is an area where local IT can often help, and is OS dependent only insomuch as local support (engineering or IT) may be trained and experienced in one OS more so than on others. Am I missing anything here?

Thanks,

Davis Gentry
 
D

Davis Gentry

Why would anyone load Outlook on a production machine?

I don't think that I have ever seen it there. And
you can load embedded Windows versions (I started with embedded NT, and have some installs running XPe) without IE. Haven't tried it under XP, but the Add/Remove Windows Components tool does offer the option - I'll try it on an old laptop after I get back home from this trip. One you missed is Windows Messenger - I think that is a pretty major
vulnerability, and one I remove on all pcs. The
latest versions of Media Player are also potentially problematic, and I remove that on production XP machines.

I do want to point out again that I am not knocking Linux - our latest and greatest motion control processor actually runs a real time Linux kernal, so I'm in process of getting back into the swing of *nix for the first time in a decade.

For Curt and the other Linux guys out there - we are including a web server on the processor with a web app that allows a GUI into the processor. What kinds of vulnerabilities should I be looking out for here?

Davis Gentry
 
M

Michael Griffin

In reply to Michael Batchelor: The following is more or less a reiteration of my previous statement on this subject (or at least of the one that was connected to the putative subject).

I believe that the SCADA network is best controlled and maintained by the people responsible for the rest of the SCADA system. Some of the discussion we have had has been directed towards how we could make the SCADA systems easier to support to make this more feasible.

The problem you are trying to address is one of how to align the interests of different parts of the company so that they all work harmoniously in a common direction. If you really have a general solution for that, then I suggest that you get out of the automation business and become a management consultant. You might not have as much fun, but the money is certainly a lot better.
 
B

Brian E Boothe

I've tried the Linux Experience on several of my Customers, and the biggest complaint is "Software installs" and Flash/Java/Adobe Configurations - Installations.

Linux is great O/S for wire-heads and Computer Geeks like myself who have been in computers since 78, but you go to a Non-Computer experienced Individual and plop Linux in their lap... sure it's free, But YOU as the Enabler will feel the burn after 2 weeks, and they'll be running to you every day with questions and Driver issues. It's really not worth the headache in my opinion. I have 3 Linux machines running in my basement. I love Linux, but I also know how to write Device drivers and shell scripts, I'm also under the Poll of "Keeping linux Geeked!" no noobs.
 
M

Michael Griffin

In reply to Curt Wuollet - I don't know if it's possible to get any further off topic, but with regards to your statement about a replacement for OPC the following might interest you.

It occurred to me several years ago that the primary function of OPC is to allow one or more programs to access data by tag names without being directly linked to the driver. When looked at this way, a tag read operation resembles a database look up. That is, the tag name is the "query key" with a value (or values) being returned similarly to a database result. The actual communications driver would be making selects and updates on one side, while the application(s) is making corresponding selects and updates on the other side. Dealing with databases is a very common and well established technology. There is very little if anything new that would need to be developed to make use of it.

While you could theoretically use a conventional database for this, I recently came across the following project for making a program's internal data structures accessible as if they were a database. It uses a subset of the Postgres protocol.

http://www.linuxappliancedesign.com/projects/rta/index.html

I think the applications for this are obvious when you realise that the service being described on the web site listed above could just as easily be a Modbus driver. Even better still would be a common server with "pluggable" drivers. I haven't tried this library, but I suspect it wouldn't take too much effort to have a working implementation up and running.
 
M

Michael Griffin

In reply to denn: The web site for the "SCADA and Control Systems Procurement Project" is: http://www.msisac.org/scada/

I am not involved with them, and I am not making any recommendations regarding them. I brought them up to point out that there are SCADA users who intend to place more emphasis on security and who intend to make the supplier responsible for providing it.

I hope that answers your questions. I have already sent you a copy of this reply directly to you as you requested. If I can be of any further help,
please let me know.
 
C

Chris Jennings

Well my background is that I have an Electronic/Computer Engineering degree and I started working for a manufacturing company in the IT department. It was great overseeing small projects that ranged from upgrading old thinwire Ethernet to UTP and deploying new versions of Microsoft Office. I got sick of this pretty quickly because it was pretty much the same thing everyday. So I took a job working as an Automation Engineer on one of the paper machines. I had the computer experience I had done control theory at uni and there were a number of very experienced engineers around who could help me learn the ropes. From this experience I was able to apply my IT skills to the automation field and I made a lot of changes that hopefully improved reliability of the process control networks and computers. I could also see the frustrations that automation people had with the IT crowd. IT have a very narrow focus and they also don't like making exceptions to their often very specific security rules. Examples:
-Admin rights for normal users
-Virus scanners MUST be installed
-Non-standard software not allowed

Just those three rules make an automation engineers job almost impossible. I forces people to have a normal work PC (so they can read e-mail) and a "real" work PC which has all the DCS stuff on it. In some cases IT won't even let you connect up to the business LAN so you need to double up network infrastructure.

So the real way to get people to understand your problems is to put the shoe on the other foot. Get some of those IT desk jockeys to come and spend a week in the plant and see things from your perspective. But the converse is also true. Spend a week on a helpdesk or managing the business infrastructure and you will soon see why they aren't keen for your wonderful new DCS to be hooked up to the business LAN, or why they don't like non-standard software installed on work computers.

Empathy is a wonderful thing.

Chris Jennings
 
M

Michael Griffin

In reply to Davis Gentry (on security configuration) - There are several sides to the security question.

1) Local access is a question of passwords and privilege levels. This sort of set up should be routine, and therefore for a "SCADA appliance" the complete system (including OS) configuration should be automatic (possibly with a choice of several usage profiles plus optional manual changes). This should be part of the SCADA system management software, and not require third party packages or low level tweaking for a typical installation.

2) Remote access is actually a more difficult problem, not a less difficult one. It's all very well to say "cut off all unnecessary services on the computer", but that can be more difficult in practice than it is in theory. Firstly, if you are using MS-Windows as the OS, it has a bad habit of turning on (and needing) all sorts of "services" (daemons) that listen to various ports even for purely local needs. There are a number of worms that have exploited this problem in the past. Furthermore, the services that *are* necessary can still be a security problem.

Secondly, software bugs can cause vulnerabilities that need to be patched regularly. There are also cases where the problem is an original misconfiguration that must be corrected. This is where the idea of the SCADA appliance vendor taking long term responsibility for the software comes into the picture. They would continuously address these problems.

However again, for configuration issues the "SCADA appliance" system management package should provide standard system configurations that address these needs for typical installations.

3) There is a problem which occurs with many MS-Windows updates or driver installations where some system settings get reset back to the defaults when unrelated changes take place (I had this happen when installing a network card for someone last week). This could happen at anytime after the original security configuration. Again, the SCADA appliance system management package could run regular audits and report on configuration changes.

Security is not something that you can do once and forget. It is an ongoing and evolving problem. For dedicated applications though (such as SCADA) it should be possible in most cases to use standard configurations. When you can use a standard configuration, it makes sense to automate the task of making (and verifying) the configuration. The manageability of the SCADA control room is improved by simplifying and standardising these factors.
 
M

Michael Griffin

In reply to Davis Gentry (with regards to removing MS-Windows programs) - You don't usually have to "load" MS-Outlook on a computer. If you buy a typical desktop computer from a major vendor it comes with MS-Outlook (and a lot of other junk) already loaded. It's more a matter of whether anyone had the time to try to remove all the excess junk before putting it into service. Computers from major OEMs are used as marketing channels for extra software and services. The OEMs get paid to put a lot of the third party junk on there. Locally built clones have less of this, and computers that you build yourself of course have the least of all.

As for removing MS-IE, that's not really possible. The MS-IE libraries are part of the basic GUI. You can remove the stub that loads the front end, but you can't remove the DLLs that make up most of MS-IE (at least without losing a lot of functionality). That's why many of third party programs will list a particular version of MS-IE as being a dependency even if they have no obvious relationship to the internet. Many of the MS-IE vulnerabilities are in the DLLs, so the problems are still there if someone can find a way to trigger them.

In many (if not most) cases, when you "remove" a program that comes with MS-Windows, you're not really removing it. You're just at most removing the loader stub. While I believe that MS-Windows XP Embedded comes with software that allows more detailed removal (which is basically the difference between it and regular XP), some of the libraries we are talking about are fairly fundamental to MS-Windows.

For most people who are using MS-Windows in an automation project, this really isn't an option. They can't remove enough of it to matter without becoming experts as to what each and every DLL does and what program needs it. The whole point of using MS-Windows was supposed to be that they could just buy a computer with it already loaded and not have to worry about that sort of thing in the first place.

========

With regards to your question about security of a web application, that
depends upon a number of questions (what web server are you using, is there
any scripting available and what kind, what add-on modules are you using, is
there a database present, etc.).

Some web site problems are due to holes in the web server itself or in add-on
modules (for encryption or server side scripting other tasks). If you are
using Apache (or a re-branded Apache) this isn't too common.

Many problems however are actually application vulnerabilities or poor
configuration. Some common application vulnerabilities are:

- SQL injection in a database.

- Failing to set the proper access permissions to the directories that hold
the web pages (allowing the user to access directories they shouldn't be able
to get to).

- Counting on cookies or Javascript being turned on to prevent the user from
otherwise doing something they shouldn't (e.g. counting on client side
re-direct).

- Trusting the contents of cookies, or the URL in a GET string. People can
(and do) modify these.

- Security by obscurity. People can guess at the name of a web page, and ask
for it directly. You can't count on them getting there by an "approved"
route.

- Leaving passwords in an accessible file.

- Allowing people to upload files without properly controlling where the files
end up. Someone could upload a file which replaces one of your own critical
web pages, in which case they can script the web page to do something for
them they don't otherwise have direct access to.

- DOS attacks. There's probably not much you can do about this except have
lots of bandwidth.

- A lot of problems these days are not due to a single vulnerability, but
rather to combinations of several.

Most of the above are really problems for publicly accessible web sites. If
your web server is embedded in a device and is being used as an MMI on a
network that isn't publicly accessible, then probably the most serious
problems would either be some sort of script injection or allowing access to
pages not intended for the user (e.g. the user can get at a system
configuration page by typing in the URL directly).
 
C

Curt Wuollet

But, Michael,

We are talking about IT vs. Maintenance. By placing themselves in the center of IT's domain, it is nearly inevitable that they will fall into the bailiwick of those entrusted with the Microsoft franchise. Management will see largely parallel efforts and most likely side with the professional reboot, reload, and upgrade crew taking care of it all. It only makes sense from their perspective. When you make MS a part of your production systems, you marry IT. The conflict is inherent and comes with the territory. It's all about territory. If you start doing house power, you get involved with electricians, hanging pipe you hang with plumbers, etc.

Regards

cww
 
C

Curt Wuollet

> In reply to Curt Wuollet: If someone wanted to use a Linux OS in their "SCADA
> appliance", probably the best strategy would be to base the OS part of it off
> a major well established "full featured" distribution. You could track that
> distribution, and just have a different standard package list (i.e. strip out
> the stuff that is not required, and add in your own). <

Exactly.

> It is important to remember that "stability" is not the same thing
> as "stasis". New releases are good if they fix bugs in the old ones or
> provide features that people are waiting for. What people are going to want
> to know is that the new release is going to continue to work on their
> hardware. The only way to really know that is to try it out and the more
> people trying it out the better. <

But, you can limit churn to truly relevant issues. And you have the benefit of letting the general interest audience for the parent distribution be the guinea pigs. No, you shouldn't lock things down for the duration, but you can ignore change for changes sake and keep change to a tolerable
level keeping in mind the end purpose for the system. If all you interact with is the application, much of the noise is irrelevant.

> The SCADA appliance market is likely to be small compared to the general
> computing market. If you track another much larger distribution, you can take
> advantage of their larger user base in testing the software you have in
> common with them (which would be most of it). <

Should be almost all of it for high level apps like SCADA. It would be akin to the "stable tree, development tree" model used in much of Linux
development. Change the stable tree only when there is need or at least a really good reason and only well tested code.

> What would differentiate the SCADA appliance distribution from the one it is
> based off would be the SCADA software itself (of course), the different
> standard package list, and the system management package (which would be
> SCADA oriented). <

Yes, a stable and known reliable working set relative to the application. Mostly a stable set of libraries and any utilities used so you don't have the Linux equivalent of dll hell.

In short, you could ensure that things always work as intended. What Linux is famous for, "It just works". Until the _customer_ wants to change. And another stable package for then.

Regards

cww
 
Top