SCADA Control Room Management

C
Hi Nathan I can answer that last question:
> Why do so many industrial software companies insist on writing their own pieces and plugins for their packages when they do a second rate job compared to existing technologies?>

Because in 5 or 10 years their own database systems will still be supportable rather than obsoleted by MS. And in the meantime, you won't have to change all the interface code that doesn't need changing 3 or 4 times. Often, these decisions are based on past experience and pragmatism. Keeping up with Windows churn is a very expensive part of an ISD's development budget. Also, MS SQL is a resource hog and very much overkill for the job at hand. IT does like to own a new server for each function though. If I were to standardize, it would be on something that fits and can't be obsoleted, perhaps with an API that would work with several other systems. Owning the source would provide even more assurance that you won't be hung out to dry.

Regards
cww
 
C
The company whose salesman tells management that this will scale flawlessly up to enterprise size should be entirely and explicitly and legally responsible for this. And yes, they should know the equipment and resources to do it. Expecting the customer to secure and protect software they know almost nothing about is ridiculous. This is almost the best possible way to produce an unsupportable mess and it's the status quo because the sale matters, not the aftermath. Perhaps the BS level would be lowered considerably if the sales types were held responsible for the promises they make.

It's just as bad as "This database is so simple your secretary can administrate it". I've seen a lot of those databases. Everyone should have to clean up their own mess.

Regards
cww
 
D

Dennis Patterson

Nathan,
Nobody said anyone shouldn’t work together. But a plumber, doesn’t wire a light bulb, or shouldn’t anyway. If there is a Cisco router to be configured, then by all means its ITs responsibility. But when they start playing with packets of data in the SCADA, because its traffic does not suit their network, hence rendering the SCADA inoperable, then they have over stepped the line.

Dennis Patterson
 
C
They are inextricably related. The choice of MS for the secretaries currently more or less dictates the choice of tools for the automation dept. And right or wrong, management sees the IT gurus as professionals and everyone else as users. This explains, in part, why I must use WXP on the job. The other part is that automation and maintenance actively and decisively put themselves in this position by choosing products that include IT property in every system that uses OPC or depends on Windows.

This was _not_ a problem when PLCs were standalone and DCS was run on whatever was the most stable. And that is enforced by automation vendors supporting only one system, the one most attractive to IT meddling. The guru vs user bit is intrinsic in using MS for anything, so why should factory systems be handled any differently? And for the most part, since IT keeps up with crashware by necessity, and control folks presumably have other things to worry about, why would you _not_ organize it that way? After all, they are MCSE's.

If control systems were control systems and not simply another Windows box, we might win this, but the way things are, the MCSE is going to get the call any time you can't fix it in 10 minutes. It's the price you pay to use Microsoft, just like IT. You are just another Windows user.

Regards
cww
 
IT departments are concerned about security in a different way than you are. IT worries about financial information being compromised and employees wasting their time looking at celebrity crotch shots, not 911.
 
D
Come on, Curt. Everyone here knows that you are a *nix man, and that you don't care much for MS. Having said that, your points are often cogent, usually well stated, and backed by a good deal of real world experience. Walt's point here is also very much to the point. The *nix OSs are not inherently any more suited to our work than MS, and tne converse is also true. In a well put together system the OS involved should be more or less immaterial to the customer. If we as coders and integrators do our homework then we can build a good, secure system be it on MS, Linux, or even BOSS. Most of the OSs available have their good points and their bad points, and we do a disservice to our customers when we blind ourselves to the facts.

Davis Gentry
 
N

Nathan Boeger

cww,

Good point about Microsoft products - they certainly have a history of short lifecycles, especially support and forward compatibility, compared to industrial hardware.

Have you used newer versions of SQL Server? It's a much better product than it used to be. I find myself more often recommending that integrators use MySQL or PostgreSQL, but as much as I'd rather not admit it, I think that MS SQL Server 2005 is as good, if not a better product. Sure, you're not going to run it on old hardware, but it does perform.

----
Nathan Boeger
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
M

Michael R. Batchelor

Curt,

Completely, absolutely true. Which is why I have been advocating for moving maintenance of the data devices into the hands of the maintenance department.

But, as I said several days ago, that would entail moving the CIO's responsibility into the engineering department. That's the change that's unlikely.

To Curt's point below, there's not a single electrical supervisor alive - at least not one who deserves the job - who sees the requirements for the 480VAC 3 phase motor that runs the conveyor chain in the same light as the bathroom outlets, but he's responsible for both. And a decent manager gets both right.

However, with the current organizational structure in most plants, the secretaries get correct service, and the control room gets shafted. So, I propose changing the organizational structure of the data services to mimic one that gets it right in another genera.

But there will be a fight.

Michael

www.ind-info.com
 
C
Those are some pretty broad statements. Let's explore a few.

> Come on, Curt. Everyone here knows that you are a
> *nix man, and that you don't care much for MS. Having
> said that, your points are often cogent, usually well
> stated, and backed by a good deal of real world
> experience. Walt's point here is also very much to
> the point. The *nix OSs are not inherently any more
> suited to our work than MS, and tne converse is also
> true. <

No, it's not. For automation and control purposes, stability is key. Flexibility is important and the ability to interface to most anything makes integration much easier. To say that a 50 million line secret binary monolith with the Worlds shakiest security and stability record is equally suited to a modular open customizable OS is a fairly interesting argument. For actual
control purposes, running the minimum amount of code necessary has always been a given as it promotes reliability and speed. The ability to run headless with no user interaction is fundamental in most control applications. Owning the source code for your control system offers a great deal of confidence that you can support it long term.

> In a well put together system the OS involved
> should be more or less immaterial to the customer. If
> we as coders and integrators do our homework then we
> can build a good, secure system be it on MS, Linux, or
> even BOSS. <

On any particular application, I'd be happy to compare my uptime on Linux with yours on Windows.

> Most of the OSs available have their good
> points and their bad points, and we do a disservice to
> our customers when we blind ourselves to the facts. <

Well, on that much we can agree. It's just that most of Windows good points relate to playing games and consumer floobydust and nearly everything not related to the serious business of
automation and control. They have institutionalized many features that are kewl, but will forever be insecure. And their direction is
towards more of the same including such things as allowing them access to your data and breaking their own file formats with every version and a myriad of other nightmares for maintainability I find it quite incredible that one can favorably compare that with design for compatibility and commonality and openness. Even from a consumer perspective, it's a stretch that forced obsolescence, deliberate incompatibility, proprietary data schema and formats, DRM, making all data part of a database, secrecy and onerous
licensing and use restrictions are beneficial. That would require a special type of blindness as to whom all the benefits of such arrangements accrue. That blindness is endemic at present, but
more and more people are being cured every year. It should be stressed that Linux was created and is being developed specifically to address those faults and steer the benefits toward the consumer. And to make development and integration easier. And more secure and practically every good thing that can be done for computer users. Windows is being developed to make MS money and maximize their control over us and sustain their monopoly.
It's really hard to see them as equals and ignore the skunk on the table.

Regards

cww
 
It will come under 'Process Control Engineers' or 'Electrical Engineering' as a standard maintenance requirement, but you will find most Electrical Engineers will not have the ability to make adjustments to SCADA or PLC ladder.
 
N

Nathan Boeger

Dennis,
The line needs to be clearly drawn, utilizing both skillsets. Clearly IT is at fault in your example of disrupting production by setting packet restrictions on a managed switch. Unfortunately most integrators/controls engineers don't really know what a managed switch is.

You seem to be straddling the fence on this issue. On one hand you mention "cooperation is good" and on 2 other posts you give extreme examples of keeping IT away from the system (lady at front desk using word processor controlling SCADA, and example of plumber wiring the light bulb). An IT persons technical background is not significantly different than your own (although their priorities can be). SCADA implementations are quickly including SQL databases, enterprise systems, and complex infrastructure - all areas where IT has experience. Do you really think that the role of a competent IT department should be limited to programming routers? Where does working with Oracle/SAP, or even administering a MySQL or Microsoft SQL Server database come in - these are things that might have to be implemented in your SCADA application? Or we could stick to single user HMI packages forever...

----
Nathan Boeger
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
C

Chris Jennings

In my experience the best way to ensure security is by having complete control of data that enters and leaves the SCADA environment. We set up our networks with a router between the SCADA network and the business network and only opened the ports that were required for each application. By doing this you have seriously improved the security of the system. The biggest
advantage I can see with Linux over Windows is to do with having a different OS for business applications to the SCADA applications. This means if a virus (most likely cause of security problems) gets on the business network the only effect it would have on the SCADA would be potential DOS attack but actual infection is unlikely. So another good example would be using Apache web server on one network and IIS on another. Diversity can be a good thing
sometimes.

> On any particular application, I'd be happy to compare my
> uptime on Linux with yours on Windows.

Just to put some numbers to the uptime argument, check out this website
http://en.uptime-project.net/page.php?page=toplist or maybe
http://uptime.netcraft.com/up/today/top.avg.html

Looks like SunOS or websites running IIS are the winners ;)

Chris Jennings
 
C
Hi Chris,

Chris Jennings wrote: <In my experience the best way to ensure security is by having complete control of data that enters and leaves the SCADA environment. We set up our networks with a router between the SCADA network and the business network and only opened the ports that were required for each application. By doing this you have seriously improved the security of the system. The biggest advantage I can see with Linux over Windows is to do with having a different OS for business applications to the SCADA applications. This means if a virus (most likely cause of security problems) gets on the business network the only effect it would have on the SCADA would be potential DOS attack but actual infection is unlikely. So another good example would be using Apache web server on one network and IIS on another. Diversity can be a good thing sometimes. >

cww: Diversity would solve most of the really horrible vulnerabilities.
The current situation of all MS plants is the _best_ possible case for virus propagation. If every other machine were running Linux or anything other than MS, most of these attacks would fail or at least slow enough to be manageable.

But there are many, many less obvious advantages to being able to use "purpose built" systems for SCADA and control. One can make many systems "appliances" with little to no potential for malware or other abuse. You can even make ROMable systems that are totally immune. This is the approach the router folks use, cycle power and you are back to uncompromised purity. You can also go the other way and dramatically lower your box count with Linux rather than the function per box approach currently seen as necessary with Windows.

Chris Jennings wrote: <Just to put some numbers to the uptime argument, check out this website http://en.uptime-project.net/page.php?page=toplist or maybe http://uptime.netcraft.com/up/today/top.avg.html >

cww: That's a little different than our arena with some special hardware and
farming going on, but not totally irrelevant. I like the fact that until quite recently, Microsoft had Linux firewalls in front of their IIS servers and were counted in the Linux camp :^) SunOS is not a bad system and the BSDs regularly appear there as well. On commodity hardware with perhaps a simple UPS, I'll still put any stable Linux up against any shrinkwrap Windows. But I will admit they have gotten dramatically better as of late, we only have a couple incidents where something is unavailable a week where I work. With 95 and 98 it was pretty hilarious in an installation of any size.

Chris Jennings wrote: <Looks like SunOS or websites running IIS are the winners ;) >

cww: Many of those are pretty high buck machines with failover, etc.

Regards
cww
 
B

Brian E Boothe

God I love these Conversations, we still ("install and Operate w/ 20yr old
technology and sell it as new,,, Dh485 and 1 PC to control an entire Water plant and stations, hows that for greatness
 
D
cww>>For automation and control purposes, stability is key.

Of course. And MS products from 2000 on have been quite stable in my experience if properly configured and administered.

>>Flexibility is important and the ability to interface to most anything makes integration much easier.

Agreed. And what OS has the most hardware drivers written for it? Yes, you can of course write your own drivers for Linux. Then your customers are really caught in a bad place. Five year from now, after a lightning strike takes down their PC and cards, and they find out that they can no longer buy the same card from their vendor, and then they find that their driver no longer works with the firmware rev of the new card. Now if they bought source code with the original install they can go out and find a programmer who is capable enough to rewrite the driver (said programmer generally NOT found in most plants in the US, or on the nearest streetcorner, either). They also find out that the Linux kernal they were running is not compatible with the tools that the developer is currently using, so they need to upgrade OS as well. Maybe at this time they also have to rewrite, or at least recompile, their HMI.

CWW>>The ability to run headless with no user interaction is fundamental in most control applications.

Yeah - and Embedded NT/XP works great for that. As well as trivializing some of the other Windows concerns - smaller kernal, very limited and fully controllable number of programs running on the machine, etc.

cww>>Owning the source code for your control system offers a great deal of confidence that you can support it long term.

See arguement above about modifying old source code. While I personally greatly prefer to have source to any system on which I work, I have seen far too many end customers who own the source and are at best a menace if they try to do anything with it. Also a few integrators.

Michael Griffen>>An operating system which is in a configuration used for typical desktop applications will not meet the security requirements laid out in the studies. If it is possible to configure a system to meet these requirements it would require more knowledge than all but a very few SCADA integrators or plant operators possess.

This is perhaps a valid point - and if the SCADA integrators cannot set up a secure system with Windows, why would you think they can with Linux?

My original point stands. A competent integrator can set up a good (good = stable, secure, and supportable) system with many of the OSs available today. I am certain that Curt can set up something at least as capable in Linux as what I can set up with XP. And I know from over fifteen years of experience with automation systems under Windows (and a few *nix systems) that I can set up a good system under Windows.

Davis Gentry
 
N

Nathan Boeger

Davis,
Your point about future supportability is exactly why I don't push custom software (and often Linux applications) in controls. End users need to know that there are viable support options for them a few years down the road.

I think you (and many others on this post) are missing cww's major security point about Linux. He's talking about trimming the machine down to an appliance as have been done with routers and other specialty devices. This eliminates (unneeded) functionality and indisputably decreases possible security holes. This point doesn't attempt to address a secured normal Windows install vs. secured Linux (with every optional package) install - you can trim down your Linux install to remove all the bloat. This isn't possible to the same extent in Windows. Could a Microsoft Programmer with the source code do the same? Possibly. Could you as an integrator? Probably not. As long as MS continues along it's current path with its software, versions like that won't be an option. That's a conscious business decision by Microsoft. They don't make appliances, they make a usable, highly backward compatible, general purpose operating system.

That said, I would support a high quality vendor's device that they've configured as a sweet Linux based appliance. Or it could be powered by a stripped, secured version of Windows. The bottom line is that the device better work like a VCR.

In my experience, end users and integrators don't have the expertise to set up workable Linux based systems, and certainly aren't able to maintain them. It's unfortunate because I'm a fan of the penguin. As it stands, I find myself recommending Windows systems most often. But Linux is getting more widely accepted and supported...

----
Nathan Boeger
Microsoft Certified Systems Engineer
http://www.inductiveautomation.com
"Design Simplicity Cures Engineered Complexity"
 
C
On Dec 29, 2006 5:21 pm, Davis Gentry wrote:
<clip>MS products from 2000 on have been quite stable in my experience if properly configured and administered. <clip> And what OS has the most hardware drivers written for it? Yes, you can of course write your own drivers for Linux. Then your customers are really caught in a bad place. Five year from now, after a lightning strike takes down their PC and cards, and they find out that they can no longer buy the same card from their vendor, and then they find that their driver no longer works with the firmware rev of the new card. Now if they bought source code with the original install they can go out and find a programmer who is capable enough to rewrite the driver (said programmer generally NOT found in most plants in the US, or on the nearest streetcorner, either).<clip>

First of all they should have received the source free with their Linux distribution
or at least had the option. And Linux drivers seldom depend on the manufacturer, they are typically written by the user base or sometimes Linux vendors.

Old hardware is far better supported under Linux and continuing support is the rule rather than the exception because, arguably, many people run old hardware on Linux. There are projects like Comedi that support cards that haven't been sold for a decade. That's because the drivers are generally written by users without a profit motive that demands that last years equipment is forgotten. I know this is the case because I use old hardware and it is very unusual that I have any sort of compatibility problem when
I load a new version of Linux. And when I have, I contacted the maintainer and so far, they have been good about revving the driver for a new kernel.

Why don't you suggest that to your ABC card company? So, you don't need a kernel programmer in your plant and I surely hope they wouldn't
be on the street corner. But they are accessible real people and will usually
be glad to help if treated nicely.

And how does using Windows alleviate this problem? Drivers are single sourced and if the company isn't interested in supporting what they have obsoleted, you are out of luck. It they think you should buy the new card you are out of luck. If they decide not to support your version of Windows you are out of luck. They are binaries and can't be revved or fixed except
by the vendor, if there's money in it. My "old" PC won't even boot new Windows due to hardware support but it will run the two current Linux
CDs I've tried. To put it another way, pick any card supported by both and I'll bet I can get a broken Linux driver fixed sooner than you can
get a broken Windows driver fixed, if ever.

Davis Gentry: <clip> They also find out that the Linux kernal they were running is not compatible with the tools that the developer is currently using, so they need to upgrade OS as well. Maybe at this time they also have to rewrite, or at least recompile, their HMI.>

Most driver writers I know use a text editor and GCC. And unlike MS developers they can simply use the tools and libraries you have because the tools came with your version of Linux and are guaranteed to work with that version because they
were used to compile the kernel and drivers on the machine. And very few Linux drivers are dropped or unmaintained because there is no profit incentive to make people upgrade. This will continue this way as long as binary drivers are discouraged by the community because they are bad for the community.

Davis Gentry: <clip> While I personally greatly prefer to have source to any system on which I work, I have seen far too many end customers who own the source and are at best a menace if they try to do anything with it. Also a few integrators. >

Perhaps, but having recourse is much better than having no recourse. I don't know that Linux developers will always be more accessible for help than Windows developers, but I can email some pretty important Linux folks and get answers. Somehow this doesn't work at Rockwell.

Michael Griffen: >An operating system which is in a configuration used for typical desktop applications will not meet the security requirements laid out in the studies. If it is possible to configure a system to meet these requirements it would require more knowledge than all but a very few SCADA integrators or plant operators possess.
>
> This is perhaps a valid point - and if the SCADA integrators cannot set up a secure system with Windows, why would you think they can with Linux?>

A Linux system in a configuration used for common desktop use doesn't have several of the most frequently exploited holes left open. And an install option will lock things up fairly well. I can enable the NSA extensions and have pretty good security out of the box. And it doesn't break everything.

Davis Gentry: >My original point stands. A competent integrator can set up a good (good = stable, secure, and supportable) system with many of the OSs available today. I am certain that Curt can set up something at least as capable in Linux as what I can set up with XP. And I know from over fifteen years of experience with automation systems under Windows (and a few *nix systems) that I can set up a good system under Windows.>

But that does nothing to deny the many other benefits folks would have with a truly Open and customizable system. Telco, automotive, military, infrastructure, machine builders, and even NASA are seeing this. Right now, you have to be able to automate without packaged PLCs or shrinkwrap software to be able to use Linux, but the reasons must be compelling to interest these people when they could take the easy way out. I think it's safe to say they all have experience
with Microsoft.

Regards
cww
 
C
On Dec 30, 2006 2:29 pm, Nathan Boeger wrote:
> Davis,
> Your point about future supportability is exactly why I don't push custom software (and often Linux applications) in controls. End users need to know that there are viable support options for them a few years down the road. <clip>


Hi Nathan,
That's been a really rocky road for automation customers. Most, if not all own unsupported systems of some type. Don't you? And, there is room for other support options. Being wholly dependent on a single shrinkwrap vendor is a fairly large liability in itself. One can read from this fora that that isn't necessarily working that great. With all the mergers, acquisitions and consolidation there are large numbers of orphans under the current model. Add to that many packages that are simply obsoleted with no recourse, and having stuff that any qualified programmer can work on begins to appear much more attractive long term. What the commercial vendors see as a viable option often hinges upon the transfer of large amounts of money from your pocket to theirs and very few other choices. Single sourcing is risky business, why do business types see that for everything but software?

Nathan: > I think you (and many others on this post) are missing cww's major security point about Linux. He's talking about trimming the machine down to an appliance as have been done with routers and other specialty devices. This eliminates (unneeded) functionality and indisputably decreases possible security holes. This point doesn't attempt to address a secured normal Windows install vs. secured Linux (with every optional package) install - you can trim down your Linux install to remove all the bloat. This isn't possible to the same extent in Windows. Could a Microsoft Programmer with the source code do the same? Possibly. Could you as an integrator? Probably not. As long as MS continues along it's current path with its software, versions like that won't be an option. That's a conscious business decision by Microsoft. They don't make appliances, they make a usable, highly backward compatible, general purpose operating system.>

If they would simply make their systems usable without IE and Outlook, it would be a great leap forward for security. Instead it seems these can operate at administrator equivalent permission levels with full system access. Not good.

Nathan: >That said, I would support a high quality vendor's device that they've configured as a sweet Linux based appliance. Or it could be powered by a stripped, secured version of Windows. The bottom line is that the device better work like a VCR.>

Or a Tivo? :^)

Nathan: >In my experience, end users and integrators don't have the expertise to set up workable Linux based systems, and certainly aren't able to maintain them. It's unfortunate because I'm a fan of the penguin. As it stands, I find myself recommending Windows systems most often. But Linux is getting more widely accepted and supported...>

In my experience these same folks don't have the expertise to deal with Windows problems either. The ones that can are certainly technical enough to deal with a _documented_ system, especially with free community help available for the asking. If you think of it that way, it may well be more likely that you can fix a Linux system. Probably not instantly, but there are a lot more people who know the answers. If folks had the same exposure to Linux, I'm fairly sure they would do as well or better. After all, there simply aren't any Linux secrets.

Regards
cww
 
M

Michael Griffin

In reply to Davis Gentry - You have replied to several messages in your posting. I will deal with the points directed towards me first.

> Davis Gentry: "This is perhaps a valid point - and if the SCADA integrators
> cannot set up a secure system with Windows, why would you think they
> can with Linux?"

You appear to have misread my statements. I didn't say that a typical integrator was more capable of setting up a secure SCADA system with Linux than they were with MS-Windows. I simply said that a typical integrator was not capable of setting up a SCADA system which met security requirements which are beginning to be expected in today's systems.

These requirements include those of the "SCADA and Control Systems Procurement Project", the members of which represent a number of large SCADA customers. These standards as presently drafted will require extensive modification to an operating system which is configured for "office desktop" applications. Furthermore, these new security standards (as listed in the current draft) require that the SCADA vendor take responsibility for maintaining the on-going security configuration of the complete system (not just the SCADA application itself), and that their responsibility for the system does not end with installation and acceptance but rather continues for years (presumably for the lifetime of the installation). While not every customer is likely to adopt these standards a number of important ones are, and others are likely to be influenced by them.

I also didn't say that any particular operating system was more suited than another to this application. I mentioned Linux only to point out the fact that this sort of full package support is exactly what the major Linux vendors do for a living (primarily for large critical enterprise applications), so this sort of requirement is not unreasonable or unprecedented. It is however new to most of the automation business. These major Linux vendors do not offer this same support for SCADA systems, but it is the business model rather than the particular companies or products that I am referring to.

For this sort of business model to be useful in SCADA applications would require the SCADA vendors to offer the sort of "single point of responsibility" that major Linux vendors do for enterprise type applications. As to what operating system a vendor may choose to base this sort of business model on is another issue altogether. Simply having an integrator install a different operating system (what ever that may be) while otherwise doing everything else the same as before doesn't really address the issue.

> Davis Gentry: "My original point stands. A competent integrator can set
> up a good (good = stable, secure, and supportable) system with many
> of the OSs available today. I am certain that Curt can set up something
> at least as capable in Linux as what I can set up with XP. And I know
> from over fifteen years of experience with automation systems under
> Windows (and a few *nix systems) that I can set up a good system
> under Windows.">

I won't argue as to whether you could initially set up a good (stable, secure, supportable) system under MS-Windows. No doubt you are much better than a typical integrator in that respect. It is however completely irrelevant for several reasons. The first reason is that most integrators are not capable of this. They are experts in manufacturing or process industry automation, not in operating systems, databases, and networking. Even for specialists in operating systems, databases, and networking, only a very small percentage of those people are security experts.

The second reason is that most integrators operate on a project by project basis. They get paid to complete specific projects and they don't maintain a security lab researching future problems. Presently the industry operates on an "ignorance is bliss" basis and an integrator would only get involved in a problem when a customer hires them to fix a specific existing problem. The new security requirement is that someone looks for security problems, finds solutions to them, and provides the customer with a solution. Note that this is not the customer looking for problems and hiring someone to fix them; this is someone doing this for the customer without prompting on a continuous basis.

Finally, many of today's security problems do not arise from problems with single software packages on their own. They result from combinations of packages when working together. That is, a flaw in one software package may not on its own pose a security problem, but it may result in a serious one when combined with a different flaw in a separate package. This means that relying on each independent package's author to be responsible for security when looking at their own package in isolation won't work. Some party has to take responsibility for how all packages work together. Today, nobody does this for typical SCADA or MMI systems.

I don't see this as an issue that integrators can properly address. Rather, I see this as an issue for SCADA and MMI package vendors, the solutions for which integrators then apply. The role of the integrator would change very little from today, with the exception that the customer could assume that the "default install" version of the base packages would meet all the required security standards. The residual security risks would then be whatever application level problems the integrator may have created in any code they may have written for the application, or in any unsupported software that may have been added to the system by the integrator. This is a much more practical scope of responsibility for the integrator and one that more closely corresponds to what most would believe they are presently assuming.

If you want a practical example, let's assume we have the following (rather simple) problem. Suppose as part of a SCADA application, we have a third party OPC server feeding data into a third party database via ODBC. All are running on the same MS-Windows operating system. Now suppose someone discovers a way to send data via that OPC server which causes the database to stop logging certain data.

Now whose responsibility is this? We have five parties involved. The SCADA vendor will say that while they are compatible with OPC and databases via ODBC running on MS-Windows, they didn't sell any of this third party software so they aren't responsible for it. The OPC vendor will say that their server is just passing through data, and it's not their responsibility as to what that data does to anything else. The database vendor will say that they've never heard of OPC, so they "don't support it". The operating system vendor will say "try reinstalling Windows". The integrator will most likely say that the software was all from the customer's approved list of vendors, so it's not his fault (if you would like to specify a different set of software though, he'd be happy to give you a quote on installing it).

Under the proposed "single point of responsibility" though, the SCADA vendor would discover the problem (or analyse it when it is reported to them), come up with a solution, and download a patch to their customers' sites in a timely fashion (before most customers even knew there was a problem). There would be no question about who is responsible; the SCADA vendor is responsible for everything in the base packages. They may not write everything, but they would supply and take responsibility for all of it.

The above addresses security and other long term support concerns. If you have any valid complaints about it, I would think they would more properly be the flexibility you would be giving up if you want to remain within the vendor's support terms. While the vendor would supply and support everything they require, that support would be limited to those packages only. This is a limitation however that many customers may be willing to live with if it solves their long term support problems.

Your reply to Curt Wuollet discussed a number of points which are essentially questions about long term support of software. I would like to point out that that is exactly the issue which I have said is not being addressed under the current model.

> Davis Gentry (in reply to Curt Wuollet): "Then your customers
> are really caught in a bad place. Five year from now, after a lightning
> strike takes down their PC and cards, and they find out that they can no
> longer buy the same card from their vendor, and then they find that their
> driver no longer works with the firmware rev of the new card. Now if they
> bought source code with the original install they can go out and find a
> programmer who is capable enough to rewrite the driver (said programmer
> generally NOT found in most plants in the US, or on the nearest
> streetcorner, either). They also find out that the Linux kernal they were
> running is not compatible with the tools that the developer is currently
> using, so they need to upgrade OS as well. Maybe at this time they also
> have to rewrite, or at least recompile, their HMI.>

This is an interesting example, because it more or less describes the problem I had with a production test system (not a SCADA system). The operating system used was an "MS product" (as you would put it) though, not Linux. The board was still working, but the old driver would not work on a new computer. Cascading software and hardware incompatibilities meant that buying a different board meant replacing the entire test system when all I really wanted to do was replace the computer hardware.

In my case the data acquisition card vendor refused to even discuss making any changes to fix their driver. Their point of view was that the card was no longer a current product, so they were not going to devote any developer hours at all to making what they had admitted was a very trivial change to their driver (basically changing a constant and recompiling). The board vendor was one of the largest and most reputable companies in the field, so it's not as if we were dealing with a "bad vendor".

In the end, I was able to get enough information from several sources to reverse engineer enough of the board to control it without using their driver. I didn't get complete functionality, but I did get the features we were actually using. This let us extend the life of the test system out a few years to match the life of the product it was testing.

The problem you are describing is actually one of proprietary drivers, not choice of operating systems. You complain that if you have the source code, it is hard to fix. I can assure you though from my own experience that it is a lot harder to fix with no source code at all. If the driver code was publicly available then it is likely that someone else would have run into this problem before I did and have already fixed it, so that I would have had to do was download the new version.

Note that this isn't an "MS versus Linux" argument. It is definitely not Microsoft's fault that a data acquisition vendor doesn't support their own products any more. It was one of the situations though that convinced me that using whatever is the current Microsoft product isn't going to guaranty that your software will continue to work over the long term.

There is a larger scale problem along these lines that is coming up in the future. I am expecting that a lot of people are going to get very seriously burned by this one. 64 bit PC hardware has been on the market for several years now, and can be considered to be mainstream. Microsoft is one of the last operating system vendors to begin supporting the hardware in 64 bit mode. (Most people are using the 64 bit hardware in 32 bit mode. Linux was ported to 64 bits years ago though, so most Linux drivers and application packages have already been ported to 64 bits and people are using them.)

There have recently been 64 bit versions of MS-Windows available, but there is virtually no software or drivers to support it. Over the next couple of years though, "mainstream" MS-Windows is expected to move to 64 bits for the standard versions.

The problem which is arising is that the 64 bit version of the new version of MS-Windows ("Vista") will require drivers to be cryptographically signed by Microsoft before the operating system will load them. To get the drivers signed by Microsoft will require that the driver author is listed as an "approved vendor" by Microsoft. Microsoft has stated that they will only approve a limited number of vendors, as they need to keep the vendor list small enough to be manageable.

The reason for this new requirement is to ensure that nobody writes a driver that can be used to bypass the DRM (copy protection) systems for music or movies. Microsoft sees entertainment as a big future market, and they are positioning themselves as a "secure platform" (from a copy protection standpoint) so they can make distribution deals with the major media outlets. They aren't going to leave any backwards compatibility "holes" open for very long, because then the music pirates would use these same holes which would defeat the whole point of signed drivers.

Where this leaves people in the automation business who need to use drivers from relatively obscure (as compared to major consumer goods) companies that aren't on the "approved vendor" list is a good question. I suspect that it will leave them nowhere at all and wondering what to do next.
 
M

Michael Griffin

In reply to Nathan Boeger: A "SCADA appliance" could be an integrated set of packages that run on PC hardware which was purchased separately. You would then install over top of (erasing) whatever operating system came with the PC (if any). There are "network security appliances" that work like this.

The key to the "SCADA appliance" would be a good system management package that takes care of all the installation and administration tasks of *all* the software in the system. Something like this should be a lot easier to manage than current SCADA systems because you are dealing with all of the software through a single consistent interface rather than half a dozen (or more) different ones.
 
Top