I’m Jason Firth.
I’ve noticed a number of discussions lately about using different technologies that are less expensive than typical controls equipment.
“Why not use Arduino? Why not use virtualized PCs? Why not use the cloud?”
These questions really trouble me.
“Cloud computing” generally relates to computing resources being pooled in third party data centers and their operation being shown as a “cloud” on network diagrams.
Three current examples of cloud computing are solutions from Microsoft Azure, Amazon AWS, and Google Cloud Platform.
Some of the benefits of cloud computing are:
You can immediately spin up or down capacity as required with no capital expenditure. For example, if you’re running a website and your website becomes more and more popular, you can press a button and Amazon will dedicate additional computing resources to your website.
Your resources aren’t tied to a single point of failure like a single server. Most virtualized computers can be swapped between different servers to allow failover without any disruption of services. This can mean that if server hardware is failing, it can be replaced with no impact on the provided services.
Another company can focus on the specifics of managing large IT infrastructure and can carry the risks of investing in individual pieces of IT infrastructure
By having many huge companies pooling their data infrastructure requirements together, the total pool increases, which provides some safety. For example, it’s very difficult to DDOS Amazon AWS compared to a single corporate file server located in a company head office. In addition, in the event of such attacks, a single company specializing in network infrastructure can have experts on hand to work on such risks, where a single company may not be able to afford an expert for a few servers.
So those are some benefits, but people honestly suggesting this solution really troubles me. The reason is that they show a lack of appreciation for the gravity of your control devices.
What do I mean by this?
Well there’s a number of things.
Before we start with any of the reasons, I need to explain the difference between two types of failure: Revealed and Unrevealed failures.
Revealed failures are failures that are immediately apparent. They shut down devices, or they immediately cause a process upset. These failures are immediately found, and likely immediately fixed.
Unrevealed failures, by contrast, are not immediately apparent.
A few years back I was facilitating Reliability Centered Maintenance analyses. In many of the scenarios we covered, a motor would overheat and fail, and the control systems would shut it down. A relay would fail, and the control system would shut it down. A boiler would reach overpressure, and the control system would shut it down.
Then we started covering different control systems. Failure wouldn’t immediately harm anything, but when other problems occurred, that’s when the problem starts. Suddenly, the control systems wouldn’t be shutting down failing systems. In fact, they might be actively working to make things worse.
I’ve witnessed the trouble caused when control systems go down. Operators rely on their control systems more than they realize to tell them exactly what’s going on. If the alarm doesn’t sound, if the screen doesn’t light up, if the flashing light doesn’t flash, then often they’ll have no way to know something bad is happening until something terrible has happened.
So now we have this critical piece of infrastructure we’ve sent to another site, perhaps thousands of kilometers away, that we’re relying on the Internet to maintain connectivity to.
The ability to spin up or spin down CPU resources is sort of a non-starter, in terms of relevant benefits. CPU resources are not usually a bottleneck in PLC or DCS systems. You might have an issue if your PLC scantimes are getting too high, but PLCs are already using CPUs that have been obsolete for 15 years — if CPU resources were a primary consideration, then that wouldn’t be the case.
The risks associated with controls infrastructure are not the same risks associated with IT infrastructure. That being the case, I don’t think companies like Microsoft, Google, or Amazon are better prepared to manage those risks than companies that deal with the real risks every day. We’re not talking about having to issue refunds to some customers, we’re potentially talking about dead people and a pile of smoking rubble where your plant used to be.
Whether the data center side of your infrastructure is safe from DDoS attack or not, your plant likely is not. Therefore, a potential attack, or a potential loss of connectivity has the capability not just of taking down your email and intranet, but your plant.
Along the same lines, industry best practices for security uses a “Defense in depth” strategy, where your core devices are protected by layers and layers of security. By contrast, a cloud strategy puts your control data directly onto the public internet. If your system has a vulnerability (and it will), then you’re going to be risking your people, and the environment, and your plant, and your reputation.
Industrial network security generally places availability at the highest priority, and security as important, but secondary. If your controller can cause an unrevealed release of toxic gas if it gets shut down, it’s important that the controller continues to function. If that controller stops controlling because a security setting told it to, then that’s going to be a question to answer for later.
A single point of failure can be a good thing, or a horrible thing. In September of this year, Amazon AWS suffered an outage. Half a dozen companies that rely on them suffered reduction in availability as a result. By contrast, not a single system in my plant realized it. What if you were using AWS to control your plant?
Now that we’ve covered cloud computing, let’s look at Arduino.
Arduino is an open source hardware platform based on ATmel microcontrollers. An arduino provides a circuit board that lets people use breadboard materials to connect to their microprocessor; a USB programming interface; and a software package that makes it relatively simple to write software for the controller and send your program to the microcontroller.
Benefits? Well, an arduino comes cheap. You can buy an arduino board for 20 Euro from arduino.cc.
Downsides? There’s a lot, actually.
Let’s start from the outside and work our way in.
Arduino I/O isn’t protected with optoisolators. The analog IO isn’t filtered. None of it is certified for industrial use, and I’m guessing any insurance company would tear you apart for directly connecting an Arduino to plant devices, especially safety related ones.
Arduino is “modular” in a sense that you can plug devices called “shields” onto the main board, but this is not modular the same way a Schneider Quantum or an Allen Bradley ControlLogix. You can get a shield onto the board. Once it’s on, it’s on until you power down, and don’t expect to get 16 different kinds of card mounted together.
Arduino isn’t hardened on the power side, either. Noisy power could cause problems, and unless you break out a soldering iron and start building a custom solution, multiple power supplies aren’t an option.
Arduino isn’t protected physically. Admittedly, most PLCs are only IP20 rated, but at least that means you can’t stick your fingers on the exposed board, which isn’t something you can say for Arduino.
Arduino has a fairly limited amount of logic capacity. The number of things you can do with one are limited to begin with, and become more limited as you add things like TCP/IP for HMI access.
Speaking of logic, don’t expect to change logic on the fly without shutting down the controller, or view logic as it is running. That’s just not possible.
With Arduino, you’re likely to be reinventing the wheel in places. Want Modbus TCP? Hope you’ve brushed up on your network coding, because you’re going to be writing a Modbus TCP routine for everything you’d like to do.
As well, congratulations on your new Arduino installation! The guy who wrote the software quit. Can a regular electrician or instrument guy muddle their way through the software?
I’m not saying you can’t build a PLC using an arduino, or that you can’t control a process with one. What I am saying is that if you’re doing industrial control, it probably isn’t the right tool for the job.
I’m going to save Virtual Machines for another time, because there’s a trick in there I want to expand on a bit more.
Thanks for reading!