Categories
Industrial IT

The Cloud, Virtual Machines, and Arduino (Oh my!)

I’m Jason Firth.

I’ve noticed a number of discussions lately about using different technologies that are less expensive than typical controls equipment.

“Why not use Arduino? Why not use virtualized PCs? Why not use the cloud?”

These questions really trouble me.

“Cloud computing” generally relates to computing resources being pooled in third party data centers and their operation being shown as a “cloud” on network diagrams.

Three current examples of cloud computing are solutions from Microsoft Azure, Amazon AWS, and Google Cloud Platform.

Some of the benefits of cloud computing are:

You can immediately spin up or down capacity as required with no capital expenditure. For example, if you’re running a website and your website becomes more and more popular, you can press a button and Amazon will dedicate additional computing resources to your website.

Your resources aren’t tied to a single point of failure like a single server. Most virtualized computers can be swapped between different servers to allow failover without any disruption of services. This can mean that if server hardware is failing, it can be replaced with no impact on the provided services.

Another company can focus on the specifics of managing large IT infrastructure and can carry the risks of investing in individual pieces of IT infrastructure

By having many huge companies pooling their data infrastructure requirements together, the total pool increases, which provides some safety. For example, it’s very difficult to DDOS Amazon AWS compared to a single corporate file server located in a company head office. In addition, in the event of such attacks, a single company specializing in network infrastructure can have experts on hand to work on such risks, where a single company may not be able to afford an expert for a few servers.

So those are some benefits, but people honestly suggesting this solution really troubles me. The reason is that they show a lack of appreciation for the gravity of your control devices.

What do I mean by this?

Well there’s a number of things.

Before we start with any of the reasons, I need to explain the difference between two types of failure: Revealed and Unrevealed failures.

Revealed failures are failures that are immediately apparent. They shut down devices, or they immediately cause a process upset. These failures are immediately found, and likely immediately fixed.

Unrevealed failures, by contrast, are not immediately apparent.

A few years back I was facilitating Reliability Centered Maintenance analyses. In many of the scenarios we covered, a motor would overheat and fail, and the control systems would shut it down. A relay would fail, and the control system would shut it down. A boiler would reach overpressure, and the control system would shut it down.

Then we started covering different control systems. Failure wouldn’t immediately harm anything, but when other problems occurred, that’s when the problem starts. Suddenly, the control systems wouldn’t be shutting down failing systems. In fact, they might be actively working to make things worse.

I’ve witnessed the trouble caused when control systems go down. Operators rely on their control systems more than they realize to tell them exactly what’s going on. If the alarm doesn’t sound, if the screen doesn’t light up, if the flashing light doesn’t flash, then often they’ll have no way to know something bad is happening until something terrible has happened.

So now we have this critical piece of infrastructure we’ve sent to another site, perhaps thousands of kilometers away, that we’re relying on the Internet to maintain connectivity to.

The ability to spin up or spin down CPU resources is sort of a non-starter, in terms of relevant benefits. CPU resources are not usually a bottleneck in PLC or DCS systems. You might have an issue if your PLC scantimes are getting too high, but PLCs are already using CPUs that have been obsolete for 15 years — if CPU resources were a primary consideration, then that wouldn’t be the case.

The risks associated with controls infrastructure are not the same risks associated with IT infrastructure. That being the case, I don’t think companies like Microsoft, Google, or Amazon are better prepared to manage those risks than companies that deal with the real risks every day. We’re not talking about having to issue refunds to some customers, we’re potentially talking about dead people and a pile of smoking rubble where your plant used to be.

Whether the data center side of your infrastructure is safe from DDoS attack or not, your plant likely is not. Therefore, a potential attack, or a potential loss of connectivity has the capability not just of taking down your email and intranet, but your plant.

Along the same lines, industry best practices for security uses a “Defense in depth” strategy, where your core devices are protected by layers and layers of security. By contrast, a cloud strategy puts your control data directly onto the public internet. If your system has a vulnerability (and it will), then you’re going to be risking your people, and the environment, and your plant, and your reputation.

Industrial network security generally places availability at the highest priority, and security as important, but secondary. If your controller can cause an unrevealed release of toxic gas if it gets shut down, it’s important that the controller continues to function. If that controller stops controlling because a security setting told it to, then that’s going to be a question to answer for later.

A single point of failure can be a good thing, or a horrible thing. In September of this year, Amazon AWS suffered an outage. Half a dozen companies that rely on them suffered reduction in availability as a result. By contrast, not a single system in my plant realized it. What if you were using AWS to control your plant?

Now that we’ve covered cloud computing, let’s look at Arduino.

Arduino is an open source hardware platform based on ATmel microcontrollers. An arduino provides a circuit board that lets people use breadboard materials to connect to their microprocessor; a USB programming interface; and a software package that makes it relatively simple to write software for the controller and send your program to the microcontroller.

Benefits? Well, an arduino comes cheap. You can buy an arduino board for 20 Euro from arduino.cc.

Downsides? There’s a lot, actually.

Let’s start from the outside and work our way in.

Arduino I/O isn’t protected with optoisolators. The analog IO isn’t filtered. None of it is certified for industrial use, and I’m guessing any insurance company would tear you apart for directly connecting an Arduino to plant devices, especially safety related ones.

Arduino is “modular” in a sense that you can plug devices called “shields” onto the main board, but this is not modular the same way a Schneider Quantum or an Allen Bradley ControlLogix. You can get a shield onto the board. Once it’s on, it’s on until you power down, and don’t expect to get 16 different kinds of card mounted together.

Arduino isn’t hardened on the power side, either. Noisy power could cause problems, and unless you break out a soldering iron and start building a custom solution, multiple power supplies aren’t an option.

Arduino isn’t protected physically. Admittedly, most PLCs are only IP20 rated, but at least that means you can’t stick your fingers on the exposed board, which isn’t something you can say for Arduino.

Arduino has a fairly limited amount of logic capacity. The number of things you can do with one are limited to begin with, and become more limited as you add things like TCP/IP for HMI access.

Speaking of logic, don’t expect to change logic on the fly without shutting down the controller, or view logic as it is running. That’s just not possible.

With Arduino, you’re likely to be reinventing the wheel in places. Want Modbus TCP? Hope you’ve brushed up on your network coding, because you’re going to be writing a Modbus TCP routine for everything you’d like to do.

As well, congratulations on your new Arduino installation! The guy who wrote the software quit. Can a regular electrician or instrument guy muddle their way through the software?

I’m not saying you can’t build a PLC using an arduino, or that you can’t control a process with one. What I am saying is that if you’re doing industrial control, it probably isn’t the right tool for the job.

I’m going to save Virtual Machines for another time, because there’s a trick in there I want to expand on a bit more.

Thanks for reading!

Categories
Management and supervision

This one is for the travellers

I’m Jason Firth.

Fly-in Fly-out jobs are becoming commonplace in Canada, because the places where resources happen to be are not the places people want to live, or places where it’s not practical to place a down, or simply because the skills you need are not in the same place as where you need them.

I’ve been flying a lot lately in Ontario, and so I’ve been flying a lot on Porter Airlines.

I noticed something interesting in their frequent flier program. They have a “goal meter” on their website for their VIPorter program. It shows the $1500 level before you get to the Passport level, and the $3000 level before you get to the Priority level, but after that it shows $10,000.

How bizzare? I decided to look further into it.

It turns out Porter has another level to their frequent flier program. It’s invitation only, and requires a $10,000 annual spend on flights. It’s called the “VIPorter First” program.

It looks like it has everything the VIPorter Priority status has, and in addition:

2 Free checked bags

Free premium seat selection

Free last seat guarantee on sold-out flights

Free same day changes to reservations

Now, another note for people who travel very frequently:

Your spending shows up on your VIPorter level when you have completed the flight, not when you book your flight. By contrast, your VIPorter level when you book the flight is the level the flight will be treated as in their computer system, not the level you’re at when you fly.

This means that if you buy 6 months of tickets in advance in January, you won’t VIPorter status for any of those flights, in spite of potentially spending more than the $3000 level with them, even once you’ve taken $3000 worth of flights. Your new status won’t kick in until you’ve purchased a flight AFTER your status has activated.

Thanks for reading!

Categories
Management and supervision

5 reasons I gave up being an instrument guy (for now)

I’m Jason Firth.

Recently, I accepted a new role within my organization for a while: I’m assisting with a project to deploy a new Enterprise Resource Planning (ERP) system. My role is to bring maintenance expertise to the project specific to the plant I’m stationed at.

When I told my team about it, my partner was confused: “Why don’t you want to be an instrument guy anymore?”, he asked.

In fact, it’s exactly because I’m an instrument guy and because I plan to continue being an instrument guy that I accepted the role!

Let’s look at some of my reasons.

1. Instrument technicians are information mongers.

There’s no two ways to put this: Instrument Techs, or at least good instrument techs, are information mongers. Every piece of information we gather is another tool in our belt that we might make use of.

This project is going to leave me and my shop with comprehensive lists regarding equipment and maintenance for everything on site. We’re going to have access to more people and more information than we ever would have had as just instrument guys. That sort of information is invaluable at moments you’d never have imagined — because until you have it, you never consider it.

Everything is connected, and the more information you have, the easier you can make the connections yourself. The more you can make those connections on your own, the more effective you can be.

2. Potential synergies between these two roles.

I always hate to use the S-word, but it’s absolutely true. I’ll explain.

Work that’s been done as part of other completely unrelated projects suddenly becomes relevant, and you don’t have to do anything but cross-reference. This means the project benefits from that work. Work I previously did on SCADA, establishing equipment taxonomies and working within constraints, suddenly becomes extremely important because it’s already done.

As well, work that’s done today may have a dramatic impact on future trades projects. In particular, having a say as to how business critical systems are set up on Day 1 means those same systems are structured in a way that might facilitate the information’s re-use later. All it takes is a little vision, and you can make everyone’s life easier in those future projects.

3. The wrong person doing this can sink entire shops.

The Computerized Maintenance Management System (CMMS) aspect of ERP is one of the most important elements of a modern shop. It facilitiates communication between operations and trades, it stores information about the history of work done, it keeps track of costs and of time spent on jobs, and it plays a key role in material management.

If the wrong person is setting up the system, communication between operations and trades may become ineffective, history may be lost or unusuable, costs can’t be tracked, and materials can’t be found. All this adds up to a skilled worker not spending time using their skills.

4. The right person doing this can let us spend more time being tradesmen.

Along the same lines, if the right person sets up the system to its potential, communcations between operations and trades (and between trades and other trades) may be enhanced, history may become a key tool in predicting failures or detecting current failures, costs and time spent on jobs can be effectively measured and managed, and materials will always be where they need to be when they’re needed.

Modern ERP systems also allow supervisors to assign jobs to certain individuals, and to allow those individuals to see their work queues on mobile devices, so the work is always at their fingertips. They can allow test results to be stored and historized immediately without additional paperwork. They can allow work completion comments to be added directly when a job is complete, increasing the speed and accuracy of the history. They can even allow documentation to be carried around for instant access to key information.

All this adds up to one thing: Tradesmen spending less time on paperwork, and more time on trades. That’s good for the business, it’s good for the tradesmen, and it makes everyone’s life a little easier.

5. Ultimately, you need a voice from the front.

There’s a lot of perfectly reasonable sounding suggestions out there.

It’s easy to sit around a desk and come up with this stuff, and the technology is amazing, you can implement anything you want. The problem is, how will it affect someone on the front line? It’s those people who are going to keep your plant running day to day, and even smart people, with the best of intentions, can make a decision that works very well in theory, but is disasterous in practice. Someone from the front line is absolutely neccessary. You need a canary to tell you when things could get bad.

Thanks for reading!

Categories
Management and supervision

Big legal in engineering technology land

I’m Jason Firth.

One part of the OACETT and CTTAM codes of conduct is a responsibility to learn the applicable laws and codes to your field. To that end, I often load up CanLII and search for information relating to the field of engineering technology, and people certified as engineering technologists.

I found this 1996 case, and it’s interesting to me.

This case relates to the construction of a paper mill in Alberta in the late 80s. There are 3 main groups: A contractor called Dilcon, an engineering company called NLK, and a corporation created solely for the purpose of the construction of this new paper mill, called ANC.

What is disputed is the amount of money owed to Dilcon by ANC. The amount disputed is quite substantial, with Dilcon claiming they are owed a further $20 million dollars, and ANC claiming they have overpaid by $10 million dollars.

At first glance, Dilcon sort of accepted a dangerously vague contract: They placed a bid on a job for which the detailed engineering was only around 10% complete. They came up with a fixed bid for a contract for which scope wasn’t even remotely defined yet. On the basic merits, this turned out to be as bad an idea as it sounds: In one area, the amount of work turned out to be literally twice as much as originally proposed, and in another area, it was 25% greater.

Luckily for them, the engineering company was set as the facilitator in contract measures, and acted in a fair and reasonable manner, allowing different additions to be considered as “extras” despite the fact that ANC ostensibly asked for a “no extras” contract.

The case provides a fairly in-depth look into how one major contract was negotiated and carried out, including a lot of detail about what looks like a fairly well done change management process. (the case centered around changes, but I don’t think the problems related to the processes in place)

In addition, it gives painstaking details about the consequences of not considering lead times.

I started my career as an engineering technologist working in an engineering department, designing and managing projects and ordering parts. Later on, as a technician, I had broad latitude to manage my projects and order parts as well.

One lesson I learned is this: Stakeholders want to ignore lead times, and will pressure you to do so. However; especially if you’re working in a remote site or remote community, failing to pay attention to lead times will result in looking like an idiot and wasting the company’s money.

In this case, NLK took too long to deliver certified engineering documents, which caused several vendors to be delayed in delivering materials. In other cases, the vendors simply took long to deliver, with one vendor taking 5 months longer than scheduled to deliver a key part. In another case, Dilcon was fully mobilized on site for months before an acceptable number of pipes had been delivered. By not properly accounting for the amount of time it would take for tasks to get done and for parts to arrive, the entire project suffered massive cost overruns in the millions of dollars.

Of course, you could say “But you should be able to plan based on the best possible information!”, which seems like a great idea on paper. In practice, if you don’t have any good reason to believe that a part will be at your site, I don’t think it’s a good idea to start lining up huge resources. All it takes is a lack of a single critical component and suddenly you’re not just not done, but you’re down.

That’s what happened in this case. The engineering wasn’t complete, the shipping wasn’t complete, the parts hadn’t arrived, and they’d just mobilized a bunch of trades workers to stand around doing nothing for months at a time.

This sort of bad planning has massive consequences. In this case, Dilcon did manage to get their contract completed in time, but at huge cost: The court found that they were owed a full 10 million dollar more by ANC — Considering the original contract was under $20M, that’s a huge cost increase — and all that was needed to avoid it was for someone to say “Slow down! We’re going to make sure we’re ready before we start signing contracts and hiring people.”

Thanks for reading!

Categories
Process Control

Modbus Plus communication to your PC without a $2000 USB Dongle

I’m Jason Firth.

A few months back, I had a discussion with someone about how to communicate with a PC over Modbus Plus without breaking the bank. I decided to take the highlights of that conversation and bring them together in a blog post so others might be able to use the information.

Modbus Plus is still a widely used protocol in industry where Schneider Modicon PLCs are in use. However; it’s not a cheap protocol to work with. A USB Modbus Plus dongle for your PC will run you over $2000!

That’s a lot of money if you just want to look at some bits in the PLC. Today, I want to look at some other options.

Modbus Plus is a completely different protocol than Modbus.

Modbus Plus uses a proprietary signalling standard, where Modbus generally uses RS-232 or RS-485. Modbus Plus supports routing, where Modbus has no networking features. Modbus Plus requires a DSP to handle the communications, where Modbus can use a standard UART. Modbus Plus is peer-to-peer, where Modbus is master/slave. Modbus Plus uses a token passing system to ensure everyone gets a turn on the line, where Modbus doesn’t have any mechanisms (hence requiring the master/slave architecture where the master dictates who will speak). You just set your addresses, and all the devices will start talking. The Bridge Multiplexer acts as a gateway between the two protocols.

Modbus Plus allows for routing between devices called Bridge Pluses. The way you do this is by defining the modbus plus address of each bridge plus you pass through. For example, if you have node 1 on a modbus plus network with a bridge plus node 64, and there’s a node 32 on the other side you want to communicate with, you’d be communicating with 64.32.0.0.0

So to make sure your modbus devices can talk over a modbus plus network, you need to map the single values to modbus plus addresses.

Modbus plus requires a DSP. This means that no matter what, you’re going to need a dedicated piece of hardware to communicate on the network. You can’t just slap an RS-232 pigtail together and hope it will work.

Here are two potential options: first, if you get a PC with an ISA slot, the cards are quite inexpensive. I found some on eBay for 100-200 USD. A PICMG backplane can use an ISA slot, and there are industrial main boards still available. The downside to this is that you may be stuck using a very slow PC just to communicate with your one PLC.

Another option is a bridge multiplexer, which converts modbus plus to modbus. It takes a bit to put together, but it should work. I found a bridge MUX on eBay for 100usd when I first investigated this option. Today, I’ve found them for under $250 on eBay.

The manual really overcomplicates things.

There are models on page 10 of the manual — nw-bm85-000; NW-bm85C000; and NW-BM85D008 which don’t need a special progran. You don’t need to make a C++ program. You connect to one of the serial ports (I think the second one) and put the bridge into programming mode by flipping a DIP switch on the back, then it gives you a fairly nice and easy menu based interface to configure what it does.

You’ll need to figure out master/slave stuff, because modbus is client/host but modbus plus is peer to peer, but it should be very doable.

The master in a modbus interaction is the device that actually sends the commands. For example, I programmed a modbus TCP library, and in that case, the device that initiates the connection is the master: you connect, then either tell the device you want to read or write a coil or register, then the slave device that you connected to will respond with either a success/fail message for a write, or the data you wanted or an error message.

The Peer-to-peer feature is relevant because it means nothing on the Modbus Plus network is a master or a slave. They’ll just take care of communications on their own. In my tests, the only problem I could cause to the modbus plus network is if you set your bridge mux to the same address modbus plus address as something else on the modbus plus network. If you set a wrong modbus setting in the text interface, all you’re going to do is not communicate with the Modbus Plus network over the Modbus port.

Here’s how I managed to talk to a PLC using my PLC software and a BM85 connected to my serial port.

I set my modbus plus address of the BM85 to 2 by turning the first switch on and the rest off.

I entered config mode using the dip switch on the back

Once the screen loaded, I entered the following commands:

V1

P1

Tm

B9600

S1

Re

Mr

Y1

V2

E1 01 20 00 00 00 00

V4

W

Press y to confirm

Turn off bm85 and return config mode to run mode.

One key thing seems to be the rs-232 cable from the PC to the BM85. I used a 990NAA26320 but the wiring diagram should work ok to make a cable similar.

So what you’ll have is:

Your PLC, and bm85 daisy chained together on the modbus plus network.

Your PC plugged into the BM85 on modbus port 1.

Your BM85 set to a free Modbus Plus port

Your PLC set to whatever it is (no change)

Your bm85 configured using the keystrokes above, in run mode.

I use Fasttrak softworks, so I open it up and set it up to look at the COM port. Next, I did a PLC connect and did a port scan. I saw the BM85 and the PLC. Next, I chose the PLC from the list and hit connect. I was talking to the PLC!

So what’s going on: In this situation, you have two types of communication going on: The Modbus Plus communication, and the Modbus communication. So the PC acts as the master in a modbus communication. It sends commands to the bridge mux. The bridge mux, while configured with the master option, is actually acting as a slave on the modbus port: It is only responding to commands the PC master provides. The BM85 receives the message, and saves it so it can send a corresponding message on the modbus plus network to the PLC.

Now, you have another network, the Modbus Plus network. What’s going on there is, each device gets a token which is its turn to speak, and it says what it has to say on the modbus plus network and passes the token to the next device.

If the PLC and the BM85 are all configured, all this is transparent behind the scenes.

So let’s talk troubleshooting.

If you seem to have all the right pieces connected in the right way, I’d start from the outside and work my way in.

Can you connect to a PLC using your directly using the serial cable? If so, then you’ve proven the converter and the cable. I know that those serial converters can be flaky — If you plug them into different USB ports, they’ll use different COM addresses. You can check device manager to make sure you’re configured for the COM port you think you’re using.

What is your BM85 Modbus Plus LED doing? The following are the flash codes for Modbus Plus:

Six flashes/second Normal operating state. All nodes on a healthy network flash this pattern.

One flash/second The node is off-line. After being in this state for 5 seconds, the node attempts to go to its normal operating state.

Two flashes, then OFF for 2 seconds The node detects the network token being passed among other nodes, but it never receives the token.

Three flashes, then OFF for 1.7 seconds The node does not detect any token passing on the network.

Four flashes, then OFF for 1.4 seconds The node has detected another node using the same address.

If the light is flashing 6 flashes per second, then it suggests that your Modbus Plus network is working correctly.

Thanks for reading!

Categories
Management and supervision

Certified Engineering Technologist? Not here, you aren’t!

I’m Jason Firth.

There’s one thing most people don’t know about the law that they should: The law isn’t the same everywhere.

Often, people will talk about “how things are”, as if their experience in their location describes everyone’s. That’s incorrect, and it’s quite dangerous.

In an earlier entry, I talked about what it takes to become a Certified Engineering Technologist, and in another entry, I talked about what it takes to become a red seal Journeyman. I know first-hand about these things because I went through the process in 2013.

However; in 2013, I also made a mistake. I applied for, and achieved, my Certified Engineering Technologist designation in Manitoba. At the time, I didn’t know if I was covered nationwide, so I called the Canadian Council of Technicians and Technologists and asked if I could use my designation across the country. They told me it was fine.

They were not being entirely truthful. In 2010, the governments of British Columbia, Alberta, Saskatchewan, and Ontario split from the Canadian Council of Technicians and Technologists to create Technology Professionals Canada, a new organization dedicated to the profession of Engineering Technology in Canada.

As a result, and as a result of the wording of Section 11 of the Ontario Association of Certified Engineering Technicians and Technologists Act, 1998, S.o. 1998 C.Pr7, the use of the CET designation is restricted and it is an offense for anyone who is not a full member of OACETT to use the title.

Not realizing that the title didn’t automatically transfer like a red seal, I used my CET title in Ontario, only to receive a Cease and Desist letter from OACETT’s lawyers.

In my case, I asked about my options as a member of CTTAM, and the lawyer told me:

1. You can maintain your primary membership in Manitoba and apply to OACETT as an out-of-province member. You will pay full dues to Manitoba. You will need to pay out-of-province member’s dues in Ontario which are one-third of what a regular member pays;

2. You can transfer your membership to Ontario; or

3. You can transfer your membership to Ontario and maintain out-of-province status with Manitoba (assuming Manitoba has this provision).

After paying a small fee, I was able to transfer my membership to Ontario without any further difficulty. It took about a month, during which I stopped using my designation in Ontario.

I ended up taking the third option, transferring my primary membership to the province I practice in, and using an out-of-province membership (at a cost of about $100/yr) in Manitoba.

Something to keep in mind!

Thanks for reading!

Categories
Management and supervision

You need your own competence!

I’m Jason Firth.

So, now that oil has dropped dramatically, oil companies are discovering something I’ve always said: You need your own competent people.

The oil and gas industry has found out in a very abstract way: They’ve realized huge cost overruns on projects since they decided to let other companies monopolize engineering competence. When they had their own people working on the projects, they didn’t have the same cost overruns.

There’s a number of reasons why this is the case.

Let’s start off with what I like to call institutional memory. This is the effect of having people who have worked in your facilities for years making decisions about those same facilities. You can’t outsource institutional memory. It’s something that either belongs to you, or is lost. When you’re making decisions that affect the short-term viability of a project or the long-term viability of a facility, there’s a million tiny facts which you need to know, and you won’t learn in a 2-day site visit. Those folks don’t know about the fifteen other solutions that were tried on a problem until the sixteenth worked. They’re stuck asking why something is done a certain way. Sometimes this means they’ll be going back to the failed solution 1.

Then, there’s the pure question of “Who do you work for?” — An outside company will always be looking out for their own bottom line. If a conflict arises between the interests of your company and theirs, they are going to be more likely to act in their best interests.

Along those same lines, if you’re dealing with your own talent, then you can make sure you’ve got the best talent on the most important jobs. I’ve seen it in the past where an outside firm will wow a client with the A-team, then once the contract is signed, send in the Z-team.

Speaking of contracts: Of course everyone agrees that scope creep is dangerous. However; there’s a certain flexibility that comes with dealing with your own people that allows economies. For example, I was working on one big project with an engineering firm, and another different project was going on at the same time. The engineering firm refused to do anything that wasn’t part of the plan without negotiating a scope change and fee increase. Meanwhile, I plugged two cables in, successfully connecting the projects together, and changed a couple lines on a CAD drawing.

The formality of dealing with another company can also be a danger when you rely on other firms to have your competence. While working for a company, there’s been a number of times when I’ve been working on some skunkworks for a company alongside the main, official jobs. Skunkworks is a term that comes from Lockheed Martin’s Skunk Works development group, which developed the U-2, the SR-71 Blackbird, the F-117 Nighthawk, and the F-22 Raptor. The designation is widely used in business, engineering, and technical fields to describe a group within an organization given a high degree of autonomy and unhampered by bureaucracy, tasked with working on advanced or secret projects. The concept of that autonomy and relaxation of bureaucracy is incompatible with the formality of a business to business arrangement.

All of these problems reduce economies and prevent innovation within companies, and facilities. You can’t outsource your institutional memory to another company. Your employees will always be the most aligned to your interests. You are only in full control of your own employees. Contract work can be inflexible and cause large cost increases. You can’t give other companies autonomy the way you can give your employees autonomy.

Thanks for reading!

Categories
Process Control

What’s inside a Modbus Plus connector?

I’m Jason Firth.

Eventually, I knew I would be writing at length about Modbus Plus.

Modbus Plus is the protocol that Modicon created to supersede the Modbus protocol. It has some superficial similarities, but it quite different under the hood.

Modbus is a Master/Slave protocol. One device is the master, and tells which Modbus slave to talk. By contrast, Modbus Plus is a peer-to-peer protocol. Each Modbus Plus device can request data from any other device.

Modbus doesn’t have any real way to manage congestion, because there should never be congestion. The master requests data, and will not request more data until the first is sent. By contrast, Modbus Plus uses a token passing mechanism, where each node in the network gets a chance to talk, then once it has finished talking it will pass the token to the next node.

Modbus generally relies on RS-232, RS-422, or RS-485 signalling to communicate. By contrast, Modbus Plus uses a proprietary signalling protocol over a single twisted pair of wires. Modbus can be implemented using a standard UART, where Modbus Plus requires a special DSP.

Physically, Modbus Plus is a bus protocol. All devices are electrically connected to every other device on the Modbus Plus network through an electrically continuous shielded twisted pair cable.

Modbus basically doesn’t have network capability by itself. However, Modbus Plus has basic networking capability. Using a device called a “Bridge Plus”, you can connect different Modbus Plus networks together. The address you use to connect to a device is actually the path of devices you follow to get to the other device. If you were connecting to device 1 on the local Modbus Plus network, then you’d connect to 1.0.0.0.0. If you were connecting through a bridge multiplexer at address 2, then you’d connect to 2.1.0.0.0. If you went a step further and connected to 1 through yet another bridge multiplexer across the network at address 3, then you’d use address 2.3.1.0.0.

The data transferred using Modbus and Modbus Plus is roughly equivalent. You can send and receive inputs and coils, inputs and registers. There are also other operation codes in the protocol for diagnostics, or programming PLCs, or a number of other functions.

There are 3 types of connector routinely used for Modbus Plus. The AS-MBKT-085 inline connector takes a Modbus Plus cable and stabs the wires, to establish continuity. The 990NAD23000 Tap takes that cable and stabs the wire into a “tap”, which connects to a moulded cable which connects to the device. The 990NAD23020 or 990NAD23021 Supertap doesn’t stab the wire, instead using screw terminals to connect the Modbus Plus cable and the moulded drop cable.

Today I want to look at the AS-MBKT-085 inline Modbus Plus connector. We’re going to look inside one.

The first thing you need to realize about these connectors is, they’re not cheap. This store is selling one for 35 USD, and that’s about what you can expect to pay. I’ve seen some online stores asking for twice that.

So, what are you getting for your money? Well, first, let’s look at what this thing does.

Here, you can see a Modbus Plus connector with wires already crimped into it.

Normally, the shield grounds to the middle pad, and the wires are held in place by the plastic back, which is itself held to the connector with a screw.

The stabs which hold the wires are fastened to the connector with screws, so we’ll pull them out.

I expected the connector to be welded shut, but it’s actually held in place with 3 pegs which push into 3 holes. Once I had a point I could leverage, I could pry the connector apart.

So here’s what we have: 3 metal block with a threaded hole, connected to a regular D9 connector.

Taking apart the connector further, I found that the ground is coupled to the chassis ground using a capacitor. That’s all there is to it!

And how about those light grey plugs? The difference is that there’s a 120 Ohm resistor across the data pairs. That’s the difference.

Thanks for reading!

Categories
Management and supervision

Connected

I’m Jason Firth.

Today, when people use the word “Connected”, they’re usually talking about ethernet networks. That’s not what I’m talking about today.

Quite often as an instrument technician or technologist, I get asked to help participate in jobs that don’t immediately appear to be related to my trade. In one case, I was asked to participate in the Reliability Centred Maintenance initiative. In another, I was asked to help set up a site’s maintenance management system. A lot of people heard this, and immediately went “That’s not in your job description!”

As much as that might be true (depending on how the job description is worded), I belive that everything we do is connected.

Let’s look at the workflow for RCM. The very first task is to get a complete list of equipment at your site. Occasionally you already have that list, but often you do not. At one site I worked at, gathering a list of equipment was a massive undertaking, requiring cross-referencing of every single piece of information available, from the existing maintenance management system, to maintenance records, to data from the PLC and SCADA, to some good old fashioned leg work. This might not be part of the job description, but it certainly makes life easier having that information. Suddenly we know exactly what we have, where we didn’t before.

The next task is to sit down with operations personnel, and get them to help define the function of every piece of equipment. You record that definition for later. Let’s look at the incredible value you get here: Not only do you personally get to sit down and learn one at a time exactly what operations believes each piece of equipment does, but you’re recording it so anyone can look it up. You can gather this information together later to improve operations documentation, or to add invaluable context to the PLC or DCS program, or to produce complete disaster planning.

Next, you determine the dominant failure modes; the scenario for each failure mode; criticality of each failure; what the consequences are of failure; and what maintenance will be done to address each failure. This might seem incredibly specific to preventative maintenance, but imagine the value in having this available on a minute-to-minute basis. Imagine getting a trouble call and knowing immediately how it can be prioritized in a situation where multiple breakdowns are occurring. Having that information closely at hand could mean the difference between a plant facing safety hazards, environmental impact, capital damage, or productive loss, and not.

Finally, by having an instrument guy in the room, sometimes new solutions to failures can be found. Systematically discussing everything with operations personnel and other maintenance personnel can be a great catalyst for innovative solutions to problems that have plagued a site for ages.

As for helping to improve a maintenance management system, that’s something you don’t realize you needed until you have it. A properly set up system will have all your different pieces of equipment, so you can build a history of things you’ve done to that equipment. History is a key information source in Reliability Centred Maintenance. A really well done MMS will fundamentally change how reactive maintenance gets done. When a tradesman recognise that a piece of equipment needs to be changed out, instead of immediately hunting for some indication of what the part they need is, they can immediately head to stores with the tag number and recieve the exact part they need from the asset Bill of Materials.

These are just two examples of how things that don’t seem to be at all connected to what we do can nonetheless make our day to day work easier. “That’s not in your job description!” is a great way to make that job harder.

Thanks for reading!

Categories
Industrial IT

Really Disappointing

[Note: This situation changed since I wrote this in 2014/2015]

I’m Jason Firth.

I want to talk a bit about a trend I’ve been seeing.

First, some background to understand where I’m coming from.

Downloading a piece of software on the Internet is a real gamble.

Of course, some software contains viruses meant to destroy your PC, but that’s a relatively small amount. To really understand where the software that will really harm you comes from, you need to follow the money. Some people make money stealing proprietary secrets from companies, or credit card numbers, or passwords for paid accounts on services like netflix. Other people make money selling advertising displayed through unscrupulous means. Still other people make their money controlling “botnets”, which are large numbers of computers (computers owned by normal people) infected with worms that take control which allow buyers to send specially crafted packets to servers intended to tie up resources like bandwidth and memory in order to prevent that server from operating, in an attack called a “Distributed Denial of Service”, or DDoS.

There are a few ways you can be infected (called ‘vectors’). In 1999, my first website was hosted on a free web host, and I visited my own website to learn it was trying to install a spyware program on my PC (I soon after started paying for ad-free hosting). In 2000, the ILOVEYOU worm moved through e-mail, and the e-mail program outlook at the time would automatically run the infection script when the e-mail was viewed. 2004, the sasser worm could infect a PC that was simply connected to the Internet, without the owner of the computer doing anything wrong. Along the way, programs that were (or appeared to be) useful eventually started being bundled with a new kind of worm, that served advertising content even when the program wasn’t running. Some applications installed this software without any indication that it was doing anything. Others would try to hide their request to install behind deceptive wording or deceptive button placement. Either way, once installed, the new program would display advertising on your PC even when you weren’t using the application that installed the ad program — in fact, sometimes even after the original program was uninstalled.

My first job was as a registered apprentice helpdesk support analyst for a school board. One of the challenges they faced at the time was removing the programs installed by peer to peer file sharing programs (it was a more innocent time before such computers were completely locked down). These programs would cause web advertisements to appear, and used resources that the old Pentium 133MHz machines with 32MB of RAM couldn’t really afford to give up.

If you want to be assured that software from the Internet is safe, you have only a few options. You can download only from trusted proprietary sources like Microsoft. You can just not download anything at all. A third option for a long time was Free and Open Source software. This software is written by hobbyists or by companies who want to build an open platform, and is licensed so that everyone who wants to use it can use it, and anyone can access the source code and use it, but only if they release any modifications for use by others. (There are other free licenses which are broader, but we’ll discuss this one for now). These programs didn’t usually come with problem software, because anyone could check the code, and compile a clean version from scratch.

Unfortunately, something has changed. The largest distributor of open source software, called Sourceforge, changed owners a few years back, and now they encourage top projects to try to install this problem software on their users computers.

I’ve contributed to open source projects over the years. I’ve written documentation, I’ve written code, I’ve even started projects. I believe in the idea. To see big projects that people like me have put their heart and soul into and to have it used to try to unscrupulously infect other people’s computers, I consider that criminal, and I consider it a tragedy.

I consider it criminal because any consent could only be possibly be gained through deception, because no reasonable person would allow such software to be installed on their computer. “Hey, want me to put advertising on every webpage you visit, even the ones without advertising? Want me to randomly pop up advertising all the time? Want all this to happen even when you’re not using my program?” — of course the answer is ‘No’. I’ve seen them get an “I agree” click by using deceptive language (“Click if you don’t want to unagree that we won’t not install the software”), or by using deceptive button placement (“next next next next next I agree to be infected with a virus”). I don’t consider this access to one’s computer legitimately authorized, and thus taking over people’s computer is a cyber-crime. Governments around the world appear to agree with me, because laws are being passed all the time to make it perfectly clear for the courts that this sort of behaviour isn’t acceptable.

As for why I consider it a tragedy, imagine the passion of people contributing to their favourite open source projects, working for no compensation. Their reward? To have the project maintainer try to take over their computer for monetary gain. That’s tragic. It really is the tragedy of the commons here, where someone realized they could burn it all to the ground to make a buck. That’s very sad.

Anyway, back to instrumentation next time. I just had to vent about the end of an era: The era where you could at least trust an open source project.

Thanks for reading!