Categories
Management and supervision

Connected

I’m Jason Firth.

Today, when people use the word “Connected”, they’re usually talking about ethernet networks. That’s not what I’m talking about today.

Quite often as an instrument technician or technologist, I get asked to help participate in jobs that don’t immediately appear to be related to my trade. In one case, I was asked to participate in the Reliability Centred Maintenance initiative. In another, I was asked to help set up a site’s maintenance management system. A lot of people heard this, and immediately went “That’s not in your job description!”

As much as that might be true (depending on how the job description is worded), I belive that everything we do is connected.

Let’s look at the workflow for RCM. The very first task is to get a complete list of equipment at your site. Occasionally you already have that list, but often you do not. At one site I worked at, gathering a list of equipment was a massive undertaking, requiring cross-referencing of every single piece of information available, from the existing maintenance management system, to maintenance records, to data from the PLC and SCADA, to some good old fashioned leg work. This might not be part of the job description, but it certainly makes life easier having that information. Suddenly we know exactly what we have, where we didn’t before.

The next task is to sit down with operations personnel, and get them to help define the function of every piece of equipment. You record that definition for later. Let’s look at the incredible value you get here: Not only do you personally get to sit down and learn one at a time exactly what operations believes each piece of equipment does, but you’re recording it so anyone can look it up. You can gather this information together later to improve operations documentation, or to add invaluable context to the PLC or DCS program, or to produce complete disaster planning.

Next, you determine the dominant failure modes; the scenario for each failure mode; criticality of each failure; what the consequences are of failure; and what maintenance will be done to address each failure. This might seem incredibly specific to preventative maintenance, but imagine the value in having this available on a minute-to-minute basis. Imagine getting a trouble call and knowing immediately how it can be prioritized in a situation where multiple breakdowns are occurring. Having that information closely at hand could mean the difference between a plant facing safety hazards, environmental impact, capital damage, or productive loss, and not.

Finally, by having an instrument guy in the room, sometimes new solutions to failures can be found. Systematically discussing everything with operations personnel and other maintenance personnel can be a great catalyst for innovative solutions to problems that have plagued a site for ages.

As for helping to improve a maintenance management system, that’s something you don’t realize you needed until you have it. A properly set up system will have all your different pieces of equipment, so you can build a history of things you’ve done to that equipment. History is a key information source in Reliability Centred Maintenance. A really well done MMS will fundamentally change how reactive maintenance gets done. When a tradesman recognise that a piece of equipment needs to be changed out, instead of immediately hunting for some indication of what the part they need is, they can immediately head to stores with the tag number and recieve the exact part they need from the asset Bill of Materials.

These are just two examples of how things that don’t seem to be at all connected to what we do can nonetheless make our day to day work easier. “That’s not in your job description!” is a great way to make that job harder.

Thanks for reading!

Categories
Industrial IT

Really Disappointing

[Note: This situation changed since I wrote this in 2014/2015]

I’m Jason Firth.

I want to talk a bit about a trend I’ve been seeing.

First, some background to understand where I’m coming from.

Downloading a piece of software on the Internet is a real gamble.

Of course, some software contains viruses meant to destroy your PC, but that’s a relatively small amount. To really understand where the software that will really harm you comes from, you need to follow the money. Some people make money stealing proprietary secrets from companies, or credit card numbers, or passwords for paid accounts on services like netflix. Other people make money selling advertising displayed through unscrupulous means. Still other people make their money controlling “botnets”, which are large numbers of computers (computers owned by normal people) infected with worms that take control which allow buyers to send specially crafted packets to servers intended to tie up resources like bandwidth and memory in order to prevent that server from operating, in an attack called a “Distributed Denial of Service”, or DDoS.

There are a few ways you can be infected (called ‘vectors’). In 1999, my first website was hosted on a free web host, and I visited my own website to learn it was trying to install a spyware program on my PC (I soon after started paying for ad-free hosting). In 2000, the ILOVEYOU worm moved through e-mail, and the e-mail program outlook at the time would automatically run the infection script when the e-mail was viewed. 2004, the sasser worm could infect a PC that was simply connected to the Internet, without the owner of the computer doing anything wrong. Along the way, programs that were (or appeared to be) useful eventually started being bundled with a new kind of worm, that served advertising content even when the program wasn’t running. Some applications installed this software without any indication that it was doing anything. Others would try to hide their request to install behind deceptive wording or deceptive button placement. Either way, once installed, the new program would display advertising on your PC even when you weren’t using the application that installed the ad program — in fact, sometimes even after the original program was uninstalled.

My first job was as a registered apprentice helpdesk support analyst for a school board. One of the challenges they faced at the time was removing the programs installed by peer to peer file sharing programs (it was a more innocent time before such computers were completely locked down). These programs would cause web advertisements to appear, and used resources that the old Pentium 133MHz machines with 32MB of RAM couldn’t really afford to give up.

If you want to be assured that software from the Internet is safe, you have only a few options. You can download only from trusted proprietary sources like Microsoft. You can just not download anything at all. A third option for a long time was Free and Open Source software. This software is written by hobbyists or by companies who want to build an open platform, and is licensed so that everyone who wants to use it can use it, and anyone can access the source code and use it, but only if they release any modifications for use by others. (There are other free licenses which are broader, but we’ll discuss this one for now). These programs didn’t usually come with problem software, because anyone could check the code, and compile a clean version from scratch.

Unfortunately, something has changed. The largest distributor of open source software, called Sourceforge, changed owners a few years back, and now they encourage top projects to try to install this problem software on their users computers.

I’ve contributed to open source projects over the years. I’ve written documentation, I’ve written code, I’ve even started projects. I believe in the idea. To see big projects that people like me have put their heart and soul into and to have it used to try to unscrupulously infect other people’s computers, I consider that criminal, and I consider it a tragedy.

I consider it criminal because any consent could only be possibly be gained through deception, because no reasonable person would allow such software to be installed on their computer. “Hey, want me to put advertising on every webpage you visit, even the ones without advertising? Want me to randomly pop up advertising all the time? Want all this to happen even when you’re not using my program?” — of course the answer is ‘No’. I’ve seen them get an “I agree” click by using deceptive language (“Click if you don’t want to unagree that we won’t not install the software”), or by using deceptive button placement (“next next next next next I agree to be infected with a virus”). I don’t consider this access to one’s computer legitimately authorized, and thus taking over people’s computer is a cyber-crime. Governments around the world appear to agree with me, because laws are being passed all the time to make it perfectly clear for the courts that this sort of behaviour isn’t acceptable.

As for why I consider it a tragedy, imagine the passion of people contributing to their favourite open source projects, working for no compensation. Their reward? To have the project maintainer try to take over their computer for monetary gain. That’s tragic. It really is the tragedy of the commons here, where someone realized they could burn it all to the ground to make a buck. That’s very sad.

Anyway, back to instrumentation next time. I just had to vent about the end of an era: The era where you could at least trust an open source project.

Thanks for reading!

Categories
Instrumentation

ProComSol DevCom2000 first impressions

I’m Jason Firth.

Modern electronic instruments are almost never purely analog devices anymore.

The reason is really simple: When you’re dealing with analog signals, you’re constantly worrying about introducing error. Every trace on a circuit board, every op-amp, every transistor has the chance to introduce non-linearity, noise, or to mess up your scaling. By contrast, once you digitize your signal, that’s it — your signal is what your signal is, and from there you can process it or analyze it however you like without degrading the signal, until you spit out your signal, either through a fieldbus or through an analog signalling standard like 4-20mA.

If you have an instrument with a processor that communicates using a 4-20mA analog signal, then you’re sort of out of luck with respect to advanced configuration and diagnostics over that line, right?

Not exactly. Since a 4-20mA signal is very slow, and most analog input devices have filtering built-in to read only that fairly slow signal, you can overlay a very fast signal on top of the 4-20mA. A communications protocol called HART overlays a 300 baud analog modem signal (Remember modems?) on top of the 4-20mA, allowing the instrument and a user interface to communicate digital data.

There are 2 main standards for device drivers for communicating with HART devices: DTMs(Device Type Managers), and DDs(Device Descriptions). DTMs function primarily with a piece of software called “PactWARE”, whereas DDs are used on Rosemount HART communicators such as the 475.

HART communicators are nice because they are single purpose devices: They do one thing, and presumably they do it well. However; there are benefits to using a PC for communicating with instruments. PCs have virtually unlimited storage space, memory, and CPU power compared to a HART communicator. They have full network capabilities. They also have a much larger screen and keyboard.

I’m a fan of instrument techs having their own fully powered laptop. Windows allows file shares to be automatically synchronized and made available off-line, so a technician can make a change in the field and save it to their file share, then that change will be available for any other technician who works on that instrument, and it’s available for archival use by engineering groups, and it can be backed up to redundant drives to ensure its future availability.

Now, two well-known options are called PactWARE and Emerson AMS. There are some problems with each.

Both share the problem that they really want to take over your plant. They’re not designed for a technician’s PC, they’re designed to be run on a server handling your entire plant using a HART enabled analog input card or a HART multiplexer, so they’re a bit unweildly. AMS is far too expensive to be practical as a simple tool for a laptop. PactWARE is free, but it takes a dozen clicks to connect to an instrument.

Another problem is drivers. The most common standard interface for HART devices is has standardized on the “dd” standard, which is platform agnostic, but PactWARE uses the DTM standard, which is a Windows program designed to fit into the software.

One possible solution is ProComSol DevCom2000. ProComSol puts this hardware out for $800, so it’s still fairly expensive, but it’s far less expensive than an Emerson 475 field communicator.

I recently acquired a copy, and figured I’d show give first impressions, out of the box. I’m going to go into a bit of ostensibly trivial detail, just to give a full idea of what you can expect.

The software is available as a digital download.

Opening the installer package starts initialization.

Then you see a standard “Let’s get started”

Next, accept the license agreement.

Select your installation folder

Confirm your settings

The installation will start working.

After it is done, it will prompt you to finish.

DevCom2000 will ask you what you’d like to do on the first install. We’re going to activate it.

Activation can be done over the phone with a representative, or online. We’re going to activate online.

Enter your code and password.

It will take a moment to connect to the server.

After connecting, activation will be complete.

When you start DevCom2000, the program immediately tries to connect to a HART device.

You can purchase a USB, RS-232, or Bluetooth dongle from ProComSol to connect to a HART device, or you can use a standard dongle like the Mactek Viator.

In addition, you can connect to a wireless HART gateway like that Emerson Smart Wireless Gateway.

The configuration is set to COM99, so it won’t connect the first time.

So we open the basic options to configure our communication device.

Here’s the second tab of the settings, regarding search.

This is the final tab of settings, advanced settings.

If you configured an IP HART gateway, upon connection you’ll be asked to select which of the instruments you’d like to use.

Once you’re connected, there’s a navigation tree on the left, and the relevant values and methods are shown on the right.

dd drivers include variables and methods. Variables are single points of data that can be read, and some can be written. Methods are simple programs which allow the automation of certain tasks. This is an example of a method running.

This is an example of some read-only variables.

Besides what I’ve shown, DevCom2000 has the ability to export the full configuration of a device as a pdf file. This pdf file can be printed to provide a hard copy of device configurations.

Remarkably, it also seems to have the ability to read back that file, and write the values into an instrument.

This was just a quick look, but it covers a lot of the main elements of the program. If I see any requests for more, I may write more on this topic later.

Thanks for reading!

Categories
Process Control

Don’t give them what they ask for

I’m Jason Firth.

This June will mark the 9th year that I’ve been working in heavy industry.

I started my career and spent four years in design and maintenance planning and another 5 years as a technician. Of all the pieces of advice I wish that I’d been given this one is probably the most important: don’t give the people what they ask for; give them what they need.

Something to remember is that as an instrument technician or an engineering technologist or an engineer you have depth of knowledge regarding instrumentation, control, and automation that regular operations and maintenance personnel don’t have. Sometimes they’re asking for one very specific piece of equipment, but if you give them that piece of equipment you’re going to get in trouble because it’s not going to work.

One very specific example came from my time in the pulp & paper industry. Creating pulp and paper takes an incredible amount of power. One of the ways that paper mills get this power without excessive costs is by burning the bark has been removed from trees before their used in the paper process. This removed bark is called hog. This hog is then send to a hog bin which is attached to a power boiler which burns the hog to produce steam, which is in used throughout the process. I was asked to put one of two extremely specific level switches on one of these hog bins. Instead I started looking at the history of the unit, and discovered that that level transmitter has already been tried and has been removed because it didn’t work. I looked at all the different environmental concerns within the bin, and decided on a completely different level measurement than what they asked for. However, this level measurement worked. By giving them what they wanted instead of what they were asking for, my internal customers were far happier with the result.

As a technician, I end up with many extremely similar circumstances. The difference is that whereas they wanted me to design a level measurement before here they would ask me to do the same maintenance task over and over and over again. The company was wasting money on me, the Operations Group wasn’t getting the control that they needed, there was no reason to keep on giving them what they were asking for. Instead, I focused on what the real problem was. In many of these cases, I was able to find a problem that no one ever knew existed. By giving them what they needed instead of what they asked for everybody won.

Let me be clear: I’m not suggesting insubordination. There is a chain of command in workplaces, and it isn’t a good career move to violate it. However; in most cases, you have some latitude to make decisions that you think will work best. If you don’t have that latitude, you are still a voice within your team, and you may be able to change minds if you provide a good reason for people to listen.

If you get this right, and manage to give people what they need and want instead of exactly what they ask for, you’re going to look like a miracle worker.

Thanks for reading!

Categories
Process Control

Troubleshooting tools for Wonderware DAServers

I’m Jason Firth.

When troubleshooting Wonderware software, most people know about the system management console’s log, but today I’m going to talk about some of the lesser known diagnostic tools available for troubleshooting DAServers while you’re using Wonderware. There’s a huge amount of literature included with Wonderware, but it can be a bit overwhelming. These are some tricks I picked up that may not be immediately obvious.

To start off, you’re going to have to open the system management console.

Under the diagnostics, there’s a tree filled with diagnostic tools.

Let’s look at what each one does.

The Client Groups tree displays all the communications methods in service, in each one, it lists all the communications that are in service.

The structures tree shows all the different topic names that are communicating at the moment. You can click each device on the list for a more detailed view of what is or is not communicating correctly.

The transactions tab ostensibly shows transactions, but on all my servers it is empty, so I can’t say what it does.

The statistics tree shows statistics for the currently selected DAServer.

The device groups tree lists the tags active in each device group, extremely similar to the structure tree.

The messages tree displays information about messages currently being processed. It updates fairly quickly.

Moving on from these diagnostic tools, there are ways to show additional debugging information in your system log.

Head to your local log viewer in the system management console, and right click “local”. Click “Log Flags”.

You’re going to see a list of services on the left, and a list of options on the right. Every global log flag you check will cause that set of debugging and status messages to be shown in the log file. Note that there’s a reason these aren’t all selected by default: If you select all log flags, then your log will be inundated by a deluge of minutae.

Here’s some options relating to DASMBTCP, a common DAServer used for communication over Ethernet using Modbus TCP.

This quick rundown isn’t fancy, but hopefully this helps other people who are trying to figure out what the DAServers are thinking when communication isn’t working.

Thanks for reading!

Categories
Industrial IT

sqlcmd- A means to doing SQL commands from the command line

I’m Jason Firth.

Last time, I posted about openness. One of the ways openness can help everyone is by providing flexibility to do things that may not have otherwise been possible.

At this point, many software packages use Microsoft SQL server as a front-end. Pi Historian and Wonderware Historian, for example, both use SQL Server as a front-end.

This provides some really neat opportunities. You can automate the retrieval or analysis of data from the historian, for example. Visual Studio Express is available for free, and includes all the APIs for communicating with an SQL server.

Let’s say you don’t want to do anything that complicated. What if you just want to run a simple query and spit out a simple table?

If you’re running a computer with SQL Server or the free to download and use SQL Server express, you can use the sqlcmd command from the command line.

You can use the command line “sqlcmd -S [protocol:]server[\instance_name][,port] -U [userid] -p [password]” to connect to a command line instance, but the really interesting part is that you can use “-i input_file[,input_file2…]” or “-o output_file” to automate the running of certain queries.

The input file is a script written in TRANSACT-SQL, (that “Select * where” stuff).

Knowing this, you can pull data and manipulate data from the command line, or from batch files. That isn’t something you may want to use for everything on a regular basis, but it’s a great little tool to have in your back pocket for those times when you have to get a little script going quickly.

Thanks for reading!