Good afternoon everyone,
I would like to share an interesting, and very annoying, problem that we have been dealing with over the last few months.
This is a rather long story, but I am sharing it in detail in the hope that it may help someone in the future.
Background
We build different machines and systems, all based on the same basic software framework. One type of these machines includes Denso robots, which are controlled via DeviceNet using an X20 DeviceNet master on a POWERLINK bus coupler.
In the past, these robots had repeatedly caused problems — or at least that was our assumption — because they would stop without giving us any useful feedback.
Troubleshooting, part 1
During troubleshooting together with the robot manufacturer, we found that one of two bits in the robot interface occasionally dropped to 0 for a very short time, although both bits should always remain TRUE.
Since we do not actively write these two bits anywhere in the application program, but simply keep them permanently set to TRUE, we initially assumed that our PLC application code was not the cause.
Later, we tried all kinds of methods to capture the error case, including traces, log entries, breakpoints, and other checks, hoping to find an incorrect address access or something similar. However, we had no success.
Long story short:
We assumed that something was wrong with the robot I/O card, while the robot manufacturer assumed that something was wrong in our PLC program.
Troubleshooting, part 2
After we had already delivered several machines (all similar but not exactly identical) the problem continued to occur very sporadically on almost all of them.
During commissioning of the latest machine, however, the same problem suddenly occurred much more regularly. On average, one of the bits dropped to 0 for one robot cycle every 3 to 6 hours.
We even tried replacing DeviceNet with PROFINET, but the same problem still occurred.
Later, we used a CAN analyzer to check when these “wrong” bits were being sent.
This time we were successful:
the incorrect I/O image was actually coming out of the DeviceNet coupler.
However, since we still could not capture the error case inside the PLC application, we eventually used Wireshark to record the POWERLINK traffic as well.
This was quite time-consuming, but with some persistence, the POWERLINK documentation, and some help from AI, we were able to isolate the exact telegram and the corresponding bytes.
And again, we found that the wrong bits were actually being sent on POWERLINK. They were always wrong for exactly one PLC cycle. In our case, this corresponded to three telegrams, because TC#1 runs with three times the POWERLINK cycle time.
Further tests revealed:
Not only one of the two bits, but bits and bytes over the whole I/O image was “wrong” from time to time. But only these two bits showed an effect consistently.
The solution(?)
For the robot communication, we use one channel with a width of OCTET[32] to send 32 bytes to the robot and receive 32 bytes from it. All commands, jobs, and status information are encoded within these bytes.
In the I/O mapping, we therefore only have to map two arrays with 32 bytes each. These arrays are declared globally and are part of a larger “IO” structure, which also contains other I/O data.
The two arrays containing the robot input and output bytes are written by a task in TC#4. However, other I/Os in the same “IO” structure are also used by tasks in other task classes.
Why is this important?
The issue seems to have been related to the task class setting of the mapped IO structure. This setting was set to “Automatic”.
The compiler also gave us warning 6779, but unfortunately we ignored it.
The warning in the help article says:
“The fastest of these task classes is used, which can produce inconsistent transfer behavior for complex variables (structures, arrays).”
We still do not fully understand why this can result in an incorrect I/O image being sent.
However, after we explicitly set the task class for this mapping to TC#4, the problem no longer occurred.
Conclusion
A few lessons learned from this issue:
- Always keep investigating! There’s a reason for every strange behaviour
- Read compiler and logger warnings carefully
- Do not be afraid of Wireshark.
Questions
Have you ever experienced something similar?
Do you have any idea what exactly happens internally in this case?
By coincidence, I stumbled accross this post today, which sounds suspiciously similar.
But I still do not fully understand how the PLC can actively send a “wrong” I/O image.
