Synchronized CRC errors across independent POWERLINK I/O zones behind a passive hub

Setup: B&R PLC (Automation Studio/APROL), single POWERLINK cable from the CPU to 3x non-manageable 8-port hub (B&R 0AC808), which fans out to 16 independent remote I/O zones (bus controllers X67BC8321.L12), each on its own cable.

Symptom: EthRxCRCError counters on several physically independent zones increase by the exact same amount, in sync, in bursts spaced days/weeks apart.

Ruled out so far:

  • POWERLINK address collision (checked in config, none)
  • Power supply issue on the segment (checked, no anomaly)
  • Process-related EMI (machine idle, no process running for 1 month, bursts still occur)
  • Cable is shielded, panels normally grounded
  • Nothing special in AR logger

One zone also logged somes EthTxCollision, which per B&R documentation should never happen on POWERLINK and points to either a wiring problem or a non-POWERLINK-capable station on the segment — no such station is knowingly present, and addressing was double-checked.

The hub has no diagnostic/error counters of its own, so I can’t get direct visibility into it. The only diag variables i can check is on X67BC.

Questions:

  1. Beyond shield continuity/ground potential testing, what would you check next on a passive POWERLINK hub suspected of intermittent corruption with no error counters available?
  2. Has anyone seen a 0AC808 (or similar passive hub) fail in a way that produces synchronized CRC bursts across all ports rather than a single degraded port?
  3. Any known failure modes on this hub model worth ruling out specifically?
  4. Can something be seen with Wireshark ?

Happy to share more logger/trend data if useful.

Thank you for help.
Best regards,
Vincent

Hi,

I haven’t any special experience with that setup, just can give some general infos about ethernet and crc errors. As Powerlink is based on standard 802.3 ethernet on layer 1 + 2, it’t maybe worth to share the following information.

The ethernet CRC is a checksum built from the ethernet chip on the ISO/OSI layer 2 including the complete content of an ethernet package before the data is sent physically out of the interface via physical layer 1.

A hub like 0AC808 is an ISO/OSI layer 1 device, that more or less just acts as a physical signal amplifier / multiplicator.
Unless like a switch which operates on layer 2, a hub is working on layer 1, does not analyse the ethernet packet, does not send / receive on dedicated ports depending on the networks hardware MAC address (the so-called layer 2 micro-channeling), and therefore does not investigates checksums or other content of the ethernet packet.

That means, if a package with a CRC error “arrives” at a hub, the hub won’t mention it and the packet is still send out to all ports as it was received.
This is, at least from the common ethernet functionality point of view, the explanation why multiple devices will all see the same packets and same CRC errors.

Ethernet CRC errors are normally always caused by layer 1 issues, that means by physical / hardware problems like shielding, grounding, bad cables / connectors, or similar hardware failures (including the ethernet hardware itself, but many more often by wiring or interference).

It’s hard to say if a wireshark trace would deliver more information. It depends on the hardware where wireshark is running.
Definitely, wireshark running on a normal hardware would not catch every packet as Powerlink is sending/receiving packets very fast. Additionally it depends on how the ethernet interface handles CRC error packets (normally wireshark runs the network interface in promiscous mode and should report every packet, but in the end it depends on if the ethernet interface firmware skips the CRC error packets received without reporting the packet content to higher layers, which is the default behavior as defined by ethernet standard)…

But nevertheless I would say it’s worth a try! Under good circumstances, maybe you’ll get more data that helps for further investigation.
I would propose tracing directly at the first hub where the Powerlink master is connected to.

Hope this information helps a bit, best regards!

Hi @Vincent_Druart is your issue solved there, can you update us with some final conlusion?

Hello,
Diag is still in progress, not 100% closed on our side.
We are now at one month of production without CRC increasing so it’s in good way for resolution.

Problem was very hard to find. On Powerlink network there was a RJ45 câble inside a cabinet who was too short, someone decided to put a standard IT RJ45 cable instead of B&R RJ45.
It was hard to find because also hidden in the cabinet. Since we replaced it there is no problem on all network.
All shield continuity was OK. Wireshark wasn’t helpfull on that case.

→ usually this is not good idea to do not use B&R RJ45, because of shielding etc. So if problem has not occur after cables replacement. I think you are on good way. I answer your answer as a solution, but feel free to update us if needed. Thanks for feedback

I know, I’ve always put B&R RJ45 in cabinets for 10 years now and it works very well. The electrician who did this didn’t have a B&R RJ45 with him, it was supposed to be a temporary solution, it was then forgotten and not raised