Could not open connection to the Host on Port 502 ( host = PLC CP3686)

Dear Experts,

I am facing a communication issue between PLCCP3686X and Siemens WinCC scada system. I have no issue in the PLC, no error shows in the PLC logs, Program is running fine.

PLC is sending the field instruments data to SCADA via Modbus TCP/IP protocol using Cluster IP.

HSBY PLC LAN cable is connected in level managed switch and then to SCADA.

scada PC is using cluster IP and everything works fine for about 1year… recently my scada lost communication with PLC cluster IP.

while checking Telnet, it says Could not open the connection to the host, Port 502: Connect Failed.

It looks like something in the network has blocked the 502 port and my scada is not getting the data from PLC.

I have isolated the network switch and connected my laptop directly with a small unmanaged 4 port switch and after a power cycle of both the PLC , my laptop is getting modbus data from plc.

once i connect back to Network switch , after a while the port is blocked again and not data appears in SCADA.

Please refer the wireshark capture on the network switch. (10.85.106.53 is the cluster IP ) ( 10.85.106.22 is the scada IP )

i think the PLC is not responding to the Scada request. after i power cycle the PLC it works for 5 mins and again gets blocked.

Question :

  1. Is there any way to track what is blocking the PLC 502 port ,
  2. If the port is blocked in network switch , how to unblock it without doing power cycle, 3)Can see any log in the PLC on how and when the PLC port is blocked.

rough network architecture.

everything was working fine for one year and suddenly you lost communication with your SCADA system…mmm…some key information are missing, Have you updated something in the last few days? IT department has updated something?

Form AI:
Yes, a network switch can block port 502 , though its ability to do so depends on whether it is a managed or unmanaged device. Port 502 is the standard port for Modbus TCP, and it is frequently blocked by security features in industrial network environments.

How Port 502 Can Be Blocked

  • Access Control Lists (ACLs): Managed switches can use ACLs to filter traffic based on Layer 4 protocols and port numbers. A switch administrator can create a rule specifically to deny traffic targeting TCP port 502.

  • Security Services: High-end industrial switches often include integrated firewalls or “advanced threat detection” that may automatically block port 502 if they detect unusual polling patterns, which can be flagged as a potential brute-force attack.

  • VLAN Isolation: If devices are on different VLANs and the switch is performing inter-VLAN routing (Layer 3), it can block traffic between those segments on specific ports.

  • Managed vs. Unmanaged:

    • Managed Switches: Can actively block port 502 through configuration settings.

    • Unmanaged Switches: Typically cannot block specific ports like 502 because they operate strictly at Layer 2 and do not inspect the TCP/IP headers of the data they pass.

Common Reasons for Blockage

  • Industrial Security: Modbus TCP is often considered “high risk” because many older PLC implementations lack authentication. IT departments often block this port by default to prevent unauthorized access to control systems.

  • Software Firewalls: Even if the switch isn’t the culprit, port 502 is frequently blocked by the Windows Firewall or other endpoint security software on the master or slave device

    Ciao
    Valerio

Hi,
can you please upload a complete wireshark trace (as zip)?
In a screenshot, we can’t see some maybe interesting details (like mac addresses).

Best case would be having a wireshark trace where the running communication and the breakdown of the communication is included.

Hi

Thanks for responding , I understand you want the complete wireshark log from start and end of the incident. i could only capture aftermath of the incident .

Please find attached ZIP of the log file i have shown in earlier screenshot. Kindly check if you get any info out of it mean while i will try to restart the PLC and get the running and port blocked logs.

PLC-SPAN-capture.zip (3.9 MB)

I am no expert in networking but i can see the PLC IP actively rejects the request , dont know why??!!

FYI, Below is the IP i have set in PLC

@arunraj.premraj, thanks for the wireshark trace!

I had a look into, and I saw following:

  • in the trace, there’s a 2nd device with IP address 10.85.106.33 which is communicating with the PLCs cluster IP via port 502 (already existing TCP connection at the beginning of the trace)
  • the IP 10.85.106.22 (which is the SCADA panel as I understood) cannot communicate with the cluster IP of the PLC, as long as a communication with 10.85.106.33 is already active (the PLCs denys the connection - I think, because there’s already a connection active)
  • There’s one section in the trace (around packet 275517) where the connection with IP 10.85.106.22 is established and working … until 10.85.106.33 get’s active again (around packet 278714), after that, communication with 10.85.106.33 is active and IP 10.85.106.22 is blocked again.
  • Additionally, it looks like that from .22 are multiple TCP channels opened on the PLC (I see that 3 different ports are resetted in the last screenshot, when the connection to .22 breaks).

So I think, something in the network has changed, some device was added, ore something else happened in the infrastructure outside of the PLC.
I think the first important question is: who is IP 10.85.106.33 and who is IP 10.85.106.22?

.33 has a Siemens MAC address, .22 has a different one (Hewlett Packard) … is it really wanted that 2 devices are communicating with the PLC? From your description, it doesn’t sounds like that.
And is it a wanted/configured behavior, that multiple streams are opened from .22? It seems that .33 just uses one stream (which I would expect with a single ModbusTCP connection).

Packet from .33

Packet from .22

Running TCP comm. on port 502 with .33

Running TCP comm. on port 502 with .22, breaking as soon as .33 connects:

Best regards!

Hi,

Thank you for your time in analyzing the packet. I too have the same doubt why .33 device is requesting the data.

I still have one doubt, is it like PLC cluster IP can only respond to one device .22 ? what if we have a requirement of giving data to multiple device for monitoring the parameters from PLC cluster IP . 53?

Thanks and regards

Hi,

no, I don’t think that the cluster IP can only communicate with one device - in the part of the wireshark trace where modbusTCP sessions are running, the cluster IP already communicates with 2 devices in parallel: .22 and .33, so I’m pretty sure that’s not the problem.

And, after some more analysis, I can see that both devices .22 and .33 are running several sessions / streams in parallel.
The difference is (at least in the small snapshot of a running communication I can see in the trace), that the device .22 sometimes does it in a way I would expect and that is wanted when using TCP: it opens a stream (TCP SYN), queries some modbus data, and then closes the connection (TCP FIN).
For example like here:

But in some other streams, I can see that this is not happening, and the PLC .53 sends a RST flag after some time where the stream is idle (no data flow at all on this stream).
As this always happened after 5 seconds of stream idle, I assume it’s the “device idle timeout” configured in the PLCs modbusTCP stack. (that was also the same situation in the screenshot of my post above, when .33 started communication)
For example here:

And: the device .33 opens some other parallel streams, but doesn’t end that streams in a way that I would expect (I haven’t seen any TCP FIN in all of the .33 initiated streams!) - instead of ending the stream by FIN, the device .33 just sends a RST flag.
For example here:

All in all, because of the combination of all things mentioned above, I could imagine that the PLC comes into some ressource bottleneck after some time with the handling of all of those parallel sessions, where some of them aren’t closed in the way TCP expects it (because all of these sessions have to have handled by the PLC in a way TCP defines the communication flow, and the number of TCP handles is limited inside a PLC).

This bottleneck could lead to the situation, that somewhen all TCP handles of the PLC / all reserved handles for server port 502 are blocked which would then lead to the situation, that no more connections are possible (until reboot).

For me, the questions are:

  • who is .33, and why does it not close the sessions in a TCP-usual way?
  • why are parallel TCP streams from the same device are querying the PLC server, is that neccessary / reconfigurable?
  • what has changed since it does not work anymore? As long as you haven’t changed software on the PLC and it ran for a year or so, I think there must have been some change in the network (for example introducing device .33?), in the firmware of the communication partners (have there been software changes / upgrades on the SCADA device for example), or somewhere else in the infrastructure → even if the PLC is not responding anymore after some time, I’m relatively sure that the PLC is not the root cause but some change in network / in the number of devices / in the devices behavior (that’s what I interprete out of the wireshark trace).
  • I don’t think that it’s directly linked to this issue, nevertheless I want to mention it: in the complete trace, I see a very high amount of so-called ARP requests (requests for the MAC address behind of a IP address) for a dedicated small number of IP addresses, much more then I would expect (97% of all captured packages are arp requests).
    • most of that requests (87.4% of all arp requests) are sent by a HMS device (10.85.106.41), is that the switch? Or is there some permanent network / IP scanner running or something like that?

I think that’s more or less everything that can be interpreted out of the wireshark trace.

Best regards.

Hi Alex,

Thank you so much for spending your time in this matter,

Below are my finding which may clear some clouds

  1. The device .33 is siemens PLC trying to capture some data from .53 PLC.. ( This is surprise to me , since nobody has informed this change to me prior testing ), I believe this is the change they have made in recent days. Initially the communication was from .53 cluster to .22 siemens SCADA, Due to some delay in network they have used .33 PLC to capture some more data
  2. I suppose the Parallel streams are captured to reduce the response delay, ( Expecting to get more data quickly in all requests all parallelly )
  3. to answer what has changed in the network: I am suspecting the introduction of Siemens plc in the network , since my plc .53 is providing a small zone of the plant and Siemens scada is collecting data from other zones of the plant from other PLCs , there seems a small time delay in SCADA to monitor all the parameters.. To reduce the timedelay the teams has splitted some data to SCADA .22 and siemens. 33 and tried to establish a quicker communication.
  4. The ARP you mentioned is IP scanner we can ignore that, it been there in the network since the stone age.

I can agree with the bottleneck formed in the .53 PLC because of the parallel sessions, But still please have some time to answer my below question

Q1. why PLC .53 is not releasing the Port 502 block once the .33 and .22 device stopped the request ( when they are removed from the network )

Q2. Is it possible to get a warning before the 502 port gets blocked. ( if it block then only a reboot is opening the port , for which a plant shutdown is required )

Q3. Do we have a limitation or standard practice like only these many connections are possible or above that it shall go to port block !!!

Thanks and regards

AJ

Hi,

regarding your questions, I apologize but I haven’t concrete answers to that.
I just can give some common background information.

Q1.
Here’s some background about the TCP protocol (independent of the device): it’s a protocol designed for flow control and data consistency. That means, that TCP has some connection + data flow control mechanisms to manage different streams and to detect if data was sent in the right order and completeness. It is designed even to hold a stream consistent and not starting data transfer from scratch, even if the underlying connection is instable, has hugh jitter, is temporary breaking down, a.s.o.
Therefore, the TCP protocol also has some dedicated mechanisms for enabling and disabling a stream (3-way handshake). So if a connection is not ended in the right way (by sending / receiving the so-called FIN packet from the partner who wants to end the stream + leaving the stream open and keep receiving data, until the communication partner acknowledges the closing of the stream), the TCP stack has to handle the stream as it’s still needed.
So: if one communication partner does not finish the stream as it’s defined by protocol, the other one just don’t know that the stream or even the partner itself is not longer available (in TCP definition, this is a so-called half-open stream).
Of course the’re are things like timeout, wether in the TCP stack or in the application protocol, but there’s no strict definition in the TCP protocol itself, how such timeouts have to be handled (for example how long they should are, a.s.o.)
And: if the B&R PLC is the TCP server (I think so, because I assume you’re a ModbusTCP slave, which means that the device is a TCP server), which makes it even more complicated to really know wether a stream should be closed or not. The TCP server definition, together with the ModbusTCP protocol, defines the servers as the communication part that has to “wait until a request is coming in, then response with the wanted data, then wait for the next request” - so the ModbusTCP server does never send data to a client from it’s own but just on request, which makes it on TCP flow control level very difficult / almost impossible for a server to detect, if a client is still active or not, and therefore if the stream has to kept active or not.

Q2: I’m sorry, I’m not aware of such warning or something similar (as described in Q1, I don’t think that the blocking is active by policy, but just by too much already opened but not right closed sockets to the TCP server).
So if you’re a ModbusTCP slave (I think so), a few things you could monitor are the datapoints in the slave IOmapping like “TimeSinceLastRequest” to get an idea if a communication is still active.

Q3: same as Q2, sorry.

About Q2 + Q3, I assume you’re using the ModbusTCP functionality that is embedded into Automation Runtime, and you’re acting as a ModbusTCP Slave (which is a TCP server), am I right? As I’m as automation engineer don’t have insights into / sources of Automation Runtime or B&R standard libary functions, unfortunately I don’t know the implementation and the limitations behind those functionality.
Have you already checked the PLCs fieldbus logger? Is there any warning / error inside about ModbusTCP protocol or the ModbusTCP interface?

So I just can recommend that you contact the support team from your local B&R office for official assistance and support, so that they can get in touch with the headquarter and define how to proceed.
If doing so, please reference to this B&R Community post so that the colleguages are aware of what we already figured out.

Best regards!

1 Like

Hi Alex,

Thank you very much for your efforts. I am seeing a black hole here, the more we dig , the deeper its going on..

to overcome this Port blocking problem i can only think of giving a timely request and acknowledge every packet closed and then initiate the next packet so that no conjuction will close the port.

I will R&D on the problem further,.

Your initial wire shark analysis has pointed me in the right direction ..

Thank you very much and i really appreciate your effort.

Thanks and regards

AJ

Hi @arunraj.premraj ,

just for the sake of completeness:
it looks like that the “missing FIN” is a known topic which is a hurdle for various systems; unfortunately, in the thread below there wasn’t any fix or workaround available (it’s already some years old, I haven’t investigated further):
S7 1500: tdiscon doesn’t close the connection cleanly - Siemens SiePortal

Best regards!

1 Like