Optimizing UDP communication in PLC

Hi B&R Community,

I am investigating the minimum achievable communication time for UDP communication from a PLC, with a target of 1 ms round-trip latency. For context, I am working on a real-time control application where low-latency communication is critical.

In my experience, UDP communication at 10 ms or 100 ms typically requires four cycle times for a send/receive operation, which is stable at these intervals. To benchmark my setup, I first tested communication with a Python script on a PC (acting as a UDP client) and achieved an average round-trip time of 1.2 ms over 1,000 requests.

My goal is to replicate this performance on a PLC. Given that AsUDP functions are asynchronous and execute during idle time, I conducted several tests to evaluate stability and latency. To calculate the communication time, the PLC sends a life counter, which the server returns. This allows me to count the number of cycles between sending and receiving.

Below are my findings for each configuration:


Case 1

Idle time
Idle task class Cyclic#1
Task class idle time 2000 µs
Cyclic #1
CPU Core Core #1
Duration 10000 µs
Tolerance 0 µs

Results (20s trace):

Metric Number of cycles Time (ms)
Average cycles (Send/Recv) 3.608 36.08
Max cycles (Send/Recv) 6 60

On average, the communication time is below my target of four cycle times.


Case 2

Idle time
Idle task class Cyclic#1
Task class idle time 2000 µs
Cyclic #1
CPU Core Core #1
Duration 3000 µs
Tolerance 0 µs

Results (20s trace):

Metric Number of cycles Time (ms)
Average cycles (Send/Recv) 4.096 12.29
Max cycles (Send/Recv) 10 30

On average, the communication time exceeds my target of four cycle times.


Case 3

Idle time
Idle task class Cyclic#1
Task class idle time 1000 µs
Cyclic #1
CPU Core Core #1
Duration 2000 µs
Tolerance 0 µs

Results (20s trace):

Metric Number of cycles Time (ms)
Average cycles (Send/Recv) 4.202 8.40
Max cycles (Send/Recv) 16 32

On average, the communication time exceeds my target of four cycle times.


Case 4

Idle time
Idle task class Cyclic#1
Task class idle time 200 µs
Cyclic #1
CPU Core Core #1
Duration 400 µs
Tolerance 0 µs

Results (20s trace):

Metric Number of cycles Time (ms)
Average cycles (Send/Recv) 7.012 2.80
Max cycles (Send/Recv) 67 26.8

At this configuration, the communication becomes highly unstable.


PLC Program

CASE State OF
    0:
        IF EnableCom THEN
            Open.enable := TRUE;
            Open.port := PlcPort;
            Open.pIfAddr := 0;
            State := 10;
        END_IF
    10:  // Open
        IF Open.status = ERR_OK THEN
            Open.enable := FALSE;
            Ident := Open.ident;
            Send.datalen := SIZEOF(outgoingdata);
            Send.pData := ADR(outgoingdata);
            Send.ident := Ident;
            Send.pHost := ADR(UnrealIPAdr);
            Send.port := UnrealPort;
            Send.enable := TRUE;
            State := 20;
        ELSIF Open.status <> ERR_FUB_BUSY THEN
            State := 255;
            ErrorID := Open.status;
        END_IF

    20: // Send
        outgoingdata.LifeCnt := outgoingdata.LifeCnt + 1;
        IF NOT EnableCom THEN
            State := 220;
        ELSIF Send.status = ERR_OK THEN
            Send.enable := FALSE;
            Receive.ident := Ident;
            Receive.datamax := SIZEOF(ingoingdata);
            Receive.pData := ADR(ingoingdata);
            Receive.pIpAddr := ADR(UnrealIPAdr);
            Receive.enable := TRUE;
            State := 30;
        ELSIF Send.status <> ERR_FUB_BUSY THEN
            State := 255;
            ErrorID := Send.status;
        END_IF;

    30: // Receive
        outgoingdata.LifeCnt := outgoingdata.LifeCnt + 1;
        IF NOT EnableCom THEN
            State := 220;

        ELSIF Receive.status = ERR_OK THEN
            UnrealPort := Receive.port;
            Receive.enable := FALSE;
            Send.datalen := SIZEOF(outgoingdata);
            Send.pData := ADR(outgoingdata);
            Send.ident := Ident;
            Send.pHost := ADR(UnrealIPAdr);
            Send.port := UnrealPort;
            Send.enable := TRUE;
            outgoingdata.ReflLifeCnt := ingoingdata.LifeCnt;
            LifeDiff := outgoingdata.LifeCnt - ingoingdata.ReflLifeCnt;
            State := 20;
        ELSIF Receive.status = udpERR_NO_DATA THEN
            Receive.enable := FALSE;
        ELSIF Receive.status = ERR_FUB_ENABLE_FALSE THEN
            Receive.enable := TRUE;
        ELSIF Receive.status <> ERR_FUB_BUSY THEN
            State := 220;
            ErrorID := Receive.status;
        END_IF

    40: // Close
        IF Close.status = ERR_OK THEN
            State := 255;
        ELSIF Close.status <> ERR_FUB_BUSY THEN
            State := 255;
            ErrorID := Close.status;
        END_IF;

    220: // Error during Send/Recv --> Close connection
        Close.ident := Ident;
        Close.enable := TRUE;
        State := 40;

    255:
        Open.enable := FALSE;
        Close.enable := FALSE;
        Send.enable := FALSE;
        Receive.enable := FALSE;
        ErrorID := 0;
        Ident := 0;
        State := 0;

END_CASE

Open();
Close();
Send();
Receive();

From profiler informations this program takes max 40us (Max Net Time) to be executed.

Questions

  1. Is it possible to use the IDLE task to achieve fast and stable UDP communication? Or is there a better approach (e.g., higher-priority task, hardware timers)?
  2. Does anyone have more information about how the IDLE task schedules asynchronous functions? Is there a way to minimize jitter?
  3. Are there any recommendations for optimizing the PLC configuration or program logic to reduce latency and improve stability?

Hardware and Software

  • AS: 6.5.0.306
  • PLC: X20CP1686X AR 6.5.1
  • Note: Only one program is running in Cyclic#1 on the PLC.

Server UDP

The UDP server is a third-party and I don’t have control over it, I can’t change the protocol or any configuration.


Apologies for the lengthy post! I hope the details are clear. Any advice or insights would be greatly appreciated.

Regards,
Florent

Have you tried sending and receiving with functions from ArApi_6.7.0 (license needed)?

Can the other side support OPC UA Pub/Sub or OPC UA FX?

Thanks for you answer!
I forgot to mention that I don’t have control over the server and can’t modify it, so I can’t change the protocol :confused: I’ll add it to my original post.

For the ArApi_6.7.0, I never heard about that, it’s a PLC library? Do you know where I can have more informations about it?

ArApi is something special. Its the “internal” API most mapp components and other libraries use as basis.
For your use-case you could create a acyclic task and then run blocking socket functions in it. For acces and license you should reach to your local support.

For speeding the UDP functionblok:
Call multiple times the send / recieve fub in the same cycle so that you read and set it active again for the next.

Thanks for this informations about ArApi, I didn’t know that was actually something ^^
I will ask my local contact more detail about that!

For UDP FUB, yeah like calling inside of the case would speeding up a bit, but I’m not sure that it will be enough to achieve what I want. I will make a test and update this thread when I have more info.

I’m pretty sure the ArApi is the solution, I will mark as solution when I got infos from my local contact.

Thanks again for your help guys!

1 Like

Hello Florent,

One change that you can make that may improve your performance is to not disable the UdpRecv() FUB when you receive the statues udpERR_NO_DATA. You don’t need to reset the function block to obtain data later. You can check the implementation of the AsUDP library sample (LibAsUDP1_ST), where the code takes no action if no data is received. This would help by saving some cycles of disabling then re-enabling the receive function block. It may also reduce your jitter by quite a bit, since you don’t miss cycles on the disable/enable loop.


LibAsUDP1_ST.

Another piece of information when working on these sorts of timing optimization problems, is to map out the critical path that you are measuring. Based on my reading of your code, I’d write out the following, and see where I could get a better idea of where the time is spent:

Store origin time [start timer], call UdpSend().
AsUDP build packet, sends to network driver.
Ethernet bytes on the wire. Collisions, other traffic being sent/delays?
Network driver of server, receive.
Interpretation and response of server. How long does this take? You have no control here, but it’s within your critical path.
Network driver of server, send.
Ethernet bytes on the wire. Collisions, other traffic being sent/delays?
AsUDP receive packet, flags UdpRecv on next call.
Program call reacts, compares the overall response [stop timer]

I’ve added some notes on ideas where you could hit delays, that I started thinking about as I worked through the process. You may be able to see the bytes on the wire with Wireshark, and get a better idea of what comprises the total communication delay time. Note that your normal workstation network driver doesn’t deal with single-microsecond timestamps, so you may not get the resolution you need on that level.

Good luck!

-Austin

2 Likes

Hello Austin,

Thanks for this answer!
For the UdpRecv you’re right! I forgot about that!

Thanks for the detail of where I can detect time spent!

I have a Python script that is kinda my reference, he’s running on my workstation and just implement the udp client. With this python script I have a stable 1.2ms com duration over 10K requests. I know it’snt the same hardware than a PLC but the goal is to try getting same perf on a PLC.

Regards,
Florent

Hi @florent.boissadier,

I’m not sure if it’s worth to mention, but I did some testing with 2 PLCs (X20CP3585 and X20EM0612, because they were available for me).

I ran one PLC the sender + receiver + measurement, on the other just receiver + sender (copying the received data).
Both PLCs used 400us TC#1 for the code. As both had enough idle time, I haven’t changed any idle time task settings (not neccessary at least in my setup).

I just sent a increasing counter as data, copied it on the other PLC, and measured on the first PLC the time between sending the counter value and again receiving it. For measurement, I used the difference between two AsIOTimestamp() values.

In this setup I reached roundtrip times of about 1.6 - 2 ms, with sometimes cases of roundtrips up to about 6ms, and trying to eliminate all other network influence (e.g. the switch runtime, I tested with a direct connection between the 2 PLCs and used the PL interfaces of those PLCs as 2nd ETH interfaces for online connection).

What was really interesting, and that’s the reason why I share it:

I reached those times just because of using more then one UdpRecv() FB instance im every cycle (called in a loop).
If working with just one instance of UdpRecv(), I wasn’t able to reach the roundtrip times above.

I also learned, that if changing any code and transferring to the PLC, you should restart both systems for a new clear measurement and to get valid information from it (as while downloading, the task class system is shortly stopped, but maybe UDP packets still arrive which destroyed at least my min/max measurement value).

I’m not sure if it’s really interesting for your use-case, but I would try to call UdpRecv() in every cycle and do the Recv() before Send() (which also could save 1 cycle in the overall roundtrip) as suggested by Austin, and call more then one instance of UdpRecv() in a loop processing the received data to see if / what is changing (in my test, 2 instance were already enough to get better timing).

Unfortunately I haven’t had the time left to dig deeper into the higher roundtrip times sometimes happening.
I can’t imagine that a roundtrip < 1 ms can be reached every time, but maybe it’s also worth trying a PLC supporting 200us base time (CP1686X should support that if I remember right? Or do you use 400us because of PL IO’s / Motion?)

Best regards!

1 Like

Hi @alexander.hefner ,

Thanks for this reply! Really intersting!
I will keep doing some tests this week, I maybe will test a setup like yours with 2 CP1686X.
For the moment I didn’t know if I will need to go with some IOs / Motion or not, so I used the classic 400us base time :smiley:
But during my tests I could take a look for a 200us base time! Maybe it will help!

After all my tests I will update this thread or maybe create a new one about what I found :slight_smile:

Regards,
Florent

1 Like

Hi,

About measurement tasks, in Germany we say: “If you measure, you measure garbage.” :wink: So maybe there’s also some misunderstanding from my side about the time measurement.

So if you want to do some more testing, I uploaded here my (short and ugly) test project as reference what I’ve tested.

Best regards!

UdpPerfTest.zip (128.9 KB)

1 Like

Hi,

Thanks for sharing this!
I never thought of using a FOR loop for the Recv part!
I will use your project as reference for my tests :smiley: But with a little modification just uncomment some of your code, I need to send a new packet when I get the response back from the server :slight_smile:

Regards,
Florent

Hi,

yes it was already prepared as I tried to find out if it makes a difference using “unchained” or “chained” packet transfer; I haven’t seen a big impact, but from my experience it’s always better if you have the “chained” situation as it’s more easy to detect packet loss, using timeout monitoring, and so on …

1 Like

Yes the “chained” situation is how I detect packet lost and react to this discontinuity!

1 Like

Hi,

Just to update this, I’m encountering a strange behavior so I have a System tick of 400us and my Cyclic#1 is running at 400us. I use the “chained” situation and I got a problem that sometimes the AR didn’t see the response so he stopped sending back a packet.

Here what I found, I used Wireshark to determine if a response was send or not, to identify if the problem is from the PLC or the server.

The last thing I see in Wireshark is the reponse from the server, a little screenshot:

  • PLC (X20CP1686X) : 192.168.0.10
  • Server : 192.168.0.200
  • Time between Send/Recv in Wireshark : 173us

The program logic is the same as @alexander.hefner shared in the project “UdpPerfTest.zip” just in another PLC and with a exchange of 220 bytes not just an UINT.

I identify that when the server response is faster than the cycle time of the Task, the packet is like destroy or just cannot be readed by AR. But I’m wondering the Ethernet interface should do some buffering (with a limited buffer)?

I try to use a system tick of 100us and a Cyclic#1 running at 100us, I didn’t have this behaviour, but I have some cycle time violation ^^’

Just changing system tick to 100us and keep Cyclic#1 to 400us didn’t solve the problem.

Is this something you already see?

Hi,

I haven’t seen it in my test, but it wasn’t running for a long time to be honest.
I totally agree, normally a received packet should be buffered even if using UDP. As UDP doesn’t have a flow control, it can happen that data sent to the same socket is overwritten if receiving more then one packet between 2 receive cycles - but this isn’t what you described.

If a UDP packet is lost for whatever reason, because of missing flow control it won’t be send again. But right now, I can’t imagine the reason why it’s lost or destroyed: in my opinion, a packet arriving at the ethernet interface should lead to a interrupt and should be copied by the interrupt service routine into the receive buffer. And because of the data size is below one ethernet MTU, there also no packet fragmention should happen. I think if a CRC error would be the case, you should see it in the Wireshark trace?
Any special error state reported at the UdpRecv() status output?

Could you please add the AsEth function block to your code, and check if one of the SG4 error counters changes after the issue occured?

Best regards!

1 Like

I think you cannot reproduce it with your setup as you are using 2 PLCs.
I will test tomorrow to check the error counters with AsEth!

On wireshark I didn’t see any errors ^^’

1 Like

Hi @alexander.hefner

I just add the EthStat FBK and no error appears on the Eth interface:

In wireshark nothing display that can be an error on the communication, here a screenshot of wireshark without filter:

For my multiple tests it the timing between both request and response, here in red rectangle : 58.429435s - 58.429279s = 0.000156s → 156us

Do you think I should report this to my local support?

Note: I tried to increase index in for loop from your project and arrays used in it, I have 50 Recv FBKs running in the for loop

Regards,
Florent

Hi @florent.boissadier,

I used my lunch break for playing around a bit more :wink:
To get some similar timing behavior I changed my setup (with “similar timing” I mean: respond packets from the “receiver” faster then the cycle time of the “sender”).
My “sender” is now running at 600us timer and TC#1, and my “receiver” runs at 200us timer and TC#1 → that means that many UDP responses from “receiver” to “sender” are send somehow faster then the “senders” taskclass time.
Additionally, I changed the send / receive data size to 220 byte (just using the same mechanism as before but filling the rest with 0) and used “chaining mode”.

And now, I also have seen that in rare cases the chain stopped sending.
Unfortunately, I have no idea right now why, but:

  • I saw in wireshark that it’s not based on the “slow” “sender” → I’ve also seen that the “fast” “receiver” stopped sending back a package (see the two screenshots below, in one the “sender” was the last one, in the other the “receiver”).
  • Because of my setting described above, I don’t think that the cause is “just timing based”, because I’m sure that I have this timing context “response comes faster then requester’s cycle time” in many / almost every packet (based on the PLC settings described above)
  • I added some “stopping condition” in my test code to check, if a unexpected function block state in UdpRecv() is popping up, but I haven’t seen anything else then SUCCESS, BUSY, NO_DATA
  • I’ve also no unexpected error counter increasing inside ETHstat().

As both communication partners do sending and receiving, I can’t imaging it’s really based just on timing (if so, in my interpretation my “receiver” must not be the last one in a trace) - because it should happen very more often in my setup, then I’ve seen it (some hundred thousand packets were gone trough between two stucks in my tests here).

But, as I said, I have no idea right now were the packet that was on the line is lost (somewhere in the ISO/OSI stack (maybe a interrupt handler conflict

Sure, you can ask your L1 support if they have any more information about it? Could be helpful!
But I think, in the end a UDP packet loss in reality can also happen for many different reasons, caused for example by “switch overload”, “CRC error”, “layer one error” and so on. And as UDP doesn’t have any flow control / consistency mechanism, a packet loss has to be handled somehow also on code / protocol base.

So or so, I’ll keep thinking about it as I find it extremely interesting to dig even deeper inside :wink:

Best regards!

PS: I played around with different numbers of Recv() instances, but for me it looked like that just having more then 1 (means 2) is already enough… at least I haven’t seen a faster reaction with higher number of instances in chained mode.

2 Likes

Thanks for your interrest about this!
I think we didn’t have exactly the same behavior due to difference in my setup and yours. I just realized that I haven’t shared my current setup:

  • My laptop is running a executable build with Unreal Engine 5 and inside of this a UDP socket is running and act as a server (wait for command from external device). It could be compared to a classic C++ socket (as inside of Unreal Engine 5 the socket is managed using C++ on a dedicated thread)
  • A X20CP1686X connected directly without switch to my laptop

I think that’s why in my case the last packet is from my laptop and not from the PLC.
With this setup I try to avoid a maximum ways to got packet lost (a single cable between PLC and my laptop) :slight_smile:
There is also the way AsUdp handle the sending on Idle task.

I will dig a bit more in wireshark analysis to check about timing, to identify if in a test run there is like a single send/recv timing that is under a timing threshold or not.

I will ask to the L1 support (given them the link to this topic as info) what they think about this behavior.

I’ll keep this thread for any update about my tests :slight_smile:

Regards,
Florent

1 Like

I can confirm your throught about it can’t be really just based on timing!
After a quick analyze with python to retreive all pair in a wireshark capture and get max/min and number of pairs that timing is under cycle time I found that:

  • 4900 pairs analyzed
  • ~ 6% of the pairs are below the 0.4ms of cycle time.
  • Average time: 1.119ms
  • Min time 0.165ms (not that the last pairs before stop is 0.173ms)
  • Max time 11.849ms
1 Like