Mapp Data SnapShot in fast Task class

Hello,

I’m currently testing out mappData for csv logging. The problem is that when you run it in a fast task class with no tolerance, cycle time violations can happen on save when too much data is recorded. That is noted in the docs too.

My question is: Is there a way (currently or planned) to do the saving part with a lower priority in another task class? So you can record in a fast class, and the save would happen in one that has higher tolerances.

I’ve thought about caching the data in a FIFO so that another task class can write that to the log. While that would most likely work, it would mess with the timestamp the dataRecorder generates. So a “built-in” solution would be preferred.

Thanks

Martin

Hi @mreiter ,
Just an idea: so far in my case, my sample rate wasn’t very fast. Have you tried calling MpDataRegPar in your fast-task class and MpDataRecorder in the slower one? Could you run a test and see in your CSV/XML file if all samples are recorded correctly?

Ciao
Valerio

Hi @valerio.manenti

That doesn’t work. RegPar is used only to register them once. You don’t even need to call them anymore after registering(as long as you don’t want to deregister).

So recording is entirely dependent on the main recorder fub.

@mreiter is correct: MpDataRegPar is used only for registration, once that is done, this FUB basically does nothing.

As to the performance topic (that’s a bit of a longer story):

MpData was never really meant for fast recording without tolerance. That is mainly because MpData stores the recorded values into a buffer (NOT a FIFO, at least not in the traditional sense). Once the buffer is full, it is transferred to a file (that is done to avoid killing the data storage with too many write requests). And here’s the problem… As long as the buffer is full, no new data can be recorded, until the information that is currently inside the buffer was removed from it.
Now, as you realised, that’s usually not a problem if you either have long task classes or tolerances, but for, let’s call them “time-critical tasks”, that creates a problem. The bigger the buffer, the more data has to be moved, first into RAM (which is usually the part triggering the violation, because it has to be done before recording the new values), then afterwards into the actual file (already done in IDLE up to my knowledge).

Unfortunately I cannot give concrete advise, but I have heard of some projects, where tinkering with the buffer size resolved such issues. However, there is no guarantee that works, the only reliable solution is to use a higher TC with tolerance (e.g. 10ms TC#8 +100ms tolerance).

Out of curiosity:

  • How much data are you recording (number of PVs, datatypes are pretty much irrelevant)?
  • How often are you recording (e.g. time-based 10ms, value change, custom trigger) and what is your task class time?
  • How big did you configure your buffer and file size (+ file type)?
  • Which “Save interval” did you configure?

PS: Regarding your thought of using a FIFO. That would most likely run into a situation that MpData does not have (or is solving)… What if the buffer (FIFO) is full, but there is no time (yet) to process the data? Are you then overwriting the oldest value, effectively losing it, or are you going to just not store anything new (also losing data) until it was processed, or do you make your FIFO bigger than it needs to be so that such situations are accounted for (technically wasting memory). MpData is trying to solve this by grabbing the buffer contents back into RAM when it is full (which unfortunately requires more time, especially for bigger buffer sizes) and then triggering a background save operation on this data, allowing new values to be recorded and stored into the buffer immediately.

PPS: I’m not saying that MpData is perfect, it was just one of the first components of mapp Services (or “mapp” as it was called back then), and it has never been extended/reworked to a more modern version. Hence I would love to hear feedback how it is used, because that makes it easier for us to plan ahead (we don’t hear a lot of complaints or feedback regarding this component).

Hello @michael.stockhammer ,

Thanks for the comprehensive answer.

As for use cases: For most of our data logging mappData is completely fine as we are only logging data infrequently or on a trigger.

But our TC1 runs on 0.8ms tick with 0 Tolerance. And sometimes we want to trace data on TC1. That can be really helpful for debugging. The variable amount varies.

Question: If I create a new TC that runs as fast as TC1 but with higher tolerances, I should only drop a few cycles on save, right? So everything I recorded before should be fine. Imagine a debug log that constantly records to memory and only exports once on a specific trigger.

Also, since I need to use the TIME type if I want a periodic trigger, I can’t go below 1ms. Which means if I want to trigger faster, I need to configure the trigger on change and then add another variable that gets incremented every cycle.

Overall i really like mappData as it makes the whole process really easy. Just trying to figure out where the limits are.

THX

Overall i really like mappData as it makes the whole process really easy.

That’s nice to hear (although I’m not the developer)! :heart: And regarding the limits… We need to improve our documentation. That’s something we also realised…

If I create a new TC that runs as fast as TC1 but with higher tolerances, I should only drop a few cycles on save, right?

I’m not entirely sure and can’t seem to find the help page anymore that I have used/referenced in the past, but from what I know/remember of the cyclic system (and TC violations / tolerances), that would/may happen, yes. In case potentially losing data from a few cycles is ok for your use-case, that might be a solution. In case losing any data is unacceptable, I’m honestly out of ideas (I tried thinking about solutions, but all of them involve way too much synchonization, copying/moving values around, …). One potential solution might be to “aggregate” data over multiple (fast) cycles and then record them in one (slow) cycle, but that would also indirectly lose data, and requires some maths to calculate e.g. Median, Mean, Min/Max, … (I’m just throwing this in as an idea, not sure if it is useful to you or anyone else)

Also, since I need to use the TIME type if I want a periodic trigger, I can’t go below 1ms.

There might be a way (but it’s been a while since I’ve tried it). I believe you can set the TIME to T#0ms (or just 0)… While you would get warning -2137866234 (same in mapp Services V5 and V6), the recording would automatically record once every cycle (and the warning could be ignored). This should happen with any time that is less than the TC the FUB runs in.

Losing a few datapoints on save should be fine.

Regarding TIME, the doc states: “The smallest number which can be represented other than 0 is T#1ms, which corresponds to one millisecond.”

Also, I just did a quick test.

Time-based with T#0s. Dropped cycles.

And triggered on value change:

Oh… That’s not what I expected… But then it seems we use TIME (or an integer type) also internally :see_no_evil_monkey: … I thought that at least internally we used a higher-precision datatype. OK, then ignore what I said about setting it to 0 (that may only work for multiples of 1ms then). I’m not sure I like that, but changing this might also be problematic… One more thing to document I guess. :sweat_smile:

Anyways, is your original question answered, or is there something still open (not sure if I may have missed a question in between)?

Yes, my original question is answered.

thx