Suggestions for Large Scale Data Recording

Ask Questions Programming
1

I was curious if anyone was doing large scale data recording and how they are implementing it. The MAPPdata recorder seems to limit file sizes to 50 mb.

2

Hi @Danny_Rodriguez ,
welcome to the community!
So far I have never worked with file so big so let’s ask AI:
Question to AI: “In B&R Automation Studio how can we implement large scale data recording and how they are implementing it?“

Answer:
In B&R Automation Studio, large-scale data recording is primarily implemented through the mapp Technology framework, specifically using the mapp Data components. This modular approach allows for high-speed, reliable data acquisition without the need for extensive custom coding.

Implementation Methods

  • mapp Data Recorder (MpDataRecorder): This is the core component for high-performance recording. It uses a “configure, don’t program” philosophy to log variable values.

    • Data Capture: You define variables (e.g., temperatures, positions) using the MpDataRegPar function block, which connects to a central recorder configuration via a mapp link.

    • Storage Management: Recorded data is typically saved as CSV files on physical storage devices like industrial PC M.2 modules (up to 1 TB) or CFast cards.

    • Large-Scale Handling: To manage massive amounts of data, the recorder can be configured to use a FIFO (First-In-First-Out) system by setting a “Maximum number of files” and enabling “Overwrite oldest file”.

  • Linux-Based Databases (exOS): For truly large-scale or complex data analytics, B&R uses exOS technology to run Linux-based database systems (e.g., SQL, InfluxDB) directly on the machine controller. This allows historical data to be stored and processed using AI or machine learning algorithms without burdening the real-time control loop.

  • IIoT Services & Cloud Connectivity: For enterprise-level data recording, B&R implements IIoT Services that aggregate machine data and stream it to higher-level IT architectures or cloud platforms for long-term storage and advanced analytics

At the end have you considered the option, at the first step, to configure “Maximum number of files“ and enable the option “Overwrite oldest file“ ? When you have new file you can copy the old one via FTP on your server? On the server side with a script (phyton?) you can merge them in a new file.

Just an idea
Ciao
Valerio

3

Not sure what ‘large scale data’ in your project means. But when working with larger files, I would move out of the PLC. I think the 50MB restriction has a reason.

1 Like
4

If you don’t need real time logging you then PVI in combination with brwatch might be a solution.

brwatch can log variables to csv files on PC side.

Since PVI is limited to a refresh time of a minimum of about 50 ms this could be a limiting factor though.

1 Like
5

interesting tool, I was not aware of it, thanks for sharing :slight_smile: