hi, guys
Just as the title indicates. The customer wants to read several relatively large CSV files to obtain data.
A csv file has approximately several hundred thousand rows, but the number of columns is fixed.
For this reason, I’m not sure if there is a good way to read CSV files instead of operating through fileIO.
Hi there,
unfortunately I think the only possible way is to handle them via FileIO. To my knowledge there is no library that can handle such a thing for you.
Best regards
Perhaps, as you said, there may not be a suitable library to do this for the time being.
But using fileIO to handle such a large csv is really unwise. To be honest, it’s just too stupid.
fileIO does not provide the readline method, and it also cannot know how many lines there are in the file.
The most crucial point is that fileIO does not offer reading by IEC type.
If it is possible to implement MpDataTableUI or MpAuditTrailUI, which can display multiple lines, turn pages, and also obtain actual text values by IEC type. However, it seems that the corresponding parsing method is not provided in AS.
I know that this is not a nice API for CSV in general, but file reading can also occur in chunks - and the processing be done so as well. (and FileReadEx also tells you when it read less than the bytes requested).
CSV is a really flexible format allowing for many different columns and rows to be stored - but therefore it’s also not easy to make generic methods that return specific data.
Both of the FUBs you mentioned make use of known formats behind everything they show - they don’t need to parse data they know nothing about.
Anyway, maybe someone has any other input or already programmed anything.
Good luck
The Loupe team (Home | Loupe) created this library (GitHub - loupeteam/CSVFileLib: AR Library for reading and writing variable values to and from CSV files) that might provide you something. I personally haven’t used it. I’ve used several other custom libraries written by B&R regional offices or B&R application teams. They aren’t publically available because as I’ll allude to, there is no CSV standard.
The issue is CSV is a more or less a formatting standard, not necessarily a type of file with a universally standard structure that can be parsed in a standardized way. There is no such thing as a universal CSV format that can be used/implemented on a per language basis. The FileIO library is a low level file access library. It is written and optomized with real time performance first, developer usability second.
To do a standard ReadLine type of function could potentially require reading multiple kilobytes of data before a newline character shows up if your CSV file contains large amounts of text. It also requires you to know the encoding of your file (ASCII, UTF-8, UTF16, etc). CSV doesn’t specificy any of those.
Since CSV doesn’t have any standards body published standard, there isn’t technically a requirement that a newline, carriage return or line feed is even present in your file. It could be just commas for column breaks and semi-colons for line breaks.
I’ve worked on projects that used extremely large CSV files, and generally they are best to be manipulated/sorted in a GPOS and passed down to the PLC in better format, especially if the source of the CSV isn’t neccessarily controled by the end user. I.e. if some third party tool can be updated more often than a customer wants the PLC updated.
I’ve even seen a CSV library that read in a csv file and treated it like a Excel worksheet, i.e. you could access the data in a per cell like design, i.e. I want Row 4 column 3.
Sorry to say you are likely going to need to create your own parser using FileIO, AsBrStr or AsBRWstr, Standard library, and AsIecCon (string to IEC type conversions).
Hello all,
long time ago when such mpTools for csv didn´t really exist i build my own system.
The most important was that when you read the file by fileio lib i put all the Data in Memory area which i tooked from tmp_alloc and after i parse the data in a double chained list in order to have access. This “double chained list” is expert mode and deep ANSI C
But i wonder using a tmp_alloc for the memory area where the data of the file are copied should be also a solution to provide the data for further use for csv Libraries.
In the end: how big is the needed memory size and how big is the size of free memory on the plc
The controller is a pair of redundant X20CP3687X
Yes, as you said, memory is also a key factor.
But the good news is that the customer’s CSV is a serialized and regular format.
And I am very familiar with the C language. I would be most grateful if you could provide this source code.
The good news is that the customer’s CSV is a serialized and regular format.
Let me take the solution provided by Loupe team
Tks