Creating an multidimensional array of measurement files dynamically

Question

Hello All,

I want to create an app for reading multiple measurement files. First of all I don't know how many files I will get. I also don't know how many measurement points one file has. The situation could be as follows:

I have 7 measurement files (e.g. CSV), first file has 300 measurement points, second has 500, third has 250. I know which variables each file has for the sake of simplicity think of one column "Voltage", second "Current". When opening the program it should read all the available files store them into (in this case) 7 objects classes.

I thought first of creating a Class with Public Fields and than declare it as a List<T>, hmm this works only like this if I read only one file.

I would like to know which subject should I dive into when using C#, do I need to use LINQ for this or some other subject? So anybody has some suggestions on a topic what I should study to do what I explained above.

In case anybody is wondering why I should do this. When analyzing all data (i.e. all files) I need to be able to scroll thru the different files easily. And if I would create a method that loads each file when selecting it I think the program will start to be slow. It would be better to load all files in memory and than scroll thru it.

Thanks for some suggestions

The situ

Answer 1

Thank you for clarifying how you intend to use the data. We'll continue the approach I recommended earlier but note that we're making some assumptions here:

• Your files aren't too big
• You don't have a lot of files
• You don't want to load the data more than once if possible

If any of these assumptions are not true then you'll need to adjust the architecture but this might be a good start. Firstly you will have a type to represent the data in a row. You seem to already have that but I'm calling my version MeasurementRecord.

   public class MeasurementRecord  
   {  
       public int SampleNo { get; set; }  
       public int UdpTime { get; set; }  
       public int RPM { get; set; }  
   }  

This is what the grid will show. Next we'll have a wrapper class MeasurementFile that represents the collection of records from a file. It is important that your UI just focus on UI stuff so anything related to how you read/write the data should be elsewhere and we're using this type for that. Later if you need to expand to other sources then this type might be the base class that you start with. For now we'll just use a CSV file though.

   public class MeasurementFile  
   {   
       public MeasurementFile ( string filePath )  
       {  
           FilePath = filePath;  
           FileName = Path.GetFileName(filePath);  
     
           _items = new Lazy<List<MeasurementRecord>>(LoadData);  
       }  
     
       public string FilePath { get; }  
       public string FileName { get; }  
     
       public IEnumerable<MeasurementRecord> GetMeasurements () => _items.Value;  
     
       private List<MeasurementRecord> LoadData ()  
       {  
          ...  
       }  
     
       private readonly Lazy<List<MeasurementRecord>> _items;  
   }  

Finally we have the UI. The UI is just responsible for displaying data and reacting to user input. When the button is clicked we get a directory from the user, create a MeasurementFile instance for each csv file and bind that to the combo box. When the user selects an item from the combo we get the associated file information and render it in the grid. This allows us to read the data files only once.

   private void button1_Click ( object sender, EventArgs e )  
   {  
       var dlg = new FolderBrowserDialog();  
     
       if (dlg.ShowDialog() != DialogResult.OK)  
           return;  
     
       //Get the CSV files in the directory  
       //create an instance of each file so we can use it later  
       var files = Directory.EnumerateFiles(dlg.SelectedPath, "*.csv", SearchOption.AllDirectories)  
                               .Select(x => new MeasurementFile(x))  
                               .ToList();  
     
       //Update the combo box with the available files  
       comboBox1.DisplayMember = nameof(MeasurementFile.FileName);  
       comboBox1.DataSource = files;                      
   }  
     
   private void comboBox1_SelectedIndexChanged ( object sender, EventArgs e )  
   {  
       var file = comboBox1.SelectedItem as MeasurementFile;  
       if (file == null)  
       {  
           //Clear out any data  
           dataGridView1.DataSource = null;  
           return;  
       };  
     
       //Bind the current file data to the grid  
       dataGridView1.DataSource = file.GetMeasurements();  
   }  

Now the UI should be working but we need to read the CSV. There are many CSV libraries available but they depend upon your CSV file format. Let's assume your CSV format is the same across files and the header names are consistent as well. We just need to map each line to the corresponding data. I'm using CsvHelper here but most CSV libraries work the same way. I'm doing a simple mapping here but you can get as complex as you need to.

   private List<MeasurementRecord> LoadData ()  
   {  
       //Load the data from the CSV and force to List so it only runs once  
       return LoadDataCore().ToList();  
   }  
     
   private IEnumerable<MeasurementRecord> LoadDataCore ()  
   {  
       using (var stream = new StreamReader(FilePath))  
       using (var reader = new CsvReader(stream, System.Globalization.CultureInfo.InvariantCulture)) {  
     
           reader.Read();  
           reader.ReadHeader();  
     
           //Doing this by hand but you could also use the map feature...  
           while (reader.Read())  
           {  
               yield return new MeasurementRecord() {  
                   SampleNo = reader.GetField<int>("SampleNo"),  
                   UdpTime = reader.GetField<int>("UdpTime"),  
                   RPM = reader.GetField<int>("RPM")  
               };  
           };  
       };  
             
   }  

The above code hides the details of reading a CSV and can be as simple or complex as you need. More importantly the class manages the lifetime of the readers so the UI doesn't have to. For performance reasons it grabs the data only when first requested and returns the same data from that point on. If you need to support editing of the file then this may no longer be a good option but for your OP this should meet the needs.

Answer 2

The screenshot below shows what I was trying to do. And as you mentioned I did not create an instance of the object

So I have created an instance as you suggested like this:

 MeasurementPoint pt = new MeasurementPoint();  
 pt.SampleNo = "10";  

If I understand it well now I assign to pt.SampleNo a string "10";

But what if I would retrieve all values from e.g. UdpTime or all SampleNo?

I hope you don't get crazy with all these questions, if so sorry about that.

Answer 3

I want to add something to my previous comment just to clarify how all is working. The csv files I use don't have a header instead of that the first row of the csv have a string with the names of each column. Than from row 2 all values appear. To fix this I'll show you how I made the Class MeasurementFile:

   public class MeasurementFile  
    {  
        public MeasurementFile (string filePath)  
        {  
            FilePath = filePath;  
            FileName = Path.GetFileName(filePath);  
  
            _items = new Lazy<List<MeasurementRecord>>(LoadData);   
        }  
  
        public string FilePath { get; }  
        public string FileName { get; }  
             
        public IEnumerable<MeasurementRecord> GetMeasurements() => _items.Value;  
  
        private List<MeasurementRecord> LoadData()  
        {  
            //Load the data from the CSV and force to List so it only runs once  
            return LoadDataCore().ToList();  
        }  
  
        private IEnumerable<MeasurementRecord> LoadDataCore()  
        {  
            using (var stream = new StreamReader(FilePath))  
            {  
                var csvConfig = new CsvConfiguration(CultureInfo.InvariantCulture)  
                {  
                    Delimiter = ";",  
                    HasHeaderRecord = false,  
                };  
  
                using (var reader = new CsvReader(stream, csvConfig))  
                {  
                    for (int i = 0; i < 1; i++)  
                    {  
                        reader.Read();                         
                    }  
  
                    //Doing this by hand but can be done with map feature...  
                    while (reader.Read())  
                    {  
                        yield return new MeasurementRecord()  
                        {  
                            SampleNo = reader.GetField(0),  
                            UdpTime = reader.GetField(1),  
                            RPM = reader.GetField(2),  
                            Section = reader.GetField(3),  
                            QgtdAtSec = reader.GetField(4),  
                            TimeAtSec = reader.GetField(5),  
                            QgtcAvgMv = reader.GetField(6),  
                            AiTorque = reader.GetField(7),  
                            QgtcAvgMa = reader.GetField(8),  
                            AiCurrent = reader.GetField(9),  
                            CurrentAtSec = reader.GetField(10),  
                            Ch10 = reader.GetField(11),  
                            BlueMv = reader.GetField(12),  
                            BlueStress = reader.GetField(13),  
                            BlueTorque = reader.GetField(14),  
                            TimeGap = reader.GetField(15)  
                        };  
                    };  
                };  
            }  
        }    
        private readonly Lazy<List<MeasurementRecord>> _items;  
    }  

What I want to do is to carry out mathematical operations on certain columns e.g. compute the max values on a column and compute Standard Deviation on certain columns. I think this should be better to do at the method comboBox1_SelectedIndexChanged and for the sake of readability create functions for those computations.

I guess these kind of things start to get way off topic. But I think it would be easier to do such operations on the columns of the datagridview, don't you think?