Creating an multidimensional array of measurement files dynamically

Question

Hello All,

I want to create an app for reading multiple measurement files. First of all I don't know how many files I will get. I also don't know how many measurement points one file has. The situation could be as follows:

I have 7 measurement files (e.g. CSV), first file has 300 measurement points, second has 500, third has 250. I know which variables each file has for the sake of simplicity think of one column "Voltage", second "Current". When opening the program it should read all the available files store them into (in this case) 7 objects classes.

I thought first of creating a Class with Public Fields and than declare it as a List<T>, hmm this works only like this if I read only one file.

I would like to know which subject should I dive into when using C#, do I need to use LINQ for this or some other subject? So anybody has some suggestions on a topic what I should study to do what I explained above.

In case anybody is wondering why I should do this. When analyzing all data (i.e. all files) I need to be able to scroll thru the different files easily. And if I would create a method that loads each file when selecting it I think the program will start to be slow. It would be better to load all files in memory and than scroll thru it.

Thanks for some suggestions

The situ

Answer 1

Thank you for clarifying how you intend to use the data. We'll continue the approach I recommended earlier but note that we're making some assumptions here:

• Your files aren't too big
• You don't have a lot of files
• You don't want to load the data more than once if possible

If any of these assumptions are not true then you'll need to adjust the architecture but this might be a good start. Firstly you will have a type to represent the data in a row. You seem to already have that but I'm calling my version MeasurementRecord.

   public class MeasurementRecord  
   {  
       public int SampleNo { get; set; }  
       public int UdpTime { get; set; }  
       public int RPM { get; set; }  
   }  

This is what the grid will show. Next we'll have a wrapper class MeasurementFile that represents the collection of records from a file. It is important that your UI just focus on UI stuff so anything related to how you read/write the data should be elsewhere and we're using this type for that. Later if you need to expand to other sources then this type might be the base class that you start with. For now we'll just use a CSV file though.

   public class MeasurementFile  
   {   
       public MeasurementFile ( string filePath )  
       {  
           FilePath = filePath;  
           FileName = Path.GetFileName(filePath);  
     
           _items = new Lazy<List<MeasurementRecord>>(LoadData);  
       }  
     
       public string FilePath { get; }  
       public string FileName { get; }  
     
       public IEnumerable<MeasurementRecord> GetMeasurements () => _items.Value;  
     
       private List<MeasurementRecord> LoadData ()  
       {  
          ...  
       }  
     
       private readonly Lazy<List<MeasurementRecord>> _items;  
   }  

Finally we have the UI. The UI is just responsible for displaying data and reacting to user input. When the button is clicked we get a directory from the user, create a MeasurementFile instance for each csv file and bind that to the combo box. When the user selects an item from the combo we get the associated file information and render it in the grid. This allows us to read the data files only once.

   private void button1_Click ( object sender, EventArgs e )  
   {  
       var dlg = new FolderBrowserDialog();  
     
       if (dlg.ShowDialog() != DialogResult.OK)  
           return;  
     
       //Get the CSV files in the directory  
       //create an instance of each file so we can use it later  
       var files = Directory.EnumerateFiles(dlg.SelectedPath, "*.csv", SearchOption.AllDirectories)  
                               .Select(x => new MeasurementFile(x))  
                               .ToList();  
     
       //Update the combo box with the available files  
       comboBox1.DisplayMember = nameof(MeasurementFile.FileName);  
       comboBox1.DataSource = files;                      
   }  
     
   private void comboBox1_SelectedIndexChanged ( object sender, EventArgs e )  
   {  
       var file = comboBox1.SelectedItem as MeasurementFile;  
       if (file == null)  
       {  
           //Clear out any data  
           dataGridView1.DataSource = null;  
           return;  
       };  
     
       //Bind the current file data to the grid  
       dataGridView1.DataSource = file.GetMeasurements();  
   }  

Now the UI should be working but we need to read the CSV. There are many CSV libraries available but they depend upon your CSV file format. Let's assume your CSV format is the same across files and the header names are consistent as well. We just need to map each line to the corresponding data. I'm using CsvHelper here but most CSV libraries work the same way. I'm doing a simple mapping here but you can get as complex as you need to.

   private List<MeasurementRecord> LoadData ()  
   {  
       //Load the data from the CSV and force to List so it only runs once  
       return LoadDataCore().ToList();  
   }  
     
   private IEnumerable<MeasurementRecord> LoadDataCore ()  
   {  
       using (var stream = new StreamReader(FilePath))  
       using (var reader = new CsvReader(stream, System.Globalization.CultureInfo.InvariantCulture)) {  
     
           reader.Read();  
           reader.ReadHeader();  
     
           //Doing this by hand but you could also use the map feature...  
           while (reader.Read())  
           {  
               yield return new MeasurementRecord() {  
                   SampleNo = reader.GetField<int>("SampleNo"),  
                   UdpTime = reader.GetField<int>("UdpTime"),  
                   RPM = reader.GetField<int>("RPM")  
               };  
           };  
       };  
             
   }  

The above code hides the details of reading a CSV and can be as simple or complex as you need. More importantly the class manages the lifetime of the readers so the UI doesn't have to. For performance reasons it grabs the data only when first requested and returns the same data from that point on. If you need to support editing of the file then this may no longer be a good option but for your OP this should meet the needs.

Answer 2

Thanks for your guidance this is much appreciated. There is not yet a real architecture regarding the app at the moment I'm figuring out how to make the app and what will work and what not.

Even though the files are not very big it is not efficient to keep reading the files as I will keep skipping through files while analyzing the data. So the most efficient IMHO is to read all files store them as MeasurementFiles into the memory and go thru them.
I found CsvHelper but if I do that and create the 'List' as you suggested it will read everytime I skip from one file to another the data. I have created a 'ComboBox' which shows all the files and by selecting a file it should show me the data. For now I think the best is to create those MeasurementFiles for each file available with all measurement points.

So I think in the case I would like to do you suggest to do it like so?

public class MeasurementFile  
{  
   public MeasurementFile (  string filename )  
   {  
      var stream = new StreamReader(filename);  
      _reader new CsvReader(reader, CultureInfo.InvariantCulture);                      
   }  
   private CvsReader _reader;  
}  

Answer 3

In fact previously I tried to fix it by myself with no luck I have to say but found out there were more issues to solve such as the delimiter and start at the correct row. What I made was this:

private void button1_Click(object sender, EventArgs e)  
        {  
            string path = "";  
            var measurementFiles = new List<MeasurementFile>();  
  
            using (FolderBrowserDialog fbd = new FolderBrowserDialog())  
            {  
                if (fbd.ShowDialog() == DialogResult.OK)  
                {  
                    path = fbd.SelectedPath;  
                    label1.Text = path;  
                }  
  
                string[] files = Directory.GetFileSystemEntries(path, "*.csv", SearchOption.AllDirectories);  
                foreach (string file in files)  
                {  
                    comboBox1.Items.Add(Path.GetFileName(file));  
                    using (var reader = new StreamReader(file))  
                    {  
                        var csvConfig = new CsvConfiguration(CultureInfo.InvariantCulture)  
                        {  
                            Delimiter = ";",  
                            HasHeaderRecord = false,  
                        };  
                        using (var csvReader = new CsvReader(reader, csvConfig))  
                        {  
                            for (int i = 0; i < 1; i++)  
                            {  
                                csvReader.Read();  
                            }  
                            var records = csvReader.GetRecords<MeasurementPoint>().ToList();  
                            measurementFiles.Add(records);  
                        }                                  
                    }  
                }  
            }      
        }  

This gave exactly the same issue but I understand now why next thing is to solve it ;)

Answer 4

I think I still miss some conceptual understanding. I'm able to show the data of each file into a 'datagridview' with this code:

        private void comboBox1_SelectedIndexChanged(object sender, EventArgs e)  
        {  
            string fileName = comboBox1.SelectedItem.ToString();  
            string filePath = path + "\\" + fileName;  
            //var measurementFiles = new List<CsvReader>();  
  
            using (var reader = new StreamReader(filePath))  
            {  
                var csvConfig = new CsvConfiguration(CultureInfo.InvariantCulture)  
                {  
                    Delimiter = ";",  
                    HasHeaderRecord = false,  
                };  
                using (var csvReader = new CsvReader(reader, csvConfig))  
                {  
                    for (int i = 0; i < 1; i++)  
                    {  
                        csvReader.Read();  
                    }  
                    var records = csvReader.GetRecords<MeasurementPoint>().ToList();  
  
                    dataGridView1.DataSource = records;  
                    label1.Text = records[6].ToString();  
                      
                }  
            }  
  
        }  

but I don't understand why I can't acces a certain measurement at a certain colomn. E.g. 'MeasurmentPoint.SampleNo'

I'm completely confused

Answer 5

First of all sorry for the late response but I'm in Brazil for holiday and my apartment lost internet so I could not answer earlier.

Second thank you so much for your detailed answer it works great now, exactly as I wanted! I wish I could understand the full code that you applied but there are techniques and coding ways that I really do not understand unfortunately. Funny thing is that you mentioned earlier to use Lazy<> but I thought "Lazy" in the way of easy and fast programming I did not know it was a constructor build in C#.

One thing I really would like to understand is the following (for the sake of learning):

We created a Class MeasurementPoint with a field SampleNo why is it impossible to retrieve data from the class like this:
MeasurementPoint.SampleNo into a lable for example?

Is that because I do not use the right instance of it? On the other hand when I write MeasurementPoint. it doesn't even show the coded fields from the class.

I would appreciate if you could explain why this is such