Creating an multidimensional array of measurement files dynamically

Question

Hello All,

I want to create an app for reading multiple measurement files. First of all I don't know how many files I will get. I also don't know how many measurement points one file has. The situation could be as follows:

I have 7 measurement files (e.g. CSV), first file has 300 measurement points, second has 500, third has 250. I know which variables each file has for the sake of simplicity think of one column "Voltage", second "Current". When opening the program it should read all the available files store them into (in this case) 7 objects classes.

I thought first of creating a Class with Public Fields and than declare it as a List<T>, hmm this works only like this if I read only one file.

I would like to know which subject should I dive into when using C#, do I need to use LINQ for this or some other subject? So anybody has some suggestions on a topic what I should study to do what I explained above.

In case anybody is wondering why I should do this. When analyzing all data (i.e. all files) I need to be able to scroll thru the different files easily. And if I would create a method that loads each file when selecting it I think the program will start to be slow. It would be better to load all files in memory and than scroll thru it.

Thanks for some suggestions

The situ

Answer 1

Thank you for clarifying how you intend to use the data. We'll continue the approach I recommended earlier but note that we're making some assumptions here:

• Your files aren't too big
• You don't have a lot of files
• You don't want to load the data more than once if possible

If any of these assumptions are not true then you'll need to adjust the architecture but this might be a good start. Firstly you will have a type to represent the data in a row. You seem to already have that but I'm calling my version MeasurementRecord.

   public class MeasurementRecord  
   {  
       public int SampleNo { get; set; }  
       public int UdpTime { get; set; }  
       public int RPM { get; set; }  
   }  

This is what the grid will show. Next we'll have a wrapper class MeasurementFile that represents the collection of records from a file. It is important that your UI just focus on UI stuff so anything related to how you read/write the data should be elsewhere and we're using this type for that. Later if you need to expand to other sources then this type might be the base class that you start with. For now we'll just use a CSV file though.

   public class MeasurementFile  
   {   
       public MeasurementFile ( string filePath )  
       {  
           FilePath = filePath;  
           FileName = Path.GetFileName(filePath);  
     
           _items = new Lazy<List<MeasurementRecord>>(LoadData);  
       }  
     
       public string FilePath { get; }  
       public string FileName { get; }  
     
       public IEnumerable<MeasurementRecord> GetMeasurements () => _items.Value;  
     
       private List<MeasurementRecord> LoadData ()  
       {  
          ...  
       }  
     
       private readonly Lazy<List<MeasurementRecord>> _items;  
   }  

Finally we have the UI. The UI is just responsible for displaying data and reacting to user input. When the button is clicked we get a directory from the user, create a MeasurementFile instance for each csv file and bind that to the combo box. When the user selects an item from the combo we get the associated file information and render it in the grid. This allows us to read the data files only once.

   private void button1_Click ( object sender, EventArgs e )  
   {  
       var dlg = new FolderBrowserDialog();  
     
       if (dlg.ShowDialog() != DialogResult.OK)  
           return;  
     
       //Get the CSV files in the directory  
       //create an instance of each file so we can use it later  
       var files = Directory.EnumerateFiles(dlg.SelectedPath, "*.csv", SearchOption.AllDirectories)  
                               .Select(x => new MeasurementFile(x))  
                               .ToList();  
     
       //Update the combo box with the available files  
       comboBox1.DisplayMember = nameof(MeasurementFile.FileName);  
       comboBox1.DataSource = files;                      
   }  
     
   private void comboBox1_SelectedIndexChanged ( object sender, EventArgs e )  
   {  
       var file = comboBox1.SelectedItem as MeasurementFile;  
       if (file == null)  
       {  
           //Clear out any data  
           dataGridView1.DataSource = null;  
           return;  
       };  
     
       //Bind the current file data to the grid  
       dataGridView1.DataSource = file.GetMeasurements();  
   }  

Now the UI should be working but we need to read the CSV. There are many CSV libraries available but they depend upon your CSV file format. Let's assume your CSV format is the same across files and the header names are consistent as well. We just need to map each line to the corresponding data. I'm using CsvHelper here but most CSV libraries work the same way. I'm doing a simple mapping here but you can get as complex as you need to.

   private List<MeasurementRecord> LoadData ()  
   {  
       //Load the data from the CSV and force to List so it only runs once  
       return LoadDataCore().ToList();  
   }  
     
   private IEnumerable<MeasurementRecord> LoadDataCore ()  
   {  
       using (var stream = new StreamReader(FilePath))  
       using (var reader = new CsvReader(stream, System.Globalization.CultureInfo.InvariantCulture)) {  
     
           reader.Read();  
           reader.ReadHeader();  
     
           //Doing this by hand but you could also use the map feature...  
           while (reader.Read())  
           {  
               yield return new MeasurementRecord() {  
                   SampleNo = reader.GetField<int>("SampleNo"),  
                   UdpTime = reader.GetField<int>("UdpTime"),  
                   RPM = reader.GetField<int>("RPM")  
               };  
           };  
       };  
             
   }  

The above code hides the details of reading a CSV and can be as simple or complex as you need. More importantly the class manages the lifetime of the readers so the UI doesn't have to. For performance reasons it grabs the data only when first requested and returns the same data from that point on. If you need to support editing of the file then this may no longer be a good option but for your OP this should meet the needs.

Answer 2

if you define a file name scheme and use a common folder, then the app could load all files in the folder (this is common log viewers). see file I/o and directory access.

in your UI you could have a picker list of files (maybe organized by type and date created) and the scrolling view of selected log file(s).

whether you preload the files will depend on size. will they be too big, or take too long to load in memory.

Answer 3

Unless performance is an issue use List<T> to be able to create a dynamically resizable array. So it sounds like your measurement points would be stored here. The files should probably be stored in a List<T> as well since you don't know how many files up front. For example you might do something like this.

   public class MeasurementFile  
   {  
      public List<MeasurementPoint> Points { get; } = new List<MeasurePoint>();  
   }  
     
   //Could be a struct if it is just a data point and some classification data...  
   public class MeasurementPoint  
   {  
      public double Value { get; set; }  
   }  

Now you just track the files.

   var files = new List<MeasurementFile>();  
     
   foreach (var filenames in GetFilesToRead())  
   {  
      var file = ReadFile(filename);  
      files.Add(file);  
   };  

Helper methods here are whatever you currently do to read the files. The point(s) go into the MeasurementFile object you create for each file you're reading.

Answer 4

Ahhh clear so as I expected a bit already I should use 2 times a List<T>. Thanks a lot!

One small extra question would I make two separate Class files, or one Class file with two Class objects?

I will play around a bit will be a new adventure :)

Answer 5

I have done the following

        private void button1_Click(object sender, EventArgs e)  
        {  
            string path = "";  
            var measurementFiles = new List<MeasurementFile>();  
              
            using (FolderBrowserDialog fbd = new FolderBrowserDialog())  
            {  
                if (fbd.ShowDialog() == DialogResult.OK)  
                {  
                    path = fbd.SelectedPath;  
                    label1.Text = path;  
                }  
  
                string[] files = Directory.GetFileSystemEntries(path, "*.csv", SearchOption.AllDirectories);  
                foreach (string file in files)  
                {  
                    comboBox1.Items.Add(Path.GetFileName(file));  
                    using (var reader = new StreamReader(file))  
                    using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture))  
                    {  
                        measurementFiles.Add(csv);  
  
                    }  
                }  
            }  
        }  

And created two classes

public class MeasurementFile  
    {  
        public List<MeasurementPoint> Measurements { get; } = new List<MeasurementPoint>();  
  
    }  

public class MeasurementPoint  
    {  
        public int SampleNo { get; set; }     
        public int UdpTime { get; set; }  
        public int RPM { get; set; }  
  
    }  

But this doesn't work as I hoped for. The measurementFiles.Add(csv); gives a compile error and states that csv is NOT NULL.

Message Error CS1503 Argument 1: cannot convert from CsvHelper.CsvReader to TestProjectBlueCal.MeasurementFile

This error I don't understand