question

ChrisManceyGroup-6193 avatar image
0 Votes"
ChrisManceyGroup-6193 asked MotoX80 commented

pnp powershell remove lines from a csv file

I have a script that iterates through a list of files in a csv file that are known to have broken inheritance using

Import-Csv $filelist | ForEach-Object {

and re-inherits the permissions.

I am wondering if there is a way, within each loop, to remove the current line in the csv file.

The idea is that as the script runs it clears out the current row in the csv file. That way if the script crashes I can simply restart the script a it will pick up where it left off. Not normally something I would worry about but the piece of work I am currently engaged with has a test library of over 50k files and the production library over a million files.

Thanks

windows-server-powershell
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MotoX80 avatar image
0 Votes"
MotoX80 answered

If you hundreds or thousands of entries in the csv, it will take some time to save that amount of data on each loop.

I would think it would be better to maintain a count in a file that indicates how many rows were processed. On restart, just skip the rows until you hit the count.

 $PointerFile = 'c:\temp\Pointer.txt'                   #   del  c:\temp\Pointer.txt   to start over   
 $csv = import-csv -Path C:\temp\test.csv
 If (test-path $PointerFile) {
     $StartAt  =  [int](get-content -path $PointerFile)
     "Restarting at line {0}" -f $StartAt 
 } else {
     $StartAt  =  0
     "Starting at first entry." 
 }
 $counter = 1
 foreach ($Row in $csv) {
     if ($counter -lt $StartAt) {
         "Skipping {0}" -f $counter
         $counter++                       # look to next row 
         continue                         # This entry has been processed 
     }
    
     # process the row here 
    
        
     $counter++ 
     $counter | out-file $PointerFile             # save the counter to our file     
     start-sleep 5                        # delay for testing, remove this line  when running for real  
 }    


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

RichMatheisen-8856 avatar image
0 Votes"
RichMatheisen-8856 answered MotoX80 commented

I just can't resist tinkering with stuff. Here's another possible take, using the suggestion @MotoX80 offered:

 $PointerFile = 'c:\temp\Pointer.txt'                   #   del  c:\temp\Pointer.txt   to start over 
 If (Test-Path $PointerFile){
     $StartAt  =  [int](get-content -path $PointerFile)
     "Restarting at line {0}" -f $StartAt 
     # since the script will skip the first "x" of number data rows in the CSV
     # get the header names from the file
     $p =    Get-Content C:\junk\server.csv |
                 Select-Object -First 2 |
                     ConvertFrom-CSV
     $h = $p.psobject.properties.name -join ","
 } else {
     $StartAt  =  0
     "Starting at first entry." 
 }
    
 If ($StartAt -gt 0){
     $c =    Get-Content C:\Junk\server.csv |
                 Select-Object -Skip ($StartAt + 1) |    # add 1 for the header!
                     ConvertFrom-Csv -Header $h
 }
 else{
     $c =    Import-CSV C:\Junk\server.csv
                 Select-Object 
 }
 ForEach ($row in $c){
     Try{
         # process this row
     }
     Catch{
         # report problem and die
         # -- or
         # place current record into an eacrption file
         # to be processed later (if possible)
     }
     # The whole "skip 'x' records may not be necessary
     # if the script never dies and an "exception" file is
     # used to work with the files that failed
     ++$StartAt | 
         Out-File $PointerFile             # save the counter
 }


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Admit it, you're just trying to out-code me. Keep it up, and I going to post S/370 Assembler code.

0 Votes 0 ·

I'll raise you a neatly wired 401 accounting machine plug board -- or maybe some IBM 1401 SPS-1 code. :-)

0 Votes 0 ·
MotoX80 avatar image MotoX80 RichMatheisen-8856 ·

You got me on that one.

0 Votes 0 ·