pnp powershell remove lines from a csv file

Question

I have a script that iterates through a list of files in a csv file that are known to have broken inheritance using

Import-Csv $filelist | ForEach-Object {

and re-inherits the permissions.

I am wondering if there is a way, within each loop, to remove the current line in the csv file.

The idea is that as the script runs it clears out the current row in the csv file. That way if the script crashes I can simply restart the script a it will pick up where it left off. Not normally something I would worry about but the piece of work I am currently engaged with has a test library of over 50k files and the production library over a million files.

Thanks

Answer 1

If you hundreds or thousands of entries in the csv, it will take some time to save that amount of data on each loop.

I would think it would be better to maintain a count in a file that indicates how many rows were processed. On restart, just skip the rows until you hit the count.

$PointerFile = 'c:\temp\Pointer.txt'                   #   del  c:\temp\Pointer.txt   to start over   
$csv = import-csv -Path C:\temp\test.csv
If (test-path $PointerFile) {
    $StartAt  =  [int](get-content -path $PointerFile)
    "Restarting at line {0}" -f $StartAt 
} else {
    $StartAt  =  0
    "Starting at first entry." 
}
$counter = 1
foreach ($Row in $csv) {
    if ($counter -lt $StartAt) {
        "Skipping {0}" -f $counter
        $counter++                       # look to next row 
        continue                         # This entry has been processed 
    }

    # process the row here 


    $counter++ 
    $counter | out-file $PointerFile             # save the counter to our file     
    start-sleep 5                        # delay for testing, remove this line  when running for real  
}    

Answer 2

I just can't resist tinkering with stuff. Here's another possible take, using the suggestion @MotoX80 offered:

$PointerFile = 'c:\temp\Pointer.txt'                   #   del  c:\temp\Pointer.txt   to start over   
If (Test-Path $PointerFile){  
    $StartAt  =  [int](get-content -path $PointerFile)  
    "Restarting at line {0}" -f $StartAt   
    # since the script will skip the first "x" of number data rows in the CSV  
    # get the header names from the file  
    $p =    Get-Content C:\junk\server.csv |  
                Select-Object -First 2 |  
                    ConvertFrom-CSV  
    $h = $p.psobject.properties.name -join ","  
} else {  
    $StartAt  =  0  
    "Starting at first entry."   
}  
  
If ($StartAt -gt 0){  
    $c =    Get-Content C:\Junk\server.csv |  
                Select-Object -Skip ($StartAt + 1) |    # add 1 for the header!  
                    ConvertFrom-Csv -Header $h  
}  
else{  
    $c =    Import-CSV C:\Junk\server.csv  
                Select-Object   
}  
ForEach ($row in $c){  
    Try{  
        # process this row  
    }  
    Catch{  
        # report problem and die  
        # -- or  
        # place current record into an eacrption file  
        # to be processed later (if possible)  
    }  
    # The whole "skip 'x' records may not be necessary  
    # if the script never dies and an "exception" file is  
    # used to work with the files that failed  
    ++$StartAt |   
        Out-File $PointerFile             # save the counter  
}