Powershell: Get-DnsServerResourceRecord -ZoneName test.com|ConvertTo-Json running this command on huge records data

Question

Hi,

I'm working on a project trying to parse the contents of the zones and records, which I will be using for some other task in my project.

I'm using the below command to get the data in the json format on windows server 2019.

For zones:
Get-DnsServerZone|ConvertTo-Json
Get-DnsServerResourceRecord -ZoneName test.com -RRtype 'SOA'|ConvertTo-Json

For records:
Get-DnsServerResourceRecord -ZoneName test.com|ConvertTo-Json

For the zones, I mainly extract values such as zonename, ttl, expiry etc. and for the records, I mainly extract the record values such as ip addr etc

Note: I'm using python to parse the json data

The command works fine for small amount of data, but when I try to extract info of a zone which consists of thousands(or maybe a million, I'm not sure of the exact count) of records, the commands gets stuck (I have waited for around 15 hours after executing the command). Also, once I encountered the below issue.

ConvertTo-Json : Exception of type 'System.OutOfMemoryException' was thrown.

At line:1 char:61

• ... nsServerResourceRecord -ZoneName test.com|ConvertTo-Json
• ~~~~~~~~~~~~~~
  • CategoryInfo : NotSpecified: (:) [ConvertTo-Json], OutOfMemoryException
  • FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.ConvertToJsonCommand

What would be the best way to get the required data in the json format, for such huge data set? Also, please let me know if there are any other viable methods to accomplish this. I would like the final output to be in json format as its easy to parse.

Answer 1

If you're working with very large data sets, the "Select-Object" probably isn't the best way to get the information. That creates a new PSCustomObject that's passed into the pipeline and that object type, while convenient, takes a while to create and it's actually pretty big (in terms of memory).

This code does essentially the same thing but uses a Here-String. Your individual JSON records are simple enough to make this pretty easy without creating any additional intermediate objects:

Get-DnsServerResourceRecord -ZoneName w2k12.local |  
    ForEach-Object{  
@"  
{  
    "RecordType":  "$($_.RecordType)",  
    "HostName":  "$($_.HostName)",  
    "RecordData":  "$($_.RecordData)"  
}  
"@          
    }  

Answer 2

Hi @Aditya G

I don't have a large enough environment to test it, but could try this approach which adds the json text for each record to the output file one at a time, rather than trying to hold the json text in memory until all the records have been returned.

foreach ($t in Get-DnsServerResourceRecord -ZoneName w2k12.local) { $t | convertto-json  |  out-file -file "json.test1.txt" -append}  

Gary.

Answer 3

This is how I solved the issue.

The ConvertTo-Json does not output the data it receives from the stream, until it is done processing all the objects.

Solution: Send each object one by one instead of sending all objects at once.

Get-DnsServerResourceRecord -ZoneName Corp.microsoft.com|Select-Object -Property "RecordType","HostName","RecordData"|Foreach-object {$_ |ConvertTo-Json}

Note: Select-Object is optional depending on your needs.

In case of large data, better to redirect the result to a file.