Powershell: Get-DnsServerResourceRecord -ZoneName test.com|ConvertTo-Json running this command on huge records data

asked 2022-08-23T06:25:26.43+00:00
Aditya G 41 Reputation points

Hi,

I'm working on a project trying to parse the contents of the zones and records, which I will be using for some other task in my project.

I'm using the below command to get the data in the json format on windows server 2019.

For zones:
Get-DnsServerZone|ConvertTo-Json
Get-DnsServerResourceRecord -ZoneName test.com -RRtype 'SOA'|ConvertTo-Json

For records:
Get-DnsServerResourceRecord -ZoneName test.com|ConvertTo-Json

For the zones, I mainly extract values such as zonename, ttl, expiry etc. and for the records, I mainly extract the record values such as ip addr etc

Note: I'm using python to parse the json data

The command works fine for small amount of data, but when I try to extract info of a zone which consists of thousands(or maybe a million, I'm not sure of the exact count) of records, the commands gets stuck (I have waited for around 15 hours after executing the command). Also, once I encountered the below issue.

ConvertTo-Json : Exception of type 'System.OutOfMemoryException' was thrown.

At line:1 char:61

  • ... nsServerResourceRecord -ZoneName test.com|ConvertTo-Json
  • ~~~~~~~~~~~~~~
    • CategoryInfo : NotSpecified: (:) [ConvertTo-Json], OutOfMemoryException
    • FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.ConvertToJsonCommand

What would be the best way to get the required data in the json format, for such huge data set? Also, please let me know if there are any other viable methods to accomplish this. I would like the final output to be in json format as its easy to parse.

Windows DHCP
Windows DHCP
Windows: A family of Microsoft operating systems that run across personal computers, tablets, laptops, phones, internet of things devices, self-contained mixed reality headsets, large collaboration screens, and other devices.DHCP: Dynamic Host Configuration Protocol (DHCP). A communications protocol that lets network administrators manage centrally and automate the assignment of Internet Protocol (IP) addresses in an organization's network.
925 questions
Windows Server PowerShell
Windows Server PowerShell
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.PowerShell: A family of Microsoft task automation and configuration management frameworks consisting of a command-line shell and associated scripting language.
4,609 questions
No comments
{count} votes

Accepted answer
  1. answered 2022-08-28T15:27:43.477+00:00
    Rich Matheisen 34,791 Reputation points

    If you're working with very large data sets, the "Select-Object" probably isn't the best way to get the information. That creates a new PSCustomObject that's passed into the pipeline and that object type, while convenient, takes a while to create and it's actually pretty big (in terms of memory).

    This code does essentially the same thing but uses a Here-String. Your individual JSON records are simple enough to make this pretty easy without creating any additional intermediate objects:

    Get-DnsServerResourceRecord -ZoneName w2k12.local |  
        ForEach-Object{  
    @"  
    {  
        "RecordType":  "$($_.RecordType)",  
        "HostName":  "$($_.HostName)",  
        "RecordData":  "$($_.RecordData)"  
    }  
    "@          
        }  
    
    No comments

2 additional answers

Sort by: Most helpful
  1. answered 2022-08-23T11:37:53.96+00:00
    Gary Reynolds 8,801 Reputation points

    Hi @Aditya G

    I don't have a large enough environment to test it, but could try this approach which adds the json text for each record to the output file one at a time, rather than trying to hold the json text in memory until all the records have been returned.

    foreach ($t in Get-DnsServerResourceRecord -ZoneName w2k12.local) { $t | convertto-json  |  out-file -file "json.test1.txt" -append}  
    

    Gary.


  2. answered 2022-08-28T12:47:09.597+00:00
    Aditya G 41 Reputation points

    This is how I solved the issue.

    The ConvertTo-Json does not output the data it receives from the stream, until it is done processing all the objects.

    Solution: Send each object one by one instead of sending all objects at once.

    Get-DnsServerResourceRecord -ZoneName Corp.microsoft.com|Select-Object -Property "RecordType","HostName","RecordData"|Foreach-object {$_ |ConvertTo-Json}

    Note: Select-Object is optional depending on your needs.

    In case of large data, better to redirect the result to a file.

    No comments