[Azure-Table-Storage-Access-RestAPI-via-PowerShell] - Invoke-RestMethod : Bytes to be written to the stream exceed the Content-Length bytes size specified

tn-57-gs 26 Reputation points
2021-04-26T11:47:14.29+00:00

Target: Fetch a list of installed applications from multiple computers to Azure Table Storage using Invoke-RestMethod.

I am trying to collect information from the computer and put it in a batch lets say 100 rows and then insert it into the table.

Prepared the script based on the https://learn.microsoft.com/en-us/rest/api/storageservices/performing-entity-group-transactions this article.

Error: "Invoke-RestMethod: Bytes to be written to the stream exceed the Content-Length bytes size specified"

Here is the code below

$tableEndpoint = 'https://****.table.core.windows.net/$batch'  
  
  
$tableName = 'InstalledApplications'  
$StorageAccount = "********"  
  
$URI = $tableEndpoint + $tableName + $SAS  
  
$cot =@()  
  
  
$array1 = Get-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\* | Select-Object DisplayName, DisplayVersion, Publisher, InstallDate -unique  
  
foreach($arr in $array1){  
  
  
$RequestBody = ConvertTo-Json -InputObject @{  
        "TagetName"= $arr.DisplayName;  
        "Message"= $arr.DisplayVersion;  
        "ComputerName"= $ENV:ComputerName;  
        "Username"= $ENV:Username;  
        "BatchName" = "_a1e9d677-b28b-435e-a89e-87e6a768a431";  
        "Content-Type" = "multipart/mixed; boundary=changeset_8a28b620-b4bb-458c-a177-0959fb14c977"  
        "PartitionKey"= "$ENV:ComputerName";  
        "RowKey"= $arr.DisplayName  
        "TableName" = "InstalledApplication"  
    }  
    $cot += $RequestBody  
}  
  
$batching = $cot[1..10]  
  
$RequestHeaders = @{  
    "x-ms-date"=(Get-Date -Format r);  
    "x-ms-version"="2009-09-19";  
    "Accept-Charset"="UTF-8";  
    "DataServiceVersion"="3.0;NetFx";  
    "MaxDataServiceVersion"="3.0;NetFx";  
    "Content-Type" = "multipart/mixed; boundary=batch_a1e9d677-b28b-435e-a89e-87e6a768a431"  
    "Content-Length" =  "1323"  
    "Authorization"  = "SharedKeyLite " + $StorageAccount + ":" + $SAS  
}  
  
Invoke-RestMethod -Method POST -Uri $tableEndpoint -Headers $RequestHeaders -Body $cot -ContentType "multipart/mixed; boundary=changeset_8a28b620-b4bb-458c-a177-0959fb14c977"  

Please let me know why content length exceeds although the returned rows are no more than 60.

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
1,549 questions
{count} votes

3 answers

Sort by: Most helpful
  1. shiva patpi 9,171 Reputation points Microsoft Employee
    2021-04-27T01:03:38.01+00:00

    Hello @SujithkumarGSuriyamurthy-7096 ,
    Thanks for your query ! I just tried your previous version of the script and was able to insert the required data to the azure table storage.
    I was able to repro your issue on your previous version by explicitly mentioning the "Content-Length" to some x value in the headers.

    • What is the reason you want to explicitly mention the content length ?
    • Did you try removing that property ?
    • If the Content-Length is not hardcoded , how are you calculating the content Length dynamically?

    Basically , content length implicitly also includes the type of encoding which we use . For example , if the actual data is 5 bytes - then over all content-length can be more than 5 bytes depending upon the encoding used. So while the data is being pushed using REST API when it goes through transit - it implicitly uses some sort of encoding - you might want to think over about that.

    ASCIIEncoding encoding = new ASCIIEncoding ();
    byte[] samplebyte= encoding.GetBytes (sampledata);

    For example: For the below string , if you try to get a normal length - it will just output 1 . But if you try to get the length through some encoding in terms of bytes those will return different outputs (2 & 1)

    var s = "é"
    var s = "e";

    Kindly let us know if you need additional help.

    No comments

  2. tn-57-gs 26 Reputation points
    2021-04-27T07:08:43.223+00:00

    Thanks for your response to my query!

    the previous version works well for me still the problem is when you run the PS command to get a list of applications it returns 90 rows, what happens is, it calls the API 90 times since the invoke-rest method resides inside the for loop. let's suppose if I use the same script on 8000 computers then you can imagine what will be the transactional cost at the end. correct me if I am wrong here, as per my understanding 1 transaction = $0.0004.

    This is the reason why I am trying to use batch insert operation.

    Initially, I was encoding the data and copy it into a variable, and pass the same variable in the header where it says "content-length = $variable. length". this way was also giving me the same error.

    As per your suggestion, today, I tried removing the "content-length" still no luck.

    therefore, please assist me if there any misconstruction in the above script.

    No comments

  3. tn-57-gs 26 Reputation points
    2021-05-02T17:37:46.737+00:00

    As per this article https://learn.microsoft.com/en-us/rest/api/storageservices/performing-entity-group-transactions#table-service-support-for-odata-batch-requests,

    I tried to insert a batch with two entities using postman but I get an error "one of the request inputs is not valid". please check out the body below I used for testing.

    --batch_f351702c-c8c8-48c6-af2c-91b809c651c4    
    Content-Type: multipart/mixed; boundary=changeset_8a28b620-b4bb-458c-a177-0959fb14c977    
        
    --changeset_8a28b620-b4bb-458c-a177-0959fb14c977    
    Content-Type: application/http    
    Content-Transfer-Encoding: binary    
        
    POST https://*****.table.core.windows.net/TableName HTTP/1.1  
    Content-Type: application/json    
    Accept: application/json;odata=minimalmetadata    
    Prefer: return-no-content    
    DataServiceVersion: 3.0;    
        
    {"PartitionKey":"Channel_19", "RowKey":"1", "Rating":9, "Text":".NET..."}    
    --changeset_8a28b620-b4bb-458c-a177-0959fb14c977    
    Content-Type: application/http    
    Content-Transfer-Encoding: binary    
        
    POST https://*****.table.core.windows.net/TableName HTTP/1.1  
    Content-Type: application/json    
    Accept: application/json;odata=minimalmetadata    
    Prefer: return-no-content    
    DataServiceVersion: 3.0;    
        
    {"PartitionKey":"Channel_19", "RowKey":"2", "Rating":9, "Text":"Azure..."}    
    --changeset_8a28b620-b4bb-458c-a177-0959fb14c977    
    Content-Type: application/http    
    Content-Transfer-Encoding: binary    
      
    --changeset_8a28b620-b4bb-458c-a177-0959fb14c977--    
    --batch_f351702c-c8c8-48c6-af2c-91b809c651c4    
    

    please assist me with the above JSON body construct.