Could not find object: linkedService.SupportDatabase

Question

Could not find object: linkedService.SupportDatabase

MiKeZZa 21

Hi all,

I'm not very experienced with PowerShell. But I've downloaded a solution which calls a .ps1 file. I've changed some small things to let it work with my local paths. But now it containts this:

$AdfPath = "C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\public"
$opt.CreateNewInstance = $true
$opt.DeleteNotInSource = $false
$opt.StopStartTriggers = $true
Publish-AdfV2FromJson -RootFolder $AdfPath `
    -ResourceGroupName $resourceGroupName `
    -DataFactoryName $dataFactoryName `
    -Location $region `
    -Option $opt `
    -Stage $configFilePath

And the output is this error:

Update-PropertiesFromFile : Could not find object: linkedService.SupportDatabase
At C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\public\Publish-AdfV2FromJson.ps1:160 char:9
+         Update-PropertiesFromFile -adf $adf -stage $Stage -option $op ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Update-PropertiesFromFile

The property 'Body' cannot be found on this object. Verify that the property exists.
At C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\private\Update-PropertiesFromFile.ps1:68 char:9
+         $json = $o.Body
+         ~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], PropertyNotFoundException
    + FullyQualifiedErrorId : PropertyNotFoundStrict

The variable '$json' cannot be retrieved because it has not been set.
At C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\private\Update-PropertiesFromFile.ps1:69 char:23
+         if ($null -eq $json) {
+                       ~~~~~
    + CategoryInfo          : InvalidOperation: (json:String) [], RuntimeException
    + FullyQualifiedErrorId : VariableIsUndefined

Update-PropertiesFromFile : Wrong path defined in config for object(path): linkedService.SupportDatabase(properties.typeProperties.connectionString.secretName)
At C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\public\Publish-AdfV2FromJson.ps1:160 char:9
+         Update-PropertiesFromFile -adf $adf -stage $Stage -option $op ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], DataException
    + FullyQualifiedErrorId : System.Data.DataException,Update-PropertiesFromFile
The variable '$json' cannot be retrieved because it has not been set.
At C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\private\Update-PropertiesFromFile.ps1:88 char:44
+                 Update-ObjectProperty -obj $json -path "properties.$p ...
+                                            ~~~~~
    + CategoryInfo          : InvalidOperation: (json:String) [], RuntimeException
    + FullyQualifiedErrorId : VariableIsUndefined

Save-AdfObjectAsFile : Cannot bind argument to parameter 'obj' because it is null.
At C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\private\Update-PropertiesFromFile.ps1:104 char:41
+         $f = (Save-AdfObjectAsFile -obj $o)
+                                         ~~
    + CategoryInfo          : InvalidData: (:) [Save-AdfObjectAsFile], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Save-AdfObjectAsFile

The variable '$f' cannot be retrieved because it has not been set.
At C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\private\Update-PropertiesFromFile.ps1:105 char:23
+         $o.FileName = $f
+                       ~~
    + CategoryInfo          : InvalidOperation: (f:String) [], RuntimeException
    + FullyQualifiedErrorId : VariableIsUndefined

I've checked C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\public\Publish-AdfV2FromJson.ps1 and it contains. Line 160 contains:

Update-PropertiesFromFile -adf $adf -stage $Stage -option $opt

Is there anybody who can help me? What is going wrong here?

Rich Matheisen 47,901 Reputation points

2020-11-01T14:51:48.75+00:00

Where is $opt defined? What does it contain at the time you use it as a parameter value in the call to PublishADFV2FromJson?

MiKeZZa 21

The code that fills #opt is situated just in front of the first code part of my opening posts:

$opt = New-AdfPublishOption
$deploymentObject = (Get-Content $deploymentFilePath) | ConvertFrom-Json 
$objectsToInclude = $deploymentObject.datasets + $deploymentObject.linkedServices + $deploymentObject.pipelines + $deploymentObject.triggers
$objectsToInclude | ForEach-Object { 
    $objName = $_.substring(1).Replace('.json', '').Replace('/', '.') 
    $opt.Includes.Add($objName, "")
}

# Deployment of ADF
$AdfPath = "C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\public"
$opt.CreateNewInstance = $true
$opt.DeleteNotInSource = $false
$opt.StopStartTriggers = $true

That's all.

3 answers

Your answer

Rich Matheisen 47,901 Reputation points

2020-11-01T14:51:48.75+00:00

Where is $opt defined? What does it contain at the time you use it as a parameter value in the call to PublishADFV2FromJson?
MiKeZZa 21 Reputation points

2020-11-01T15:36:32.467+00:00

The code that fills #opt is situated just in front of the first code part of my opening posts:

$opt = New-AdfPublishOption $deploymentObject = (Get-Content $deploymentFilePath) | ConvertFrom-Json $objectsToInclude = $deploymentObject.datasets + $deploymentObject.linkedServices + $deploymentObject.pipelines + $deploymentObject.triggers $objectsToInclude | ForEach-Object { $objName = $_.substring(1).Replace('.json', '').Replace('/', '.') $opt.Includes.Add($objName, "") } # Deployment of ADF $AdfPath = "C:\Program Files\WindowsPowerShell\Modules\azure.datafactory.tools\0.17.0\public" $opt.CreateNewInstance = $true $opt.DeleteNotInSource = $false $opt.StopStartTriggers = $true

That's all.

Answer 1

There's an awful lot to unpack in your problem. I don't have the code from the module you're using, but from the exception error the function you call in turn calls another function (at line 160 in Publish-AdfV2FromJson) named Update-PropertiesFromFile. That function has encountered WriteErrorException, but there's no indication as to what file it was trying to write to, or what the reason was for the exception (e.g., was it permission problem? was it trying to write to a read-only file? or was it something else?).

Another problem is that I don't understand what you're doing when you load the $objectsToInclude variable because I don't know what sort of objects $deploymentObject.datasets, etc. are. Are they arrays, or are the strings? If they're arrays then it's okay. OTOH if they're strings you're concatenating them.

When you're removing the string ".json" and turning forward slashes into periods, why are you starting at the second position of the string? Also, the "." in ".json" represents ANY character, not a period. Use "." instead. It probably isn't hurting anything the way it is, but it's a poor habit to acquire!

The remainder of the exceptions are likely the result of the script continuing to run after encountering what should have been treated as a terminating error by the module's author.

Answer 2

MiKeZZa 21

Hi @Rich Matheisen ,

It's happening when running this script: https://github.com/mrpaulandrew/procfwk/blob/master/DeploymentTools/DataFactory/DeployProcFwkComponents.ps1. That's part of a ADF framework which I'm trying to get it working.

Maybe that this whole script makes more sense to you?

Rich Matheisen 47,901 Reputation points

2020-11-02T19:25:29.177+00:00

If the contents of your ProcFwkComponents.json file are correct you should be okay. The only unknown is the content of the environment variables the script (in the link you posted) uses.

But what you changed in that script isn't known to me.
MiKeZZa 21 Reputation points

2020-11-02T19:38:55.303+00:00

The content of my ProcFwkComponents.json is identically to what you can see on https://github.com/mrpaulandrew/procfwk/blob/master/DeploymentTools/DataFactory/ProcFwkComponents.json. That must be correct I guess.

What I've changed in the PS1 script is remove the first 14 lines and added this:

$tenantId = "mytenantid"
$subscriptionId = "myscubscriptionid"
$spId = "azureclientid"
$spKey = "azureclientsecret"
$resourceGroupName = "myresourcename"
$dataFactoryName = "myadfname"
$region = "West Europe" (but also tried "WestEurope" or "West-Europe")

Rich Matheisen 47,901

Okay, but what about "SQLDatabase" environment variable????

Right before your script's "$opt = New-AdfPublishOption", there's this (note the last line!):

# Get Deployment Objects and Params files
$scriptPath = Join-Path -Path (Get-Location) -ChildPath "\DeploymentTools\DataFactory"
$deploymentFilePath = Join-Path -Path $scriptPath -ChildPath "ProcFwkComponents.json"
$configFilePath = Join-Path -Path $scriptPath -ChildPath "config-all.csv"
$Env:SQLDatabase = "<secretKeyToDbConnectionString>"

MiKeZZa 21

It seems to be a variable that must also be filled, but is located at a strange place... I have filled this with this:

Server=tcp:adfframework.database.windows.net,1433;Initial Catalog=frameworkmetadatadb;Persist Security Info=False;User ID=MYUSERNAME;Password=MYPASSWORD;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;

But that makes no difference.

Rich Matheisen 47,901 Reputation points

2020-11-03T20:26:50.673+00:00

I have no idea where the environment variable "SQLDatabase" is used, but its name wouldn't seem to imply that it's purpose is to hold the entire connection string. If I had to guess it's supposed to contain "key", but what the heck a "key" is I have no idea! A password? In plain-text? As a secure string?

I think there's nothing more I can do for you since this no longer seems to be a PowerShell problem.

Answer 3

Kamil Nowinski 1 MVP

@MiKeZZa , the first bug I can see in your code is in the first line.
$AdfPath shouldn't point to PowerShell module but onto Azure Data Factory folder with all subfolders with JSON files.
Could you create a new issue on GitHub repo of azure.datafactory.tools?
Then, I believe we can find out the root cause of your issue pretty quickly...

Share via

Could not find object: linkedService.SupportDatabase

3 answers

Your answer