ADF Data Flows: Could not load resource. Please ensure no mistakes in the JSON and that referenced resources exist.

STAKV 36 Reputation points
2021-01-18T09:09:25.99+00:00

I have created a pipeline with data flow activity inside. The activity is used to flatten json file with complex hierarchical structure. It was working fine for a month and then suddenly the code got invalid. The error message is "Could not load resource. Please ensure no mistakes in the JSON and that referenced resources exist." I found an article in:

https://social.msdn.microsoft.com/Forums/en-US/d630eeb9-6881-436b-aecd-4e5882f0abce/could-not-load-resource-please-ensure-no-mistakes-in-the-json-and-that-referenced-resources-exist?forum=AzureDataFactory

and it turns out that this had been a production issue back in 2019. Other similar pipelines have also crashed at the same time. I have tried to build them from scratch and managed to publish but then they crashed again. I always use the graphical interface. I have checked all the referenced resources and they are fine.
I am pasting the JSON code for reference.

{
"id": "/subscriptions/5323a6fa-3843-4a66-a555-51972ecf84ea/resourceGroups/MIT-B2B-EDI-Analytics-DEV/providers/Microsoft.Synapse/workspaces/zncswb2bedianalyticsdev/dataflows/df_stg_json_to_synapse_seedelfor",
"name": "df_stg_json_to_synapse_seedelfor",
"type": "Microsoft.Synapse/workspaces/dataflows",
"etag": "9e004ef8-0000-0400-0000-5ffec4d70000",
"properties": {
"type": "MappingDataFlow",
"typeProperties": {
"sources": [
{
"dataset": {
"referenceName": "ds_edilanding_json",
"type": "DatasetReference"
},
"name": "JSONSeedelfor"
}
],
"sinks": [
{
"dataset": {
"referenceName": "ds_stg_edidocumentheaderseedelfor",
"type": "DatasetReference"
},
"name": "EDIDocumentHeaderSeedelfor"
},
{
"dataset": {
"referenceName": "ds_stg_itemseedelfor",
"type": "DatasetReference"
},
"name": "ItemSeedelfor"
},
{
"dataset": {
"referenceName": "ds_stg_grpseedelfor",
"type": "DatasetReference"
},
"name": "Grp"
},
{
"dataset": {
"referenceName": "ds_stg_msgseedelfor",
"type": "DatasetReference"
},
"name": "MsgSeedelfor"
},
{
"dataset": {
"referenceName": "ds_stg_msgtextseedelfor",
"type": "DatasetReference"
},
"name": "MsgTextSeedelfor"
}
],
"transformations": [
{
"name": "FlattenHeader"
},
{
"name": "PivotProp"
},
{
"name": "MapDrifted",
"description": "Creates an explicit mapping for each drifted column"
},
{
"name": "FlattenItem"
},
{
"name": "CastItem"
},
{
"name": "FlattenGrp"
},
{
"name": "FlattenMsgSeedelfor"
},
{
"name": "CastMsgSeedelfor"
},
{
"name": "CastHeader"
},
{
"name": "CastGrp"
},
{
"name": "SelectMsgTxt"
}
],
"script": "source(output(\n\t\t{_id} as ({$oid} as string),\n\t\tprop as (k as string, v as string)[],\n\t\tseeag as (copyright as string, creationdate as string, message as string, mapping as string, version as string),\n\t\tcnt as (ic as (transmissiondate as string, transmissiontime as string, interchangecontrolnumber as string, sender as string, senderqualifier as string, reverseroutingaddress as string, receiver as string, receiverqualifier as string, routingaddress as string, testflag as string, receiverpassword as string, receiverpasswordqualifier as string, responserequired as string, applicationreference as string, processprio as string, communicationagreement as string, syntaxidentifier as string, syntaxversion as string, additionalinfo as string, ref as string), filterset as (key01 as string, key02 as string, key03 as string, key04 as string, key05 as string, key06 as string, key07 as string, key08 as string, key09 as string, key10 as string)[], grp as (fg as (identifier as string, sender as string, senderqualifier as string, receiver as string, receiverqualifier as string, transmissiondate as string, transmissiontime as string, groupcontrolnumber as string, agency as string, messageversionnumber as string, associationassignedcode as string, applicationpasswort as string, messagereleasenumber as string, additionalinfo as string, ref as string), msg as (me as (messagenumber as string, messagetype as string, messagetypeversion as string, messagetyperelease as string, controllingagency as string, assocassignedcode as string, commonaccessreference as string, sequenceoftransfer as string, firstandlasttransfer as string, additionalinfo as string), add as (qualifier as string, identifier as string, agencycode as string, name1 as string, name2 as string, name3 as string, name4 as string, name5 as string, partyname1 as string, partyname2 as string, partyname3 as string, partyname4 as string, partyname5 as string, street1 as string, street2 as string, street3 as string, street4 as string, pobox as string, city as string, state as string, postalcode as string, countrycode as string, country as string, additionaldata as string, contact as string, accountinformation as string, txt as string)[], ref as (qualifier as string, reference as string, referencedate as string, referencetime as string, additionalinfo as string)[], weightandvolume as string, qty as string, txt as (qualifier as string, languagecoded as string, textline as string[])[], date as (qualifier as string, date as string, time as string, additionalinfo as string)[], currency as (qualifier as string, currency as string, additionalqualifier as string, currencyratebase as string, rateofexchange as string, currencymarketexchange as string)[], tax as string, termsofpayment as (qualifier as string, timereferencecoded as string, timerelationcoded as string, periodtype as string, numberofperiods as string, percentage as string, date as string, txt as (qualifier as string, languagecoded as string, textline as string[])[])[], allowancecharge as (qualifier as string, signallowancecharge as string, conditionkey as string, conditiontext as string, conditionpercentage as string, conditionrate as string, amountdetails as (qualifier as string, amount as string, currency as string, priceunit as string, measureunitqualifier as string)[], txt as (qualifier as string, languagecoded as string, textline as string[])[])[], amountdetails as string, transportdetails as (transportnumber as string, modeoftransport as string, additionaldetails as (qualifier as string, details as string)[], date as string), seeitem as (itempos as (lineitemnumber as string, linesubitemnumber as string, countryoforigin as string, itemdescription as string[]), itemnumber as (qualifier as string, itemnumber as string, additionalinfo as string)[], qty as (qualifier as string, quantity as string, quantityunit as string)[], amountdetails as (qualifier as string, amount as string, currency as string, priceunit as string, measureunitqualifier as string)[], ref as (qualifier as string, reference as string, referencedate as string, referencetime as string, additionalinfo as string)[], date as (qualifier as string, date as string, time as string, additionalinfo as string)[], add as (qualifier as string, identifier as string, agencycode as string, name1 as string, name2 as string, name3 as string, name4 as string, name5 as string, partyname1 as string, partyname2 as string, partyname3 as string, partyname4 as string, partyname5 as string, street1 as string, street2 as string, street3 as string, street4 as string, pobox as string, city as string, state as string, postalcode as string, countrycode as string, country as string, additionaldata as string, contact as string, accountinformation as string, txt as string)[], weightandvolume as string, txt as (qualifier as string, languagecoded as string, textline as string[])[], packagedetails as string, tax as string, allowancecharge as string, qtydateref as string, schedule as (startdate as string, starttime as string, enddate as string, endtime as string, quantity as string, cumquantity as string, quantityunit as string, forecastqualifier as string, timingqualifier as string, numberofworkingdays as string, originaldatetime as string, originaldatetimecode as string, timeperiodcoded as string, deliveryrequirementscoded as string, despatchpatterntimingcoded as string, ref as string, date as string, txt as string)[])[])[])[])[]\n\t),\n\tallowSchemaDrift: false,\n\tvalidateSchema: false,\n\tignoreNoFilesFound: false,\n\trowUrlColumn: 'File_Name',\n\tdocumentForm: 'documentPerLine',\n\tbackslashEscape: true,\n\twildcardPaths:['data3/set01/seexml_3_000*.json']) ~> JSONSeedelfor\nJSONSeedelfor foldDown(unroll(prop),\n\tmapColumn(\n\t\tfile_name = File_Name,\n\t\tprop_k = prop.k,\n\t\tprop_v = prop.v,\n\t\tseeag,\n\t\tcnt\n\t),\n\tskipDuplicateMapInputs: false,\n\tskipDuplicateMapOutputs: false) ~> FlattenHeader\nFlattenHeader pivot(groupBy(file_name,\n\t\tseeag_copyright = toString(seeag.copyright),\n\t\tseeag_creationdate = toString(seeag.creationdate),\n\t\tseeag_message = toString(seeag.message),\n\t\tseeag_mapping = toString(seeag.mapping),\n\t\tseeag_version = toString(seeag.version),\n\t\tic_transmissiondate = replace(replace(replace(toString(cnt.ic.transmissiondate),'[',''),']',''),'\"',''),\n\t\tic_transmissiontime = replace(replace(replace(toString(cnt.ic.transmissiontime),'[',''),']',''),'\"',''),\n\t\tic_interchangecontrolnumber = replace(replace(replace(toString(cnt.ic.interchangecontrolnumber),'[',''),']',''),'\"',''),\n\t\tic_sender = replace(replace(replace(toString(cnt.ic.sender),'[',''),']',''),'\"',''),\n\t\tic_senderqualifier = replace(replace(replace(toString(cnt.ic.senderqualifier),'[',''),']',''),'\"',''),\n\t\tic_reverseroutingaddress = replace(replace(replace(toString(cnt.ic.reverseroutingaddress),'[',''),']',''),'\"',''),\n\t\tic_receiver = replace(replace(replace(toString(cnt.ic.receiver),'[',''),']',''),'\"',''),\n\t\tic_receiverqualifier = replace(replace(replace(toString(cnt.ic.receiverqualifier),'[',''),']',''),'\"',''),\n\t\tic_routingaddress = replace(replace(replace(toString(cnt.ic.routingaddress),'[',''),']',''),'\"',''),\n\t\tic_testflag = replace(replace(replace(toString(cnt.ic.testflag),'[',''),']',''),'\"',''),\n\t\tic_receiverpassword = replace(replace(replace(toString(cnt.ic.receiverpassword),'[',''),']',''),'\"',''),\n\t\tic_receiverpasswordqualifier = replace(replace(replace(toString(cnt.ic.receiverpasswordqualifier),'[',''),']',''),'\"',''),\n\t\tic_responserequired = replace(replace(replace(toString(cnt.ic.responserequired),'[',''),']',''),'\"',''),\n\t\tic_applicationreference = replace(replace(replace(toString(cnt.ic.applicationreference),'[',''),']',''),'\"',''),\n\t\tic_processprio = replace(replace(replace(toString(cnt.ic.processprio),'[',''),']',''),'\"',''),\n\t\tic_communicationagreement = replace(replace(replace(toString(cnt.ic.communicationagreement),'[',''),']',''),'\"',''),\n\t\tic_syntaxidentifier = replace(replace(replace(toString(cnt.ic.syntaxidentifier),'[',''),']',''),'\"',''),\n\t\tic_syntaxversion = replace(replace(replace(toString(cnt.ic.syntaxversion),'[',''),']',''),'\"',''),\n\t\tic_additionalinfo = replace(replace(replace(toString(cnt.ic.additionalinfo),'[',''),']',''),'\"',''),\n\t\tic_ref = toString(cnt.ic.ref),\n\t\tic_filterset = toString(cnt.filterset)),\n\tpivotBy(prop_k),\n\tprop_ = max(prop_v),\n\tcolumnNaming: '$N$V',\n\tlateral: true) ~> PivotProp\nPivotProp derive(file_name = file_name,\n\t\tprop_mtprefix = toString(byName('prop_MT_PREFIX')),\n\t\tprop_trackid = toString(byName('prop_TRACKID')),\n\t\tprop_adapter = toString(byName('prop_ADAPTER')),\n\t\tprop_magnaplant = toString(byName('prop_MAGNAPLANT')),\n\t\tprop_partner = toString(byName('prop_PARTNER')),\n\t\tprop_docgroup = toString(byName('prop_DOCGROUP')),\n\t\tprop_doctype = toString(byName('prop_DOCTYPE')),\n\t\tprop_direction = toString(byName('prop_DIRECTION')),\n\t\tprop_starttime = toString(byName('prop_START_TIME'))) ~> MapDrifted\nJSONSeedelfor foldDown(unroll(cnt.grp.msg.seeitem),\n\tmapColumn(\n\t\tfile_name = File_Name,\n\t\tme_messagenumber = cnt.grp.msg.me.messagenumber,\n\t\titempos_lineitemnumber = cnt.grp.msg.seeitem.itempos.lineitemnumber,\n\t\titempos_linesubitemnumber = cnt.grp.msg.seeitem.itempos.linesubitemnumber,\n\t\titempos_countryoforigin = cnt.grp.msg.seeitem.itempos.countryoforigin,\n\t\titempos_itemdescription = cnt.grp.msg.seeitem.itempos.itemdescription,\n\t\titem_itemnumber = cnt.grp.msg.seeitem.itemnumber,\n\t\titem_qty = cnt.grp.msg.seeitem.qty,\n\t\titem_amountdetails = cnt.grp.msg.seeitem.amountdetails,\n\t\titem_ref = cnt.grp.msg.seeitem.ref,\n\t\titem_date = cnt.grp.msg.seeitem.date,\n\t\titem_add = cnt.grp.msg.seeitem.add,\n\t\titem_weightandvolume = cnt.grp.msg.seeitem.weightandvolume,\n\t\titem_txt = cnt.grp.msg.seeitem.txt,\n\t\titem_packagedetails = cnt.grp.msg.seeitem.packagedetails,\n\t\titem_tax = cnt.grp.msg.seeitem.tax,\n\t\titem_allowancecharge = cnt.grp.msg.seeitem.allowancecharge,\n\t\titem_qtydateref = cnt.grp.msg.seeitem.qtydateref,\n\t\titem_schedule = cnt.grp.msg.seeitem.schedule\n\t),\n\tskipDuplicateMapInputs: false,\n\tskipDuplicateMapOutputs: false) ~> FlattenItem\nFlattenItem derive(itempos_itemdescription = toString(array(itempos_itemdescription)),\n\t\titem_itemnumber = toString(array(item_itemnumber)),\n\t\titem_qty = toString(array(item_qty)),\n\t\titem_amountdetails = toString(array(item_amountdetails)),\n\t\titem_ref = toString(array(item_ref)),\n\t\titem_date = toString(array(item_date)),\n\t\titem_add = toString(array(item_add)),\n\t\titem_txt = toString(array(item_txt)),\n\t\titem_schedule = toString(array(item_schedule))) ~> CastItem\nJSONSeedelfor foldDown(unroll(cnt.grp),\n\tmapColumn(\n\t\tfile_name = File_Name,\n\t\tgrp_fg = cnt.grp.fg\n\t),\n\tskipDuplicateMapInputs: false,\n\tskipDuplicateMapOutputs: false) ~> FlattenGrp\nJSONSeedelfor foldDown(unroll(cnt.grp.msg),\n\tmapColumn(\n\t\tfile_name = File_Name,\n\t\tme_messagenumber = cnt.grp.msg.me.messagenumber,\n\t\tme_messagetype = cnt.grp.msg.me.messagetype,\n\t\tme_messagetypeversion = cnt.grp.msg.me.messagetypeversion,\n\t\tme_messagetyperelease = cnt.grp.msg.me.messagetyperelease,\n\t\tme_controllingagency = cnt.grp.msg.me.controllingagency,\n\t\tme_assocassignedcode = cnt.grp.msg.me.assocassignedcode,\n\t\tme_commonaccessreference = cnt.grp.msg.me.commonaccessreference,\n\t\tme_sequenceoftransfer = cnt.grp.msg.me.sequenceoftransfer,\n\t\tme_firstandlasttransfer = cnt.grp.msg.me.firstandlasttransfer,\n\t\tme_additionalinfo = cnt.grp.msg.me.additionalinfo,\n\t\tmsg_add = cnt.grp.msg.add,\n\t\tmsg_ref = cnt.grp.msg.ref,\n\t\tmsg_weightandvolume = cnt.grp.msg.weightandvolume,\n\t\tmsg_qty = cnt.grp.msg.qty,\n\t\tmsg_txt = cnt.grp.msg.txt,\n\t\tmsg_date = cnt.grp.msg.date,\n\t\tmsg_currency = cnt.grp.msg.currency,\n\t\tmsg_tax = cnt.grp.msg.tax,\n\t\tmsg_termsofpayment = cnt.grp.msg.termsofpayment,\n\t\tmsg_allowancecharge = cnt.grp.msg.allowancecharge,\n\t\tmsg_amountdetails = cnt.grp.msg.amountdetails,\n\t\tmsg_transportdetails_transportnumber = cnt.grp.msg.transportdetails.transportnumber,\n\t\tmsg_transportdetails_modeoftransport = cnt.grp.msg.transportdetails.modeoftransport,\n\t\tmsg_transportdetails_additionaldetails = cnt.grp.msg.transportdetails.additionaldetails,\n\t\tmsg_transportdetails_date = cnt.grp.msg.transportdetails.date\n\t),\n\tskipDuplicateMapInputs: false,\n\tskipDuplicateMapOutputs: false) ~> FlattenMsgSeedelfor\nFlattenMsgSeedelfor derive(msg_add = toString(array(msg_add)),\n\t\tmsg_ref = toString(array(msg_ref)),\n\t\tmsg_weightandvolume = toString(array(msg_weightandvolume)),\n\t\tmsg_qty = toString(array(msg_qty)),\n\t\tmsg_txt = toString(array(msg_txt)),\n\t\tmsg_date = toString(array(msg_date)),\n\t\tmsg_currency = toString(array(msg_currency)),\n\t\tmsg_tax = toString(array(msg_tax)),\n\t\tmsg_termsofpayment = toString(array(msg_termsofpayment)),\n\t\tmsg_allowancecharge = toString(array(msg_allowancecharge)),\n\t\tmsg_amountdetails = toString(array(msg_amountdetails)),\n\t\tmsg_transportdetails_additionaldetails = toString(array(msg_transportdetails_additionaldetails))) ~> CastMsgSeedelfor\nMapDrifted derive(ic_ref = toString(array(ic_ref)),\n\t\tic_filterset = toString(array(ic_filterset))) ~> CastHeader\nFlattenGrp derive(grp_fg = toString(array(grp_fg))) ~> CastGrp\nCastMsgSeedelfor select(mapColumn(\n\t\tfile_name,\n\t\tmsg_txt\n\t),\n\tskipDuplicateMapInputs: true,\n\tskipDuplicateMapOutputs: true) ~> SelectMsgTxt\nCastHeader sink(input(\n\t\tFile_Name as string,\n\t\tProp_MtPrefix as string,\n\t\tProp_Trackid as string,\n\t\tProp_Adapter as string,\n\t\tProp_Magnaplant as string,\n\t\tProp_Partner as string,\n\t\tProp_Docgroup as string,\n\t\tProp_Doctype as string,\n\t\tProp_Direction as string,\n\t\tProp_StartTime as string,\n\t\tSeeag_Copyright as string,\n\t\tSeeag_CreationDate as string,\n\t\tSeeag_Message as string,\n\t\tSeeag_Mapping as string,\n\t\tSeeag_Version as string,\n\t\tIc_TransmissionDate as string,\n\t\tIc_TransmissionTime as string,\n\t\tIc_InterchangeControlNumber as string,\n\t\tIc_Sender as string,\n\t\tIc_SenderQualifier as string,\n\t\tIc_ReverseRoutingAddress as string,\n\t\tIc_Receiver as string,\n\t\tIc_ReceiverQualifier as string,\n\t\tIc_RoutingAddress as string,\n\t\tIc_Testflag as string,\n\t\tIc_ReceiverPassword as string,\n\t\tIc_ReceiverPasswordQualifier as string,\n\t\tIc_ResponseRequired as string,\n\t\tIc_ApplicationReference as string,\n\t\tIc_ProcessPrio as string,\n\t\tIc_CommunicationAgreement as string,\n\t\tIc_SyntaxIdentifier as string,\n\t\tIc_SyntaxVersion as string,\n\t\tIc_AdditionalInfo as string,\n\t\tIc_Ref as string,\n\t\tIc_FilterSet as string\n\t),\n\tallowSchemaDrift: false,\n\tvalidateSchema: false,\n\tdeletable:false,\n\tinsertable:true,\n\tupdateable:false,\n\tupsertable:false,\n\ttruncate:true,\n\tformat: 'table',\n\tstaged: true,\n\tallowCopyCommand: true,\n\tmapColumn(\n\t\tFile_Name = file_name,\n\t\tProp_MtPrefix = prop_mtprefix,\n\t\tProp_Trackid = prop_trackid,\n\t\tProp_Adapter = prop_adapter,\n\t\tProp_Magnaplant = prop_magnaplant,\n\t\tProp_Partner = prop_partner,\n\t\tProp_Docgroup = prop_docgroup,\n\t\tProp_Doctype = prop_doctype,\n\t\tProp_Direction = prop_direction,\n\t\tProp_StartTime = prop_starttime,\n\t\tSeeag_Copyright = seeag_copyright,\n\t\tSeeag_CreationDate = seeag_creationdate,\n\t\tSeeag_Message = seeag_message,\n\t\tSeeag_Mapping = seeag_mapping,\n\t\tSeeag_Version = seeag_version,\n\t\tIc_TransmissionDate = ic_transmissiondate,\n\t\tIc_TransmissionTime = ic_transmissiontime,\n\t\tIc_InterchangeControlNumber = ic_interchangecontrolnumber,\n\t\tIc_Sender = ic_sender,\n\t\tIc_SenderQualifier = ic_senderqualifier,\n\t\tIc_ReverseRoutingAddress = ic_reverseroutingaddress,\n\t\tIc_Receiver = ic_receiver,\n\t\tIc_ReceiverQualifier = ic_receiverqualifier,\n\t\tIc_RoutingAddress = ic_routingaddress,\n\t\tIc_Testflag = ic_testflag,\n\t\tIc_ReceiverPassword = ic_receiverpassword,\n\t\tIc_ReceiverPasswordQualifier = ic_receiverpasswordqualifier,\n\t\tIc_ResponseRequired = ic_responserequired,\n\t\tIc_ApplicationReference = ic_applicationreference,\n\t\tIc_ProcessPrio = ic_processprio,\n\t\tIc_CommunicationAgreement = ic_communicationagreement,\n\t\tIc_SyntaxIdentifier = ic_syntaxidentifier,\n\t\tIc_SyntaxVersion = ic_syntaxversion,\n\t\tIc_AdditionalInfo = ic_additionalinfo,\n\t\tIc_Ref = ic_ref,\n\t\tIc_FilterSet = ic_filterset\n\t),\n\tskipDuplicateMapInputs: true,\n\tskipDuplicateMapOutputs: true,\n\tsaveOrder: 0) ~> EDIDocumentHeaderSeedelfor\nCastItem sink(input(\n\t\tFile_Name as string,\n\t\tItemPos_LineItemNumber as string,\n\t\tItemPos_LineSubItemNumber as string,\n\t\tItemPos_CountryOfOrigin as string,\n\t\tItemPos_ItemDescription as string,\n\t\tItem_ItemNumber as string,\n\t\tItem_Qty as string,\n\t\tItem_AmountDetails as string,\n\t\tItem_Ref as string,\n\t\tItem_Date as string,\n\t\tItem_Add as string,\n\t\tItem_WeightAndVolume as string,\n\t\tItem_Txt as string,\n\t\tItem_PackageDetails as string,\n\t\tItem_Tax as string,\n\t\tItem_AllowanceCharge as string,\n\t\tItem_QtyDateRef as string,\n\t\tItem_Schedule as string\n\t),\n\tallowSchemaDrift: false,\n\tvalidateSchema: false,\n\tdeletable:false,\n\tinsertable:true,\n\tupdateable:false,\n\tupsertable:false,\n\ttruncate:true,\n\tformat: 'table',\n\tstaged: true,\n\tallowCopyCommand: true,\n\tmapColumn(\n\t\tFile_Name = file_name,\n\t\tItemPos_LineItemNumber = itempos_lineitemnumber,\n\t\tMe_MessageNumber = me_messagenumber,\n\t\tItemPos_LineSubItemNumber = itempos_linesubitemnumber,\n\t\tItemPos_CountryOfOrigin = itempos_countryoforigin,\n\t\tItemPos_ItemDescription = itempos_itemdescription,\n\t\tItem_ItemNumber = item_itemnumber,\n\t\tItem_Qty = item_qty,\n\t\tItem_AmountDetails = item_amountdetails,\n\t\tItem_Ref = item_ref,\n\t\tItem_Date = item_date,\n\t\tItem_Add = item_add,\n\t\tItem_WeightAndVolume = item_weightandvolume,\n\t\tItem_Txt = item_txt,\n\t\tItem_PackageDetails = item_packagedetails,\n\t\tItem_Tax = item_tax,\n\t\tItem_AllowanceCharge = item_allowancecharge,\n\t\tItem_QtyDateRef = item_qtydateref,\n\t\tItem_Schedule = item_schedule\n\t),\n\tskipDuplicateMapInputs: true,\n\tskipDuplicateMapOutputs: true,\n\tsaveOrder: 2) ~> ItemSeedelfor\nCastGrp sink(input(\n\t\tFile_Name as string,\n\t\tGrp_Fg as string\n\t),\n\tallowSchemaDrift: false,\n\tvalidateSchema: false,\n\tdeletable:false,\n\tinsertable:true,\n\tupdateable:false,\n\tupsertable:false,\n\ttruncate:true,\n\tformat: 'table',\n\tstaged: true,\n\tallowCopyCommand: true,\n\tmapColumn(\n\t\tFile_Name = file_name,\n\t\tGrp_Fg = grp_fg\n\t),\n\tskipDuplicateMapInputs: true,\n\tskipDuplicateMapOutputs: true,\n\tsaveOrder: 0) ~> Grp\nCastMsgSeedelfor sink(input(\n\t\tFile_Name as string,\n\t\tMe_MessageNumber as string,\n\t\tMe_MessageType as string,\n\t\tMe_MessageTypeVersion as string,\n\t\tMe_MessageTypeRelease as string,\n\t\tMe_ControllingAgency as string,\n\t\tMe_AssocAssignedCode as string,\n\t\tMe_CommonAccessReference as string,\n\t\tMe_SequenceOfTransfer as string,\n\t\tMe_FirstAndLastTransfer as string,\n\t\tMe_AdditionalInfo as string,\n\t\tMsg_Add as string,\n\t\tMsg_Ref as string,\n\t\tMsg_WeightAndVolume as string,\n\t\tMsg_Qty as string,\n\t\tMsg_Txt as string,\n\t\tMsg_Date as string,\n\t\tMsg_Currency as string,\n\t\tMsg_Tax as string,\n\t\tMsg_TermsOfPayment as string,\n\t\tMsg_AllowanceCharge as string,\n\t\tMsg_AmountDetails as string,\n\t\tMsg_TransportDetails_TransportNumber as string,\n\t\tMsg_TransportDetails_ModeOfTransport as string,\n\t\tMsg_TransportDetails_ShippingMode as string,\n\t\tMsg_TransportDetails_AdditionalDetails as string,\n\t\tMsg_TransportDetails_Date as string\n\t),\n\tallowSchemaDrift: false,\n\tvalidateSchema: false,\n\tdeletable:false,\n\tinsertable:true,\n\tupdateable:false,\n\tupsertable:false,\n\ttruncate:true,\n\tformat: 'table',\n\tstaged: true,\n\tallowCopyCommand: true,\n\tmapColumn(\n\t\tFile_Name = file_name,\n\t\tMe_MessageNumber = me_messagenumber,\n\t\tMe_MessageType = me_messagetype,\n\t\tMe_MessageTypeVersion = me_messagetypeversion,\n\t\tMe_MessageTypeRelease = me_messagetyperelease,\n\t\tMe_ControllingAgency = me_controllingagency,\n\t\tMe_AssocAssignedCode = me_assocassignedcode,\n\t\tMe_CommonAccessReference = me_commonaccessreference,\n\t\tMe_SequenceOfTransfer = me_sequenceoftransfer,\n\t\tMe_FirstAndLastTransfer = me_firstandlasttransfer,\n\t\tMe_AdditionalInfo = me_additionalinfo,\n\t\tMsg_Add = msg_add,\n\t\tMsg_Ref = msg_ref,\n\t\tMsg_WeightAndVolume = msg_weightandvolume,\n\t\tMsg_Qty = msg_qty,\n\t\tMsg_Txt = msg_txt,\n\t\tMsg_Date = msg_date,\n\t\tMsg_Currency = msg_currency,\n\t\tMsg_Tax = msg_tax,\n\t\tMsg_TermsOfPayment = msg_termsofpayment,\n\t\tMsg_AllowanceCharge = msg_allowancecharge,\n\t\tMsg_AmountDetails = msg_amountdetails,\n\t\tMsg_TransportDetails_TransportNumber = msg_transportdetails_transportnumber,\n\t\tMsg_TransportDetails_ModeOfTransport = msg_transportdetails_modeoftransport,\n\t\tMsg_TransportDetails_AdditionalDetails = msg_transportdetails_additionaldetails,\n\t\tMsg_TransportDetails_Date = msg_transportdetails_date\n\t),\n\tskipDuplicateMapInputs: true,\n\tskipDuplicateMapOutputs: true,\n\tsaveOrder: 1) ~> MsgSeedelfor\nSelectMsgTxt sink(input(\n\t\tFile_Name as string,\n\t\tMsg_Txt as string\n\t),\n\tallowSchemaDrift: false,\n\tvalidateSchema: false,\n\tdeletable:false,\n\tinsertable:true,\n\tupdateable:false,\n\tupsertable:false,\n\ttruncate:true,\n\tformat: 'table',\n\tstaged: true,\n\tallowCopyCommand: true,\n\tmapColumn(\n\t\tFile_Name = file_name,\n\t\tMsg_Txt = msg_txt\n\t),\n\tskipDuplicateMapInputs: true,\n\tskipDuplicateMapOutputs: true) ~> MsgTextSeedelfor"
}
}
}

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions
{count} vote

Accepted answer
  1. HarithaMaddi-MSFT 10,136 Reputation points
    2021-01-21T05:23:10.11+00:00

    Hi @STAKV ,

    Thanks for your patience. Product team observed an technical issue that caused this error across few subscriptions. They prioritized and deployed a hotfix yesterday and this issue should be fixed at your end now. Please let us know for further queries and we will be glad to assist.

    --

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.
    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful