Azure Data Factory Workflow stopped working in automatic mode when started via trigger

Question

Hi, My pipeline includes workflow process and running via trigger, then start every night. But the last two days ago, pipeline crashed. I am getting the following error:

"Operation on target DWH_FACT_GeneralLadger failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Column name used in expression is unavailable or invalid","Details":"at Select 'SelectFieldsGL'(Line 295/Col 11): Column 'SAPBKPF@MANDT' not found. The stream is either not connected or column is unavailable"}"

If I just run it not through a trigger everything works!!!!!
Could you help me and understand what the problem is?

Script:
source(output(
MANDT as integer,
BUKRS as string,
BELNR as string,
GJAHR as integer,
MONAT as string,
BUDAT as string,
WAERS as string,
KURST as string,
AEDAT as string,
GRPID as string,
KNUMV as string,
XBLNR as string,
BLART as string,
USNAM as string,
CPUTM as string,
BLDAT as string,
TCODE as string,
CPUDT as string,
UPDDT as string,
WWERT as string,
BVORG as string,
DBBLG as string,
STBLG as string,
STJAH as integer,
BKTXT as string,
KURSF as decimal(9,5),
KZWRS as string,
KZKRS as decimal(9,5),
BSTAT as string,
XNETB as string,
FRATH as decimal(13,5),
XRUEB as string,
GLVOR as string,
DOKID as string,
ARCID as string,
IBLAR as string,
AWTYP as string,
AWKEY as string,
FIKRS as string,
HWAER as string,
HWAE2 as string,
HWAE3 as string,
KURS2 as decimal(9,5),
KURS3 as decimal(9,5),
BASW2 as string,
BASW3 as string,
UMRD2 as string,
UMRD3 as string,
XSTOV as string,
STODT as string,
XMWST as string,
CURT2 as string,
CURT3 as string,
KUTY2 as string,
KUTY3 as string,
XSNET as string,
AUSBK as string,
XUSVR as string,
DUEFL as string,
AWSYS as string,
TXKRS as decimal(9,5),
LOTKZ as string,
XWVOF as string,
STGRD as string,
PPNAM as string,
BRNCH as string,
NUMPG as integer,
ADISC as string,
BATCH as string,
RLDNR as string,
SNAME as string,
CCINS as string,
CCNUM as string,
{/SAPF15/STATUS} as string,
VATDATE as string,
PSOBT as string,
PSOZL as string,
LDGRP as string,
XREF1_HD as string,
XREF2_HD as string,
XREVERSAL as string,
REINDAT as string,
PROPMANO as string,
XBLNR_ALT as string,
DOCCAT as string,
PSOKS as string,
PSOSG as string,
PSOFN as string,
INTFORM as string,
INTDATE as string,
PSODT as string,
PSOTM as string,
FM_UMART as string,
EXCLUDE_FLAG as string,
BLIND as string,
OFFSET_STATUS as string,
OFFSET_REFER_DAT as string,
PENRC as string
),
allowSchemaDrift: false,
validateSchema: true,
ignoreNoFilesFound: false,
isolationLevel: 'READ_UNCOMMITTED',
format: 'table') ~> SAPBKPF
source(output(
MANDT as integer,
BUKRS as string,
BELNR as string,
GJAHR as integer,
BUZEI as string,
BUZID as string,
AUGDT as string,
AUGCP as string,
AUGBL as string,
BSCHL as string,
KOART as string,
SHKZG as string,
GSBER as string,
MWSKZ as string,
DMBTR as decimal(13,2),
WRBTR as decimal(13,2),
PSWBT as decimal(13,2),
PSWSL as string,
MWART as string,
KTOSL as string,
VALUT as string,
ZUONR as string,
SGTXT as string,
VBUND as string,
BEWAR as string,
ALTKT as string,
VORGN as string,
KOKRS as string,
KOSTL as string,
AUFNR as string,
VBELN as string,
VBEL2 as string,
POSN2 as string,
ETEN2 as string,
ANBWA as string,
BZDAT as string,
PERNR as string,
SAKNR as string,
HKONT as string,
KUNNR as string,
LIFNR as string,
FILKD as string,
XBILK as string,
GVTYP as string,
HZUON as string,
ZFBDT as string,
ZTERM as string,
ZBD1T as decimal(3,0),
ZBD2T as decimal(3,0),
ZBD3T as decimal(3,0),
ZBD1P as decimal(5,3),
ZBD2P as decimal(5,3),
SKFBT as decimal(13,2),
SKNTO as decimal(13,2),
WSKTO as decimal(13,2),
ZLSPR as string,
DMBT1 as decimal(13,2),
WRBT1 as decimal(13,2),
MWSK2 as string,
DMBT2 as decimal(13,2),
WRBT2 as decimal(13,2),
MWSK3 as string,
DMBT3 as decimal(13,2),
WRBT3 as decimal(13,2),
LANDL as string,
MATNR as string,
WERKS as string,
MENGE as decimal(13,3),
MEINS as string,
ERFMG as decimal(13,3),
ERFME as string,
BPMNG as decimal(13,3),
BPRME as string,
EBELN as string,
EBELP as string,
ELIKZ as string,
VPRSV as string,
PEINH as decimal(5,0),
BWKEY as string,
BWTAR as string,
REWRT as decimal(13,2),
REWWR as decimal(13,2),
BUALT as decimal(13,2),
PSALT as string,
NPREI as decimal(11,2),
STCEG as string,
EGBLD as string,
EGLLD as string,
PRCTR as string,
VERTT as string,
VERTN as string,
VBEWA as string,
KSTRG as string,
AUFPL as string,
APLZL as string,
PROJK as string,
PAOBJNR as string,
BTYPE as string,
XEGDR as string,
LOKKT as string,
PPRCT as string,
KBLPOS as string,
FKBER as string,
KKBER as string,
EMPFB as string,
KIDNO as string,
KONTT as string,
KONTL as string,
BUPLA as string,
LSTAR as string,
PRZNR as string,
PENDAYS as integer,
FKBER_LONG as string,
SEGMENT as string,
PSEGMENT as string,
KSTAR as string,
BUDGET_PD as string
),
allowSchemaDrift: false,
validateSchema: true,
ignoreNoFilesFound: false,
isolationLevel: 'READ_UNCOMMITTED',
format: 'table') ~> SAPBSEG
source(output(
Client as integer,
CompanyCode as string,
DocumentType as string,
DocumentNumber as string,
NumberOfLineItemWithinAccountingDocument as string,
FiscalYear as integer,
FiscalPeriod as string,
PostingDateInTheDocument as string,
DocumentDateInDocument as string,
ProfitCenter as string,
SegmentForSegmentalReporting as string,
Plant as string,
OrderNumber as string,
MaterialNumber as string,
CustomerNumber as string,
AccountNumberOfVendorOrCreditor as string,
ControllingArea as string,
GLAccountNumber as string,
GeneralLedgerAccount as string,
AlternativeAccountNumberInCompanyCode as string,
DebitCreditIndicator as string,
AmountInDocumentCurrency as decimal(13,2),
CurrencyKey as string,
AmountInLocalCurrency as decimal(13,2),
LocalCurrency as string,
Quantity as decimal(13,3),
PurchasingDocumentNumber as string,
ItemNumberOfPurchasingDocument as string,
UpdateCurrencyForGLTransactionFigures as string,
ValueDate as string,
ItemText as string,
SupplyingCountryForDeliveryOfGoods as string,
BillingDocument as string,
WBSElement as string,
CostElement as string,
CashDiscountDays1 as decimal(3,0),
BaselineDateForDueDateCalculation as string,
PaymentReference as string,
CompanyIdOfTradingPartner as string,
ReferenceDocumentNumber as string,
ExchangeRate as decimal(9,5),
DocumentStatus as string,
NameOfUserWhoParkedThisDocument as string,
TransactionCode as string,
UserName as string,
UploadDate as timestamp,
FunctionalArea as string,
FunctionalAreaLong as string,
CostCenter as string,
ReferenceKey as string
),
allowSchemaDrift: false,
validateSchema: true,
ignoreNoFilesFound: false,
isolationLevel: 'READ_UNCOMMITTED',
format: 'table',
staged: false) ~> StoredDataToDWH
SAPBKPF, SAPBSEG join(SAPBKPF@MANDT == SAPBSEG@MANDT
&& SAPBKPF@BUKRS == SAPBSEG@BUKRS
&& SAPBKPF@GJAHR == SAPBSEG@GJAHR
&& SAPBKPF@BELNR == SAPBSEG@BELNR,
joinType:'inner',
broadcast: 'auto')~> LookupSAPBKPFBSEG
LookupSAPBKPFBSEG select(mapColumn(
Client = SAPBKPF@MANDT,
CompanyCode = SAPBKPF@BUKRS,
DocumentNumber = SAPBKPF@BELNR,
NumberOfLineItemWithinAccountingDocument = BUZEI,
FiscalYear = SAPBKPF@GJAHR,
FiscalPeriod = MONAT,
PostingDateInTheDocument = BUDAT,
ProfitCenter = PRCTR,
SegmentForSegmentalReporting = SEGMENT,
OrderNumber = AUFNR,
MaterialNumber = MATNR,
Plant = WERKS,
CustomerNumber = KUNNR,
AccountNumberOfVendorOrCreditor = LIFNR,
ControllingArea = KOKRS,
GLAccountNumber = SAKNR,
GeneralLedgerAccount = HKONT,
AlternativeAccountNumberInCompanyCode = LOKKT,
DebitCreditIndicator = SHKZG,
AmountInDocumentCurrency = WRBTR,
CurrencyKey = WAERS,
AmountInLocalCurrency = DMBTR,
LocalCurrency = HWAER,
Quantity = MENGE,
PurchasingDocumentNumber = EBELN,
ItemNumberOfPurchasingDocument = EBELP,
UpdateCurrencyForGLTransactionFigures = PSWSL,
ValueDate = VALUT,
ItemText = SGTXT,
SupplyingCountryForDeliveryOfGoods = EGLLD,
BillingDocument = VBELN,
WBSElement = PROJK,
CostElement = KSTAR,
CashDiscountDays1 = ZBD1T,
BaselineDateForDueDateCalculation = ZFBDT,
PaymentReference = KIDNO,
CompanyIdOfTradingPartner = VBUND,
ReferenceDocumentNumber = XBLNR,
DocumentType = BLART,
DocumentDateInDocument = BLDAT,
ExchangeRate = KURSF,
TransactionCode = TCODE,
NameOfUserWhoParkedThisDocument = PPNAM,
DocumentStatus = BSTAT,
UserName = USNAM,
FunctionalArea = FKBER,
FunctionalAreaLong = FKBER_LONG,
CostCenter = KOSTL,
ReferenceKey = AWKEY
),
skipDuplicateMapInputs: true,
skipDuplicateMapOutputs: true) ~> SelectFieldsGL
SelectFieldsGL derive(UploadDate = fromUTC(currentUTC(), 'Europe/Berlin')) ~> AddUploadDate
AddUploadDate, StoredDataToDWH exists(SelectFieldsGL@Evan ! == StoredDataToDWH@Evan !
&& SelectFieldsGL@CompanyCode == StoredDataToDWH@CompanyCode
&& SelectFieldsGL@DocumentNumber == StoredDataToDWH@DocumentNumber
&& SelectFieldsGL@FiscalYear == StoredDataToDWH@FiscalYear
&& SelectFieldsGL@NumberOfLineItemWithinAccountingDocument == StoredDataToDWH@NumberOfLineItemWithinAccountingDocument,
negate:true,
broadcast: 'auto')~> CheckExistData
CheckExistData sink(input(
Client as integer,
CompanyCode as string,
DocumentType as string,
DocumentNumber as string,
NumberOfLineItemWithinAccountingDocument as string,
FiscalYear as integer,
FiscalPeriod as string,
PostingDateInTheDocument as string,
DocumentDateInDocument as string,
ProfitCenter as string,
SegmentForSegmentalReporting as string,
Plant as string,
OrderNumber as string,
MaterialNumber as string,
CustomerNumber as string,
AccountNumberOfVendorOrCreditor as string,
ControllingArea as string,
GLAccountNumber as string,
GeneralLedgerAccount as string,
AlternativeAccountNumberInCompanyCode as string,
DebitCreditIndicator as string,
AmountInDocumentCurrency as decimal(13,2),
CurrencyKey as string,
AmountInLocalCurrency as decimal(13,2),
LocalCurrency as string,
Quantity as decimal(13,3),
PurchasingDocumentNumber as string,
ItemNumberOfPurchasingDocument as string,
UpdateCurrencyForGLTransactionFigures as string,
ValueDate as string,
ItemText as string,
SupplyingCountryForDeliveryOfGoods as string,
BillingDocument as string,
WBSElement as string,
CostElement as string,
CashDiscountDays1 as decimal(3,0),
BaselineDateForDueDateCalculation as string,
PaymentReference as string,
CompanyIdOfTradingPartner as string,
ReferenceDocumentNumber as string,
ExchangeRate as decimal(9,5),
DocumentStatus as string,
NameOfUserWhoParkedThisDocument as string,
TransactionCode as string,
UserName as string,
UploadDate as timestamp,
FunctionalArea as string,
FunctionalAreaLong as string,
CostCenter as string,
ReferenceKey as string
),
allowSchemaDrift: false,
validateSchema: true,
deletable:false,
insertable:true,
updateable:false,
upsertable:false,
format: 'table',
staged: true,
mapColumn(
Client,
CompanyCode,
DocumentType,
DocumentNumber,
NumberOfLineItemWithinAccountingDocument,
FiscalYear,
FiscalPeriod,
PostingDateInTheDocument,
DocumentDateInDocument,
ProfitCenter,
SegmentForSegmentalReporting,
Plant,
OrderNumber,
MaterialNumber,
CustomerNumber,
AccountNumberOfVendorOrCreditor,
ControllingArea,
GLAccountNumber,
GeneralLedgerAccount,
AlternativeAccountNumberInCompanyCode,
DebitCreditIndicator,
AmountInDocumentCurrency,
CurrencyKey,
AmountInLocalCurrency,
LocalCurrency,
Quantity,
PurchasingDocumentNumber,
ItemNumberOfPurchasingDocument,
UpdateCurrencyForGLTransactionFigures,
ValueDate,
ItemText,
SupplyingCountryForDeliveryOfGoods,
BillingDocument,
WBSElement,
CostElement,
CashDiscountDays1,
BaselineDateForDueDateCalculation,
PaymentReference,
CompanyIdOfTradingPartner,
ReferenceDocumentNumber,
ExchangeRate,
DocumentStatus,
NameOfUserWhoParkedThisDocument,
TransactionCode,
UserName,
UploadDate,
FunctionalArea,
FunctionalAreaLong,
CostCenter,
ReferenceKey
),
skipDuplicateMapInputs: true,
skipDuplicateMapOutputs: true) ~> StoreToDWHGeneralLadger

Answer 1

Hi, Haritha
I have made a minor modification in data flow and run again in not debug mode, everything fine. Thanks a lot for the answer.

Br,
Mike

Answer 2

Hi @Yachnik Mykhailo ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

Since it is running fine in debug mode but not in trigger mode, can you please make minor modification to pipeline and publish it again and then retry running it. Sometimes the published ARM template might have incomplete code though publishing shows completed on ADF UI. Please let us know if it works. If issue persists, kindly share the data factory name, region and pipeline run id when triggered as it is hard to take it from snaps above.

Looking forward for your response!