U-SQL activity is loading partial data in ADLS using Azure Data factory

amikm 11

I am executing the U-SQL script using U-SQL activity inside Azure Data Factory, It is loading partial data into ADLS ( 25 GB/50 GB)
And when I execute the script using Visual Studio (U-SQL enabled), it is loading complete data (50 GB/50 GB), even If I execute the same script using ADLA (Azure Data Lake analytic a job, It is loading full data)

In the case of ADF, I have used Degree of Parallelism =50
In the case of Visual Studio or ADLA job I have used AU Hours- 50

Also, It is not extracting all the files using ADF but extracts all files while using directly in ADLA

Any reason for this behavior?
is there any limitation for U-SQL activity using ADF it not taking latest files to extract and load?

ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2022-09-16T14:05:28.777+00:00

Hi @amikm ,

Thank you for posting query in Microsoft Q&A Platform.

U-SQL Activity in ADF, will actually run your U-SQL on your configured ADLA only.

While configuring U-SQL activity, under ADLA account tab, Are you using same ADLA account which you was running good? Also is the script is same which running via ADF and other medium?

Usually ADF role here is only to Submit job on ADLA thats it. Kindly check the job execution details on ADLA if that gives any clue. Thank you.

Please let me know how it goes.
amikm 11 Reputation points

2022-09-16T14:50:25.373+00:00

@ShaikMaheer-MSFT
it's the same ADLA account, it's running by loading the partial files using U-SQL activity inside ADF