ADF Get-MetaData activity file size option not visible in case of Binary data set

Question

ADF Get-MetaData activity file size option not visible in case of Binary data set

Shiv Kumar 40

I have a requirement where I need to copy non-Zero compressed files from SFTP server. I have created a binary data set. But when I am trying to get the file size from Get-MetaData activity, FIleSize option not visible.

How i can get the zipped/Compressed file size in ADF or Synapse pipelines.

Please Suggest.

Thanks,
Shiv

Samy Abdul 3,376 Reputation points

2023-05-04T12:13:32.4133333+00:00

Hi @Shiv Kumar ,try to use one more GetMetadata activity inside the if conditional activity so as to get the file size .Thanks
Shiv Kumar 40 Reputation points

2023-05-05T02:25:43.0233333+00:00

@Samy Abdul ; I have a compressed file and file size option is not applicable for binary data set.

Please check my attached screenshot, file size option not here also please notice I have used binary data set to read compressed file. I know how to use if and filleter the things.
If you have any work around, please suggest.

Thanks,

Shiv
Vinodh247 32,086 Reputation points MVP

2023-05-05T10:11:35.0166667+00:00

Hi

Thanks for reaching out to Microsoft Q&A.

I think you don't necessarily need the filesize in the fieldlist argument for this, you can also setup condition to omit files below certain size using getmetadata output.

Please refer the link below and let me know if this worked for you.

https://stackoverflow.com/questions/66130799/ignore-files-less-than-200kb-in-azure-data-factory-pipeline

Please Upvote and Accept as answer if the reply was helpful, this will be helpful to other community members.
KranthiPakala-MSFT 46,627 Reputation points Microsoft Employee

2023-05-12T22:44:45.7533333+00:00

@Shiv Kumar Thanks for using Microsoft Q&A forum and posting your query.

I don't have SFTP environment to test it this scenario, but I just tried to verify if the size property from GetMetadata is available for Zip files and noticed it is available.

As per the getMetada documenation, file size is supported for both SFTP and FTP. In that case you should be able to see the file size option for your Zipped file.

If you are not seeing the option, then it seems something is not correct. I recommend filing support case for deeper investigation because it could be a specific issue to your SFTP provider.

Please do file a support case and do share the support ticket number with us here.

Thank you

1 answer

Your answer

Samy Abdul 3,376 Reputation points

2023-05-04T12:13:32.4133333+00:00

Hi @Shiv Kumar ,try to use one more GetMetadata activity inside the if conditional activity so as to get the file size .Thanks
Shiv Kumar 40 Reputation points

2023-05-05T02:25:43.0233333+00:00

@Samy Abdul ; I have a compressed file and file size option is not applicable for binary data set.

Please check my attached screenshot, file size option not here also please notice I have used binary data set to read compressed file. I know how to use if and filleter the things.
If you have any work around, please suggest.

Thanks,

Shiv
Vinodh247 32,086 Reputation points MVP

2023-05-05T10:11:35.0166667+00:00

Hi

Thanks for reaching out to Microsoft Q&A.

I think you don't necessarily need the filesize in the fieldlist argument for this, you can also setup condition to omit files below certain size using getmetadata output.

Please refer the link below and let me know if this worked for you.

https://stackoverflow.com/questions/66130799/ignore-files-less-than-200kb-in-azure-data-factory-pipeline

Please Upvote and Accept as answer if the reply was helpful, this will be helpful to other community members.
KranthiPakala-MSFT 46,627 Reputation points Microsoft Employee

2023-05-12T22:44:45.7533333+00:00

@Shiv Kumar Thanks for using Microsoft Q&A forum and posting your query.

I don't have SFTP environment to test it this scenario, but I just tried to verify if the size property from GetMetadata is available for Zip files and noticed it is available.

As per the getMetada documenation, file size is supported for both SFTP and FTP. In that case you should be able to see the file size option for your Zipped file.

If you are not seeing the option, then it seems something is not correct. I recommend filing support case for deeper investigation because it could be a specific issue to your SFTP provider.

Please do file a support case and do share the support ticket number with us here.

Thank you

Answer 1

Diego Menese 1

It's a bit weird that "Size" does not show up in the drop-down list, but I found that you can add it manually and it works fine. Simply add a new field and select "Add dynamic content" and then in the dynamic content window type "Size":

User's image

And then when debugging it works fine (returning the size in bytes):

User's image

Hope it helps!

Thanks,
Diego.

Share via

ADF Get-MetaData activity file size option not visible in case of Binary data set

1 answer

Your answer