question

$$ANON_USER$$ avatar image
0 Votes"
$$ANON_USER$$ asked PRADEEPCHEEKATLA-MSFT commented

Load data from SFTP to ADLS in azure synapse using data flow activity?

Hi All, I am loading data from SFTP server to ADLS blob storage. I am using data flow activity, the reason is I want to move files from my ftp source folder to archieve folder. Also I have multiple csv files at source level. I am able to move files from sftp source folder to archieve folder but I am getting problem while loading these files to ADLS. First thing the file names are not coming correctly in ADLS. for eg: ![220943-image.png][1] Like thispart-00000-4bdb94ba-0994-4232-9a03-036e115d471b-c000.csv [1]: /answers/storage/attachments/220943-image.png Second same file is getting created twice in ADLS. Kindly help! Thanks in advance.

azure-data-factoryazure-synapse-analyticsazure-monitorazure-databricksazure-data-lake-storage
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Can I Please get any advice on this from someone.

0 Votes 0 ·

Hello anonymous user,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered

Hello anonymous user,

Thanks for the question and using MS Q&A platform.

In order to get output to the single file, make sure to the set the partition option to single partition under optimize section:

Note: Single partition combines all the distributed data into a single partition. This is a very slow operation that also significantly affects all downstream transformation and writes. This option is strongly discouraged unless there is an explicit business reason to use it.

221799-image.png

For more details, refer to Mapping data flows performance and tuning guide.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


image.png (158.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.