move files to DFS

Abhilash Reddy Pathireddy-External 1

how can we tranfer only required files from emr cluster(pyspark code) to Microsoft DFS

Vinodh247-1375 11,046 Reputation points

2022-09-21T11:26:07.787+00:00

Hi,

Thanks for reaching out to microsoft Q&A.

Can you provide more details? do you mean copy files to adls ?
Abhilash Reddy Pathireddy-External 1 Reputation point

2022-09-21T11:32:33.643+00:00

thank you for the replay, i mean i want to transfer (or copy or move ) a pdf file which is in my local machine with the help of pyspark code on EMR cluster (notebook) to microsoft DFS location.
PRADEEPCHEEKATLA-MSFT 76,836 Reputation points Microsoft Employee

2022-09-26T05:47:51.443+00:00
Hello @Abhilash Reddy Pathireddy-External ,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.
Abhilash Reddy Pathireddy-External 1 Reputation point

2022-09-26T08:37:51.687+00:00

it may needed but not in my case, thank you for replay. and i tried to connecte with SMB server machine after gathering essential details then i am getting connection refused and not connected error .It will be helpful if you can help me on this. Thank you
PRADEEPCHEEKATLA-MSFT 76,836 Reputation points Microsoft Employee

2022-10-10T04:00:42.327+00:00

Hello @Abhilash Reddy Pathireddy-External ,

Apologies for the delay in response.

The scope of your question is different compared to the original scope of the question asked.

I would recommend creating a new thread on the same forum with as much details about your issue as possible. That would make sure that your issue has better visibility in the community.

1 answer

PRADEEPCHEEKATLA-MSFT 76,836 Reputation points Microsoft Employee

2022-09-22T05:50:17.607+00:00
Hello @Abhilash Reddy Pathireddy-External ,

Thanks for the question and using MS Q&A platform.

Unfortuantely, you cannot move data from EMR cluster to Microsoft DBF location.

Possible way is since the pdf file which is available in your local machine, you can launch databricks workspace and then upload the data by selecting the DBFS => Upload

For more details, refer to Manage the DBFS file browser.

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators
Please sign in to rate this answer.
Abhilash Reddy Pathireddy-External 1 Reputation point

2022-09-22T13:16:32.827+00:00

can we move from s3 to microsoft DFS, if yes how can we do that ?

PRADEEPCHEEKATLA-MSFT 76,836 Reputation points Microsoft Employee

2022-09-23T05:40:05.61+00:00

Hello @Abhilash Reddy Pathireddy-External ,

Databricks recommends against using DBFS and mounted cloud object storage for most use cases in Unity Catalog-enabled Azure Databricks workspaces.

Important Note: Even though the DBFS root is writeable, we recommend that you store data in mounted object storage rather than in the DBFS root.

From Azure Databricks Best Practices: Do not Store any Production Data in Default DBFS Folders.

PRADEEPCHEEKATLA-MSFT 76,836 Reputation points Microsoft Employee

2022-10-12T06:29:11.21+00:00

Hello @Abhilash Reddy Pathireddy-External ,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.
Sign in to comment