Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,561 questions
How to copy files to working directory in the azure databricks?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 44%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sharath
20
Reputation points
I'm using the databricks api to submit/create a job, please find the payload as below. and then manually running the job from the UI. When I run the job, basically it's a jar which talks to Hana data lake.
{
"name": "sharath-sparkconf-{{$timestamp}}",
"existing_cluster_id": "0602-xxxxx-yyyy",
"libraries": [
{
"jar": "<path to jar>-order-core.jar"
}
],
"spark_jar_task": {
"main_class_name": "streaming.xxx",
"parameters": [
"fs.azure.enable.flush=false",
"fs.azure.account.key=,
"fs.azure.account.name=",
"spark.executorEnv.CLUSTER=v",
"spark.executorEnv.NAME_SPACE=",
"spark.executorEnv.AZURE_ACCOUNT_NAME=",
"spark.executorEnv.KAFKA_HOST= ",
"spark.executorEnv.KAFKA_PORT= ",
"spark.executorEnv.KAFKA_USER= ",
"spark.executorEnv.KAFKA_PASSWD= ",
"spark.executorEnv.HANA_DATA_LAKE_FILE_SYSTEM_URI=hdlfs://ondemand.com",
"spark.yarn.submit.waitAppCompletion=false",
"spark.hadoop.fs.azure.enable.flush=false",
"spark.sql.legacy.parquet.int96RebaseModeInRead=LEGACY",
"spark.sql.legacy.parquet.int96RebaseModeInWrite=LEGACY",
"spark.sql.legacy.parquet.datetimeRebaseModeInRead=LEGACY",
"spark.sql.legacy.parquet.datetimeRebaseModeInWrite=LEGACY",
"spark.sql.legacy.timeParserPolicy=LEGACY",
"spark.executorEnv.HANA_DATA_LAKE_PASSWORD= !",
"spark.executorEnv.HANA_DATA_LAKE_PK12_LOCATION=/dbfs/Filestore/tables/.p12"
]
}
}
HanaDatalake uses .p12 to establish the communication. My question is how can we upload the .pk12 certificate to working directory of the spark? I tried adding the certificate to dbf file and provided the file path, but ended up with the error show below.
Caused by: java.io.IOException: java.nio.file.NoSuchFileException: ./client-keystore.p12
at com.sap.hana.datalake.files.HdlfsConnectionConfigurator.<init>(HdlfsConnectionConfigurator.java:71)
... 72 more
Caused by: java.nio.file.NoSuchFileException: ./client-keystore.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.apache.hadoop.security.ssl.ReloadingX509KeystoreManager.loadKeyManager(ReloadingX509KeystoreManager.java:139)
at org.apache.hadoop.security.ssl.ReloadingX509KeystoreManager.<init>(ReloadingX509KeystoreManager.java:76)
Earlier I was using Livy api where I can pass the dependency certification as the parameter using --file which used to make it available in the working directory for the spark.
how can we achieve same using databricks api?
Azure Databricks
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT 46,647 Reputation points Microsoft Employee Moderator2023-06-08T22:46:42.1333333+00:00 @Sharath Welcome to Microsoft Q&A forum and thanks for reaching out here.
Have you uploaded the file to DBFS first before using it? If you haven't then please try upload ing the file to Databricks DBFS by running this CLI command -
databricks fs cp <local-file-path> dbfs:/<dbfs-file-path>.Once after adding the file to DBFS, then in your job configuration, add the file to the
librariesarray with thedbfsprefix, as shown below:{ "name": "sharath-sparkconf-{{$timestamp}}", "existing_cluster_id": "0602-xxxxx-yyyy", "libraries": [ { "jar": "<path to jar>-order-core.jar" }, { "dbfs": "dbfs:/<dbfs-file-path>/client-keystore.p12" } ], "spark_jar_task": { "main_class_name": "streaming.xxx", "parameters": [ ... "spark.executorEnv.HANA_DATA_LAKE_PASSWORD= !", "spark.executorEnv.HANA_DATA_LAKE_PK12_LOCATION=dbfs:/<dbfs-file-path>/client-keystore.p12" ] } }When you add a file to the
librariesarray in the job configuration with thedbfsprefix, Databricks will automatically download the file from DBFS and make it available in the working directory of your Spark job. This is necessary because the working directory of a Spark job is not the same as the local file system of the machine running the job. This should make the file available in the working directory of your Spark job.Hope this info helps. Let us know how it goes.
Thank you
-
Sharath 20 Reputation points
2023-06-08T23:28:40.61+00:00 @KranthiPakala-MSFT , Thank you for the response. Yes I have already uploaded the .p12 file to dbfs. My doubt is after Databricks automatically downloading the file from DBFS and make it available in the working directory, what's the path of this working directory? because I need to pass this path as param to the 3rd party library file system.
Sign in to comment
Sign in to answer