Execute Databricks init script in ADF located in Workspace

Question

Execute Databricks init script in ADF located in Workspace

Tom Leeflang 10

Hi all,

Is it possible to run an init script from ADF pipeline that is located in the Databricks Workspace?

2023-05-27_10h49_26

It seems that the default pathing to a configured init script is dbfs. I namely receive the error below:

2023-05-27_10h51_17

Thank you in advance!

Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2023-05-27T09:47:04.9466667+00:00

Hi

Thanks for reaching out to Microsoft Q&A.

Check if this will work for you. There is a answer to a similar question.

https://stackoverflow.com/questions/76220211/create-new-databricks-cluster-from-adf-linked-service-with-initscripts-from-abfs

Please Upvote and Accept as answer if the reply was helpful, this will be helpful to other community members.

Tom Leeflang 10

Hi @Vinodh247 ,

Thank you for your response! The answer to the other question is somewhat in line with my question, however misses the key component. Namely, how to (if possible at all) refer to an init script that is located in the Workspace of Databricks.


{
    "num_workers": null,
    "autoscale": {
        "min_workers": 2,
        "max_workers": 8
    },
    "cluster_name": "cluster1",
    "spark_version": "7.3.x-scala2.12",
    "spark_conf": {},
    "node_type_id": "Standard_D3_v2",
    "custom_tags": {},
    "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
    },
    "autotermination_minutes": 120,
    "init_scripts": <script_path_here_as_specified_in_document> # <-- How to refer to the Workspace in this line
}

1 answer

Your answer

Vinodh247 34,661 Reputation points MVP Volunteer Moderator

2023-05-27T09:47:04.9466667+00:00

Hi

Thanks for reaching out to Microsoft Q&A.

Check if this will work for you. There is a answer to a similar question.

https://stackoverflow.com/questions/76220211/create-new-databricks-cluster-from-adf-linked-service-with-initscripts-from-abfs

Please Upvote and Accept as answer if the reply was helpful, this will be helpful to other community members.
Tom Leeflang 10 Reputation points

2023-05-30T09:19:12.5266667+00:00

Hi @Vinodh247 ,

Thank you for your response! The answer to the other question is somewhat in line with my question, however misses the key component. Namely, how to (if possible at all) refer to an init script that is located in the Workspace of Databricks.

{ "num_workers": null, "autoscale": { "min_workers": 2, "max_workers": 8 }, "cluster_name": "cluster1", "spark_version": "7.3.x-scala2.12", "spark_conf": {}, "node_type_id": "Standard_D3_v2", "custom_tags": {}, "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }, "autotermination_minutes": 120, "init_scripts": <script_path_here_as_specified_in_document> # <-- How to refer to the Workspace in this line }

Answer 1

PRADEEPCHEEKATLA 90,641 Moderator

@Tom Leeflang - Thanks for the question and using MS Q&A platform.

Init script path should be a folder in the dbfs - example: dbfs:/databricks/scripts/initscript.sh

As per the repo, I had created a init script named odbc-install.shand upload to the folder named dbfs:/databricks/scriptsand path looks like this:

dbfs:/databricks/scripts/odbc-install.sh

User's image

Now pass the same path of the init scripts in the ADF linked service as shown below.

User's image

When I ran the Databricks Notebook activity, I'm able to install the initscript succesfully without any issue.

User's image

Here is the Event logs of the cluster:

User's image

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Tom Leeflang 10 Reputation points

2023-05-30T09:13:26.2566667+00:00

@PRADEEPCHEEKATLA , thank you for the response!
For DBFS located init scripts it worked for me as well. However, a couple of weeks ago we've received a notification from Azure that: "Existing cluster-scoped init scripts stored on DBFS should be migrated to the alternative workspace files location". We've taken this into account and moved our init script to the Workspace. However, we still need to this refer to this location in ADF, but it seems that the Workspace option isn't possible (hence my question above).

hopefully this makes my query clearer!

With kind regards,
Tom
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-05-31T05:45:40.8266667+00:00

@Tom Leeflang - As explained in the above answer, Init script must is existed path from the dbfs folder in databricks workspace.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Please feel free to take a survey on the relevant answer. You can help us improve by leaving feedback verbatim.

Have a good day!

Regards,

PRADEEPCHEEKATLA-MSFT
Tom Leeflang 10 Reputation points

2023-05-31T09:20:34.7566667+00:00

@PRADEEPCHEEKATLA , once again thank you so much for your time! However I do have one final question. In documentation Microsoft states that we should move the init scripts to the Home directory (or subfolders) of the Workspace. They also state that DBFS will be deprecated for init scripts.

When we follow these steps in Databricks all works well and the init script gets properly executed as well as the cluster being started.

However in Azure Data Factory there seems to be no option (yet) for init scripts to be configured, located in the Home directory (or subfolders) of the Workspace. It gets rerouted to the default DBFS location.
Shouldn't this be made available since the other option is deprecated?

Thank you!
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-06-05T04:52:16.58+00:00

@Tom Leeflang - All init scripts stored in DBFS should be migrated to workspace files. For migration instructions, see Cluster-named init script migration notebook in the Databricks Knowledge Base.

After migrating the init script to home directory under the /users directory you can use the same path in the data factory linked service to configure the init scripts.

Hope this helps.
Tom Leeflang 10 Reputation points

2023-06-06T11:33:37.33+00:00

Hi @PRADEEPCHEEKATLA ,

That is what I thought as well, but look at this:
Whenever we create a cluster within Databricks, all works well. So the picture below works properly.

Now in order to make ADF run the same init script, we need to refer to the same folderpath, like this:

This setting fails, and when we check why, we can see that the cluster that is created via ADF refers to DBFS:

Any ideas on how to refer to the init script in the home directory under /users?
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2023-06-14T04:12:30.3066667+00:00

@Tom Leeflang - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
Michal Piekarczyk 0 Reputation points

2023-09-27T20:03:35.5933333+00:00

@Tom Leeflang @Tom Leeflang , just in case you have found an answer here, could you please share? Can "newClusterInitScripts" some special undocumented syntax to help recognize Workspace resident init scripts perhaps? Thanks !

Share via

Execute Databricks init script in ADF located in Workspace

1 answer

Your answer