question

HansGeurtsen-2054 avatar image
0 Votes"
HansGeurtsen-2054 asked HansGeurtsen-2054 commented

Third party Python package installed on Databricks cluster gives different results than other Python stacks

We get a Python package developed by a third party. The package implements a standard mathematical model, no machine learning, no randomization. The model turned out to return incorrect results when installed on a Databricks cluster. We tried different runtime versions, including 6.2, 6.6 and even 7.0. When tested on other Python stacks (Python 3.8 on Windows 10, Python 3.8 on Ubuntu 20.04 and Python 3.7 on Ubuntu 16.04 (identical to Databricks)), it works as it should. Does anyone have an explanation why a package installed on a Databricks cluster behaves differently than the same package installed on other Python stacks?

azure-databricks
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello HansGeurtsen-2054 ,

Just wanted to know if the below response helped you or if you can give more info and we will try to help you on this . Anyways let us know either ways

Thanks
Himanshu

0 Votes 0 ·

Hello Himanshu,

First of all, thanks for trying to help me. Unfortunately, no, it doesn't help me. What should I consider with regards to 'the ADb driver-worker architecture design'? If I configure a Databricks cluster with a certain runtime, defining Ubuntu + Python + package versions, and install a Python package on the cluster, what makes that package work differently from a identical Python stack on an identical Ubuntu image? I don't know what the Databricks driver-worker architecture has to do with that? Can you clarify your remark and give me any further hints?

Thanks, Hans.

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered HimanshuSinha-MSFT edited

Hello @HansGeurtsen-2054 ,
Welcome to the Q&A .


Without knowing more about the the package , i think it will very tough to tell a very definitive answer to that . But since the Adb works driver-worker architecture design , I think that may be something worth considering .




Thanks & stay safe



Himanshu


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.