Can we upgrade commons-io in HDInsight

Question

Hello Team,

We're currently running HDInsight 5.0 for the spark runtime. We're facing the issue wrt to the common-io lib which is available as the part of HDInsight 5.0 since it has lower version of it which 2.5v.

In our project, we're dependent on a lib which is using common-io version 2.8 or above and it has been marked as PROVIDED scope due to which it's depending on the runtime libs.

My question is can we upgrade the commons-io lib to 2.8 or above in the HDInsight 5.0 without any issues?

Please let me know.

Regards,

Sharath

Answer 1

It is generally not recommended to upgrade the version of libraries that come pre-installed with a platform or tool, as it can cause compatibility issues with other components that rely on those specific versions.

In your case, upgrading the commons-io library in HDInsight 5.0 may cause issues with other components that rely on the pre-installed version of the library. It is recommended to first test the compatibility of the upgraded library with all other components in your project before making any changes.

One option would be to create a custom HDInsight cluster with the desired version of the commons-io library pre-installed. This would allow you to use the upgraded library without affecting other components that rely on the pre-installed version.

Another option would be to include the upgraded commons-io library as part of your project and manually configure your dependencies to use the upgraded version instead of the provided one. However, this may require additional configuration and may not be feasible in all cases.

Overall, it is recommended to carefully evaluate the potential impact of upgrading a pre-installed library in a platform like HDInsight before making any changes.