Edit

Share via

Configuration management in HDInsight on AKS with Apache Spark™ cluster

  • Article

Important

This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.

Azure HDInsight on AKS is a managed cloud-based service for big data analytics that helps organizations process large amounts data. This tutorial shows how to use configuration management in Azure HDInsight on AKS with Apache Spark™ cluster.

Configuration management is used to add specific configurations into the Apache Spark cluster.

When user updates a configuration in the management portal the corresponding service is restarted in rolling manner.

Steps to update Configurations

  1. Click on the cluster name in the cluster pool and navigate to cluster overview page.

  2. Click on the configuration management tab in the left pane.

    Screenshot showing how to select configuration tab.

  3. This step takes you to the Spark configurations, which are provided.

    Screenshot showing spark configuration.

  4. Click on the configuration tabs that need to be changed.

    Screenshot showing how to change spark configuration 2.

    Screenshot showing how to change spark configuration 3.

  5. To change any configuration, replace the given values on the text box with the desired values, click on OK and then click Save.

    Screenshot showing how to change spark configuration value-1.

    Screenshot showing how to change spark configuration value-2.

    Screenshot showing how to change spark configuration value-3.

  6. To add a new parameter, which isn't provided by default click on “Add” in the bottom right.

    Screenshot showing how to add configuration.

  7. Add the desired configuration and click on “Ok” and then click on "Save".

    Screenshot showing how to save configuration.

  8. The configurations are updated and the cluster is restarted.

  9. To delete the configurations, click on the delete icon next to the textbox.

    Screenshot showing how to delete configuration.

  10. Click on “Ok” and then click on “Save”.

    Screenshot showing how to save configuration changes.

    Note

    Selecting Save will restart the clusters. It is advisable not to have any active jobs while making configuration changes, since restarting the cluster may impact the active jobs.

Reference

Next steps