Azure HDInsight release notes
This article provides information about the most recent Azure HDInsight release updates. For information on earlier releases, see HDInsight Release Notes Archive.
Azure HDInsight is one of the most popular services among enterprise customers for open-source analytics on Azure. If you would like to subscribe on release notes, watch releases on this GitHub repository.
Release date: December 12, 2022
This release applies to HDInsight 4.0. and 5.0 HDInsight release is made available to all regions over several days.
HDInsight uses safe deployment practices, which involve gradual region deployment. It may take up to 10 business days for a new release or a new version to be available in all regions.
- HDInsight 4.0: Ubuntu 18.04.5 LTS Linux Kernel 5.4
- HDInsight 5.0: Ubuntu 18.04.5 LTS Linux Kernel 5.4
For workload specific versions, see here.
- Log Analytics - Customers can enable classic monitoring to get the latest OMS version 14.19. To remove old versions, disable and enable classic monitoring.
- Ambari user auto UI logout due to inactivity. For more information, see here
- Spark - A new and optimized version of Spark 3.1.3 is included in this release. We tested Apache Spark 3.1.2(previous version) and Apache Spark 3.1.3(current version) using the TPC-DS benchmark. The test was carried out using E8 V3 SKU, for Apache Spark on 1-TB workload. Apache Spark 3.1.3 (current version) outperformed Apache Spark 3.1.2 (previous version) by over 40% in total query runtime for TPC-DS queries using the same hardware specs. The Microsoft Spark team added optimizations available in Azure Synapse with Azure HDInsight. For more information, please refer to Speed up your data workloads with performance updates to Apache Spark 3.1.2 in Azure Synapse
- Qatar Central
- Germany North
HDInsight has moved away from Azul Zulu Java JDK 8 to Adoptium Temurin JDK 8, which supports high-quality TCK certified runtimes, and associated technology for use across the Java ecosystem.
HDInsight has migrated to reload4j. The log4j changes are applicable to
- Apache Hadoop
- Apache Zookeeper
- Apache Oozie
- Apache Ranger
- Apache Sqoop
- Apache Pig
- Apache Ambari
- Apache Kafka
- Apache Spark
- Apache Zeppelin
- Apache Livy
- Apache Rubix
- Apache Hive
- Apache Tez
- Apache HBase
- Apache Pheonix
HDInsight will implement TLS1.2 going forward, and earlier versions will be updated on the platform. If you're running any applications on top of HDInsight and they use TLS 1.0 and 1.1, upgrade to TLS 1.2 to avoid any disruption in services.
For more information, see How to enable Transport Layer Security (TLS)
End of support for Azure HDInsight clusters on Ubuntu 16.04 LTS from 30 November 2022. HDInsight had begun release of cluster images using Ubuntu 18.04 from June 27, 2021. We recommend our customers who are running clusters using Ubuntu 16.04 is to rebuild their clusters with the latest HDInsight images by 30 November 2022.
For more information on how to check Ubuntu version of cluster, see here
Execute the command “lsb_release -a” in the terminal.
If the value for “Description” property in output is “Ubuntu 16.04 LTS”, then this update is applicable to the cluster.
- Support for Availability Zones selection for Kafka and HBase (write access) clusters.
Open source bug fixes
Hive bug fixes
|Bug Fixes||Apache JIRA|
|HIVE-26127||INSERT OVERWRITE error - File Not Found|
|HIVE-24957||Wrong results when subquery has COALESCE in correlation predicate|
|HIVE-24999||HiveSubQueryRemoveRule generates invalid plan for IN subquery with multiple correlations|
|HIVE-24322||If there's direct insert, the attempt ID has to be checked when reading the manifest fails|
|HIVE-23363||Upgrade DataNucleus dependency to 5.2|
|HIVE-26412||Create interface to fetch available slots and add the default|
|HIVE-26173||Upgrade derby to 10.14.2.0|
|HIVE-25920||Bump Xerce2 to 2.12.2.|
|HIVE-26300||Upgrade Jackson data bind version to 22.214.171.124+ to avoid CVE-2020-36518|