Phoenix in Azure HDInsight

Apache Phoenix is an open source, massively parallel relational database layer built on HBase. Phoenix allows you to use SQL like queries over HBase. Phoenix uses JDBC drivers underneath to enable users to create, delete, alter SQL tables, indexes, views and sequences, upset rows individually and in bulk. Phoenix uses NOSQL native compilation rather than using MapReduce to compile queries, enabling the creation of low-latency applications on top of HBase.

Apache Phoenix enables OLTP and operational analytics in Hadoop for low latency applications by combining the best of both worlds. In Azure HDInsight Apache Phoenix is delivered as a 1st class Open Source framework.

Why use Apache Phoenix in Azure HDInsight?

HDInsight is the best place for you to run Apache Phoenix and other Open Source Big Data Applications. HDInsight makes Apache Phoenix even better in following ways:

Out of the box highly tuned Apache Phoenix cluster in minutes

In Azure, several large customers runs their mission critical HBase/Phoenix workloads, over the period of time services becomes more and more intelligent about right configurations for running the HBase workloads as efficiently as possible. This intelligence is than brought to you in the form of highly tuned clusters that will meet your needs. You can create clusters within minutes manually with Azure Portal or by automating the creation workflow with Azure JSON templates, Powershell, REST based API or Azure client SDK.

Decoupled Storage and CPU

HDInsight changes the game with seemingly simple , yet very powerful cloud construct where compute and CPU are decoupled. This is very powerful as you have inexpensive abundant cloud storage that could be mounted to a smallest HBase cluster. When you don't need to read/write, you can delete the cluster completely and still retain the data. This flexibility helps our customers achieve best price/performance.

Delivered as a service, yet not compromising on control

HDInsight delivers Phoenix as a service so you don't have to worry about setup, patching, upgrading , maintaining etc. Moreover, you get financially backed SLA of 99.9 percent as well as support, yet it doesn't take away any control. You have the option to further fine tune your cluster as well as install additional components and make further customizations.

Best suited for mission critical production workloads

As Microsoft's roots are in enterprises, you will find HDInsight Phoenix fitting very nicely into your enterprise architecture. You can host Phoenix clusters in a private virtual network in order to protect your valuable data. You can take advantage of Azure infrastructure to achieve high availability and disaster recovery. You can also find Azure and HDInsight constantly update their compliance status @Azure Trust Center.

What is Apache Zeppelin?

Apache Zeppelin is open source Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and many other languages. It helps data developers & data scientists in developing, organizing, executing, and sharing data code and visualizing results without referring to the command line or needing the cluster details.

Integration with Apache Phoenix

Now HDInsight customers can use Apache Zeppelin to query your Phoenix tables. Apache Zeppelin is integrated with HDInsight cluster and there are no additional steps to use it.

Simply create a Zeppelin Notebook with JDBC interpreter and start writing your Phoenix SQL queries

Try HDInsight now

We hope you will take full advantage Apache Zeppelin with Apache Phoenix. We are excited to see what you will build with Azure HDInsight. Read this developer guide and follow the quick start guide to learn more about implementing these pipelines and architectures on Azure HDInsight. Stay up-to-date on the latest Azure HDInsight news and features by following us on Twitter #HDInsight and @AzureHDInsight. For questions and feedback, please reach out to AskHDInsight@microsoft.com.

About HDInsight

Azure HDInsight is Microsoft’s premium managed offering for running open source workloads on Azure. Azure HDInsight powers mission critical applications ranging in a wide variety of sectors including, manufacturing, retail education, nonprofit, government, healthcare, media, banking, telecommunication, insurance, and many more industries ranging in use cases from ETL to Data Warehousing, from Machine Learning to IoT, and more.

Additional resources

• Apache Phoenix in HDInsight.
• Use SQLLine with HDInsight Phoenix.
• Learn more about Azure HDInsight.
• Open Source component guide on HDInsight.
• HDInsight release notes.
• Ask HDInsight questions on Msdn forums.
• Ask HDInsight questions on stackoverflow.

Traditional approach to fast interactive BI

Deep analytical queries processed on Hadoop systems have traditionally been slow. MapReduce jobs or hive queries are used for heavy processing of large datasets, however, not suitable for the fast response time required by interactive BI usage.

Faced with user dissatisfaction due to lack of query interactivity, data architects used techniques such as building OLAP cubes on top of Hadoop. An OLAP cube is a mechanism to store all the different dimensions, measures and hierarchies up front. Processing the cube usually takes place at the pre-specified interval. Post processing, results are available in advance, so once the BI tool queries the cube it just needs to locate the result, thereby limiting the query response time and making it a fast and interactive one. Since all measures get pre-aggregated by all levels and categories in the dimension, it is highly suitable for interactivity and fast response time. This approach is especially suitable if you need to light up summary views.

Above approach works for certain scenarios but not all. It tends to break down easily with large big data implementations, especially with use cases where many power users and data scientists are writing many ad-hoc queries.

Here are the key challenges:

• OLAP cubes require precomputations for creating aggregates which introduces latency. Businesses across all industries are demanding more from their reporting and analytics infrastructure within shorter business timeframes. OLAP cubes can’t deliver real-time analysis.
• In big data analytics, precomputation puts heavy burden on underlying Hadoop system creating unsustainable pressure on entire big data pipeline which severely hampers performance, reliability and stability of entire pipeline.
• This type of architecture forces large dataset movement between different systems which works well at small scale. However, falls apart at large data scale. Keeping data hot and fresh across multiple tiers is challenging.
• Power users and data scientists requires a lot more agility and freedom in terms of their ability to experiment using sophisticated ad-hoc queries that puts additional burden on overall system.

Azure HDInsight Interactive query overview

One of the most exciting new features of Hive 2 is Low Latency Analytics Processing (LLAP), which produces significantly faster queries on raw data stored in commodity storage systems such as Azure Blob store or Azure Data Lake Store.

This reduces the need to introduce additional layers to enable fast interactive queries.

Key benefits of introducing Interactive Query in your big data BI architecture:

Extremely fast Interactive Queries: Intelligent caching and optimizations in Interactive Query produces blazing-fast query results on remote cloud storage, such as Azure Blob and Azure Data Lake Store. Interactive Query enables data analysts to query data interactively in the same storage where data is prepared, eliminating the need for moving data from storage to another analytical engine for analytical needs. Refer to Azure HDInsight Performance Benchmarking: Interactive Query, Spark, and Presto to understand HDInsight Interactive Query performance expectations @ 100TB scale.

HDInsight Interactive Query (LLAP) leverages set of persistent daemons that execute fragments of Hive queries. Query execution on LLAP is very similar to Hive without LLAP, except that worker tasks run inside LLAP daemons, and not in containers.

Lifecycle of a query: After client submits the JDBC query, query arrives at Hive Server 2 Interactive which is responsible for query planning, optimization, as well as security trimming. Since each query is submitted via Hive Server 2, it becomes the single place to enforce security policies.

File format versatility and Intelligent caching: Fast analytics on Hadoop have always come with one big catch: they require up-front conversion to a columnar format like ORCFile, Parquet or Avro, which is time-consuming, complex and limits your agility.

Please read Hive Metastore in HDInsight to learn more about sharing metastore across clusters and cluster types in Azure HDInsight.

Simplified and scalable architecture with HDInsight Interactive Query

By introducing Interactive Query to your architecture, you can now route power users, data scientists, and data engineers to hit Interactive Query directly. This architectural improvement will reduce the overall burden from the BI system as well as increases user satisfaction due to fast interactive query response as well as increases flexibility to run ad-hoc queries at will.

In above described architecture, users who wants to see the summary views can still be served with OLAP cubes. However, all other users leverage Interactive Query for submitting their queries.

For OLAP based applications on Azure HDInsight, please see solutions such as AtScale and Apache Kyligence.

Security model

User adoption with familiar tools

In big data analytics, organizations are increasingly concerned that their end users aren’t getting enough value out of the analytics systems because it is often too challenging and requires using unfamiliar and difficult-to-learn tools to run the analytics. HDInsight Interactive Query addresses this issue by requiring minimal to no new user training to get insight from the data. Users can write SQL queries (hql) in the tools they already use and love the most. HDInsight Interactive query out of the box supports BI tools such as Visual Studio Code, Power BI, Apache Zeppelin, Visual Studio, Ambari Hive View, Beeline, and Hive ODBC.

To learn more about these tools, please read Azure HdInsight Interactive Query: Ten tools to analyze big data faster.

Built to complement Spark, Hive, Presto, and other big data engines

HDInsight Interactive query is designed to work well with popular big data engines such as Apache Spark, Hive, Presto, and more. This is especially useful because your users may choose any one of these tools to run their analytics. With HDInsight’s shared data and metadata architecture, users can create multiple clusters with the same or different engine pointing to same underlying data and metadata. This is very powerful concept as you are no longer bounded by one technology for analytics.

Try HDInsight now

We hope you will take full advantage fast query capabilities of HDInsight Interactive Query. We are excited to see what you will build with Azure HDInsight. Read this developer guide and follow the quick start guide to learn more about implementing these pipelines and architectures on Azure HDInsight. Stay up-to-date on the latest Azure HDInsight news and features by following us on Twitter #HDInsight and @AzureHDInsight. For questions and feedback, please reach out to AskHDInsight@microsoft.com.

About HDInsight

Azure HDInsight is Microsoft’s premium managed offering for running open source workloads on Azure. Azure HDInsight powers mission critical applications ranging in a wide variety of sectors including, manufacturing, retail education, nonprofit, government, healthcare, media, banking, telecommunication, insurance, and many more industries ranging in use cases from ETL to Data Warehousing, from Machine Learning to IoT, and more.

Additional resources

• Get started with HDInsight Interactive Query Cluster in Azure
• Azure HDInsight Performance Benchmarking: Interactive Query, Spark and Presto
• Learn more about Azure HDInsight
• Use Hive on HDInsight
• Open Source component guide on HDInsight
• HDInsight release notes
• Ask HDInsight questions on Msdn forums
• Ask HDInsight questions on stackoverflow

Apache Zeppelin

Visual Studio Code

Visual Studio

Ambari Hive View

• Browse databases.
• Write queries or browse query results in full-screen mode, which can be particularly helpful with complex queries or large query results.
• Manage query execution jobs and history.
• View existing databases, tables, and their statistics.
• Create/upload tables and export table DDL to source control.
• View visual explain plans to learn more about query plan.

Beeline

Hive ODBC

Tableau

Apache DBeaver

Excel

Try HDInsight now

About HDInsight

Additional resources

• Get started with HDInsight Interactive Query Cluster in Azure.
• Zero ETL analytics with LLAP in Azure HDInsight.
• Azure HDInsight Performance Benchmarking: Interactive Query, Spark and Presto.
• Learn more about Azure HDInsight.
• Use Hive on HDInsight.
• Open Source component guide on HDInsight.
• HDInsight release notes.
• Ask HDInsight questions on Msdn forums.
• Ask HDInsight questions on stackoverflow.

jdbc:hive2://<yourclustername>.azurehdinsight.net:443/default;ssl=true?hive.server2.transport.mode=http;hive.server2.thrift.http.path=/hive2

• Use Power BI See Visualize Interactive Query Hive data with Power BI in Azure HDInsight See Visualize big data with Power BI in Azure HDInsight.
• Use Zeppelin See Use Zeppelin to run Hive queries in Azure HDInsight .
• Use Visual Studio See Connect to Azure HDInsight and run Hive queries using Data Lake Tools for Visual Studio.
• Use Visual Studio Code See Use Visual Studio Code for Hive, LLAP or pySpark.
• Run Hive by using Ambari Hive View. See Use Hive View with Hadoop in Azure HDInsight.
• Run Hive by using Beeline. See Use Hive with Hadoop in HDInsight with Beeline. You can use Beeline from either the head node or from an empty edge node. We recommend using Beeline from an empty edge node. For information about creating an HDInsight cluster by using an empty edge node, see Use empty edge nodes in HDInsight.
• Run Hive by using Hive ODBC. See Connect Excel to Hadoop with the Microsoft Hive ODBC driver.

Before you migrate please note that there may be incompatibilities between HBase Major/Minor version and below steps only works if there is no version compatibility issues between source and destination cluster. We recommend you to review HBase book before undertaking an upgrade.

Scenario:

Migrate HDInsight 3.5 to 3.6 with same HBase major version or Migrate HDInsight between point versions e.g (3.6.2 to 3.6.3) of Azure HDInsight

Step1: Make sure that your application works with the new version.

Step2: Create HDInsight Cluster on same storage but with different container name

Step3: Flush your source HBase cluster https://raw.githubusercontent.com/Azure/hbase-utils/master/scripts/flush_all_tables.sh

Step4: Stop the ingestion to HBase cluster

Step5: Flush the cluster again with https://raw.githubusercontent.com/Azure/hbase-utils/master/scripts/flush_all_tables.sh [This will ensure that any remaining data in memstore is flushed]

Step6: Stop HBase services from Ambari

Step7: Go to new HDInsight cluster /Ambari HDFS settings and change the container name to original container –>Ambari –>HDFSàAdvanced –>Advanced core-site

Step8: Restart all required services [Ambari will indicate which services requires restart]

Step9: Point your application to the new cluster.

Step10: Start the ingestion to see if everything is working well

Step11: delete the original cluster after you see everything is working well.]]> https://docs.microsoft.com/archive/blogs/ashish/interactive-ad-hoc-analysis-at-petabyte-scale-with-hdinsight-interactive-query Fri, 11 May 2018 00:53:30 GMT https://blogs.msdn.microsoft.com/ashish/?p=2245 here ]]> https://docs.microsoft.com/archive/blogs/ashish/what-is-azure-hdinsight Tue, 08 May 2018 14:57:37 GMT https://blogs.msdn.microsoft.com/ashish/?p=2235 https://docs.microsoft.com/archive/blogs/ashish/hdinsight-hive-frequently-asked-questions Fri, 27 Apr 2018 17:57:41 GMT https://blogs.msdn.microsoft.com/ashish/?p=2205 How do I check DAG counters for a query? Go to Tez View. Find the DAG and click on Dag counters tab. How do I check that amount of data being read/writted/shuffled by a query? Go to Dag counters and look for HDFS_BYTES_READ/HDFS_BYTES_WRITTEN/SHUFFLE_BYTES. How do I download all YARN container logs? yarn logs -applicationId <application_id> > logs How do I download YARN container logs for a particular container? yarn logs -applicationId <application_id> -containerId <container_id> > containerlogs How do I download YARN container logs for all Application Master (AM) container? yarn logs -applicationId <application_id> -am ALL > amlogs How do I download YARN container logs for only the latest Application Master (AM) container? yarn logs -applicationId <application_id> -am -1 > latestamlogs How do I download YARN container logs for first 2 Application Master (AM) container? yarn logs -applicationId <application_id> -am 1,2 > first2amlogs Where is hive client logs? /tmp/<username>/hive.log Where is hive metastore logs? /var/log/hive/hivemetastore.log Where is hiveserver logs? /var/log/hive/hiveserver2.log How to I start hive shell with debug logging enabled on console? hive -hiveconf hive.root.logger=ALL,console How do I use tez engine? set hive.execution.engine=tez How do I download tez DAG data? From commandline: hadoop jar /usr/hdp/current/tez-client/tez-history-parser-*.jar org.apache.tez.history.ATSImportTool -downloadDir . -dagId <DagId> From Ambari tez view: Go to Ambari --> Go to Tez view (hidden under tiles icon in upper right corner) --> Click on the dag you are interested in --> Click on Download data. How do I get CrticalPath for a Tez DAG? hadoop jar /usr/hdp/current/tez-client/tez-job-analyzer-*.jar CriticalPath --saveResults --dagId <DagId> --eventFileName <DagData.zip> What are the other analyzers available for Tez DAG? You can list all the available analyzers with hadoop jar /usr/hdp/current/tez-client/tez-job-analyzer-*.jar. Currently it has the following analyzers available:

• ContainerReuseAnalyzer: Print container reuse details in a DAG
• CriticalPath: Find the critical path of a DAG
• LocalityAnalyzer: Print locality details in a DAG
• ShuffleTimeAnalyzer: Analyze the shuffle time details in a DAG
• SkewAnalyzer: Analyze the skew details in a DAG
• SlowNodeAnalyzer: Print node details in a DAG
• SlowTaskIdentifier: Print slow task details in a DAG
• SlowestVertexAnalyzer: Print slowest vertex details in a DAG
• SpillAnalyzer: Print spill details in a DAG
• TaskAssignmentAnalyzer: Print task-to-node assignment details of a DAG
• TaskConcurrencyAnalyzer: Print the task concurrency details in a DAG
• VertexLevelCriticalPathAnalyzer: Find critical path at vertex level in a DAG

1. DFS : You have lot of data on HDFS, which is consuming disk space. The solution is to delete the data from the HDFS that you no longer require. FOr example, you can use hdfs dfs -rm -r -skipTrash hdfs://mycluster/tmp to remove the /tmp HDFS directory.
2. Non DFS : You have lot of intermediate job data that is consuming the disk space. You have to kill the application(s) responsible for writing lots of data. You can check the values of FILE_BYTES_WRITTEN counter to infer which is the bad application(s). Once you have killed the bad application(s), the cluster should become healthy in few minutes. If you re-run the hive query without fixing it to not to produce large intermediate data, the cluster will again end up in the same situation. One of the very common example of queries which result into a cross join. If you do explain of the query, the cross join would be flagged as [WARNING].