HIVE doesn't create/manage storage for EXTERNAL tables. If the directory structure is similar to format "{parititionname}={value}", then you can use MSCK REPIR TABLE command. Otherwise you are left with loading data into managed table or the option of manually adding the partitions which is tiresome.

For example if the directory is like below

You need to create statements like

ALTER TABLE PartitionedTable ADD PARTITION (log_date='2015-08-02') location .../2015-08-02';

ALTER TABLE PartitionedTable ADD PARTITION (log_date='2015-08-03') location .../2015-08-03';

...

In such cases python script shared at https://github.com/kirankumarkolli/hdinsightsamples/blob/master/LoadPartitions.py can be used to generate them given a table name.

    Usage: LoadPartitions.py <tablename>

Run it inside cluster on a command prompt.

(NOTE: YARN UI is currently available only on newly created clusters)

Click on “Query Console” and click-through the highlighted RED sections.

“Query Console” asks for cluster credentials (credentials specified at the time of cluster creation), please enter them. 

It’s always better to open YARN UI in a new window for good experience. Click “Open in new tab”.

You have YANR status page now available in portal. See below metrics of a job outside of cluster.

If you want a job level granularity PIG CLI doesn't support it yet. But can achieve it if the jobs are submitted through Templeton.

Example of PowerShell usage

New-AzureHDInsightPigJobDefinition -Query $QueryString -StatusFolder $statusFolder -Arguments @("-x”, “tez")

 Example of CURL usage

curl -s -d file=<file name> -d arg=-v -d arg=-x -d arg=tez 'https://<dnsname.azurehdinsight.net>/templeton/v1/pig?user.name=admin'

Beeline is a command shell which can be used to interact with hive server. On HdInsight if you try it you might see below failure

hdiuser@headnode0:~$ beeline

Beeline version 0.14.0.2.2.7.1-10 by Apache Hive

beeline> !connect jdbc:hive2://headnodehost:10001 admin

scan complete in 12ms

Connecting to jdbc:hive2://headnodehost:10001

Enter password for jdbc:hive2://headnodehost:10001: **********

….

Java heap space

0: jdbc:hive2://headnodehost:10001 (closed)> !quit

Java heap space

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

        at org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:181)

        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:288)

        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:190)

        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)

        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)

        at java.sql.DriverManager.getConnection(DriverManager.java:571)

        at java.sql.DriverManager.getConnection(DriverManager.java:187)

        at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:138)

        at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:179)

        at org.apache.hive.beeline.Commands.close(Commands.java:916)

        at org.apache.hive.beeline.Commands.closeall(Commands.java:898)

        at org.apache.hive.beeline.BeeLine.close(BeeLine.java:814)

        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:764)

        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:476)

        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:459)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

hdiuser@headnode0:~$

HDInsihgt hive server is configured in HTTP mode whereas beeline is trying to communicate in TCP mode resulting this failure. Using transportMode in connect parameters will resolve the issue. Here is the sample example

hdiuser@headnode0:~$ beeline --verbose=true

Beeline version 0.14.0.2.2.7.1-10 by Apache Hive

beeline> !connect jdbc:hive2://headnodehost:10001/;transportMode=http admin

scan complete in 13ms

Connecting to jdbc:hive2://headnodehost:10001/;transportMode=http

Enter password for jdbc:hive2://headnodehost:10001/;transportMode=http: **********

Connected to: Apache Hive (version 0.14.0.2.2.7.1-10)

Driver: Hive JDBC (version 0.14.0.2.2.7.1-10)

Transaction isolation: TRANSACTION_REPEATABLE_READ

0: jdbc:hive2://headnodehost:10001/>

Here is the stackoverflow post for reference https://stackoverflow.com/questions/32661771/apache-beeline-throws-java-heap-space-error-on-hdinsight-clusters/32661772#32661772

Script action is currently shared at  https://kkhdiprodonly.blob.core.windows.net/scriptactions/MySqlSqoopConnector.ps1

CAUTION:

1. Above script action is only for testing purposes. Otherwise recommendation is to copy and re-host it.

2. Update the MySql version according to your needs. Currently Its authored to install mysql-connector-java-5.0.8 version.

How to use it:

1. Select “Custom Create” option

2. 

3. On last screen of the wizard Fill the script URI (NOTE: Make sure “All nodes” is selected for “NODE TYPE”