Partager via

HIVE external tables loading partitions

  • Article

With HIVE managed tables, you can use MSCK REPAIR TABLE. Managed tables creates a directory for each partition with format "{parititionname}={value}".

HIVE doesn't create/manage storage for EXTERNAL tables. If the directory structure is similar to format "{parititionname}={value}", then you can use MSCK REPIR TABLE command. Otherwise you are left with loading data into managed table or the option of manually adding the partitions which is tiresome.

 

For example if the directory is like below

You need to create statements like

ALTER TABLE PartitionedTable ADD PARTITION (log_date='2015-08-02') location .../2015-08-02';

ALTER TABLE PartitionedTable ADD PARTITION (log_date='2015-08-03') location .../2015-08-03';

...

 

In such cases python script shared at https://github.com/kirankumarkolli/hdinsightsamples/blob/master/LoadPartitions.py can be used to generate them given a table name.

    Usage: LoadPartitions.py <tablename>

Run it inside cluster on a command prompt.