Json SerDe hive query failing with HDInsight version 4.0
Hi All,
I have a hive query which is as follows:
ADD JAR ${hiveconf:JsonSerde};
set hive.execution.engine=tez;
DROP TABLE IF EXISTS Test;
CREATE EXTERNAL TABLE Test(
Results array<
struct<requestid:string,result:array<
struct<id:int, activityDate:string, activityTypeId:int, AttributeValueId:int, AttributeValue:string, attributes:array<
struct<name:string, value:string>
>
>
>
>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED AS TEXTFILE LOCATION '${hiveconf:Test}';
This jar file is referenced, json-serde-1.3.6-jar-with-dependencies.jar
This jar file and Test file is placed in a blob location.
In ADF, i have an on demand HDInsight Linked service with following configurations:
Cluster Type -- hadoop
Cluster size -- 2
Version -- 3.6
My pipeline has Hive activity and this works successfully with the above configurations.
But when i update the linked service configuration to Version -- 4.0 my pipeline is failing and after checking the error status log file placed in blob location following error is mentioned,
INFO : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe
INFO : Completed executing command(queryId=hive_2022........); Time taken: 0.104 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe (state=08S01,code=1)
Do we need to make changes to Jar file when version is upgraded from 3.6 to 4.0?
Is there some thing else that needs to be changed for 4.0 version?