Json SerDe hive query failing with HDInsight version 4.0

Priya Jha 866 Reputation points
2022-10-13T14:30:10.807+00:00

Hi All,

I have a hive query which is as follows:

ADD JAR ${hiveconf:JsonSerde};
set hive.execution.engine=tez;

DROP TABLE IF EXISTS Test;
CREATE EXTERNAL TABLE Test(
Results array<
struct<requestid:string,result:array<
struct<id:int, activityDate:string, activityTypeId:int, AttributeValueId:int, AttributeValue:string, attributes:array<
struct<name:string, value:string>

            >  

          >  

        >  

      >  

)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED AS TEXTFILE LOCATION '${hiveconf:Test}';

This jar file is referenced, json-serde-1.3.6-jar-with-dependencies.jar

This jar file and Test file is placed in a blob location.

In ADF, i have an on demand HDInsight Linked service with following configurations:
Cluster Type -- hadoop
Cluster size -- 2
Version -- 3.6

My pipeline has Hive activity and this works successfully with the above configurations.

But when i update the linked service configuration to Version -- 4.0 my pipeline is failing and after checking the error status log file placed in blob location following error is mentioned,

INFO : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe
INFO : Completed executing command(queryId=hive_2022........); Time taken: 0.104 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe (state=08S01,code=1)

Do we need to make changes to Jar file when version is upgraded from 3.6 to 4.0?

Is there some thing else that needs to be changed for 4.0 version?

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,692 questions
{count} votes