Json SerDe hive query failing with HDInsight version 4.0

Question

Hi All,

I have a hive query which is as follows:

ADD JAR ${hiveconf:JsonSerde};
set hive.execution.engine=tez;

DROP TABLE IF EXISTS Test;
CREATE EXTERNAL TABLE Test(
Results array<
struct<requestid:string,result:array<
struct<id:int, activityDate:string, activityTypeId:int, AttributeValueId:int, AttributeValue:string, attributes:array<
struct<name:string, value:string>

            >  

          >  

        >  

      >  

)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED AS TEXTFILE LOCATION '${hiveconf:Test}';

This jar file is referenced, json-serde-1.3.6-jar-with-dependencies.jar

This jar file and Test file is placed in a blob location.

In ADF, i have an on demand HDInsight Linked service with following configurations:
Cluster Type -- hadoop
Cluster size -- 2
Version -- 3.6

My pipeline has Hive activity and this works successfully with the above configurations.

But when i update the linked service configuration to Version -- 4.0 my pipeline is failing and after checking the error status log file placed in blob location following error is mentioned,

INFO : Starting task [Stage-0:DDL] in serial mode
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe
INFO : Completed executing command(queryId=hive_2022........); Time taken: 0.104 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe (state=08S01,code=1)

Do we need to make changes to Jar file when version is upgraded from 3.6 to 4.0?

Is there some thing else that needs to be changed for 4.0 version?