Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,444 questions
Data Factory throwing "java.io.IOException:Could not read footer: java.io.IOException" while preview data of parquet file in HDFS
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 22%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHD%3C/text%3E%3C/svg%3E)
Himanshu Devrani
11
Reputation points
I am trying to fetch data from On-Premise HDFS through copy activity of Azure data Factory. I have successfully connected with HDFS and while created dataset as well but while previewing data in copy activity I am getting " ****An error occurred when invoking java, message: java.io.IOException:Could not read footer: java.io.IOException: Could not read footer for file FileStatus{path=cibil_risk_1007_fw_fn_cc_20210731mis.parquet; isDirectory=false; length=84932951248; replication=-1; blocksize=2147483647; modification_time=0; access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false}****"
Can anyone help me to provide the resolution for this ???
I am attaching the snippet of the error as well for reference.
{count} votes
-
ShaikMaheer-MSFT 38,521 Reputation points Microsoft Employee2022-02-27T17:02:14.033+00:00 Hi @Himanshu Devrani ,
Thank you for posting query in Microsoft Q&A Platform.
As per my understanding you are trying to preview data for parquet file which is from On-Premise HDFS and ending with above issue. Please correct me if my understanding is wrong.
I did some research on this it seems this kind of error may occur because of may be parquet version problem or missing some required libraries on system etc..
You can also consider opening parquet file in local to check version of parquet file. At this point of time its hard to guess what is reason behind this error by seeing above screenshot. Could you please check if you can get parquet version details and share here?
I have also observed in some cases this issue may happen if we use SHIR version less than 5.3.7718.1. Hence, Could you please make sure to use SHIR version greater than 5.3.7718.1(Download Microsoft Integration Runtime from Official Microsoft Download Center).
Please let us know how it goes.
-
Himanshu Devrani 11 Reputation points
2022-02-28T07:01:13.393+00:00 Thanks for taking up the question.
I have checked the version of SHIR and it is greater than 5.3 which you have mentioned.
I am also getting the error while running the pipeline when I am trying to copy data from On premise Hadoop to data Lake storage.
As I am using ADF so what kind of libraries are required to fetch data because ADF is itself an ETL tool. So can you just elaborate more about this ???
Request you to provide a working solution asap because this is something very urgent. I am also attaching the snippet of error which I am getting after Executing the pipeline.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 75%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Pushpalatha Senapathi 1 Reputation point2022-03-02T08:31:20.333+00:00 Hi,
We are also facing the same issue. We are using Azure IR.
ErrorCode=ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.io.IOException:Could not read footer: java.io.IOException: Could not read footer for file FileStatus{path=dealer_target_group.parquet; isDirectory=false; length=6961; replication=-1; blocksize=2147483647; modification_time=0; access_time=0; owner=; group=; permission=rw-rw-rw-; isSymlink=false}
total entry:7java.io.IOException:can not read class org.apache.parquet.format.FileMetaData: Cannot read. Remote side has closed. Tried to read 1 bytes, but only got 0 bytes. (This is often indicative of an internal error on the server side. Please check your server logs.)
total entry:15 -
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 44%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Prieto Montes de Oca, Pablo (Allianz Technology) 0 Reputation points2023-10-18T12:25:09.6266667+00:00 Was this ever fixed? We're facing the same issue.
Sign in to comment
Sign in to answer