Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 38%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGB%3C/text%3E%3C/svg%3E)
i have same issue with .parquet files any solution tutorial videos for blob parquet files to azure sql?
Copy activity is failing while copying data from ADLS to Sqldb
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 56.00000000000001%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sagar
21
Reputation points
Could you please help me to solve this issue
ErrorCode=ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.OutOfMemoryError:Unable to retrieve Java exception.
total entry:6
sun.misc.Unsafe.allocateMemory(Native Method)
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
com.microsoft.datatransfer.bridge.parquet.ParquetBatchReaderBridge.<init>(ParquetBatchReaderBridge.java:87)
com.microsoft.datatransfer.bridge.parquet.ParquetBatchReaderBridge.open(ParquetBatchReaderBridge.java:63)
com.microsoft.datatransfer.bridge.parquet.ParquetFileBridge.createReader(ParquetFileBridge.java:22)
.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'
Azure Data Factory
3 answers
Sort by: Most helpful
-
Sagar 21 Reputation points
2022-08-03T05:26:16.233+00:00 Today it's showing same error as below
ErrorCode=ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.io.IOException:Could not read footer: java.lang.OutOfMemoryError
total entry:7
org.apache.parquet.hadoop.ParquetFileReader.readAllFootersInParallel(ParquetFileReader.java:269)
org.apache.parquet.hadoop.ParquetFileReader.readAllFootersInParallelUsingSummaryFiles(ParquetFileReader.java:210)
org.apache.parquet.hadoop.ParquetReader.<init>(ParquetReader.java:112)
org.apache.parquet.hadoop.ParquetReader.<init>(ParquetReader.java:45)
org.apache.parquet.hadoop.ParquetReader$Builder.build(ParquetReader.java:202)
com.microsoft.datatransfer.bridge.parquet.ParquetBatchReaderBridge.open(ParquetBatchReaderBridge.java:62)
com.microsoft.datatransfer.bridge.parquet.ParquetFileBridge.createReader(ParquetFileBridge.java:22)
java.lang.OutOfMemoryError:null
total entry:16
sun.misc.Unsafe.allocateMemory(Native Method)
java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
com.microsoft.datatransfer.bridge.io.parquet.BridgeInputFileStream.<init>(BridgeInputFileStream.java:21)
com.microsoft.datatransfer.bridge.io.parquet.CSharpStreamFile.getInputStream(CSharpStreamFile.java:29)
com.microsoft.datatransfer.bridge.io.parquet.BridgeFileSystem.open(BridgeFileSystem.java:167)
com.microsoft.datatransfer.bridge.io.parquet.BridgeFileSystem.open(BridgeFileSystem.java:174)
org.apache.parquet.hadoop.util.HadoopInputFile.newStream(HadoopInputFile.java:65)
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:459)
org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:437)
org.apache.parquet.hadoop.ParquetFileReader$2.call(ParquetFileReader.java:259)
org.apache.parquet.hadoop.ParquetFileReader$2.call(ParquetFileReader.java:255)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'-
ShaikMaheer-MSFT 38,636 Reputation points Microsoft Employee Moderator2022-08-05T06:37:29.58+00:00 Hi @Sagar ,
Thanks for posting query in Microsoft Q&A Platform.
Is kind of error will come if some wrong with parquet file i guess. especially with version.
Is your getting this error with any specific file or all files? I tried to copy some dummy parquet file my end and I dont see any issues. I feel its more about something wrong with your file.
If other parquet files are good to process and then see version of them and try to get error file also in similar version to process.
Please let us know how it goes. Thank you.
-
ShaikMaheer-MSFT 38,636 Reputation points Microsoft Employee Moderator
2022-08-08T06:32:15.527+00:00 Hi @Sagar , Just checking if you get chance to share updates. Thank you.
Sign in to comment -
-
Nasreen Akter 10,896 Reputation points Volunteer Moderator2022-08-02T20:24:18.247+00:00 Hi @Sagar ,
What is the current
Data integration unitfor the CopyActivity? Would you please increase the units and try again. Thanks!
