Azure pipeline failure while coping data to parquet from SFTP

Question

Azure pipeline failure while coping data to parquet from SFTP

Ashu Anand 31

I am facing an issue while copying data from a SFTP foder to parquet file while using ADF pipelines and SHIR. The same integration runtime is running perfectly fine while copying data from on-premise SQL database. The issue is coming while copying from a SFTP source.

Error Message : ErrorCode=JniException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking Java Native Interface.,Source=Microsoft.DataTransfer.Common,''Type=JNI.JniReturnException,Message=Cannot create JVM: JNI return code [-1][JNI call failed due to unknown JNI error.],Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'

Any idea what the issue could be?

AnnuKumari-MSFT 19,116 Reputation points Microsoft Employee

2023-01-09T09:37:50.087+00:00

Hi @Ashu Anand ,
Thankyou for your confirmation on the current status on the issue you were facing. Glad to know that you are no longer landing on the error anymore. I would like you to kindly consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well.

AnnuKumari-MSFT 19,116 Reputation points Microsoft Employee

2023-01-09T09:37:50.087+00:00

Hi @Ashu Anand ,
Thankyou for your confirmation on the current status on the issue you were facing. Glad to know that you are no longer landing on the error anymore. I would like you to kindly consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well.

Answer 1

AnnuKumari-MSFT 19,116 Microsoft Employee

Hi @Ashu Anand ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

As I understand your issue, you are facing JNIException issue while copying data from SFTP to ADLS as parquet file. Please let me know if my understanding is incorrect.

If the error message contains "Cannot create JVM" , the possible cause is that JVM can't be created because some illegal (global) arguments are set.

Log in to the machine that hosts each node of your self-hosted integration runtime. Check to ensure that the system variable is set correctly, as follows: _JAVA_OPTIONS "-Xms256m -Xmx16g" with memory bigger than 8G. Restart all the integration runtime nodes, and then rerun the pipeline.

Source : Troubleshoot guide for Error code: JniException

Additional resource for troubleshoot: Using Self-hosted Integration Runtime

Hope this will help. Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you.
Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Ashu Anand 31 Reputation points

2023-01-02T05:33:10.477+00:00

HI @AnnuKumari-MSFT

Thanks for taking out your time and replying on the above issue being faced. Your understanding on the issue is correct.
I have shared the error message and I cant see any possible cause as some illegal arguments. The message just says JNI return code [-1][JNI call failed due to unknown JNI error.]
Also this issue is being faced by only some random files. If I have to copy around 100 files from SFTP, some random 50-60 files are facing the above error and other are copying perfectly fine.
I think if the issue would be with IR, then all the copy activity would have failed. Please do let me know your thoughts on the same.

Thanks
AnnuKumari-MSFT 19,116 Reputation points Microsoft Employee

2023-01-03T09:51:35.703+00:00

Hi @Ashu Anand ,

Have you checked the System Variables for the JVM/JRE are returning correctly? The issue might be with the memory limits on the JRE, but frankly usually you get an error message that is more specific. How big are the files? And what format are they?
Ashu Anand 31 Reputation points

2023-01-04T07:39:36.85+00:00

Hi @AnnuKumari-MSFT

Actually we dont have the access for the client machine on which the IRT is setup, so we cant check the system variables or restart the machine. However, I tried to rerun the copy activity after some time and the failed activities were successful after one or two tries later. It seems some intermittent issue, not sure exactly.
All the files which were giving error were csv files and the size were in order of around some MBs (max around 100 MB).
I will update again on next Friday if we some issue again as this activity happens on weekly cadence every Friday.

Thanks
Ashu Anand 31 Reputation points

2023-01-06T10:22:18.093+00:00

Hi @AnnuKumari-MSFT

There was no issue in today's pipeline run and copy activity.

Thanks
AnnuKumari-MSFT 19,116 Reputation points Microsoft Employee

2023-01-13T09:05:56.1466667+00:00

Hi @Ashu Anand ,

Since you are no longer facing the issue, requesting you to kindly do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

Azure pipeline failure while coping data to parquet from SFTP

0 additional answers

Activity