gRPC connection errors in copy activity

Hamza Outa 61 Reputation points
2024-09-20T07:24:28.6333333+00:00

I have a copy data activity in ADF that copies from Google BigQuery to Blob storage. Occasionaly I get one of these errors:

"grpc_message":"End of TCP stream"
"grpc_message":"keepalive watchdog timeout"

We are running multiple copy datas parallel through a foreach loop, and sometimes this happens. So of the 5 parallell copies only 1 of them fails every so often.

we checked on BigQuery and don't see any issues.

Why is this happening and how can it be fixed?

here's the full error:

ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'CSV_NAME_00060.csv'.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DI.Connector.GoogleBigQuery.ExceptionUtils.GoogleBigQueryConnectorException,Message=Error occurred when reading, ErrorCode: UnknownError,Source=Microsoft.DI.Connector.GoogleBigQuery,''Type=Grpc.Core.RpcException,Message=Status(StatusCode="Unavailable", Detail="End of TCP stream", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1726772029.383000000","description":"Error received from peer ipv4:<IP:PORT>","file":"..\..\..\src\core\lib\surface\call.cc","file_line":953,"grpc_message":"End of TCP stream","grpc_status":14}"),Source=mscorlib,''Type=Grpc.Core.Internal.CoreErrorDetailException,Message={"created":"@1726772029.383000000","description":"Error received from peer ipv4:<same_IP:same_PORT>","file":"..\..\..\src\core\lib\surface\call.cc","file_line":953,"grpc_message":"End of TCP stream","grpc_status":14},Source=,'

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,715 questions
{count} votes

1 answer

Sort by: Most helpful
  1. phemanth 10,555 Reputation points Microsoft Vendor
    2024-09-20T11:23:22.6933333+00:00

    @Hamza Outa

    Thanks for reaching out to Microsoft Q&A

    It sounds like you’re encountering intermittent network issues when copying data from Google BigQuery to Blob storage. The errors “End of TCP stream” and “keepalive watchdog timeout” suggest that the gRPC connection between your ADF pipeline and BigQuery is being interrupted.

    Possible Causes

    Network Instability: Temporary network issues can cause the gRPC connection to drop.

    BigQuery Connection Limits: BigQuery might be closing idle connections or hitting connection limits.

    Resource Contention: Running multiple parallel copy activities might be causing resource contention, leading to timeouts.

    Solutions

    1. Retry Logic: Implement retry logic in your ADF pipeline to handle transient errors. This can help mitigate the impact of temporary network issues.
    2. Increase Timeout Settings: Adjust the timeout settings for your gRPC connections to allow more time for the operations to complete.
    3. Optimize Parallelism: Reduce the number of parallel copy activities to see if it alleviates the issue. You can gradually increase the parallelism to find a balance.
    4. Monitor Network Health: Use network monitoring tools to check for any instability or issues in your network that might be causing the interruptions.
    5. BigQuery API Usage: Ensure you are using the most efficient API for your use case. The BigQuery Storage Read API might offer better performance and reliability for large data transfers.

    Example Retry Logic in ADF

    {
        "name": "CopyData",
        "type": "Copy",
        "policy": {
            "retry": 3,
            "retryIntervalInSeconds": 30
        },
        "source": {
            "type": "BigQuerySource",
            "query": "SELECT * FROM your_table"
        },
        "sink": {
            "type": "BlobSink",
            "blobPath": "your_blob_path"
        }
    }
    
    
    

    This configuration will retry the copy operation up to three times with a 30-second interval between attempts.

    Hope this helps. Do let us know if you any further queries.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.