How to fix the copy activity error while copying data from databricks delta table to datalake in csv format

Question

There are some error tables in Databricks delta table . Those tables need to extracted as csv and load in azure data lake , inside the folder of the container.

Staging has been enabled in the copy activity since it is 2 step process.

Approx row count of the tables - 50k.

While the copy activity runs , it fails with the error .

ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Only 'http' and 'https' schemes are allowed.

Parameter name: value. Account: 'amledpstoragedev'. FileSystem: 'edp-dev'. Path: 'bronze/temp/9aeaa750-ea7c-40e0-8b76-eefbad013ae0/AzureDatabricksDeltaLakeExportCommand'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.ArgumentException,Message=Only 'http' and 'https' schemes are allowed.

Parameter name: value,Source=System.Net.Http,'

could you please advise to resolve the issue.

Answer 1

Can you verify if your Linked Service in ADF for ADLS Gen2 is correctly configured with the proper URL scheme (https) ?

It should look something like this:

https://<account_name>.dfs.core.windows.net

Here's an example JSON configuration snippet for the copy activity, ensuring that https is used:

{

    "name": "CopyDeltaToADLS",

    "type": "Copy",

    "typeProperties": {

        "source": {

            "type": "DeltaLakeSource"

        },

        "sink": {

            "type": "DelimitedTextSink",

            "storeSettings": {

                "type": "AzureBlobFSWriteSettings",

                "container": "your-container-name",

                "path": "bronze/temp/",

                "formatSettings": {

                    "type": "DelimitedTextFormat",

                    "columnDelimiter": ",",

                    "rowDelimiter": "\n"

                }

            }

        },

        "enableStaging": true,

        "stagingSettings": {

            "linkedServiceName": {

                "referenceName": "StagingLinkedService",

                "type": "LinkedServiceReference"

            },

            "path": "staging/temp/"

        }

    },

    "linkedServiceName": {

        "referenceName": "AzureDataLakeLinkedService",

        "type": "LinkedServiceReference"

    }

}