Loading CSV File From ADLS to Azure Sql As a table and update data in csv file it should in Azure SQL as well without repetating the previous data which i loaded?

Veerareddy Gangula 1 Reputation point
2022-08-19T15:14:46.823+00:00

The Query is

I am loading CSV file from ADLS Gen 2 lake to Azure SQL aa Table

It is working fine

After that when i added a row or updated values in CSV file

The data which i loaded at start it is repeating gain and updating one also saving

Example of above situation

For example

I having a CSV file like below

Name Age
Veera ,22
Mahesh, 47

When i loaded it above file it is loaded into Azure SQL as a table

When i add a row to above SQL file like below

Maheer 27

Then the table in Azure looks like

Name Age
Veera, 22
Mahesh,47
Veera,22
Mahesh 47
Maheer 27

Above one is problem

I want like this

Name Age
Veera,22
Mahesh,47
Maheer,27

I want like above table

Can you please look into it and if possible share any vedio or links

Thanks in Advance

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,082 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Veerareddy Gangula 1 Reputation point
    2022-08-25T05:29:58.717+00:00

    {
    "referenceName": "DS_SRC",
    "type": "DatasetReference",
    "parameters": {
    "filename": {
    "value": "@item().name",
    "type": "Expression"
    }

    @
    }
    }
    ],
    "outputs": [
    {
    "referenceName": "AzureSqlTable1",
    "type": "DatasetReference",
    "parameters": {
    "tablename": {
    "value": "@split(item().name,'.')[0]",
    "type": "Expression"
    }
    }
    }
    ]
    }
    ]
    }
    }
    ],
    "variables": {
    "tablename": {
    "type": "String"
    },
    "query": {
    "type": "String"
    }
    },
    "annotations": [],
    "lastPublishTime": "2022-08-23T05:03:01Z"
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
    }

    @Nandan Hegde @ShaikMaheer-MSFT

    0 comments No comments