Hello @Andrew Macinnes ,
Thanks for the question and using MS Q&A platform.
*1) Configure the D365 sink to do it. e.g configure th sink in order to ignore records that match on a certain column? Is it possible to configure the sink in some way to accomplish this? *
Copy Activity: Yes, for this you can use Copy activity and use writeBehavior
properties as Upsert
in your sink settings. Please refer to below section of the docs
Ref Doc: Dynamics as a sink type
Mapping Data flow: If you would like to use Mapping Data flow, you will have to use alter transformation and then in sink transformation you can use Update method
as upsert
(please note by default it will be insert
)
Ref doc: Mapping data flow Sink transformation
Here are few demo videos by community volunteers that will be helpful:
- Azure Data Factory - Implement Upsert logic in Mapping data flow.
- Alter Row Transformation in Mapping Data Flow in Azure Data Factory
*2) Pull in the D365 table/entity as a source and use it to filter my source CSV to remove user records that already exist in D365 perhaps by using a common field such as fullname to identify such records. This would ensure I only try to import new users. *
For this implementation, you can use the Change data capture logic using Mapping data flow as described in this demo by Mark Krome from ADF Product team: Data Flows: How to capture changed data
Hope this will help. Please let us know if any further queries.
------------------------------
- Please don't forget to click on
or upvote
button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
- Want a reminder to come back and check responses? Here is how to subscribe to a notification
- If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators