Schema drift is a bit antithetical to the concepts behind a well-defined schema format like CDM, so it is not currently enabled in ADF for CDM scenarios. That being said, we are looking at opening up ways to allow schema changes in a more dynamic CDM implementation. Will announce some changes in this regard soon.
Common Data Model as inline dataset in ADF mapping data flow does not support dynamic schema
It appears that there is not a way to ingest from Common Data Model as a source in mapping data flow using dynamic schema. The "Allow Schema Drift" option is not exposed for the Common Data Model inline dataset, and validation does not allow the Projection to be blank. Validation will provide error similar to:
Is this a bug or just a missing feature? Ideally, we want to by dynamic / metadata-driven. Is this not possibly when using Common Data Model as a source? Must we create a separate explicit mapping data flow for each entity we wish to ingest from the Common Data Model?
Azure Data Factory
3 answers
Sort by: Most helpful
-
-
Scott Hietpas 11 Reputation points
2021-05-10T12:24:39.763+00:00 The real goal is to iterate over the entities using an automatic/dynamic schema mapping. Otherwise we must create an explicit mapping for each entity. For most ingestion, we attempt to be table-driven or metadata-driven so new entities can easily be added/removed via configuration data without requiring a new deployment. For now we can workaround using an explicit mapping data flow per entity. I just wanted to confirm that this was a current limitation before moving away from the dynamic approach.
-
Andrey Grigorev 0 Reputation points
2023-06-12T07:58:16.4433333+00:00 Hello, is it possible to use parameters in data source of type "Inline set Common Data Service" for settings of that service? I mean - its schema remais same, but I want to be capable to configure location of the CDM and all releant folders by parameters.