Azure Data Factory Managed Airflow integration with Purview to get Lineage

Question

I am trying to use Azure Data Factory managed Airflow for my use cases. I am trying to use Azure Purview as data governance tool and want to use airflow to emit lineage to it. I am trying to follow the Microsoft Document here. However, what i am stuck at is after i create event hub how do i configure the .yml file as Azure does it for me and i have no access to it. When i enter openlineage-airflow==1.2.0 that time airflow scheduler stopped working. and how can i add yml file in managed airflow

Answer 1

As it is mentioned in the documentation you need to configure your Azure Event Hubs instance as the target to which OpenLineage sends the events.Create an ‘openlineage.yml’ file under your Airflow root path. The content of the file is as below:

transport:
  type: "kafka"
  config:
    bootstrap.servers: "{EVENTHUB_SERVER}:9093"
    security.protocol: "SASL_SSL"
    sasl.mechanism: "PLAIN"
    sasl.username: "$ConnectionString"
    sasl.password: "{PASSWORD}"
    client.id: "airflow-client"
  topic: "microsoft_internal_openlineage"
  flash: True