Is it possible to create a dataflow without giving script lines using azure sdk in python? if yes please share a sample code creating a dataflow contains source, transformation and sink.

Question

Is it possible to create a dataflow without giving script lines using azure sdk in python? if yes please share a sample code creating a dataflow contains source, transformation and sink.

maruthi phanendhra 0

I need following dataflow as output using azure sdk in python.

User's image

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-06-06T09:38:15.9066667+00:00

Hi maruthi phanendhra, Just checking if below answer helps. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.

1 answer

Your answer

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-06-06T09:38:15.9066667+00:00

Hi maruthi phanendhra, Just checking if below answer helps. If yes, please consider hitting Accept Answer button. Accepted answers help community as well. Please let me know if any further queries. Thank you.

Answer 1

ShaikMaheer-MSFT 38,546 Microsoft Employee Moderator

Hi maruthi phanendhra,

Thank you for posting query in Microsoft Q&A Platform.

We have Python SDK which helps us to create dataflow with transformations. Kindly check below SDK document link and explore the modules and class and functions there and start your development.

https://learn.microsoft.com/en-us/python/api/azure-mgmt-datafactory/azure.mgmt.datafactory.operations.dataflowsoperations?view=azure-python

Above link points to dataflow operations SDK in python. Feel free to browse other menus there to explore other operations available too.

Between, could you please share details on what you mean by "without giving script lines"?

Please let me know how it goes. Thank you.

maruthi phanendhra 0

I have visited that website, but they provided scriptlines manually. Is it feasible to generate those scriptlines automatically in the dataflow?

I need to generate this script lines using azure sdk .

And also i am unable to connect source and sink using azure sdk .

code:

data_flow_name = "dataflow2"
    ls_name = 'AzureBlobStorage'
    ds_name = 'datain'
    ds_ls = LinkedServiceReference(type="LinkedServiceReference", reference_name=ls_name)

    data_flow_source_name = "source1"
    data_flow_source_description = "sourceDescription"

    data_flow_sink_name = "sink1"
    data_flow_sink_description = "sinkDescription"

    blob_path = 'newappcontainer/input/'
    blob_filename = 'student.xlsx'
    ds_azure_blob = DatasetResource(properties=AzureBlobDataset(
        linked_service_name=ds_ls, folder_path=blob_path, file_name=blob_filename))

    ds_in = client.datasets.get(resource_group_name, factory_name, ds_name, ds_azure_blob)

    source_datasetRefernce = DatasetReference(reference_name=ds_name, type="DatasetReference",
parameters={
'type': 'Excel',
'schema': [{'type': 'String'}, {'type': 'String'}],
"linked_service_name": {
'additional_properties': {},
'type': 'LinkedServiceReference',
'reference_name': 'AzureBlobStorage',
'parameters': None
},
"location": {'additional_properties': None,
'type': 'AzureBlobStorageLocation',
'folder_path': 'input', 'file_name': 'stu.csv',
'container': 'newappcontainer'}
}
)
    sink_datasetRefernce = DatasetReference(reference_name=ds_name, type="DatasetReference")

    linkedservice = LinkedServiceReference(type="LinkedServiceReference", reference_name="AzureBlobStorage")
    source = DataFlowSource(name=data_flow_source_name, description=data_flow_source_description,
dataset=source_datasetRefernce)
    print(source.dataset)
    sink = DataFlowSink(name=data_flow_sink_name, description=data_flow_sink_description,
                        dataset=sink_datasetRefernce)  
    # cond = "Age=='30'"
    filter = FilterActivity(name="filter4",condition=None, items=[source])
    trans = Transformation(name="filter3")

    mp = MappingDataFlow(
        type="MappingDataFlow",
        sources=[
            source
        ],
        sinks=[
            sink
        ],

        transformations=[
            source,
            trans,sink
        ],
        script_lines=[source,trans,sink],
        script=[] script_lines=[]

    )
    dataflowresource = DataFlowResource(
        name="dataflow1",
        properties=mp
    )
    response = client.data_flows.create_or_update(
        resource_group_name,
        factory_name,
        data_flow_name,
        data_flow=dataflowresource
    )
    response

Output script lines:


 "scriptLines": [
                "source(allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     ignoreNoFilesFound: false) ~> ",
                " sink(allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     input(",
                "          Column_1 as string,",
                "          Column_2 as string",
                "     ),",
                "     skipDuplicateMapInputs: true,",
                "     skipDuplicateMapOutputs: true) ~> sink1"
            ]
        }
    }

Required script lines:

 "source(output(",
                "          Column_1 as string,",
                "          Column_2 as string",
                "     ),",
                "     allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     ignoreNoFilesFound: false) ~> source1",
                "source1 filter(Column_1=='30') ~> filter1",
                "filter1 sink(allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     input(",
                "          Name as string,",
                "          Age as string",
                "     ),",
                "     skipDuplicateMapInputs: true,",
                "     skipDuplicateMapOutputs: true) ~> sink1"

ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-06-01T10:32:40.76+00:00

Hi maruthi phanendhra,

Sorry Its little unclear when you say script lines, what it actually means. If you mean the dataflow script which gets generated after your code deploys dataflow? If yes, then it's completely owned by platform, If gets generated based on how to developed your dataflow.

When you say "not able to connect source and sink", it means you are seeing any error? Kindly share error details as well.
Please note, we need to slowly understand SDK modules and functions and start writing code accordingly.
maruthi phanendhra 0 Reputation points

2023-06-01T14:36:39.6433333+00:00

Yes, the script lines you are referring to are the dataflow scripts that are generated after your code deploys a dataflow.is it possible to modify or generate those script using azure sdk.

error message:
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-06-03T10:40:24.9366667+00:00

Hi maruthi phanendhra, If i am not wrong, you cannot generate that scripts before your run your code and deploy dataflow. After you deploy dataflow only you can see script of that dataflow inside azure data factory studio.
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator

2023-06-05T09:23:11.88+00:00

Hi maruthi phanendhra, Just checking if you get chance to share updates on above? Thank you.

Share via

Is it possible to create a dataflow without giving script lines using azure sdk in python? if yes please share a sample code creating a dataflow contains source, transformation and sink.

1 answer

Your answer