columnMapping attribute in AtlasProcess not taking effect

Question

columnMapping attribute in AtlasProcess not taking effect

Sunita 62 Microsoft Employee

I referred this link https://datasmackdown.com/oss/pyapacheatlas/entities-with-lineage-column-mapping.htmlto setup columnMapping for an entity in purview using pycheatlas library, however, the mappings don't show up in Purview UI. All I can see is the columnMapping attribute show up in Properties section.

Below is my code snippet


type_spark_job = EntityTypeDef(
  name="synapse_spark_job_process",
  attributeDefs=[
    AtlasAttributeDef(name="columnMapping")
 superTypes = ["Process"])

col_map=[
    {"columnMapping":[
        {"Source":"QueueName", "Sink":"QName"},
        {"Source":"QueueMessage", "Sink":"QMessage"}],
     "DatasetMapping"  :{
        "Source":atlas_input_df.qualifiedName, "Sink":atlas_final_df.qualifiedName}
    }
]
process=AtlasProcess(
name="Synapse spark notebook",
typeName="synapse_spark_job_process",
qualified_name="test",
description="processes Queue Dimension",
guid=guid.get_guid(),
attributes = {"columnMapping":json.dumps(col_map)},
inputs = [{"guid":"<guid-1>"},{"guid":"<guid-2>"}],
outputs = [{"guid":"<guid-op>"}] 
)

client.upload_entities([process])

Sunita 62 Reputation points Microsoft Employee

2023-09-12T02:08:41.6866667+00:00

https://datasmackdown.com/oss/pyapacheatlas/entities-with-lineage-column-mapping.html
PRADEEPCHEEKATLA 91,866 Reputation points

2023-09-13T10:53:24.59+00:00

@Sunita - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .
Mateusz Wackowski 0 Reputation points

2023-11-09T15:34:28.68+00:00
Hey, I'm facing similar issue - when trying to insert a relationship with columnMapping attribute, a following error is returned:

HttpResponseError: Operation returned an invalid status 'Internal Server Error' Content: {"requestId":"xxx","errorCode":"InternalServerError","errorMessage":"There was an error processing your request. This could be a transient issue and you may re-run the operation. If it fails again continuously, contact customer support with ID xxx"}

Code:

from azure.purview.catalog import PurviewCatalogClient client = PurviewCatalogClient(endpoint=..., credential=..., logging_enable=True) relationhip = { "end1": { "guid": 'first_obj_guid', }, "end2": { "guid": 'sec_obj_guid', }, "typeName": "lineage_dataset", "attributes": { "datasetMapping": { "Source": "source_table_qualifiedName", "Sink": 'source_table_qualifiedName', }, "columnMapping": [ {"Source": "source_col_qualifiedName", "Sink": "sink_col_qualifiedName"} ], }, } client.relationship.create(relationship=relationhip)

Changing 'columnMapping' into 'ColumnMapping' makes the request pass, but no mapping is added.

The same code without 'attributes' field succeeds

2 answers

Your answer

Sunita 62 Reputation points Microsoft Employee

2023-09-12T02:08:41.6866667+00:00

https://datasmackdown.com/oss/pyapacheatlas/entities-with-lineage-column-mapping.html
PRADEEPCHEEKATLA 91,866 Reputation points

2023-09-13T10:53:24.59+00:00

@Sunita - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .
Mateusz Wackowski 0 Reputation points

2023-11-09T15:34:28.68+00:00

Hey, I'm facing similar issue - when trying to insert a relationship with columnMapping attribute, a following error is returned:

HttpResponseError: Operation returned an invalid status 'Internal Server Error' Content: {"requestId":"xxx","errorCode":"InternalServerError","errorMessage":"There was an error processing your request. This could be a transient issue and you may re-run the operation. If it fails again continuously, contact customer support with ID xxx"}

Code:

from azure.purview.catalog import PurviewCatalogClient client = PurviewCatalogClient(endpoint=..., credential=..., logging_enable=True) relationhip = { "end1": { "guid": 'first_obj_guid', }, "end2": { "guid": 'sec_obj_guid', }, "typeName": "lineage_dataset", "attributes": { "datasetMapping": { "Source": "source_table_qualifiedName", "Sink": 'source_table_qualifiedName', }, "columnMapping": [ {"Source": "source_col_qualifiedName", "Sink": "sink_col_qualifiedName"} ], }, } client.relationship.create(relationship=relationhip)

Changing 'columnMapping' into 'ColumnMapping' makes the request pass, but no mapping is added.

The same code without 'attributes' field succeeds

Answer 1

@Sunita - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Issue: columnMapping attribute in AtlasProcess not taking effect

Solution: Got this working by fixing the casing for columnMapping element(supposed to be ColumnMapping).

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Sunita 62 Microsoft Employee

got this working by fixing the casing for columnMapping element(supposed to be ColumnMapping)

0 comments

Share via

columnMapping attribute in AtlasProcess not taking effect

2 answers

Your answer