Data flow - Could not load resource. Please ensure no mistakes in the JSON and that referenced resources exist.

Question

The JSON file which was working fine for creating a Data flow is not working now. It throws the error,
"Could not load resource. Please ensure no mistakes in the JSON and that referenced resources exist." on clicking OK to create the Data Flow.

{
    "name": "user_profiles_to_datalake",
    "properties": {
        "type": "MappingDataFlow",
        "typeProperties": {
            "sources": [
                {
                    "dataset": {
                        "referenceName": "asal400_ecommerce_userprofiles_source",
                        "type": "DatasetReference"
                    },
                    "name": "EcommerceUserProfiles"
                },
                {
                    "dataset": {
                        "referenceName": "asal400_customerprofile_cosmosdb",
                        "type": "DatasetReference"
                    },
                    "name": "UserProfiles"
                }
            ],
            "sinks": [
                {
                    "linkedService": {
                        "referenceName": "asadatalakeSUFFIX",
                        "type": "LinkedServiceReference"
                    },
                    "name": "DataLake"
                }
            ],
            "transformations": [
                {
                    "name": "userId"
                },
                {
                    "name": "UserTopProducts"
                },
                {
                    "name": "DerivedProductColumns"
                },
                {
                    "name": "UserPreferredProducts"
                },
                {
                    "name": "JoinTopProductsWithPreferredProducts"
                },
                {
                    "name": "DerivedColumnsForMerge"
                },
                {
                    "name": "Filter1"
                }
            ],
            "script": "source(output(\n\t\tvisitorId as string,\n\t\ttopProductPurchases as (productId as string, itemsPurchasedLast12Months as string)[]\n\t),\n\tallowSchemaDrift: true,\n\tvalidateSchema: false,\n\tignoreNoFilesFound: false,\n\tdocumentForm: 'arrayOfDocuments',\n\twildcardPaths:['online-user-profiles-02/*.json']) ~> EcommerceUserProfiles\nsource(output(\n\t\tcartId as string,\n\t\tpreferredProducts as integer[],\n\t\tproductReviews as (productId as integer, reviewDate as string, reviewText as string)[],\n\t\tuserId as integer\n\t),\n\tallowSchemaDrift: true,\n\tvalidateSchema: false,\n\tformat: 'document') ~> UserProfiles\nEcommerceUserProfiles derive(visitorId = toInteger(visitorId)) ~> userId\nuserId foldDown(unroll(topProductPurchases),\n\tmapColumn(\n\t\tvisitorId,\n\t\tproductId = topProductPurchases.productId,\n\t\titemsPurchasedLast12Months = topProductPurchases.itemsPurchasedLast12Months\n\t),\n\tskipDuplicateMapInputs: false,\n\tskipDuplicateMapOutputs: false) ~> UserTopProducts\nUserTopProducts derive(productId = toInteger(productId),\n\t\titemsPurchasedLast12Months = toInteger(itemsPurchasedLast12Months)) ~> DerivedProductColumns\nUserProfiles foldDown(unroll(preferredProducts),\n\tmapColumn(\n\t\tpreferredProductId = preferredProducts,\n\t\tuserId\n\t),\n\tskipDuplicateMapInputs: false,\n\tskipDuplicateMapOutputs: false) ~> UserPreferredProducts\nDerivedProductColumns, UserPreferredProducts join(visitorId == userId,\n\tjoinType:'outer',\n\tpartitionBy('hash', 30,\n\t\tproductId\n\t),\n\tbroadcast: 'left')~> JoinTopProductsWithPreferredProducts\nJoinTopProductsWithPreferredProducts derive(isTopProduct = toBoolean(iif(isNull(productId), 'false', 'true')),\n\t\tisPreferredProduct = toBoolean(iif(isNull(preferredProductId), 'false', 'true')),\n\t\tproductId = iif(isNull(productId), preferredProductId, productId),\n\t\tuserId = iif(isNull(userId), visitorId, userId)) ~> DerivedColumnsForMerge\nDerivedColumnsForMerge filter(!isNull(productId)) ~> Filter1\nFilter1 sink(allowSchemaDrift: true,\n\tvalidateSchema: false,\n\tformat: 'delta',\n\tcompressionType: 'snappy',\n\tcompressionLevel: 'Fastest',\n\tfileSystem: 'wwi-02',\n\tfolderPath: 'top-products',\n\ttruncate:true,\n\tmergeSchema: false,\n\tautoCompact: false,\n\toptimizedWrite: false,\n\tvacuum: 0,\n\tdeletable:false,\n\tinsertable:true,\n\tupdateable:false,\n\tupsertable:false,\n\tmapColumn(\n\t\tvisitorId,\n\t\tproductId,\n\t\titemsPurchasedLast12Months,\n\t\tpreferredProductId,\n\t\tuserId,\n\t\tisTopProduct,\n\t\tisPreferredProduct\n\t),\n\tskipDuplicateMapInputs: true,\n\tskipDuplicateMapOutputs: true) ~> DataLake"
        }
    }
}

Answer 1

This problem is happening by no reason. IT happens with Dataflows and with Pipelines.... and it does not have anything to do with the name of resources, or unexisting resources or anything.

Azure automatically creates some "id", "type" and "etag" which doesnt work. Breaking Dataflows and Pipelines, and making users waste a lot of time and money.

Yesterday 11 new data flows we created were ok, and today morning 6 of them are failing with this error. Same shit with the pipelines. The funniest thing is that when you try to import them, or add in JSON editor, it does not work either. With the id/type/etag or without. That feature is not working at all, so there is no way to create them fast.

PLEASE PROVIDE A SOLUTION AS SOON AS POSSIBLE, THIS IS NOT SERIOUS, IT IS A BUG IN AZURE SYNAPSE, PROBABLY AFFECTING THOUSANDS OF USERS.

THAT SHOULD HAVE BEEN REPORTED TO USERS AND SOLVED

Answer 2

Hey, the problem keeps happening. :(