This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Do you see why I get the error in the first Join which is Join1?
'Join1': Non-equality joins should have broadcasted one of the join sides
Thank you
Non-equi joins require broadcast join in Spark in order to resolve the non-equality. Go into the Optimize tab on your join and select Broadcast Fixed, then choose the Fixed broadcast option.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 12%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOK%3C/text%3E%3C/svg%3E)
Hi, I don't understand why the error says that the join is Non-equal, it is equal indeed. I'm having the same error when trying to execute the same task (joining 2 files), the error appears on every join type I select (inner, left, etc.). I want to pass parameters from pipeline, here just set them in the df: ![149447-image.png][1] Join step: ![149456-image.png][2] If I specify column name explicitly the join works fine. Is this due to some limitation related to parameters using in the condition? Please advise. [1]: /api/attachments/149447-image.png?platform=QnA [2]: /api/attachments/149456-image.png?platform=QnA
For anyone who stumbles upon here trying to find an answer:
The underlying problem (and thus the real question) is how to correctly use dynamic columns in lookup or join activities in Data Flow transformations. The following screenshot answers this question.
You need to use expr and concat functions to correctly use parameters. The 'Source@" and 'Destination@' strings refer to the names of the Data Flow activities that feed into the Join/Lookup.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Does the error clear when you set the broadcasting on?
yes, only when I mark both left and right broadcasted, but in Data preview I observe that it produces cartesian..
It looks like error message is re-using an overly broad string. If ADF is observing/returning a cartesian product, that would be why it is requiring you to explicitly set a broadcast. It is essentially forcing you to ack that is what you intended.
The case is that the result is not supposed to be cartesian - when I set the field names explicitly it returns expected outcome of several rows;
I don't mind using broadcast if it's required for my case, but it doesn't explain why the result is wrong
It is likely that your parameters are not being set properly, or perhaps ADF is not interpreting them correctly, resulting in the cartesian product. Can you open a support case from the Azure portal so that we can gather your support files to have a look?
Source datasets are pretty simple - similar fields set: ID + descriptive fields from both sides, I will narrow them just having ID field and will retest trying to find why parameters values, which hold ID value, are treated incorrectly, and will post about results.
I believe it is not the issue for support case, the answer should be hidden somewhere nearby :)
Itlooks like, AKOBYLKA has hijacked my thread? :-)
Yes, no copyrights existed here :)
As for the issue I found workaround: put the interim Derived Column step, pass parameters for IDs into it and then use output for Join condition.
In this case equal join works as intended.