Join error

arkiboys 9,686 Reputation points
2021-10-08T07:46:46.64+00:00

Do you see why I get the error in the first Join which is Join1?
'Join1': Non-equality joins should have broadcasted one of the join sides

Thank you

138843-image.png
138835-image.png
138855-image.png
138856-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,131 questions
0 comments No comments
{count} votes

Accepted answer
  1. MarkKromer-MSFT 5,206 Reputation points Microsoft Employee
    2021-10-08T18:43:22.577+00:00

    Non-equi joins require broadcast join in Spark in order to resolve the non-equality. Go into the Optimize tab on your join and select Broadcast Fixed, then choose the Fixed broadcast option.


2 additional answers

Sort by: Most helpful
  1. Ali Asgar 11 Reputation points Microsoft Employee
    2022-03-16T17:35:08.957+00:00

    For anyone who stumbles upon here trying to find an answer:

    The underlying problem (and thus the real question) is how to correctly use dynamic columns in lookup or join activities in Data Flow transformations. The following screenshot answers this question.

    183817-image.png

    You need to use expr and concat functions to correctly use parameters. The 'Source@" and 'Destination@' strings refer to the names of the Data Flow activities that feed into the Join/Lookup.

    2 people found this answer helpful.
    0 comments No comments

  2. Mark Kromer MSFT 1,146 Reputation points
    2021-11-15T17:46:00.517+00:00

    Does the error clear when you set the broadcasting on?