Error saying column is missing during join but column exists

Question

Error saying column is missing during join but column exists

John Neubecker 6

Spark job failed: {
"text/plain": "{\"runId\":\"32774401-5cff-4c65-a3aa-16ec5605c599\",\"sessionId\":\"587d97f6-a884-4b45-854b-7458906c413a\",\"status\":\"Failed\",\"payload\":{\"statusCode\":400,\"shortMessage\":\"DF-EXPR-010 at Join 'JoinWithExistingEntities'(Line 28/Col 95): Column 'extract_set_id' not found. The stream is either not connected or column is unavailable\",\"detailedMessage\":\"Failure 2022-08-09 19:41:41.145 failed DebugManager.processJob, run=32774401-5cff-4c65-a3aa-16ec5605c599, errorMessage=DF-EXPR-010 at Join 'JoinWithExistingEntities'(Line 28/Col 95): Column 'extract_set_id' not found. The stream is either not connected or column is unavailable\"}}\n"
} - RunId: 32774401-5cff-4c65-a3aa-16ec5605c599

The extract_set_id does exist. It's part of the left side of the join and is available in the dropdown and shows up in the data preview for the left side stream. I've tried deleting and recreating the join transformation and get the same error.

John Neubecker 6 Reputation points

2022-08-09T19:47:34.69+00:00

This was also working and then stopped working with me making no changes to the data flow.
John Neubecker 6 Reputation points

2022-08-09T19:56:03.577+00:00

The same issue is happening with other data flows that have been working and haven't been updated in weeks, the factory is backed by git so I can see that nothing was changed. Was an update released recently that could break existing data flows?
John Neubecker 6 Reputation points

2022-08-09T20:36:48.763+00:00

If I add a derive column transformation before the join and use byName it works. It doesn't work if byName isn't used. Even though extract_set_id is in the column list of the input schema. Without byName I get the same error I posted above in the derived column preview.
John Neubecker 6 Reputation points

2022-08-09T20:51:37.3+00:00

After some more testing it looks like the issue is that any data that is mapped as input to a flowlet and then output cannot be used in any subsequent transformation unless it is "readded" via a derived column or join to a transformation from before the flowlet ran. It's not specific to join transformations. The only explanation for this is that there was an update recently to the underlying code for flowlets that caused this bug.

1 answer

Your answer

John Neubecker 6 Reputation points

2022-08-09T19:47:34.69+00:00

This was also working and then stopped working with me making no changes to the data flow.
John Neubecker 6 Reputation points

2022-08-09T19:56:03.577+00:00

The same issue is happening with other data flows that have been working and haven't been updated in weeks, the factory is backed by git so I can see that nothing was changed. Was an update released recently that could break existing data flows?
John Neubecker 6 Reputation points

2022-08-09T20:36:48.763+00:00

If I add a derive column transformation before the join and use byName it works. It doesn't work if byName isn't used. Even though extract_set_id is in the column list of the input schema. Without byName I get the same error I posted above in the derived column preview.
John Neubecker 6 Reputation points

2022-08-09T20:51:37.3+00:00

After some more testing it looks like the issue is that any data that is mapped as input to a flowlet and then output cannot be used in any subsequent transformation unless it is "readded" via a derived column or join to a transformation from before the flowlet ran. It's not specific to join transformations. The only explanation for this is that there was an update recently to the underlying code for flowlets that caused this bug.

Answer 1

SSingh-MSFT 16,371 Moderator

Hi @John Neubecker ,

Thanks for posting question in Microsoft Q&A Platform.

Recently there was a bug identified in flowlets which got fixed. I believe that has impact here. Hope you did necessary changes to fix it. Please check below link where complete discussion of bug and resolution available.
https://learn.microsoft.com/en-us/answers/questions/911086/index.html

Hope this helps. Please let us know if know if any further queries.

------------

Please don't forget to click on or upvote button whenever the information provided helps you.
Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification

John Neubecker 6 Reputation points

2022-08-10T17:20:29.29+00:00

This is a different problem. The flowlet has a single output. The output columns can be selected to be used in other transformations but that results in the error I posted unless the columns are explicitly derived again as I describe above. And as I mention in the comments this is something that had been working for the past couple of weeks until yesterday. If you haven't read my comments they provide much more detail.
SSingh-MSFT 16,371 Reputation points Moderator

2022-08-11T11:29:52.89+00:00
Hi @John Neubecker ,

Thanks for using MS Q&A platform.

Could you please provide us with the below details of the pipeline run?

source and destination datasets

dataflow transformations

helpful screenshots

Looking forward for your reply to troubleshoot.
John Neubecker 6 Reputation points

2022-08-11T13:13:30.847+00:00

The source data

I've removed one column from the workaround so that extract_set_id isn't being rederived.

The join settings

And the error I get when I preview the join output

Spark job failed: {
"text/plain": "{\"runId\":\"bea34f83-df5a-4a8c-a797-57cacaf5c192\",\"sessionId\":\"61bd0b8e-2683-483a-a2e0-2531553d63ea\",\"status\":\"Failed\",\"payload\":{\"statusCode\":400,\"shortMessage\":\"DF-EXPR-010 at Join 'JoinWithExistingEntities'(Line 30/Col 8): Column 'extract_set_id' not found. The stream is either not connected or column is unavailable\",\"detailedMessage\":\"Failure 2022-08-11 13:06:31.837 failed DebugManager.processJob, run=bea34f83-df5a-4a8c-a797-57cacaf5c192, errorMessage=DF-EXPR-010 at Join 'JoinWithExistingEntities'(Line 30/Col 8): Column 'extract_set_id' not found. The stream is either not connected or column is unavailable\"}}\n"
} - RunId: bea34f83-df5a-4a8c-a797-57cacaf5c192

If extract_set_id is added to the workaround transformation shown in the second screenshot the error goes away.
John Neubecker 6 Reputation points

2022-08-16T18:05:04.52+00:00

I tested this again just now and the bug has been fixed.
SSingh-MSFT 16,371 Reputation points Moderator

2022-08-17T04:56:30.277+00:00

Hi @John Neubecker ,

It is great to know that you could now execute the code. We expect you to keep using this forum and motivate others to do the same. You can always help other community members by answering their queries.

Share via

Error saying column is missing during join but column exists

1 answer

Your answer