The same issue here, if I designe the table in Dataverse using simple text column as alternative key, everything works as expected and upsert is working as expected within the Dataflow.
If I do the same but I will create composite key using e.g. standard text field and a lookup Im not able to save anymore the data as datflow is throwing errors that the lookup field value is missing in the payload.
The oinly workaround that works for me is to sink data to dummy json file from Dataflow and in next pipeline action use CopyActivity to upload json to Dataverse.
I already opened a case within MS to get a clear message why this is happening, they are still investigating the issue.