@Imran Mondal
I suspect the slowdown is caused by the "unique rowkey" option. My hypothesis, is that since ADF uses the "Insert or Merge" and "Insert or Replace" operations, instead of the "Insert" operation, the task of finding a new unique rowkey value is harder. It would have to guess a value, then query the table to find out if it is already in use or not. If rowkey is already in use, guess again, if not in use, insert. The more rows in the table, the more likely it will have to guess again.
If this is the case, then adding a unique value to your data, and specifying that as rowkey would speed things up. The table storage locates a row as a combination of partitionkey and rowkey.