copy activity - parquet

arkiboys 9,386 Reputation points
2021-12-03T07:53:40.453+00:00

Hello.
The copy activity:
source tab is pulling details about the name of the pipeline, rows processed.
As there has to be a dataset for source, I am pointing it to sql server and in query option I have placed "select 1".
This way, I only pull the name of the pipeline and rows processed.
At present, sink has a dataset pointing to dataset which is pointing to a blobstorage where the name of the pipeline and rows processed get saved into a parquet file.
Question:
How is it possible to not use source as sql server? I tried using source as blobstorage but if I do that, then copy activity just copies the source file which is not what I want. I only want to get the pipeline name and rows processed which I do so by tricking the source tab by just pointing to sql server and doing a select 1

Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,073 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 25,991 Reputation points
    2021-12-03T17:58:42.303+00:00

    Hi again @arkiboys . So if I understand right, you just want to get pipeline name and some details from copy activity, and write this to a parquet using another copy activity and the 'additional columns' feature.

    When I do something like this, I do almost the same, except for source I point to an almost blank delimited text file. By almost blank, I mean it only has whitespace. This way, the only real contents will be the 'additional columns' feature.