copy activity - parquet

arkiboys 9,691 Reputation points
2021-12-03T07:53:40.453+00:00

Hello.
The copy activity:
source tab is pulling details about the name of the pipeline, rows processed.
As there has to be a dataset for source, I am pointing it to sql server and in query option I have placed "select 1".
This way, I only pull the name of the pipeline and rows processed.
At present, sink has a dataset pointing to dataset which is pointing to a blobstorage where the name of the pipeline and rows processed get saved into a parquet file.
Question:
How is it possible to not use source as sql server? I tried using source as blobstorage but if I do that, then copy activity just copies the source file which is not what I want. I only want to get the pipeline name and rows processed which I do so by tricking the source tab by just pointing to sql server and doing a select 1

Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,840 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,091 Reputation points
    2021-12-03T17:58:42.303+00:00

    Hi again @arkiboys . So if I understand right, you just want to get pipeline name and some details from copy activity, and write this to a parquet using another copy activity and the 'additional columns' feature.

    When I do something like this, I do almost the same, except for source I point to an almost blank delimited text file. By almost blank, I mean it only has whitespace. This way, the only real contents will be the 'additional columns' feature.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.