Azure Data factor connection to google big query

Question

Dear All,

We need to connect to Google Big query to pull some data . However we can't use service authentication for now.

We are using user authentication and issue , is we dont understand when refresh token actual expires.

Last time , it expired in 3-4 days , so i created a new one .

This time , i didnt run that pipeline for 3 days and when i executed again , it didnt work.

Could some one please guide on this please?

Answer 1

Regarding the issue with the Google BigQuery OAuth token expiring, it's important to know that the standard expiration time for access tokens is 1 hour, but refresh tokens are long-lived and do not expire on their own. However, they can stop working under certain conditions, such as if the user's credentials change, the token is revoked, or the token hasn't been used for six months. Also, a user can have a maximum of 50 refresh tokens, and issuing a new one after reaching this limit will cause the oldest token to stop working.

If you're using Azure Data Factory to copy data from Google BigQuery, you can use the "Bulk Copy from Database to Azure Data Explorer" template. This template is designed for copying large amounts of data from databases like SQL Server and Google BigQuery to Azure Data Explorer. The process involves creating a ControlTableDataset, setting up the pipeline, running pipelines in parallel for efficiency, validating the pipeline, and running the pipeline. It's important to note that you need an Azure subscription, an Azure Data Explorer cluster and database, a data factory, and a source of data (in your case, Google BigQuery) to begin this process