how to cancel the multiple jobs created by the '.set-or-append async' command in Kusto

Question

Hi Dear fellows,

I have created plenty of failure ingestion jobs with individual operations by the '. set-or-append async' command in Kusto, and those jobs continue growing every day, thousands of times a day. I have no idea how to cancel Infinity jobs every day. Even though I tried to block the principle, those jobs were still generated by days. We have checked, and there is no other source to trigger those jobs; it was triggered only once on the initial day.

Thanks,

Answer 1

Hi Max_Gu,

Thank you for posting query in Microsoft Q&A Platform.

It sounds like you are experiencing an issue with infinite job creation in Kusto due to the use of the . set-or-append async command in your ingestion jobs. This can happen if the ingestion job is not properly terminated or if there is an issue with the job's configuration.

Here are some steps you can take to troubleshoot and resolve this issue:

Check the status of the ingestion jobs in Kusto by running the following command:

.

This command will show you a list of all ingestion jobs that have failed or are currently in progress.

Identify the ingestion jobs that are causing the issue by looking for jobs that have been running for an unusually long time or have a high number of failures.

Cancel the ingestion jobs that are causing the issue by running the following command:

.cancel &

Replace <job-id> with the ID of the ingestion job you want to cancel.

Review the configuration of your ingestion jobs to ensure that they are properly configured and terminated after completion. You can use the .show ingestion command to view the configuration of your ingestion jobs.

Consider using the . set-or-replace async command instead of . set-or-append async to avoid creating duplicate data in your tables.

If the issue persists, contact Microsoft support for further assistance.

It's also worth noting that Kusto has a built-in feature called "auto-cleanup" that automatically deletes failed ingestion jobs after a certain period of time. You can configure this feature by setting the IngestionMaxFailedJobs and IngestionMaxFailedJobsAgeInDays properties in your Kusto cluster's configuration.

Answer 2

It seems the .cancel command is not shown well in the post.

It should look like:

.cancel operation OperationId [with ( reason = ReasonPhrase )]

like:

.cancel operation 078b2641-f10d-4694-96f8-1ee2b75dda48 with(Reason="Command canceled by me")

Example is taken from the documentation.