I am getting Negsignal.SIGKILL Airflow

Chirag Lalwani 20

I have multiple integration running but for one integration when I extract zip file size of zip is 2-3 gbs, extract and upload each file to our storage.It shows Negsignal.SIGKILL. It was running fine a month ago but now its this error. The file is same always where it fails.

I am using Airflow for Azure Data Factory. When I run it like 1 more time it will get successfully run It seems to be issue with resources but I cant don’t know how much resouces airflow is using and how to increase it.

Is there a way to increase resources in DAG or will adding multiple workers solve the issue?or will I had to add concurrency in case of multiple works.

Dag code is simple we are using PythonOperator

Chandra Boorla 4,445 Reputation points Microsoft Vendor

2024-11-22T08:18:00.9833333+00:00

Hi @Chirag Lalwani

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

The Negsignal.SIGKILL error you are encountering in Airflow typically indicates that the process was terminated by the operating system, often due to resource constraints such as memory limits. Since you mentioned that the issue occurs with a large zip file (2-3 GB) and that it runs successfully upon retry, it does suggest that the resources allocated to your Airflow task may not be sufficient for the size of the data being processed.

Here are some troubleshooting steps that might help you in resolving the issue:

I am using Airflow for Azure Data Factory. When I run it like 1 more time it will get successfully run It seems to be issue with resources but I cant don’t know how much resouces airflow is using and how to increase it.

For more details, please refer: Workflow Orchestration Manager in Azure Data Factory offers a range of powerful features.

Is there a way to increase resources in DAG or will adding multiple workers solve the issue?or will I had to add concurrency in case of multiple works.

Yes, you can increase the resources allocated to your DAG by adding more workers or increasing the concurrency. Adding more workers will increase the number of parallel tasks that can be executed, while increasing the concurrency will allow more tasks to be executed simultaneously.

To add more workers, you can increase the value of the worker_count parameter in your Airflow configuration file. Similarly, to increase the concurrency, you can increase the value of the parallelism parameter in your DAG definition.

However, keep in mind that adding more workers or increasing the concurrency will also increase the resource utilization and cost. Therefore, it is important to optimize your DAG code to reduce the memory footprint and minimize the resource usage. Since your DAG code is simple and you are using PythonOperator, you can try optimizing your Python code to reduce the memory usage.

For additional information, please refer: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html

I hope this information helps. Please do let us know if you have any further queries.

Thank you.
Chirag Lalwani 20 Reputation points

2024-11-22T11:28:14.2933333+00:00

Thanks @Chandra Boorla for quick reply.

Can this type of error occur due to Zombie or Undead task?

Also when i checked Airflow Browse -> Dag Runs I can find some logs are wronged.
For Example a dags was trigger on 20 nov but in log it shows start and end date as 19 nov but when I open that details it shows correctly.

Is there any way to increase memory only for one worker rather than data factory level (if you have code/resource please share).

Also What is job? what is LocalTaskJob and SchedulerJob?

Please share resources and python if any.

Also I went to browse-> job and found my that dagid is still running it takes just 6 min and status shows as completed when but why can I still find it as running End data is blank?

I also need good course and resources and need meaning of Latest Heartbeat, Start Date, End data is blank and how to kill a job ? I think its becoming zombie task..

Best Regards,
Chirag
Chandra Boorla 4,445 Reputation points Microsoft Vendor

2024-11-25T07:39:14.3033333+00:00
@Chirag Lalwani

Thank you for sharing the details. Here are the suggestions for your queries.

Can this type of error occur due to Zombie or Undead task?

Yes, zombie tasks (tasks that are in a "running" state even though they've completed or failed) can sometimes cause issues. These tasks can cause delays or unexpected behavior, as Airflow might still be considering them as active tasks, which can affect scheduling and resource allocation. To check for zombie tasks:

You can view your DAGs in the Airflow UI, specifically under Browse > Tasks or Browse > Dag Runs, and check if any tasks are not properly finishing.

You can manually clear tasks or kill them if needed. More details on how to do that are explained below.

Also when i checked Airflow Browse -> Dag Runs I can find some logs are wronged. For Example a dags was trigger on 20 nov but in log it shows start and end date as 19 nov but when I open that details it shows correctly.

If you’re seeing incorrect start and end dates in the logs (e.g., logs showing a task run starting and ending on the wrong dates), but the task details show correctly when you open them, this could be a UI synchronization issue. Sometimes, the Airflow UI or the backend database might lag or not refresh the logs properly.

Possible causes:

Database sync issues: Airflow UI might show cached data, and it can sometimes take time for the database to sync, especially in a large workflow.

Scheduler delays: The scheduler might delay recording task completion, especially if the system is under heavy load or if there are many DAGs running.

Is there any way to increase memory only for one worker rather than data factory level (if you have code/resource please share).

CeleryExecutor is one of the ways you can scale out the number of workers. For this to work, you need to setup a Celery backend (RabbitMQ, Redis, …) and change your airflow.cfg to point the executor parameter to CeleryExecutor and provide the related Celery settings.

For more details, please refer the below documentations:

Scaling Out with Celery

How Apache Airflow Distributes Jobs on Celery workers

Also What is job? what is LocalTaskJob and SchedulerJob?

Job: In Airflow, a job refers to a unit of work that is scheduled and executed. Jobs can refer to the tasks within a DAG or can refer to internal Airflow processes, such as the scheduler or worker processes.

LocalTaskJob: This is the job that Airflow creates to execute a single task in a DAG. When you run a task using a local executor (or other executors), each task is wrapped in a LocalTaskJob which executes the task on the available worker.

SchedulerJob: The scheduler job is responsible for scheduling and managing task execution in the system. It ensures that tasks are executed according to their schedules and dependencies. This job is crucial for managing the flow of tasks in a DAG.

Also I went to browse-> job and found my that dagid is still running it takes just 6 min and status shows as completed when but why can I still find it as running End data is blank?

This could be indicative of several issues:

Stale status: Sometimes, the status in the UI might not update immediately, especially in distributed environments or in setups with a heavy load (many tasks running). This could be a UI refresh issue.

Zombie Task: If the task took less time to run but shows as "Completed" with an empty end date, it might indicate that the task was forcefully terminated or failed without updating its status, which is often a sign of a zombie task.

In Airflow, you can check the logs and see if there are any exceptions or failure messages that were not properly reported. Sometimes, tasks can finish executing but not update the status if the worker or scheduler failed to notify the database correctly.

I also need good course and resources and need meaning of Latest Heartbeat, Start Date, End data is blank

Latest Heartbeat: This is the most recent time that a worker or scheduler communicated with the Airflow system. If a worker’s heartbeat is missing or very outdated, it might mean that the worker is stuck, unresponsive, or has failed.

Start Date: This is the timestamp when a particular task run or DAG run was triggered.

End Date: The end date of a task shows when the task actually completed. If the end date is blank, it could indicate that the task is still running, or that there was an issue updating its status.

For details, please refer: Checking Airflow Health Status

how to kill a job ?

Airflow is a popular open-source platform for managing and scheduling data pipelines. It is used by many organizations to automate the processing and movement of data, making it a critical component of their data infrastructure. One challenge that can arise when using Airflow is the presence of zombie tasks. In this blog, we will explore what zombie tasks are, how they can impact your data pipelines, and how to handle them.

For additional information, please refer: Zombie Tasks in Airflow: What They Are and How to Handle Them

Refer the below documentations that might help you:

https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html
https://airflow.apache.org/docs/apache-airflow/stable/index.html

I hope this information helps.

Thank you.
Chandra Boorla 4,445 Reputation points Microsoft Vendor

2024-11-26T08:11:10.9366667+00:00

@Chirag Lalwani

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Thank you.
Chirag Lalwani 20 Reputation points

2024-11-26T08:47:13.2+00:00

Chandra Boorla I have currently solved the issue by changing time of my integration and it seems to be working fine now. Their might be more usage of resources at particular time which cause integration to fail.

Thanks for your help.
Chandra Boorla 4,445 Reputation points Microsoft Vendor

2024-11-27T07:58:42.8966667+00:00

@Chirag Lalwani

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Accepted answer

Chandra Boorla 4,445 Reputation points Microsoft Vendor

2024-11-27T07:58:07.7233333+00:00

@Chirag Lalwani

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

I have multiple integration running but for one integration when I extract zip file size of zip is 2-3 gbs, extract and upload each file to our storage.It shows Negsignal.SIGKILL. It was running fine a month ago but now its this error. The file is same always where it fails.

I am using Airflow for Azure Data Factory. When I run it like 1 more time it will get successfully run It seems to be issue with resources but I cant don’t know how much resouces airflow is using and how to increase it.

Is there a way to increase resources in DAG or will adding multiple workers solve the issue?or will I had to add concurrency in case of multiple works.

Dag code is simple we are using PythonOperator

Solution:

I have currently solved the issue by changing time of my integration and it seems to be working fine now. Their might be more usage of resources at particular time which cause integration to fail.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.

1 person found this answer helpful.
Chandra Boorla 4,445 Reputation points Microsoft Vendor

2024-11-28T10:02:19.9666667+00:00

@Chirag Lalwani

Following up to see if the above answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

I am getting Negsignal.SIGKILL Airflow

0 additional answers

Your answer