How to Forcefully stop hanging Foreman tasks

Method I

Run the following command in the Foreman/Satellite server to access the foreman console.

[root@satellite ~]# foreman-rake console
Loading production environment (Rails 6.0.6)
irb(main):001:0> 

Cancel all running Sync tasks:

irb(main):001:0> ForemanTasks::Task.where(label: "Actions::Katello::Repository::Sync", state: "running", result: "pending").map(&:cancel)

E, [2023-08-07T09:09:36.710865 #1848999] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25-1, sender_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 999a5104-d572-4092-8564-2a6749116680, step_id: 4, event: Dynflow::Action::Cancellable::Cancel, time: , optional: false]] (Dynflow::Error)
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:149:in dispatch_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:118:in  (2 levels) in publish_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:212:in track_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:117:in  in publish_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:254:in with_ping_request_caching'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:116:in publish_request'
[ concurrent-ruby ]

...

Cancel all running Sync tasks that have been running for more than a day:

irb(main):007:0> ForemanTasks::Task.where(label: "Actions::Katello::Repository::Sync", state: "running", result: "pending").where("started_at < ?", 1.day.ago).map(&:cancel)

E, [2023-08-07T09:11:23.182400 #1848999] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25-447, sender_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 999a5104-d572-4092-8564-2a6749116680, step_id: 4, event: Dynflow::Action::Cancellable::Cancel, time: , optional: false]] (Dynflow::Error)
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:149:in dispatch_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:118:in  (2 levels) in publish_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:212:in track_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:117:in  in publish_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:254:in with_ping_request_caching'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:116:in publish_request'
[ concurrent-ruby ]
E, [2023-08-07T09:11:23.208784 #1848999] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25-449, sender_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 1042b069-38e1-4d57-a6ed-b25d255b4e9a, step_id: 4, event: Dynflow::Action::Cancellable::Cancel, time: , optional: false]] (Dynflow::Error)
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:149:in dispatch_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:118:in  (2 levels) in publish_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:212:in track_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:117:in  in publish_request'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:254:in with_ping_request_caching'
/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:116:in publish_request'
[ concurrent-ruby ]

...

irb(main):011:0> ForemanTasks::Task.where(label: ["Actions::Katello::Host::UploadPackageProfile", "Actions::Katello::Host::GenerateApplicability"], state: "running", result: "pending").where("started_at < ?", 1.day
.ago).map(&:cancel)
=> []

Export tasks  that have been running for more than 3 days:

[root@satellite ~]# foreman-rake foreman_tasks:export_tasks TASK_SEARCH='' TASK_DAYS=3

Exporting all tasks matching filter started_at > "2023-08-04 06:30:02"
Gathering 241 tasks.
1/241
2/241
3/241
4/241
5/241
6/241
7/241
8/241
9/241
10/241
11/241
12/241
13/241
14/241
15/241
16/241
17/241

...

If the above method doesnt work, you can try the below method.

Method II

List all orphaned task using the command below:

[root@satellite ~]# su - postgres

Last login: Mon Aug  7 09:35:13 EAT 2023 on pts/0
-bash-4.4$ 
-bash-4.4$ psql foreman
psql (12.12)
Type "help" for help.

foreman=# 
foreman=# select id, label, started_at, state, result from foreman_tasks_tasks where state != 'stopped';

                  id                  |                     label                      |       started_at        |   state   | result  
--------------------------------------+------------------------------------------------+-------------------------+-----------+---------
 38880d17-b9e1-4224-ab2d-fa068f948c7a | Actions::Katello::Repository::Sync             | 2023-05-15 21:14:13.865 | running   | warning
 14b57e5b-8f34-40e1-9a59-a87aa2b69b3c | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:52.306 | running   | pending
 2d32451f-6e26-43dc-8803-a4cbb35d32e2 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:25.63  | running   | pending
 7f4ae918-f06b-4403-8546-1297d09cd667 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:08.707 | running   | pending
 34983e1b-9b3a-41cb-b6a2-e80ee1935930 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:03.864 | running   | pending
 867ec86c-a825-4f4c-82b5-d6a7afa5bb64 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:20.719 | running   | warning
 2fc6e036-3e77-436c-b95f-cb91c3f343de | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:07.807 | running   | warning
 4bd963fd-2ae5-412e-8ec6-72c2b69c0822 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:22.942 | running   | warning
 2869dd7c-6b76-46cb-afff-d39a576470fc | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:28.417 | running   | warning
 199c1ca7-7443-405b-b315-d32fad832e26 | Actions::Katello::Repository::Sync             | 2023-05-18 21:05:03.588 | running   | pending
 0ccee398-16d4-469d-9290-cdf635663116 | Actions::Katello::Repository::Sync             | 2023-05-18 21:04:15.948 | running   | pending
 142ae2f8-07ee-430c-b3c2-a8e30b499c57 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:48.194 | running   | pending
 73b78001-6b50-4f84-9c92-a62333777c25 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:43.038 | running   | pending
 b21f1048-f52b-49c7-9cfe-cfe1082b6df9 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:38.61  | running   | pending
 aaf3af82-49ad-4194-a4cd-7df7a1567d42 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:34.771 | running   | pending
 b989bb65-1761-4f33-a5bd-8bda06fd8f4a | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:20.713 | running   | pending
 7e72525a-d68f-4c13-bf7f-a5549f3e3181 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:05.537 | running   | pending
 ddc736a4-ffce-4666-8eaf-0efe7457cb3e | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:02.833 | running   | pending
 0ad3265c-1dc0-436c-9291-3d7b9c3ccf3f | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:00.058 | running   | warning
 876822ad-cda1-4446-9b7b-dff734fde0cf | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:50.969 | running   | pending
 03efea46-2ed4-49e2-b4a4-e7eee5d35b50 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:18.416 | running   | pending
 c05781e0-4f7e-4cf5-9920-012969b2ae42 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:14.159 | running   | pending
 bbb4aa19-466e-41cd-888e-df8621057644 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:12.107 | running   | pending
 5f90948f-5d2b-489b-85e2-bc470202a9a4 | Actions::Katello::Repository::Sync             | 2023-05-18 21:04:26.247 | running   | pending
 ce7f42cc-84c8-4bf3-a46e-013e762ef6bd | Actions::Katello::Repository::Sync             | 2023-05-18 21:04:52.863 | running   | pending
 05b347ab-bb64-49a0-af5c-47b3a166ea79 | Actions::Katello::Repository::Sync             | 2023-05-18 21:04:20.944 | running   | pending
 82d135cb-c322-4140-a173-657ad0080fdd | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:10.53  | running   | pending

.....

Kill the orphaned tasks using the command below:

[root@satellite ~]# foreman-rake console
Loading production environment (Rails 6.0.6)

irb(main):001:0> ForemanTasks::Task.find("14b57e5b-8f34-40e1-9a59-a87aa2b69b3c").destroy

=> #<ForemanTasks::Task::DynflowTask id: "14b57e5b-8f34-40e1-9a59-a87aa2b69b3c", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Katello::Repository::Sync", started_at: "2023-05-18 21:02:52", ended_at: nil, state: "running", result: "pending", external_id: "999a5104-d572-4092-8564-2a6749116680", parent_task_id: "86f5c3f3-f343-496e-9394-43a2fad584d9", start_at: "2023-05-18 21:02:52", start_before: nil, action: "Synchronize repository 'aodh-base'; product 'RHOSP...", state_updated_at: "2023-05-30 21:15:29", user_id: 1>

irb(main):004:0> ForemanTasks::Task.find("38880d17-b9e1-4224-ab2d-fa068f948c7a").destroy

=> #<ForemanTasks::Task::DynflowTask id: "38880d17-b9e1-4224-ab2d-fa068f948c7a", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Katello::Repository::Sync", started_at: "2023-05-15 21:14:13", ended_at: nil, state: "running", result: "warning", external_id: "7307ce0e-1e2d-4aec-b058-19bbf0a92577", parent_task_id: "bfd68219-2a87-4174-b585-2aefea4e1eb4", start_at: "2023-05-15 21:14:13", start_before: nil, action: "Synchronize repository 'neutron-dhcp-agent'; produ...", state_updated_at: "2023-05-30 21:15:41", user_id: 1>
irb(main):005:0> 

irb(main):009:0> ForemanTasks::Task.find("4bd963fd-2ae5-412e-8ec6-72c2b69c0822").destroy

=> #<ForemanTasks::Task::DynflowTask id: "4bd963fd-2ae5-412e-8ec6-72c2b69c0822", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Katello::Repository::Sync", started_at: "2023-05-18 21:00:22", ended_at: "2023-08-07 07:06:00", state: "stopped", result: "warning", external_id: "0ff5701d-e634-43bf-86ae-40ac60b92724", parent_task_id: "86f5c3f3-f343-496e-9394-43a2fad584d9", start_at: "2023-05-18 21:00:22", start_before: nil, action: "Synchronize repository 'Red Hat OpenStack Platform...", state_updated_at: "2023-08-07 07:06:00", user_id: 1>
irb(main):010:0> 

It is crucial that we check and fix database integrity

[root@satellite ~]# foreman-rake katello:reimport

Importing Katello::Subscription
Importing Katello::Pool
Importing Katello::Content
Importing Activation Key Subscriptions