More

    How to Forcefully stop hanging Foreman tasks

    Method I

    Run the following command in the Foreman/Satellite server to access the foreman console.

    [root@satellite ~]# foreman-rake console
    Loading production environment (Rails 6.0.6)
    irb(main):001:0> 
    

     

     

    Cancel all running Sync tasks:

    irb(main):001:0> ForemanTasks::Task.where(label: "Actions::Katello::Repository::Sync", state: "running", result: "pending").map(&:cancel)
    
    E, [2023-08-07T09:09:36.710865 #1848999] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25-1, sender_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 999a5104-d572-4092-8564-2a6749116680, step_id: 4, event: Dynflow::Action::Cancellable::Cancel, time: , optional: false]] (Dynflow::Error)
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:149:in dispatch_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:118:in  (2 levels) in publish_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:212:in track_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:117:in  in publish_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:254:in with_ping_request_caching'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:116:in publish_request'
    [ concurrent-ruby ]
    
    ...
    
    
    

     

    Cancel all running Sync tasks that have been running for more than a day:

    irb(main):007:0> ForemanTasks::Task.where(label: "Actions::Katello::Repository::Sync", state: "running", result: "pending").where("started_at < ?", 1.day.ago).map(&:cancel)
    
    E, [2023-08-07T09:11:23.182400 #1848999] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25-447, sender_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 999a5104-d572-4092-8564-2a6749116680, step_id: 4, event: Dynflow::Action::Cancellable::Cancel, time: , optional: false]] (Dynflow::Error)
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:149:in dispatch_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:118:in  (2 levels) in publish_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:212:in track_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:117:in  in publish_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:254:in with_ping_request_caching'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:116:in publish_request'
    [ concurrent-ruby ]
    E, [2023-08-07T09:11:23.208784 #1848999] ERROR -- /client-dispatcher: Could not find an executor for Dynflow::Dispatcher::Envelope[request_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25-449, sender_id: a709e781-c24b-4f15-a3dc-2ef6b4294b25, receiver_id: Dynflow::Dispatcher::UnknownWorld, message: Dynflow::Dispatcher::Event[execution_plan_id: 1042b069-38e1-4d57-a6ed-b25d255b4e9a, step_id: 4, event: Dynflow::Action::Cancellable::Cancel, time: , optional: false]] (Dynflow::Error)
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:149:in dispatch_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:118:in  (2 levels) in publish_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:212:in track_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:117:in  in publish_request'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:254:in with_ping_request_caching'
    /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/dispatcher/client_dispatcher.rb:116:in publish_request'
    [ concurrent-ruby ]
    
    ...

     

     

    Cancel all Upload Package Profile tasks and Generate Applicability tasks that have been running for more than a day.

    irb(main):011:0> ForemanTasks::Task.where(label: ["Actions::Katello::Host::UploadPackageProfile", "Actions::Katello::Host::GenerateApplicability"], state: "running", result: "pending").where("started_at < ?", 1.day
    .ago).map(&:cancel)
    => []
    

     

    Export tasksĀ  that have been running for more than 3 days:

    [root@satellite ~]# foreman-rake foreman_tasks:export_tasks TASK_SEARCH='' TASK_DAYS=3
    
    Exporting all tasks matching filter started_at > "2023-08-04 06:30:02"
    Gathering 241 tasks.
    1/241
    2/241
    3/241
    4/241
    5/241
    6/241
    7/241
    8/241
    9/241
    10/241
    11/241
    12/241
    13/241
    14/241
    15/241
    16/241
    17/241
    
    ...

     

    If the above method doesnt work, you can try the below method.

    Method II

    List all orphaned task using the command below:

    [root@satellite ~]# su - postgres
    
    Last login: Mon Aug  7 09:35:13 EAT 2023 on pts/0
    -bash-4.4$ 
    -bash-4.4$ psql foreman
    psql (12.12)
    Type "help" for help.
    
    foreman=# 
    foreman=# select id, label, started_at, state, result from foreman_tasks_tasks where state != 'stopped';
    
                      id                  |                     label                      |       started_at        |   state   | result  
    --------------------------------------+------------------------------------------------+-------------------------+-----------+---------
     38880d17-b9e1-4224-ab2d-fa068f948c7a | Actions::Katello::Repository::Sync             | 2023-05-15 21:14:13.865 | running   | warning
     14b57e5b-8f34-40e1-9a59-a87aa2b69b3c | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:52.306 | running   | pending
     2d32451f-6e26-43dc-8803-a4cbb35d32e2 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:25.63  | running   | pending
     7f4ae918-f06b-4403-8546-1297d09cd667 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:08.707 | running   | pending
     34983e1b-9b3a-41cb-b6a2-e80ee1935930 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:03.864 | running   | pending
     867ec86c-a825-4f4c-82b5-d6a7afa5bb64 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:20.719 | running   | warning
     2fc6e036-3e77-436c-b95f-cb91c3f343de | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:07.807 | running   | warning
     4bd963fd-2ae5-412e-8ec6-72c2b69c0822 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:22.942 | running   | warning
     2869dd7c-6b76-46cb-afff-d39a576470fc | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:28.417 | running   | warning
     199c1ca7-7443-405b-b315-d32fad832e26 | Actions::Katello::Repository::Sync             | 2023-05-18 21:05:03.588 | running   | pending
     0ccee398-16d4-469d-9290-cdf635663116 | Actions::Katello::Repository::Sync             | 2023-05-18 21:04:15.948 | running   | pending
     142ae2f8-07ee-430c-b3c2-a8e30b499c57 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:48.194 | running   | pending
     73b78001-6b50-4f84-9c92-a62333777c25 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:43.038 | running   | pending
     b21f1048-f52b-49c7-9cfe-cfe1082b6df9 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:38.61  | running   | pending
     aaf3af82-49ad-4194-a4cd-7df7a1567d42 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:34.771 | running   | pending
     b989bb65-1761-4f33-a5bd-8bda06fd8f4a | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:20.713 | running   | pending
     7e72525a-d68f-4c13-bf7f-a5549f3e3181 | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:05.537 | running   | pending
     ddc736a4-ffce-4666-8eaf-0efe7457cb3e | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:02.833 | running   | pending
     0ad3265c-1dc0-436c-9291-3d7b9c3ccf3f | Actions::Katello::Repository::Sync             | 2023-05-18 21:02:00.058 | running   | warning
     876822ad-cda1-4446-9b7b-dff734fde0cf | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:50.969 | running   | pending
     03efea46-2ed4-49e2-b4a4-e7eee5d35b50 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:18.416 | running   | pending
     c05781e0-4f7e-4cf5-9920-012969b2ae42 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:14.159 | running   | pending
     bbb4aa19-466e-41cd-888e-df8621057644 | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:12.107 | running   | pending
     5f90948f-5d2b-489b-85e2-bc470202a9a4 | Actions::Katello::Repository::Sync             | 2023-05-18 21:04:26.247 | running   | pending
     ce7f42cc-84c8-4bf3-a46e-013e762ef6bd | Actions::Katello::Repository::Sync             | 2023-05-18 21:04:52.863 | running   | pending
     05b347ab-bb64-49a0-af5c-47b3a166ea79 | Actions::Katello::Repository::Sync             | 2023-05-18 21:04:20.944 | running   | pending
     82d135cb-c322-4140-a173-657ad0080fdd | Actions::Katello::Repository::Sync             | 2023-05-18 21:00:10.53  | running   | pending
    
    .....
    
    

     

    Kill the orphaned tasks using the command below:

    [root@satellite ~]# foreman-rake console
    Loading production environment (Rails 6.0.6)
    
    irb(main):001:0> ForemanTasks::Task.find("14b57e5b-8f34-40e1-9a59-a87aa2b69b3c").destroy
    
    => #<ForemanTasks::Task::DynflowTask id: "14b57e5b-8f34-40e1-9a59-a87aa2b69b3c", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Katello::Repository::Sync", started_at: "2023-05-18 21:02:52", ended_at: nil, state: "running", result: "pending", external_id: "999a5104-d572-4092-8564-2a6749116680", parent_task_id: "86f5c3f3-f343-496e-9394-43a2fad584d9", start_at: "2023-05-18 21:02:52", start_before: nil, action: "Synchronize repository 'aodh-base'; product 'RHOSP...", state_updated_at: "2023-05-30 21:15:29", user_id: 1>
    
    irb(main):004:0> ForemanTasks::Task.find("38880d17-b9e1-4224-ab2d-fa068f948c7a").destroy
    
    => #<ForemanTasks::Task::DynflowTask id: "38880d17-b9e1-4224-ab2d-fa068f948c7a", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Katello::Repository::Sync", started_at: "2023-05-15 21:14:13", ended_at: nil, state: "running", result: "warning", external_id: "7307ce0e-1e2d-4aec-b058-19bbf0a92577", parent_task_id: "bfd68219-2a87-4174-b585-2aefea4e1eb4", start_at: "2023-05-15 21:14:13", start_before: nil, action: "Synchronize repository 'neutron-dhcp-agent'; produ...", state_updated_at: "2023-05-30 21:15:41", user_id: 1>
    irb(main):005:0> 
    
    irb(main):009:0> ForemanTasks::Task.find("4bd963fd-2ae5-412e-8ec6-72c2b69c0822").destroy
    
    => #<ForemanTasks::Task::DynflowTask id: "4bd963fd-2ae5-412e-8ec6-72c2b69c0822", type: "ForemanTasks::Task::DynflowTask", label: "Actions::Katello::Repository::Sync", started_at: "2023-05-18 21:00:22", ended_at: "2023-08-07 07:06:00", state: "stopped", result: "warning", external_id: "0ff5701d-e634-43bf-86ae-40ac60b92724", parent_task_id: "86f5c3f3-f343-496e-9394-43a2fad584d9", start_at: "2023-05-18 21:00:22", start_before: nil, action: "Synchronize repository 'Red Hat OpenStack Platform...", state_updated_at: "2023-08-07 07:06:00", user_id: 1>
    irb(main):010:0> 
    
    
    
    
    

     

    It is crucial that we check and fix database integrity

    [root@satellite ~]# foreman-rake katello:reimport
    
    Importing Katello::Subscription
    Importing Katello::Pool
    Importing Katello::Content
    Importing Activation Key Subscriptions

     

    Recent Articles

    Related Articles

    Leave A Reply

    Please enter your comment!
    Please enter your name here