4.16 kvm storage issues by ravening · Pull Request #5783 · apache/cloudstack

ravening · 2021-12-16T09:19:55Z

Description

This PR provides multiple options to handle storage issue on kvm.

Ported from #4708

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

DaanHoogland · 2021-12-16T18:10:19Z

@ravening sounds like this could go on 4.16. Would you?

ravening · 2021-12-16T18:37:19Z

@ravening sounds like this could go on 4.16. Would you?

@DaanHoogland it's a new feature. So it will goto main

sureshanaparti · 2021-12-17T04:13:22Z

@blueorangutan package

blueorangutan · 2021-12-17T04:14:05Z

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-12-17T05:27:40Z

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✔️ suse15. SL-JID 1930

DaanHoogland

code looks good, but needs testing

sureshanaparti · 2021-12-17T11:31:41Z

@blueorangutan package

blueorangutan · 2021-12-17T11:32:03Z

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-12-17T12:15:29Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 1937

sureshanaparti · 2021-12-17T12:20:39Z

@blueorangutan test

blueorangutan · 2021-12-17T12:21:03Z

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-12-17T22:15:15Z

Trillian test result (tid-2678)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34214 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5783-t2678-kvm-centos7.zip
Smoke tests completed. 90 look OK, 1 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_hostha_enable_ha_when_host_disabled	`Error`	4.66	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	302.80	test_hostha_kvm.py

GutoVeronezi · 2021-12-21T14:34:42Z

server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

            "Indicates whether the VM can be migrated to different cluster if no host is found in same cluster",true, ConfigKey.Scope.Zone, null);

+    public static final ConfigKey<Long> KVM_HEARTBEAT_UPDATE_MAX_RETRIES_CK =  new ConfigKey<>("Advanced", Long.class, KVM_HEARTBEAT_UPDATE_MAX_RETRIES, "5",
+            "The maximum retries of kvm heartbeat to write to storage",


Suggested change

"The maximum retries of kvm heartbeat to write to storage",

"The maximum retries of KVM heartbeat to write to storage.",

server/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java

GutoVeronezi · 2021-12-21T15:11:55Z

agent/src/main/java/com/cloud/agent/properties/AgentProperties.java

-    public static final Property<Boolean> REBOOT_HOST_AND_ALERT_MANAGEMENT_ON_HEARTBEAT_TIMEOUT
-        = new Property<Boolean>("reboot.host.and.alert.management.on.heartbeat.timeout", true);


If we gonna remove this property, we need to explicit the option in the new feature that behave the same way and advise operators on how to configure it.

Also, we should remove from agent.properties too and wherever it is used.

The default action is to reboot the system using the -c flag. so i will just remove it from agent.properties file

GutoVeronezi · 2021-12-21T15:13:36Z

engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java

+        protected void runInContext() {
+            try {
+                ManagementServerHostVO msHost = _msHostDao.findOneInUpState(new Filter(ManagementServerHostVO.class, "id", true, 0L, 1L));
+                if (msHost == null || (msHost.getMsid() != _nodeId)) {


Suggested change

if (msHost == null || (msHost.getMsid() != _nodeId)) {

if (msHost == null || msHost.getMsid() != _nodeId) {

GutoVeronezi · 2021-12-21T15:14:35Z

engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java

+                        scheduleInvestigationTask(host.getId());
+                    }
+                }
+            } catch (final Exception e) {


Do we need a catch Pokémon (catch them all) here?

GutoVeronezi · 2021-12-21T15:15:43Z

engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java

+                    s_logger.info("Cancelling investigation on host " + hostId + " which is Down");
+                    cancelInvestigationTask(hostId);
+                }
+            } catch (final Exception e) {


Do we need a catch Pokémon (catch them all) here?

GutoVeronezi · 2021-12-21T15:16:32Z

engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java

+                future.cancel(false);
+                s_logger.info("Cancelled a task to investigate host " + hostId);
+                _investigateTasksMap.remove(hostId);
+            } catch (Exception e) {


Do we need a catch Pokémon (catch them all) here?

yadvr · 2022-07-27T09:27:56Z

@blueorangutan package

blueorangutan · 2022-07-27T09:29:02Z

@rohityadavcloud a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-07-27T10:22:53Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3836

DaanHoogland · 2022-08-01T10:28:42Z

@blueorangutan test

blueorangutan · 2022-08-01T10:29:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-08-01T21:37:34Z

Trillian test result (tid-4586)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 38793 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5783-t4586-kvm-centos7.zip
Smoke tests completed. 100 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2022-08-02T07:04:01Z

@ravening can you look at @GutoVeronezi 's comments? I think we are about done here.

yadvr · 2022-10-08T06:49:25Z

@blueorangutan package

blueorangutan · 2022-10-08T06:50:03Z

@rohityadavcloud a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-10-08T10:41:10Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 4391

DaanHoogland · 2022-10-12T06:46:31Z

@blueorangutan package

blueorangutan · 2022-10-12T06:47:03Z

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-10-12T07:42:40Z

Packaging result: ✔️ el7 ✔️ el8 ✖️ debian ✖️ suse15. SL-JID 4435

DaanHoogland · 2022-10-12T18:58:07Z

@blueorangutan test

blueorangutan · 2022-10-12T18:59:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-10-13T06:50:45Z

Trillian test result (tid-5105)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41145 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5783-t5105-kvm-centos7.zip
Smoke tests completed. 102 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_upgrade_kubernetes_ha_cluster	`Failure`	590.11	test_kubernetes_clusters.py
test_hostha_enable_ha_when_host_disabled	`Error`	3.71	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	304.92	test_hostha_kvm.py

github-actions · 2022-12-22T12:04:10Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland · 2022-12-22T13:22:33Z

@ravening can you look at the conflicts, please?

DaanHoogland · 2023-06-09T11:43:43Z

@ravening is this still relevant?

ravening mentioned this pull request Dec 16, 2021

kvm: Handle storage issue on NFS/KVM in multiple ways #4708

Closed

12 tasks

sureshanaparti added the component:kvm label Dec 16, 2021

DaanHoogland approved these changes Dec 17, 2021

View reviewed changes

GutoVeronezi reviewed Dec 21, 2021

View reviewed changes

ustcweizhou and others added 6 commits December 23, 2021 09:13

kvm: Handle storage issue on NFS/KVM in multiple ways

9029118

kvm: Schedule investigate tasks for disconnected hosts

cf8f18d

kvm: Add option 'noaction' for storage issue handling

35bf994

code refactor

1a6ea9e

remove unused agent property

f5d2789

code feedback

4fe0b19

ravening force-pushed the 4.16-kvm-storage-issues branch from 77869f4 to 4fe0b19 Compare December 23, 2021 08:24

nvazquez force-pushed the main branch from c56220f to 0f4c3af Compare June 6, 2022 15:14

github-actions bot added the status:has-conflicts label Dec 22, 2022

weizhouapache closed this Jun 9, 2023

	"The maximum retries of kvm heartbeat to write to storage",
	"The maximum retries of KVM heartbeat to write to storage.",

		public static final Property<Boolean> REBOOT_HOST_AND_ALERT_MANAGEMENT_ON_HEARTBEAT_TIMEOUT
		= new Property<Boolean>("reboot.host.and.alert.management.on.heartbeat.timeout", true);

	if (msHost == null \|\| (msHost.getMsid() != _nodeId)) {
	if (msHost == null \|\| msHost.getMsid() != _nodeId) {

Conversation

ravening commented Dec 16, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

DaanHoogland commented Dec 16, 2021

Uh oh!

ravening commented Dec 16, 2021

Uh oh!

sureshanaparti commented Dec 17, 2021

Uh oh!

blueorangutan commented Dec 17, 2021

Uh oh!

blueorangutan commented Dec 17, 2021

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Dec 17, 2021

Uh oh!

blueorangutan commented Dec 17, 2021

Uh oh!

blueorangutan commented Dec 17, 2021

Uh oh!

sureshanaparti commented Dec 17, 2021

Uh oh!

blueorangutan commented Dec 17, 2021

Uh oh!

blueorangutan commented Dec 17, 2021

Uh oh!

GutoVeronezi Dec 21, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GutoVeronezi Dec 21, 2021

Choose a reason for hiding this comment

Uh oh!

ravening Dec 23, 2021

Choose a reason for hiding this comment

Uh oh!

GutoVeronezi Dec 21, 2021

Choose a reason for hiding this comment

Uh oh!

GutoVeronezi Dec 21, 2021

Choose a reason for hiding this comment

Uh oh!

GutoVeronezi Dec 21, 2021

Choose a reason for hiding this comment

Uh oh!

GutoVeronezi Dec 21, 2021

Choose a reason for hiding this comment

Uh oh!

yadvr commented Jul 27, 2022

Uh oh!

blueorangutan commented Jul 27, 2022

Uh oh!

blueorangutan commented Jul 27, 2022

Uh oh!

DaanHoogland commented Aug 1, 2022

Uh oh!

blueorangutan commented Aug 1, 2022

Uh oh!

blueorangutan commented Aug 1, 2022

Uh oh!

DaanHoogland commented Aug 2, 2022

Uh oh!

yadvr commented Oct 8, 2022

Uh oh!

blueorangutan commented Oct 8, 2022

Uh oh!

blueorangutan commented Oct 8, 2022

Uh oh!