problem on adding new node to kubernetes #8943
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
Kubernetes
CLOUDSTACK VERSION
4.19.0.1
CONFIGURATION
Network offering: DefaultNetworkOfferingforKubernetesService
OS / ENVIRONMENT
N/A
SUMMARY
While resizing k8s cluster, we're getting error related to adding firewall rules.
STEPS TO REPRODUCE
We're reproducing that constantly on our installed cluster. Please, share which information do you need to help you to reproduce that.
Here is our firewall and portforwarding rules for that SNAT IP:
mysql> select * from firewall_rules where ip_address_id = 4433;
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
| id | uuid | ip_address_id | start_port | end_port | state | protocol | purpose | account_id | domain_id | network_id | xid | created | icmp_code | icmp_type | related | type | vpc_id | traffic_type | display |
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
| 157 | 160544dc-303a-4449-9ee5-9aa7c74ecd8e | 4433 | 6443 | 6443 | Active | tcp | Firewall | 23 | 1 | 267 | 592c3df8-3804-4ccb-8030-f1a16c77e431 | 2023-05-27 08:18:26 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 159 | 53a32814-ab9a-413d-9fba-834c60cbe41c | 4433 | 6443 | 6443 | Active | tcp | LoadBalancing | 23 | 1 | 267 | 8bfbb10f-5577-4a22-b20e-19f7a6c155ff | 2023-05-27 08:18:29 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 169 | 11c33409-788b-4639-9e77-39f124a97cd8 | 4433 | 3222 | 3222 | Active | tcp | PortForwarding | 23 | 1 | 267 | b3c13c77-0d29-4fe1-9d6d-6e023acab91c | 2023-05-29 20:28:54 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 171 | 97f9a76f-36f1-4f2b-80da-6fedf100edcb | 4433 | 3222 | 3222 | Active | tcp | Firewall | 23 | 1 | 267 | aac0f2b6-488b-4e6b-aff7-47e26dc588f8 | 2023-05-29 20:32:47 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 202 | ede86b93-2fab-4662-a75f-a7eaee50c736 | 4433 | 2222 | 2222 | Active | tcp | PortForwarding | 23 | 1 | 267 | e1645b58-8430-42e4-9108-2f474e71dc34 | 2023-05-31 13:12:51 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 203 | 1ae4c78a-5736-4a14-b22d-5dc4876f3ebc | 4433 | 2223 | 2223 | Active | tcp | PortForwarding | 23 | 1 | 267 | 35d2cbb1-9644-4a2c-ab59-e9de8cf32219 | 2023-05-31 13:12:52 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 204 | 2ae6750d-5d51-4812-b60f-abdba5c8cc37 | 4433 | 2224 | 2224 | Active | tcp | PortForwarding | 23 | 1 | 267 | d20b18a2-9287-415a-bcd2-69cf4c08414b | 2023-05-31 13:12:54 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 205 | 315e1cb9-f0d2-438c-92b2-babbb346b8b9 | 4433 | 2225 | 2225 | Active | tcp | PortForwarding | 23 | 1 | 267 | a971ffbe-cf56-4191-8589-8f660116124c | 2023-05-31 13:12:55 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 206 | 3cc5707a-a5f0-4eaa-98db-31190e4f777a | 4433 | 2226 | 2226 | Active | tcp | PortForwarding | 23 | 1 | 267 | de06f445-9db3-4652-8ef4-6dc82e2f36ef | 2023-05-31 13:12:56 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 207 | a568da68-a0a8-4eae-9e12-8ed0ebd55a8a | 4433 | 2227 | 2227 | Active | tcp | PortForwarding | 23 | 1 | 267 | 4e7bebda-591c-4051-bab3-a829ee3e888c | 2023-05-31 13:12:57 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 210 | 878750f8-cdaa-42bf-af6a-46a5d531eab3 | 4433 | 3223 | 3223 | Active | tcp | Firewall | 23 | 1 | 267 | 4a497f27-af34-4ff5-9d8f-77be06cf080f | 2023-05-31 17:10:58 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 211 | 40785ce1-ca7d-46f3-ac87-45c52dccd078 | 4433 | 3223 | 3223 | Active | tcp | PortForwarding | 23 | 1 | 267 | 45fff41a-8d5e-446f-9766-03ce2ffc522a | 2023-05-31 17:11:29 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 259 | 227e9e80-008f-4598-8474-f983d559d355 | 4433 | 3224 | 3224 | Active | tcp | PortForwarding | 23 | 1 | 267 | 3086de53-2d84-49a5-b52d-b62a6ac23943 | 2023-06-21 13:57:34 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 260 | f93c69ce-598e-4d5d-b9d6-a122bcb937da | 4433 | 3224 | 3224 | Active | tcp | Firewall | 23 | 1 | 267 | b9d22186-6bbd-4eed-a864-97fe3e44698e | 2023-06-21 13:58:08 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 281 | 21962db8-97f1-4709-8085-70bfa5cfa47f | 4433 | 35432 | 35432 | Active | tcp | PortForwarding | 23 | 1 | 267 | 5eddfc85-fec9-48b3-b4e6-8f2a44669c43 | 2023-06-23 09:32:01 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 282 | a2e99cd4-a3e9-493f-a461-d3827b5b8c6d | 4433 | 35432 | 35432 | Active | tcp | Firewall | 23 | 1 | 267 | 416c60de-3b8a-4891-8bd7-729b6ea31fb1 | 2023-06-23 09:39:29 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 284 | 8505e874-127b-4523-af26-0c4462608a3f | 4433 | NULL | NULL | Active | icmp | Firewall | 23 | 1 | 267 | 5f2e4b46-91f6-4c1f-a24f-5480cb63d052 | 2023-07-02 10:25:40 | -1 | -1 | NULL | User | NULL | Ingress | 1 |
| 1470 | ae9602cb-ad43-4251-8b1d-9ca0341a6109 | 4433 | 32727 | 32727 | Active | tcp | LoadBalancing | 23 | 1 | 267 | 11ace4c3-7c0e-4ec7-ae22-44dd06e8ce5c | 2023-07-08 13:20:19 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 1471 | 6a2753e3-3f87-4183-b541-3274dac0c724 | 4433 | 32727 | 32727 | Active | tcp | Firewall | 23 | 1 | 267 | ed955550-d415-4f8d-8460-4983f5b8fb29 | 2023-07-08 13:20:19 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 3211 | 144fbdaa-b95e-4a86-ad8b-bc777e87e7c3 | 4433 | 5601 | 5601 | Active | tcp | PortForwarding | 23 | 1 | 267 | f39579fb-8d84-448b-86e9-c77939b42e95 | 2023-07-14 12:19:22 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 3213 | 1bc0cef8-dace-48f8-b0ae-bda9a9451797 | 4433 | 1 | 65535 | Active | tcp | Firewall | 23 | 1 | 267 | c3bd9b08-9957-481e-a81a-ef470ac744f6 | 2023-07-14 12:21:51 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 3214 | b5b6b139-839a-47d0-80da-8f6b775855c4 | 4433 | 1 | 65535 | Active | tcp | Firewall | 23 | 1 | 267 | ef6eb373-b177-4f5b-bc42-3b9604629f67 | 2023-07-14 12:22:11 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 3223 | cee5dc3d-a53e-43a1-ab52-cfadfd223382 | 4433 | 1 | 65535 | Active | tcp | Firewall | 23 | 1 | 267 | 0ca97b93-9784-4c5b-b06d-22079340f8dc | 2023-07-14 13:04:01 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 15760 | 4fafe230-fc83-4540-ba11-271c0d717c75 | 4433 | 2228 | 2230 | Active | tcp | Firewall | 23 | 1 | 267 | 48f8bf55-cc62-4faa-a95d-bd244dbfcf98 | 2023-09-14 08:25:01 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 39225 | 9087fdc2-1f6c-4ec3-902e-a7cd1b7b8b08 | 4433 | 2228 | 2228 | Active | tcp | PortForwarding | 23 | 1 | 267 | 69417616-fd55-40f6-bfcd-14b2a21b6137 | 2023-12-05 10:02:08 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 48706 | 910ced9a-965e-40cf-94d4-1cb40507e576 | 4433 | 1 | 65535 | Active | tcp | Firewall | 23 | 1 | 267 | 43342855-4472-4686-b44b-7709545163a2 | 2024-01-09 02:40:01 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 48707 | 4ed50fde-a0c9-4b4b-8bad-7d6d9eb95948 | 4433 | 2929 | 2929 | Active | tcp | Firewall | 23 | 1 | 267 | f33949bd-bb92-4ead-85df-1b18ae26ad0f | 2024-01-17 03:23:08 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 48708 | 8b351e99-4c26-402b-971c-11decac854ff | 4433 | 500 | 500 | Staged | udp | Vpn | 23 | 1 | 267 | bc287688-aedc-4911-a7fb-6911327855c8 | 2024-01-21 06:13:47 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 48709 | af332d08-8cb7-4d78-b466-fadc001cc26c | 4433 | 500 | 500 | Active | udp | Firewall | 23 | 1 | 267 | 27884745-0c75-455f-9a75-ce6e6d88c325 | 2024-01-21 06:13:47 | NULL | NULL | 48708 | User | NULL | Ingress | 1 |
| 48710 | 6d7c444b-1044-48b6-99b1-a7c125fa479c | 4433 | 1701 | 1701 | Staged | udp | Vpn | 23 | 1 | 267 | 433eaf8d-a020-4899-8aab-1c2777871e5f | 2024-01-21 06:13:47 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 48711 | 3fa7d224-1608-4125-83e4-7107388ad66c | 4433 | 1701 | 1701 | Active | udp | Firewall | 23 | 1 | 267 | 9b486a5b-a359-47dc-88cc-f760dfd44a2d | 2024-01-21 06:13:47 | NULL | NULL | 48710 | User | NULL | Ingress | 1 |
| 48712 | bd0aa221-32f0-48d5-8f7a-5bbe4e3a8dbe | 4433 | 4500 | 4500 | Staged | udp | Vpn | 23 | 1 | 267 | 580ceedd-d867-41d7-93f2-e50737f9ed97 | 2024-01-21 06:13:47 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 48713 | b7c9636c-5421-45fc-b779-e600e1c24db8 | 4433 | 4500 | 4500 | Active | udp | Firewall | 23 | 1 | 267 | bfe7a1bc-c862-41a1-bdd6-e9274716310b | 2024-01-21 06:13:47 | NULL | NULL | 48712 | User | NULL | Ingress | 1 |
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
33 rows in set (0.00 sec)
EXPECTED RESULTS
Fix the problem
ACTUAL RESULTS
2024-04-18 22:07:11,414 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 392, job origin: 4626
2024-04-18 22:07:11,415 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Done executing com.cloud.vm.VmWorkStart for job-4627
2024-04-18 22:07:11,417 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Remove job-4627 from job monitoring
2024-04-18 22:07:11,421 INFO [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-7a5b66c2 job-4626 ctx-9e48efab) (logid:e142ff5b) Started VM : mediatech-node-18ef29acb54 in the Kubernetes cluster : mediatech
2024-04-18 22:07:11,424 INFO [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-7a5b66c2 job-4626 ctx-9e48efab) (logid:e142ff5b) Provisioned node VM : mediatech-node-18ef29acb54 in to the Kubernetes cluster : mediatech
2024-04-18 22:07:11,435 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-1:ctx-7a5b66c2 job-4626) (logid:e142ff5b) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd
java.lang.NullPointerException
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.removeSshFirewallRule(KubernetesClusterResourceModifierActionWorker.java:520)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterIsolatedNetworkRules(KubernetesClusterScaleWorker.java:124)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterNetworkRules(KubernetesClusterScaleWorker.java:174)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleUpKubernetesClusterSize(KubernetesClusterScaleWorker.java:376)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterSize(KubernetesClusterScaleWorker.java:412)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClusterScaleWorker.java:465)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterManagerImpl.java:1548)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKubernetesClusterCmd.java:156)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-04-18 22:07:11,448 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-7a5b66c2 job-4626) (logid:e142ff5b) Complete async job-4626, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530"}