Skip to content

SPLAT-2649: Added vSphere Day 2 logic to CRDs #2784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vr4manta wants to merge 3 commits into
base: master
Choose a base branch
from
Open

SPLAT-2649: Added vSphere Day 2 logic to CRDs #2784

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5f653bb
Added vSphere Day 2 logic to CRDs
vr4manta Mar 27, 2026
09e2b6d
Added vSphere Day 2 tests
vr4manta Apr 2, 2026
ed1ace3
Modified parent logic to not check size, but only changes in existenc…
vr4manta Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
774 changes: 774 additions & 0 deletions config/v1/tests/infrastructures.config.openshift.io/VSphereMultiVCenterDay2.yaml
View file
Open in desktop

Large diffs are not rendered by default.

16 changes: 9 additions & 7 deletions config/v1/types_infrastructure.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,8 @@ type ExternalPlatformSpec struct {
// PlatformSpec holds the desired state specific to the underlying infrastructure provider
// of the current cluster. Since these are used at spec-level for the underlying cluster, it
// is supposed that only one of the spec structs is set.
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.vsphere) && has(self.vsphere) ? size(self.vsphere.vcenters) < 2 : true",message="vcenters can have at most 1 item when configured post-install"
// +openshift:validation:FeatureGateAwareXValidation:featureGate="",rule="!has(oldSelf.vsphere) && has(self.vsphere) ? (has(self.vsphere.vcenters) && size(self.vsphere.vcenters) < 2) : true",message="vcenters can have at most 1 item when configured post-install"
// +openshift:validation:FeatureGateAwareXValidation:featureGate=VSphereMultiVCenterDay2,rule="oldSelf.?vsphere.vcenters.hasValue() ? self.?vsphere.vcenters.hasValue() : true",message="vcenters is required once set and cannot be removed"
type PlatformSpec struct {
// type is the underlying infrastructure provider for the cluster. This
// value controls whether infrastructure automation such as service load
Expand Down Expand Up @@ -1641,21 +1642,22 @@ type VSpherePlatformNodeNetworking struct {
// use these fields for configuration.
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.apiServerInternalIPs) || has(self.apiServerInternalIPs)",message="apiServerInternalIPs list is required once set"
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.ingressIPs) || has(self.ingressIPs)",message="ingressIPs list is required once set"
// +kubebuilder:validation:XValidation:rule="!has(oldSelf.vcenters) && has(self.vcenters) ? size(self.vcenters) < 2 : true",message="vcenters can have at most 1 item when configured post-install"
type VSpherePlatformSpec struct {
// vcenters holds the connection details for services to communicate with vCenter.
// Currently, only a single vCenter is supported, but in tech preview 3 vCenters are supported.
// Currently, up to 3 vCenters are supported.
// Once the cluster has been installed, you are unable to change the current number of defined
// vCenters except in the case where the cluster has been upgraded from a version of OpenShift
// where the vsphere platform spec was not present. You may make modifications to the existing
// vCenters except when 1.) the cluster has been upgraded from a version of OpenShift
// where the vsphere platform spec was not present or 2.) in TechPreview you are able to add and
// remove vCenters but may not remove all vCenters. You may make modifications to the existing
// vCenters that are defined in the vcenters list in order to match with any added or modified
// failure domains.
// ---
// + If VCenters is not defined use the existing cloud-config configmap defined
// + in openshift-config.
// +kubebuilder:validation:MinItems=0
// +kubebuilder:validation:MinItems=1
Copy link
Copy Markdown
Contributor

@everettraven everettraven Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When making this change, let's make sure we add some test cases similar to this

api/example/v1/tests/stableconfigtypes.example.openshift.io/AAA_ungated.yaml

Lines 902 to 964 in 464776f

- name: Should allow changing other fields when a persisted value is no longer valid
initialCRDPatches:
- op: remove
path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/nonZeroDefault/minimum # Minimum is normally 8
initial: |
apiVersion: example.openshift.io/v1
kind: StableConfigType
spec:
nonZeroDefault: 1
immutableField: foo
updated: |
apiVersion: example.openshift.io/v1
kind: StableConfigType
spec:
nonZeroDefault: 1
immutableField: foo
optionalImmutableField: foo
expected: |
apiVersion: example.openshift.io/v1
kind: StableConfigType
spec:
nonZeroDefault: 1
immutableField: foo
optionalImmutableField: foo
- name: Should allow updating a persisted value that is no longer valid to a valid value
initialCRDPatches:
- op: remove
path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/nonZeroDefault/minimum # Minimum is normally 8
initial: |
apiVersion: example.openshift.io/v1
kind: StableConfigType
spec:
nonZeroDefault: 1
immutableField: foo
updated: |
apiVersion: example.openshift.io/v1
kind: StableConfigType
spec:
nonZeroDefault: 8
immutableField: foo
expected: |
apiVersion: example.openshift.io/v1
kind: StableConfigType
spec:
nonZeroDefault: 8
immutableField: foo
- name: Should not allow updating a persisted value that is no longer valid to a still invalid value
initialCRDPatches:
- op: remove
path: /spec/versions/0/schema/openAPIV3Schema/properties/spec/properties/nonZeroDefault/minimum # Minimum is normally 8
initial: |
apiVersion: example.openshift.io/v1
kind: StableConfigType
spec:
nonZeroDefault: 1
immutableField: foo
updated: |
apiVersion: example.openshift.io/v1
kind: StableConfigType
spec:
nonZeroDefault: 2
immutableField: foo
expectedError: "spec.nonZeroDefault: Invalid value: 2: spec.nonZeroDefault in body should be greater than or equal to 8"

Having tests like that will help verify that this validation change is ratcheting successfully.

Copy link
Copy Markdown
Contributor Author

@vr4manta vr4manta Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can do another pass at the tests. I thought they were covering what we expected. If you are seeing any missing, that would help me narrow down the scope. Is the concern that we need to add more "AAA_ungated" tests? I always assumed those were the existing tests for the type to make sure current functionality does not break.

Copy link
Copy Markdown
Contributor

@everettraven everettraven Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because the validation change is ungated, more AAA_ungated tests would be what I'm looking for.

Specifically these scenarios:

  • Modifying other fields
  • Updating invalid value to now valid value

when vcenters: [] is explicitly set.

// +kubebuilder:validation:MaxItems=3
// +kubebuilder:validation:XValidation:rule="size(self) != size(oldSelf) ? size(oldSelf) == 0 && size(self) < 2 : true",message="vcenters cannot be added or removed once set"
// +openshift:validation:FeatureGateAwareXValidation:featureGate="",rule="size(self) != size(oldSelf) ? size(oldSelf) == 0 && size(self) < 2 : true",message="vcenters cannot be added or removed once set"
Copy link
Copy Markdown
Contributor

@JoelSpeed JoelSpeed Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I completely change the configuration of the vcenter, but keep the number the same, is that ok? Effectively like adding an item to the list, then removing the old item, but in one transaction?

Copy link
Copy Markdown
Contributor Author

@vr4manta vr4manta Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. that would be be ok now going forward since we'll be cleaning up older configs from the cloud config via the other PRs; however, the older releases this would not be allowed. Its the configuration of vcenters in the ini / yaml file that we are protecting. Older cluster installs (before multi vcenter was introduced) had ini files and that only allowed 1 vcenter. Newer installs now leverage YAML cloud config and we can configure more than 1. So even replacing the existing one w/ a new one will invalidate the INI file and cluster will be in bad shape.

Copy link
Copy Markdown
Contributor

@everettraven everettraven Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In another comment, you mention backporting - what is potentially being backported here? Do we need to account for this difference in older vs newer clusters?

Copy link
Copy Markdown
Contributor Author

@vr4manta vr4manta Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tldr

we are backporting the ability to allow adding vcenters day 2. Many customers want to migrate to a new vcenter or add new ones but not reinstall to get multi vcenter support. This involves feature gate backport and all changes related to this feature.

The long story:

Many of the checks I originally had in place account for all past states of clusters. Back in the beginning, the infrastructure spec did not have a vsphere.vcenters field. So any super old cluster will not have this. That is why we check for if initial did not exist and how we only allow it adding 1 MAX. With our feature gate change, we want to allow them to add 1 or more to the vcenters field. We will never allow the field returning to zero or not set.

Joel is recommending we change this field to be max of 1. This is because once the field is set, the minimum value should be 1 forever. A cluster with no vcenters configured is invalid and will break. For older clusters where vsphere.vcenters is not configured, we just honor whats currently configured in the old school cloud config ini file.

Another good point Joel made was a situation where user has 1 vcenter defined and tries to replace it w/ a new one. This means size is still 1 after the update. In this case, we would need validation to make sure that the failure domains defined have a vcenter configured that matches its config. We do have a few places that validate this. I am not sure if CEL is the best place for this. Not sure who owns validating the infrastructure spec from a hook perspective, but we do have the storage vsphereproblemdetector who does some validations of cluster config.

Lastly, once this is backported and functionality is tested and approved, we want to GA all the way back so customers can now migrate their old clusters to new clusters or add additional vcenters to the mix. In my mind, the featureGate="" validations would be removed and we'd only be keeping the ones where the feature gate exists (since when its GA, the newer fields will be the only one exercised).

I originally set up a meeting with Joel for yesterday but he was out. We can set up a meeting with the three of us next week to further discuss. I wanted to do that to make sure everything is clear w/o reviews taking a long time. Let me know your thoughts. Thanks!

Copy link
Copy Markdown
Contributor

@everettraven everettraven Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A meeting seems like a good way to make sure we are on the same page here.

The minimum of 1 change seems fine to me.

I am not sure if CEL is the best place for this. Not sure who owns validating the infrastructure spec from a hook perspective, but we do have the storage vsphereproblemdetector who does some validations of cluster config.

If you are wanting to make sure that all old configurations are represented in the new list, you should be able to enforce that via CEL with something like:

oldSelf.all(x, x in self)

// +openshift:validation:FeatureGateAwareXValidation:featureGate=VSphereMultiVCenterDay2,rule="true",message=""
// +listType=atomic
// +optional
VCenters []VSpherePlatformVCenterSpec `json:"vcenters,omitempty"`
Expand Down
22 changes: 9 additions & 13 deletions ...nerated.crd-manifests/0000_10_config-operator_01_infrastructures-CustomNoUpgrade.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1031,10 +1031,11 @@ spec:
vcenters:
description: |-
vcenters holds the connection details for services to communicate with vCenter.
Currently, only a single vCenter is supported, but in tech preview 3 vCenters are supported.
Currently, up to 3 vCenters are supported.
Once the cluster has been installed, you are unable to change the current number of defined
vCenters except in the case where the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present. You may make modifications to the existing
vCenters except when 1.) the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present or 2.) in TechPreview you are able to add and
remove vCenters but may not remove all vCenters. You may make modifications to the existing
vCenters that are defined in the vcenters list in order to match with any added or modified
failure domains.
items:
Expand Down Expand Up @@ -1079,27 +1080,22 @@ spec:
- server
type: object
maxItems: 3
minItems: 0
minItems: 1
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
- message: vcenters cannot be added or removed once set
rule: 'size(self) != size(oldSelf) ? size(oldSelf) == 0
&& size(self) < 2 : true'
- rule: "true"
type: object
x-kubernetes-validations:
- message: apiServerInternalIPs list is required once set
rule: '!has(oldSelf.apiServerInternalIPs) || has(self.apiServerInternalIPs)'
- message: ingressIPs list is required once set
rule: '!has(oldSelf.ingressIPs) || has(self.ingressIPs)'
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vcenters) && has(self.vcenters) ? size(self.vcenters)
< 2 : true'
type: object
x-kubernetes-validations:
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vsphere) && has(self.vsphere) ? size(self.vsphere.vcenters)
< 2 : true'
- message: vcenters is required once set and cannot be removed
rule: 'oldSelf.?vsphere.vcenters.hasValue() ? self.?vsphere.vcenters.hasValue()
: true'
type: object
status:
description: status holds observed values from the cluster. They may not
Expand Down
16 changes: 7 additions & 9 deletions ...v1/zz_generated.crd-manifests/0000_10_config-operator_01_infrastructures-Default.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -965,10 +965,11 @@ spec:
vcenters:
description: |-
vcenters holds the connection details for services to communicate with vCenter.
Currently, only a single vCenter is supported, but in tech preview 3 vCenters are supported.
Currently, up to 3 vCenters are supported.
Once the cluster has been installed, you are unable to change the current number of defined
vCenters except in the case where the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present. You may make modifications to the existing
vCenters except when 1.) the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present or 2.) in TechPreview you are able to add and
remove vCenters but may not remove all vCenters. You may make modifications to the existing
vCenters that are defined in the vcenters list in order to match with any added or modified
failure domains.
items:
Expand Down Expand Up @@ -1013,7 +1014,7 @@ spec:
- server
type: object
maxItems: 3
minItems: 0
minItems: 1
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
Expand All @@ -1026,14 +1027,11 @@ spec:
rule: '!has(oldSelf.apiServerInternalIPs) || has(self.apiServerInternalIPs)'
- message: ingressIPs list is required once set
rule: '!has(oldSelf.ingressIPs) || has(self.ingressIPs)'
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vcenters) && has(self.vcenters) ? size(self.vcenters)
< 2 : true'
type: object
x-kubernetes-validations:
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vsphere) && has(self.vsphere) ? size(self.vsphere.vcenters)
< 2 : true'
rule: '!has(oldSelf.vsphere) && has(self.vsphere) ? (has(self.vsphere.vcenters)
&& size(self.vsphere.vcenters) < 2) : true'
type: object
status:
description: status holds observed values from the cluster. They may not
Expand Down
22 changes: 9 additions & 13 deletions ...ted.crd-manifests/0000_10_config-operator_01_infrastructures-DevPreviewNoUpgrade.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1031,10 +1031,11 @@ spec:
vcenters:
description: |-
vcenters holds the connection details for services to communicate with vCenter.
Currently, only a single vCenter is supported, but in tech preview 3 vCenters are supported.
Currently, up to 3 vCenters are supported.
Once the cluster has been installed, you are unable to change the current number of defined
vCenters except in the case where the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present. You may make modifications to the existing
vCenters except when 1.) the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present or 2.) in TechPreview you are able to add and
remove vCenters but may not remove all vCenters. You may make modifications to the existing
vCenters that are defined in the vcenters list in order to match with any added or modified
failure domains.
items:
Expand Down Expand Up @@ -1079,27 +1080,22 @@ spec:
- server
type: object
maxItems: 3
minItems: 0
minItems: 1
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
- message: vcenters cannot be added or removed once set
rule: 'size(self) != size(oldSelf) ? size(oldSelf) == 0
&& size(self) < 2 : true'
- rule: "true"
type: object
x-kubernetes-validations:
- message: apiServerInternalIPs list is required once set
rule: '!has(oldSelf.apiServerInternalIPs) || has(self.apiServerInternalIPs)'
- message: ingressIPs list is required once set
rule: '!has(oldSelf.ingressIPs) || has(self.ingressIPs)'
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vcenters) && has(self.vcenters) ? size(self.vcenters)
< 2 : true'
type: object
x-kubernetes-validations:
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vsphere) && has(self.vsphere) ? size(self.vsphere.vcenters)
< 2 : true'
- message: vcenters is required once set and cannot be removed
rule: 'oldSelf.?vsphere.vcenters.hasValue() ? self.?vsphere.vcenters.hasValue()
: true'
type: object
status:
description: status holds observed values from the cluster. They may not
Expand Down
16 changes: 7 additions & 9 deletions config/v1/zz_generated.crd-manifests/0000_10_config-operator_01_infrastructures-OKD.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -965,10 +965,11 @@ spec:
vcenters:
description: |-
vcenters holds the connection details for services to communicate with vCenter.
Currently, only a single vCenter is supported, but in tech preview 3 vCenters are supported.
Currently, up to 3 vCenters are supported.
Once the cluster has been installed, you are unable to change the current number of defined
vCenters except in the case where the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present. You may make modifications to the existing
vCenters except when 1.) the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present or 2.) in TechPreview you are able to add and
remove vCenters but may not remove all vCenters. You may make modifications to the existing
vCenters that are defined in the vcenters list in order to match with any added or modified
failure domains.
items:
Expand Down Expand Up @@ -1013,7 +1014,7 @@ spec:
- server
type: object
maxItems: 3
minItems: 0
minItems: 1
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
Expand All @@ -1026,14 +1027,11 @@ spec:
rule: '!has(oldSelf.apiServerInternalIPs) || has(self.apiServerInternalIPs)'
- message: ingressIPs list is required once set
rule: '!has(oldSelf.ingressIPs) || has(self.ingressIPs)'
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vcenters) && has(self.vcenters) ? size(self.vcenters)
< 2 : true'
type: object
x-kubernetes-validations:
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vsphere) && has(self.vsphere) ? size(self.vsphere.vcenters)
< 2 : true'
rule: '!has(oldSelf.vsphere) && has(self.vsphere) ? (has(self.vsphere.vcenters)
&& size(self.vsphere.vcenters) < 2) : true'
type: object
status:
description: status holds observed values from the cluster. They may not
Expand Down
16 changes: 7 additions & 9 deletions ...ed.crd-manifests/0000_10_config-operator_01_infrastructures-TechPreviewNoUpgrade.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1031,10 +1031,11 @@ spec:
vcenters:
description: |-
vcenters holds the connection details for services to communicate with vCenter.
Currently, only a single vCenter is supported, but in tech preview 3 vCenters are supported.
Currently, up to 3 vCenters are supported.
Once the cluster has been installed, you are unable to change the current number of defined
vCenters except in the case where the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present. You may make modifications to the existing
vCenters except when 1.) the cluster has been upgraded from a version of OpenShift
where the vsphere platform spec was not present or 2.) in TechPreview you are able to add and
remove vCenters but may not remove all vCenters. You may make modifications to the existing
vCenters that are defined in the vcenters list in order to match with any added or modified
failure domains.
items:
Expand Down Expand Up @@ -1079,7 +1080,7 @@ spec:
- server
type: object
maxItems: 3
minItems: 0
minItems: 1
type: array
x-kubernetes-list-type: atomic
x-kubernetes-validations:
Expand All @@ -1092,14 +1093,11 @@ spec:
rule: '!has(oldSelf.apiServerInternalIPs) || has(self.apiServerInternalIPs)'
- message: ingressIPs list is required once set
rule: '!has(oldSelf.ingressIPs) || has(self.ingressIPs)'
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vcenters) && has(self.vcenters) ? size(self.vcenters)
< 2 : true'
type: object
x-kubernetes-validations:
- message: vcenters can have at most 1 item when configured post-install
rule: '!has(oldSelf.vsphere) && has(self.vsphere) ? size(self.vsphere.vcenters)
< 2 : true'
rule: '!has(oldSelf.vsphere) && has(self.vsphere) ? (has(self.vsphere.vcenters)
&& size(self.vsphere.vcenters) < 2) : true'
type: object
status:
description: status holds observed values from the cluster. They may not
Expand Down
1 change: 1 addition & 0 deletions config/v1/zz_generated.featuregated-crd-manifests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,7 @@ infrastructures.config.openshift.io:
- OnPremDNSRecords
- VSphereHostVMGroupZonal
- VSphereMultiNetworks
- VSphereMultiVCenterDay2
FilenameOperatorName: config-operator
FilenameOperatorOrdering: "01"
FilenameRunLevel: "0000_10"
Expand Down
Loading