Create a Windows workload cluster

Overview

CAPOCI enables users to create and manage Windows workload clusters in Oracle Cloud Infrastructure (OCI). This means that the Kubernetes Control Plane will be Linux and the nodes will be Windows. First, users build the Windows image using image-builder, then use the Windows flavor template from the latest release. Finally, install the Calico CNI Provider and OCI Cloud Controller Manager.

Known Limitations

The Windows workload cluster has known limitations:

• Limited to Standard Bare Metal (BM) shapes
• Limited to OCI provided platform images. See image-build documentation for more details
• Custom image MUST be built using the same shape of Bare Metal the worker nodes will run
• CNI provider support is Calico in VXLAN mode
• Block volumes are not currently supported
• Bring Your Own License (BYOL) is not supported
• See Calico windows docs for their limitations

Licensing

BYOL is currently not supported using CAPOCI. For more info on Windows Licensing see the Compute FAQ documentation.

Build Windows image

NOTE: It is recommended to check shape availability before building image(s)

In order to launch Windows instances for the cluster a Windows image, using image-builder, will need to be built. It is important to make sure the same shape is used to build and launch the instance.

Example: If a BM.Standard2.52 is used to build then the OCI_NODE_MACHINE_TYPE MUST be BM.Standard2.52

Check shape availability

Make sure the OCI CLI is installed. Then set the AD information if using muti-AD regions.

NOTE: Use the OCI Regions and Availability Domains page to figure out which regions have multiple ADs.

oci iam availability-domain list --compartment-id=<your compartment> --region=<region>

Using the AD name from the output above start searching for BM shape availability.

oci compute shape list --compartment-id=<your compartment> --profile=DEFAULT --region=us-ashburn-1 --availability-domain=<your AD ID> | grep BM
 
"shape": "BM.Standard.E3.128"
"shape-name": "BM.Standard2.52"
"shape-name": "BM.Standard.E3.128"
"shape": "BM.Standard.E2.64"
"shape-name": "BM.Standard2.52"
"shape-name": "BM.Standard3.64"
"shape": "BM.Standard1.36"

NOTE: If the output is empty then the compartment for that region/AD doesn't have BM shapes. If you are unable to locate any shapes you may need to submit a service limit increase request

Create a new Windows workload cluster

It is recommended to have the following guides handy:

• Windows Cluster Debugging
• Windows Container Debugging

When using clusterctl to generate the cluster use the windows-calico example flavor.

The following command uses the OCI_CONTROL_PLANE_MACHINE_TYPE and OCI_NODE_MACHINE_TYPE parameters to specify bare metal shapes instead of using CAPOCI's default virtual instance shape. The OCI_CONTROL_PLANE_PV_TRANSIT_ENCRYPTION and OCI_NODE_PV_TRANSIT_ENCRYPTION parameters disable encryption of data in flight between the bare metal instance and the block storage resources.

NOTE: The OCI_NODE_MACHINE_TYPE_OCPUS must match the OPCU count of the BM shape. See the Compute Shapes page to get the OCPU count for the specific shape.

OCI_COMPARTMENT_ID=<compartment-id> \
OCI_CONTROL_PLANE_IMAGE_ID=<linux-custom-image-id> \
OCI_NODE_IMAGE_ID=<windows-custom-image-id> \
OCI_SSH_KEY=<ssh-key>  \
NODE_MACHINE_COUNT=1 \
OCI_NODE_MACHINE_TYPE=BM.Standard.E4.128 \
OCI_NODE_MACHINE_TYPE_OCPUS=128 \
OCI_NODE_PV_TRANSIT_ENCRYPTION=false \
OCI_CONTROL_PLANE_MACHINE_TYPE_OCPUS=3 \
OCI_CONTROL_PLANE_MACHINE_TYPE=VM.Standard3.Flex \
CONTROL_PLANE_MACHINE_COUNT=3 \
OCI_SHAPE_MEMORY_IN_GBS= \
KUBERNETES_VERSION=<k8s-version> \
clusterctl generate cluster <cluster-name> \
--target-namespace default \
--flavor windows-calico | kubectl apply -f -

Access workload cluster Kubeconfig

Execute the following command to list all the workload clusters present:

kubectl get clusters -A

Execute the following command to access the kubeconfig of a workload cluster:

clusterctl get kubeconfig <cluster-name> -n default > <cluster-name>.kubeconfig

Install Calico CNI Provider and OCI Cloud Controller Manager

Install Calico

The Calico for Windows getting started guide should be read for better understand of the CNI on Windows. It is recommended to have the following guides handy:

• Windows Calico Troubleshooting

The steps to follow:

On the management cluster

1. Run

   kubectl get OCICluster <cluster-name> -o jsonpath='{.spec.controlPlaneEndpoint.host}'
   

   to get the KUBERNETES_SERVICE_HOST info that will be used in later steps

On the workload cluster

1. Download the v3.24.5 calico release

   curl -L https://github.com/projectcalico/calico/releases/download/v3.24.5/release-v3.24.5.tgz -o calico-v3.24.5.tgz
   

2. Uncompress the downloaded file and locate the calico-vxlan.yaml, calico-windows-vxlan.yaml and windows-kube-proxy.yaml files in the manifests dir
3. Edit the calico-vxlan.yaml and modify the follow variables to allow Calico running on the nodes use VXLAN
   • CALICO_IPV4POOL_IPIP - set to "Never"
   • CALICO_IPV4POOL_VXLAN - set to "Always"

4. kubectl apply -f calico-vxlan.yaml
   

5. Wait for the IPAMConfig to be loaded

6. kubectl patch IPAMConfig default --type merge --patch='{"spec": {"strictAffinity": true}}'
   

7. Edit the calico-windows-vxlan.yaml and modify the follow variables to allow Calico running on the nodes to talk to the Kubernetes control plane
   • KUBERNETES_SERVICE_HOST - the IP address from the management cluster step
   • KUBERNETES_SERVICE_PORT- the port from step the management cluster step
   • K8S_SERVICE_CIDR - The service CIDR set in the cluster template
   • DNS_NAME_SERVERS - the IP address from dns service

     kubectl  get svc kube-dns -n kube-system  -o jsonpath='{.spec.clusterIP}'
     

   • Change the namespace from calico-system to kube-system
   • add the following env to the container named node

     - name: VXLAN_ADAPTER
       value: "Ethernet 2"
     

8. kubectl apply -f calico-windows-vxlan.yaml
   

   (it takes a bit for this to pass livenessprobe)
9. Edit the windows-kube-proxy.yaml
   • update the kube-proxy container environment variable K8S_VERSION to the version of kubernetes you are deploying
   • update the image version for the container named kube-proxy and make sure to set the correct windows nanoserver version example: ltsc2019

10. kubectl apply -f windows-kube-proxy.yaml
    

Install OCI Cloud Controller Manager

By default, the OCI Cloud Controller Manager (CCM) is not installed into a workload cluster. To install the OCI CCM, follow these instructions.

Scheduling Windows containers

With the cluster in a ready state and CCM installed, an example deployment can be used to test that pods are scheduled. Accessing the deployed pods and using nslookup you can test that the cluster DNS is working.