3.2. Common instance types and preferences

Discover and use common instance types and preferences provided by KubeVirt

The KubeVirt project provides a set of common instance types and preferences . In this lab, we are going to have a look at these and how they can help us create virtual machines.

Deployment of common instance types and preferences

The common instance types and preferences are not available by default. They have to be deployed manually or by using one of the KubeVirt Operator’s feature gates.

You can check the configuration on the KubeVirt CustomResource:

kubectl get kubevirt kubevirt --namespace kubevirt -o jsonpath={.spec.configuration.developerConfiguration.featureGates}

or

kubectl get kubevirt kubevirt --namespace kubevirt -o yaml

The relevant section on the CustomResource is the following:

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
spec:
  configuration:
    developerConfiguration:
      featureGates:
        - CommonInstancetypesDeploymentGate
[...]

With the feature gate enabled, the Operator itself takes care of deploying the cluster wide common instance types and preferences.

Manually deploy the common instance types and preferences

This step has been done for your on the lab cluster. How to deploy the common instance types and preferences on your own cluster can be found in the documentation .

List and inspect instance types by cli

Be aware that the common instance types and preferences are cluster-wide resources. They are called VirtualMachineClusterInstancetype and VirtualMachineClusterPreference.

You can list the available instance types using:

kubectl get virtualmachineclusterinstancetype

A shortened output of the command above:

NAME          AGE
cx1.2xlarge   10m
[...]
gn1.2xlarge   10m
[...]
m1.2xlarge    10m
[...]
n1.2xlarge    10m
[...]
o1.2xlarge    10m
[...]
u1.2xlarge    10m
u1.4xlarge    10m
u1.8xlarge    10m
u1.large      10m
u1.medium     10m
u1.micro      10m
u1.nano       10m
u1.small      10m
u1.xlarge     10m

As you see the instance types follow a specific naming schema:

instanceTypeName = seriesName , "." , size;

seriesName = ( class | vendorClass ) , version;

class = "u" | "o" | "cx" | "g" | "m" | "n" | "rt";
vendorClass = "g" , vendorHint;
vendorHint = "n" | "i" | "a";
version = "1";

size = "nano" | "micro" | "small" | "medium" | "large" | [( "2" | "4" | "8" )] , "xlarge";

The class u, o, cx, g, m, n and rt mean the following:

  • U (universal) - Provides resources for general purpose applications. VMs will share CPU cores on a time-slice basis.
  • O (overcommitted) - Based on u with the only difference that memory will be overcommitted. Allows higher workload density.
  • CX (compute exclusive) - Provides exclusive compute resources for compute intensive applications.
  • GN (gpu nvidia) - Instance type for VMs which consume attached NVIDIA GPUs.
  • M (memory) - Provides resources for memory-intensive applications.
  • N (network) - Provide resources for network-intensive DPDK1 applications like Virtual Network Functions (VNF).
  • RT (realtime) - Provide resources for realtime-intensive applications.

We therefore can say that the classes u and o are agnostic to the workload. The other classes are optimized for specific workload.

You may see the details of an instance type by describing the resource:

kubectl describe virtualmachineclusterinstancetype o1.nano

As an example you will see that the instance type o1.nano has 1 CPU, 512Mi memory and overcommits memory by 50%. The following output is shortened:

Name:         o1.nano
Namespace:    
Labels:       app.kubernetes.io/component=kubevirt
              app.kubernetes.io/managed-by=virt-operator
              instancetype.kubevirt.io/class=overcommitted
              instancetype.kubevirt.io/common-instancetypes-version=v1.0.1
              instancetype.kubevirt.io/cpu=1
              instancetype.kubevirt.io/icon-pf=pficon-virtual-machine
              instancetype.kubevirt.io/memory=512Mi
              instancetype.kubevirt.io/vendor=kubevirt.io
              instancetype.kubevirt.io/version=1
Annotations:  instancetype.kubevirt.io/description:
                The O Series is based on the U Series, with the only difference
                being that memory is overcommitted.
                
                *O* is the abbreviation for "Overcommitted".
              instancetype.kubevirt.io/displayName: Overcommitted
API Version:  instancetype.kubevirt.io/v1beta1
Kind:         VirtualMachineClusterInstancetype
Spec:
  Cpu:
    Guest:  1
  Memory:
    Guest:               512Mi
    Overcommit Percent:  50
[...]

List and inspect preferences

kubectl get virtualmachineclusterpreference

A shortened output of the command above:

NAME                     AGE
alpine                   10m
centos.7                 10m
[...]

You may see the details of a preference by describing the resource:

kubectl describe virtualmachineclusterpreference cirros

As an example you’ll see that the instance type cirros has the requirements of 1 CPU, 256Mi memory. The following output is shortened:

Name:         cirros
Namespace:    
Labels:       app.kubernetes.io/component=kubevirt
              app.kubernetes.io/managed-by=virt-operator
              instancetype.kubevirt.io/common-instancetypes-version=v1.0.1
              instancetype.kubevirt.io/os-type=linux
              instancetype.kubevirt.io/vendor=kubevirt.io
Annotations:  iconClass: icon-cirros
              tags: hidden,kubevirt,cirros
API Version:  instancetype.kubevirt.io/v1beta1
Kind:         VirtualMachineClusterPreference
Metadata:
  [...]
Spec:
  Devices:
    Preferred Disk Bus:         virtio
    Preferred Interface Model:  virtio
  Requirements:
    Cpu:
      Guest:  1
    Memory:
      Guest:  256Mi
[...]

Querying for specific instance types and preferences with labels

Instancetypes

These instance types are labeled according to their specification. You can use labels to find the correct instance type.

Instance types are known to use the following labels:

instancetype.kubevirt.io/common-instancetypes-version: The version of the common-instancetypes project used to generate these resources.
instancetype.kubevirt.io/vendor: The vendor of the resource, this is always kubevirt.io upstream and should be changed by downstream vendors consuming the project.
instancetype.kubevirt.io/icon-pf: The suggested patternfly icon to use when displaying the resource.
instancetype.kubevirt.io/deprecated: If the resource has been deprecated ahead of removal in a future release of the common-instancetypes project.
instancetype.kubevirt.io/version: The version of instance type class the resources has been generated from.
instancetype.kubevirt.io/class: The class of the instance type.
instancetype.kubevirt.io/cpu: The number of vCPUs provided by the instance type.
instancetype.kubevirt.io/memory: The amount of memory provided by the instance type.
instancetype.kubevirt.io/numa: If NUMA guestmappingpassthrough is enabled by the instance type.
instancetype.kubevirt.io/dedicatedCPUPlacement: If dedicatedCPUPlacement is enabled by the instance type.
instancetype.kubevirt.io/isolateEmulatorThread: If isolateEmulatorThread is enabled by the instance type.
instancetype.kubevirt.io/hugepages: If hugepages are requested by the instance type.
instancetype.kubevirt.io/gpus: If GPUs are requested by the instance type.

As an example, you can query for 4 CPUs:

kubectl get virtualmachineclusterinstancetype --selector instancetype.kubevirt.io/cpu=4

Output will list all instance types with 4 CPUs:

NAME            AGE
cx1.xlarge      10m
cx1.xlarge1gi   10m
m1.xlarge       10m
m1.xlarge1gi    10m
n1.large        10m
n1.medium       10m
o1.xlarge       10m
rt1.xlarge      10m
u1.xlarge       10m

Preferences

Just like instance types, preferences are labeled with the following labels:

instancetype.kubevirt.io/common-instancetypes-version: The version of the common-instancetypes project used to generate these resources.
instancetype.kubevirt.io/vendor: The vendor of the resource, this is always kubevirt.io upstream and should be changed by downstream vendors consuming the project.
instancetype.kubevirt.io/icon-pf: The suggested patternfly icon to use when displaying the resource.
instancetype.kubevirt.io/deprecated: If the resource has been deprecated ahead of removal in a future release of the common-instancetypes project.
instancetype.kubevirt.io/os-type: The underlying type of the workload supported by the preference, current values are linux or windows.
instancetype.kubevirt.io/arch: The underlying architecture of the workload supported by the preference, current values are `arm64` or `amd64`.

We can use these labels to query preferences:

kubectl get virtualmachineclusterpreference --selector instancetype.kubevirt.io/os-type=linux

The output will list all preferences targeting the operating system linux (output shortened):

NAME                     AGE
alpine                   10m
centos.7                 10m
[...]
cirros                   10m
fedora                   10m
[...]
rhel.9.dpdk              10m
ubuntu                   10m
[...]

Understanding the different ways of specifying resources

During this lab, you might notice that there are different ways of specifying resources for VirtualMachines. This might be confusing at first, but it all comes down to the different layers we have when deploying a VM on top of a Kubernetes (or OpenShift) cluster.

The cluster layer

The cluster running beneath your virtual machine, be it Kubernetes or OpenShift, needs to be able to schedule its workloads on its nodes. To do this, the scheduler needs to know how many resources the workload is expected to use when it runs. The scheduler knows how much of each resource is available on its nodes and, with the resource specification of the workload, it tries to find the optimal fit between workloads and nodes. This specification defines how many resources are allocated to your workload containing the virtual machine. If you do not specify anything else, these resources will be passed down to the virtual machine.

You can specify the resources as follows:

kind: VirtualMachineInstance
spec:
  domain:
    [...]
    resources:
      requests:
        memory: 4Gi
        cpu: 2
      limits:
        memory: 4Gi
        cpu: 2

The virtualization layer

This is the outside perspective of your virtual machine. With these values, you can specify what the CPU topology will look like so that the virt-launcher process will be able to provision your virtual machine correctly. Instead of just “2 cores,” you use spec.domain.cpu to define sockets, cores, and threads.

You can specify the values like this:

spec:
  domain:
    cpu:
      sockets: 2
      cores: 1
      threads: 1

The guest layer

Finally, your virtual machine is often referred to as the “guest.” The specification you define for this layer is what your virtual machine sees from the inside, whereas the values from the virtualization layer are what is passed from the outside. If you don’t set the workload resources, the values are derived from the virtual machine specifications. It’s possible to set these values to a higher value than what will actually be available; this is also called “overcommitting.” If you set domain.resources.requests.memory to 2Gi but domain.memory.guest to 4Gi, you are effectively overcommitting memory.

You can specify the guest resources as follows:

spec:
  domain:
    memory:
      guest: 4Gi

Aligning the specifications

There are helpful default behaviors in place to ensure that your resource definitions don’t clash with each other. For example, if you don’t set the resource requests, KubeVirt will automatically set them based on the domain resource specifications and a small overhead for the virt-launcher process. If you don’t set anything for the virtual machine, the values are passed down from the workload resources.

However, if you do set a memory limit on the workload and then tell the virtual machine that it has much more memory available, it is likely that it will exceed the limit and your workload containing the virtual machine will be killed. You also shouldn’t set domain.cpu.cores to a higher number than your resources.limits.cpu. If you’re doing this on purpose because you are trying to overcommit the CPU, the VM will experience massive “steal time” and perform terribly because the high amount of virtual threads are fighting for fewer physical time slots. If you’re not trying to overcommit CPU, KubeVirt may try to adjust your values to ensure that they’re aligned.

Kubevirt will try to respect the values that you set as you’re setting them for a reason. It does try to optimize the given values and if values are not set, it will calculate the optimal value.

You don’t need to understand this in depth for this lab. It generally makes sense to set the virtualization layer and you can leave the rest to KubeVirt.

Task 3.2.1: Find matching instance type and preference

You want to find the optimal configuration of instance type and preference for a Windows 11 64-bit virtual machine. According to Microsoft, the Windows 11 system requirements2 are the following:

Processor: 1 gigahertz (GHz) or faster processor or SoC
RAM: 4 GB for 64-bit
Hard disk space: 64 GB for 64-bit OS

Try to find the best matching instance type and preference for a Windows 11 minimal installation using label selectors.

Task hint

You can query instance types as follows:

kubectl get virtualmachineclusterinstancetype \
   --selector instancetype.kubevirt.io/cpu=1,instancetype.kubevirt.io/memory=2Gi

Above query will return something like:

NAME            AGE
cx1.medium      10m
cx1.medium1gi   10m
o1.small        10m
rt1.small       10m
u1.small        10m

You would most likely pick o1.small or u1.small as your instance type.

For preferences, you can use the following query:

kubectl get virtualmachineclusterpreference --selector instancetype.kubevirt.io/os-type=windows

Which outputs:

NAME                  AGE
windows.10            10m
windows.10.virtio     10m
windows.11            10m
windows.11.virtio     10m
windows.2k12          10m
windows.2k12.virtio   10m
windows.2k16          10m
windows.2k16.virtio   10m
windows.2k19          10m
windows.2k19.virtio   10m
windows.2k22          10m
windows.2k22.virtio   10m

Task 3.2.2: Deploy two Cirros VMs with minimal requirements

Deploy two VMs with different instance types:

  • Deploy a cirros VM using a u class instance type and a matching preference
    • Write the VM specification in vm_lab03-u1-cirros.yaml
  • Deploy a cirros VM using a o class instance type and the same preference
    • Write the VM specification in vm_lab03-o1-cirros.yaml
Solution

labs/lab03/vm_lab03-u1-cirros.yaml

**Virtual Machine** lab03-u1-cirros:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: lab03-u1-cirros
spec:
  runStrategy: Halted
  instancetype:
    kind: VirtualMachineClusterInstancetype
    name: u1.nano
  preference:
    kind: VirtualMachineClusterPreference
    name: cirros
  template:
    metadata:
      labels:
        kubevirt.io/size: nano
        kubevirt.io/domain: lab03-u1-cirros
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
            - name: cloudinitdisk
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/kubevirt/cirros-container-disk-demo
        - name: cloudinitdisk
          cloudInitNoCloud:
            userDataBase64: SGkuXG4=

labs/lab03/vm_lab03-o1-cirros.yaml

**Virtual Machine** lab03-o1-cirros:
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: lab03-o1-cirros
spec:
  runStrategy: Halted
  instancetype:
    kind: VirtualMachineClusterInstancetype
    name: o1.nano
  preference:
    kind: VirtualMachineClusterPreference
    name: cirros
  template:
    metadata:
      labels:
        kubevirt.io/size: nano
        kubevirt.io/domain: lab03-o1-cirros
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
            - name: cloudinitdisk
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/kubevirt/cirros-container-disk-demo
        - name: cloudinitdisk
          cloudInitNoCloud:
            userDataBase64: SGkuXG4=

Apply and start both VMs using:

kubectl apply -f labs/lab03/vm_lab03-u1-cirros.yaml --namespace lab-<username>
kubectl apply -f labs/lab03/vm_lab03-o1-cirros.yaml --namespace lab-<username>
virtctl start lab03-u1-cirros --namespace lab-<username>
virtctl start lab03-o1-cirros --namespace lab-<username>

Task 3.2.3: Instance type differences

The main difference in the instance types u and o are memory overcommitting. Let’s inspect our two VMs. What exactly means overcommitting in the scope of a VirtualMachine instance? Overcommitting means that we assign more memory to a VM than we requested from the cluster by setting spec.domain.memory.guest to a higher value than spec.domain.resources.requests.memory.

What would you expect from both VMs?

Task hint
  • u class should have equal requests for spec.domain.memory.guest and spec.domain.resources.requests.memory
  • o class should have higher requests for spec.domain.memory.guest than spec.domain.resources.requests.memory

For both VMs we would expect the guest OS to have approximately 512Mi of memory.

Check the expectations about memory settings of both VirtualMachine instances lab03-u1-cirros and lab03-o1-cirros. Do they match our expectations?

Task hint

Describe both VirtualMachine instances using:

kubectl get vmi lab03-u1-cirros -o yaml --namespace lab-<username>
kubectl get vmi lab03-o1-cirros -o yaml --namespace lab-<username>

The u1-cirros instance:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  name: lab03-u1-cirros
spec:
  domain:
    resources:
      requests:
        memory: 512Mi
    memory:
      guest: 512Mi
[...]

The o1-cirros instance:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  name: lab03-o1-cirros
spec:
  domain:
    resources:
      requests:
        memory: 256Mi
    memory:
      guest: 512Mi
[...]

As we can see in the difference between spec.domain.memory.guest and spec.domain.resources.requests.memory, the o class actually overcommits by 50% as defined in the o1.nano instance type:

apiVersion: instancetype.kubevirt.io/v1beta1
kind: VirtualMachineClusterInstancetype
metadata:
  name: o1.nano
spec:
  cpu:
    guest: 1
  memory:
    guest: 512Mi
    overcommitPercent: 50
[...]

The .status.memory of both VirtualMachine instances shows that the guest was assigned 512Mi memory.

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
status:
  memory:
    guestAtBoot: 512Mi
    guestCurrent: 512Mi
    guestRequested: 512Mi
[...]

End of lab

  1. Data Plane Development Kit (DPDK)  ↩︎

  2. Windows 11 system requirements  ↩︎

Previous
Next