7.2. Perform a live migration

Migrate running VMs to other nodes for maintenance or other reasons

This lab demonstrates how to perform a live migration of a running virtual machine.

Task 7.2.1: Creating a virtual machine

For this lab we are going to create a Fedora Cloud VM and provide a cloud-init userdata configuration to initialize our VM.

We will install

an nginx Webserver running on port 8080
and configure it to return a predefined response (Hello KubeVirt) when we send requests to it.

Create a new file livemigration-secret.yaml in the folder labs/lab07/ with the following content:

apiVersion: v1
kind: Secret
metadata:
  name: lab07-cloudinit
type: Opaque
stringData:
  userdata: |
    #cloud-config
    password: kubevirt
    chpasswd: { expire: False }
    packages:
      - nginx
    timezone: Europe/Zurich
    write_files:
      - content: |
          user nginx;
          worker_processes auto;
          error_log /var/log/nginx/error.log;
          pid /run/nginx.pid;

          events {
            worker_connections 1024;
          }

          http {
              log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
              '$status $body_bytes_sent "$http_referer" '
              '"$http_user_agent" "$http_x_forwarded_for"';

              access_log  /var/log/nginx/access.log  main;

              sendfile            on;
              tcp_nopush          on;
              tcp_nodelay         on;
              keepalive_timeout   65;
              types_hash_max_size 4096;

              include             /etc/nginx/mime.types;
              default_type        text/plain;

              server {
                  listen       8080;
                  server_name  _;
                  root         /usr/share/nginx/html;

                  # Load configuration files for the default server block.
                  include /etc/nginx/default.d/*.conf;

                  location /health {
                    return 200 'ok';
                  }

                  location / {
                    set $response 'Hello KubeVirt (${hostname})';
                    return 200 '${response}';
                  }
              }
          }      
        path: /etc/nginx/nginx.conf
    runcmd:
      - systemctl enable nginx
      - systemctl start nginx

Create a new file livemigration.yaml in the folder labs/lab07/ with the following content:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: lab07-livemigration
spec:
  runStrategy: Halted
  template:
    metadata:
      labels:
        kubevirt.io/size: small
        kubevirt.io/domain: lab07-livemigration
    spec:
      domain:
        devices:
          disks:
            - name: containerdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
        resources:
          requests:
            memory: 2Gi
      livenessProbe:
        initialDelaySeconds: 120
        periodSeconds: 20
        httpGet:
          port: 8080
        timeoutSeconds: 10
      readinessProbe:
        initialDelaySeconds: 0
        periodSeconds: 5
        timeoutSeconds: 5
        httpGet:
          port: 8080
        failureThreshold: 10
        successThreshold: 1
      networks:
      - name: default
        pod: {}
      volumes:
        - name: containerdisk
          containerDisk:
            image: quay.io/containerdisks/fedora:43
        - name: cloudinitdisk
          cloudInitNoCloud:
            secretRef:
              name: lab07-cloudinit

Apply it to the cluster to create the virtual machine.

Task hint: apply command

kubectl apply -f labs/lab07/livemigration.yaml --namespace lab-<username>

Start the virtual machine:

virtctl start lab07-livemigration --namespace lab-<username>

Once the VM has started successfully, execute the following command to get the node the VMI is running on:

kubectl get vmi --namespace lab-<username>

Which will result in:

NAME                   AGE    PHASE     IP              NODENAME                 READY
lab07-livemigration    45m    Running   10.244.6.103    training-baremetal-<x>   True

During the virtual machine’s startup, KubeVirt determines whether a VMI is live-migratable or not. The information can be found in the status.conditions section of the VMI resource.

kubectl get vmi lab07-livemigration --namespace lab-<username> -o yaml

Above commands output should show:

[...]
 conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-10-05T14:51:36Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: null
    status: "True"
    type: LiveMigratable
[...]

Task 7.2.2: Start a simple process in your VM

Create the following Kubernetes Service resource (file: service-livemigration.yaml folder: labs/lab07):

apiVersion: v1
kind: Service
metadata:
  name: lab07-livemigration
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    kubevirt.io/domain: lab07-livemigration
  type: ClusterIP

Apply it using:

kubectl apply -f labs/lab07/service-livemigration.yaml --namespace lab-<username>

Go back to the webshell and open a new terminal to test the service:

curl -s lab07-livemigration.lab-<username>.svc.cluster.local:8080

Task 7.2.3: Perform a live migration

To perform a live migration we have the following options:

use the virtctl migrate command
or create a VirtualMachineInstanceMigration CustomResource

Before we perform the live migration we prepare two terminals:

1. watch the VirtualMachineInstance resource

kubectl get vmi lab07-livemigration -w --namespace lab-<username>

2. execute a while loop sending requests to the webserver running in the virtual machine. You should use the Web Console for this command because the service does not have an Ingress definition and is unknown to your local machine.

while true; do sleep 1; echo -n `date +"[%H:%M:%S,%3N] "`; echo -n " "; curl --max-time 1 --connect-timeout 0.8 lab07-livemigration.lab-<username>.svc.cluster.local:8080; echo ""; done

Now we can perform the live migration, by executing the following command:

virtctl migrate lab07-livemigration --namespace lab-<username>

The live migration will start immediately. Observe the execution in the previously opened terminals, which will show a similar output (notice the changing worker nodename):

NAME                  AGE    PHASE     IP            NODENAME            READY
lab07-livemigration   101m   Running   10.129.2.57   training-worker-1   True
lab07-livemigration   101m   Running   10.129.2.57   training-worker-1   True
lab07-livemigration   101m   Running   10.129.2.57   training-worker-1   True
lab07-livemigration   101m   Running   10.129.2.57   training-worker-1   True
lab07-livemigration   101m   Running   10.129.2.57   training-worker-1   True
lab07-livemigration   101m   Running   10.130.3.23   training-worker-1   True
lab07-livemigration   101m   Running   10.130.3.23   training-worker-2   False
lab07-livemigration   101m   Running   10.130.3.23   training-worker-2   True

And

...
[21:42:58,487] Hello KubeVirt (lab07-livemigration)
[21:42:59,775] Hello KubeVirt (lab07-livemigration)
[21:43:01,175] Hello KubeVirt (lab07-livemigration)
[21:43:02,656] curl: (28) Connection timed out after 800 milliseconds

[21:43:04,866] Hello KubeVirt (lab07-livemigration)
[21:43:06,238] Hello KubeVirt (lab07-livemigration)
[21:43:07,633] Hello KubeVirt (lab07-livemigration)
...

Details about the migration state will be represented in the VMI:

kubectl describe vmi lab07-livemigration --namespace lab-<username>

Or it can also be displayed in the corresponding VirtualMachineInstanceMigration resource. List the migrations to find the name of the migration with:

kubectl get VirtualMachineInstanceMigration --namespace lab-<username>

And then get the details for the migration like this:

kubectl get VirtualMachineInstanceMigration kubevirt-migrate-vm-hfcvd -o yaml --namespace lab-<username>

Cancel a live migration

Warning

This lab is not meant to be executed.

A live migration can also be canceled by deleting the VirtualMachineInstanceMigration object:

kubectl delete vmim lab07-livemigration-job --namespace lab-<username>

virtctl can also be used:

virtctl migrate-cancel lab07-livemigration --namespace lab-<username>

A successfully canceled migration will show the following states:

Abort Requested: true
Abort Status: Succeeded
Completed: true
Failed: true

Task 7.2.4: (Optional) Perform migrations using the VirtualMachineInstanceMigration CustomResource

We can also perform a live migration by creating a VirtualMachineInstanceMigration resource on the cluster, referencing the VMI to migrate. Create a new file livemigration-job.yaml in the folder labs/lab07/ with the following content:

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstanceMigration
metadata:
  name: lab07-livemigration-job
spec:
  vmiName: lab07-livemigration

Apply VirtualMachineInstanceMigration manifest to the cluster:

kubectl apply -f labs/lab07/livemigration-job.yaml --namespace lab-<username>

This will automatically start and perform the live migration. Use the following command to see the migration’s status:

kubectl get VirtualMachineInstanceMigration lab07-livemigration-job -w --namespace lab-<username>

Try to cancel a live migration, and verify it in the status section of the VMI manifest.

Task 7.2.5: (Optional) Migration policies

In addition to the live migration feature’s cluster-wide configuration, the concept of migration policies has been introduced. Migration policies are currently (v1.30) an alpha feature and the API might not be stable.

Explore the official documentation at https://kubevirt.io/user-guide/cluster_admin/migration_policies/ .

End of lab

Cleanup resources

You have reached the end of this lab. Please stop your running virtual machines to save resources on the cluster.

Stop the VirtualMachineInstance:

virtctl stop lab07-livemigration --namespace lab-<username>