Live migration of VM with config drive fails

[sureshanaparti]

problem

Live migration of VM with config drive fails with error: java.lang.NullPointerException: Cannot invoke "com.cloud.storage.Storage$StoragePoolType.toString()" because "type" is null.

The first migration attempt removes the config drive from the xml, and the second attempt succeeds, but no config drive attached to it (after migration)

MS log:

2025-08-11 06:01:10,887 DEBUG [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-8:[ctx-9716983a, job-46/job-47, ctx-0e302ca4]) (logid:115dd2bf) Seq 4-6100688645226770330: Routed from 32985852150177
2025-08-11 06:01:10,891 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:[ctx-9716983a, job-46/job-47, ctx-0e302ca4]) (logid:115dd2bf) Seq 1-6100688645226770330: Sending  { Cmd , MgmtId: 32985852150177, via: 1(ref-trl-9195-k-Mol8-suresh-anaparti-kvm1), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.MigrateCommand":{"vmName":"i-2-5-VM","destIp":"10.0.32.165","migrateStorageManaged":"false","migrateNonSharedInc":"false","autoConvergence":"false","hostGuid":"39ed092a-9e7b-32fd-b99f-8412af2d12e5-LibvirtComputingResource","isWindows":"false","vmTO":{"id":"5","name":"i-2-5-VM","state":"Running","type":"User","cpus":"1","minSpeed":"250","maxSpeed":"500","minRam":"(512.00 MB) 536870912","maxRam":"(512.00 MB) 536870912","arch":"x86_64","os":"CentOS 5.5 (64-bit)","platformEmulator":"CentOS 5.5","bootArgs":"","enableHA":"false","limitCpuUse":"false","enableDynamicallyScaleVm":"false","params":{"configDriveLocation":"SECONDARY","cpuOvercommitRatio":"2.0"},"uuid":"8bb0d860-5b61-43c2-a128-21d467db0dc3","enterHardwareSetup":"false","disks":[{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"configdrive/i-2-5-VM.iso","uuid":"8bb0d860-5b61-43c2-a128-21d467db0dc3","id":"0","format":"ISO","accountId":"0","hvm":"false","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"NFS://10.0.32.4/acs/secondary/ref-trl-9195-k-Mol8-suresh-anaparti/ref-trl-9195-k-Mol8-suresh-anaparti-sec1","_role":"Image"}},"bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"4","path":"configdrive/i-2-5-VM.iso","type":"ISO"},{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"080f64dd-d627-42f4-8281-60efc9140835","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"cf1bf14f-0193-3fab-bb25-69f5793330e6","name":"ref-trl-9195-k-Mol8-suresh-anaparti-kvm-pri2","id":"2","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9195-k-Mol8-suresh-anaparti/ref-trl-9195-k-Mol8-suresh-anaparti-kvm-pri2","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9195-k-Mol8-suresh-anaparti/ref-trl-9195-k-Mol8-suresh-anaparti-kvm-pri2/?ROLE=Primary&STOREUUID=cf1bf14f-0193-3fab-bb25-69f5793330e6","isManaged":"false"}},"name":"ROOT-5","size":"(8.00 GB) 8589934592","path":"080f64dd-d627-42f4-8281-60efc9140835","volumeId":"8","vmName":"i-2-5-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"2","id":"8","deviceId":"0","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"0","path":"080f64dd-d627-42f4-8281-60efc9140835","type":"ROOT","_details":{"storageHost":"10.0.32.4","managed":"false","storagePort":"2049","storage.pool.disk.wait":"60","volumeSize":"(8.00 GB) 8589934592"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":"0","format":"ISO","accountId":"0","hvm":"false","bootable":"false","directDownload":"false","deployAsIs":"false","followRedirects":"false"}},"diskSeq":"3","type":"ISO"}],"nics":[{"deviceId":"0","networkRateMbps":"200","defaultNic":"true","pxeDisable":"false","nicUuid":"a8d499fe-c315-4109-a1d0-ee74e41e3e99","details":{"ForgedTransmits":"true","internetProtocol":"IPv4","PromiscuousMode":"false","MacAddressChanges":"true","MacLearning":"false"},"dpdkEnabled":"false","networkId":"204","networkSegmentName":"D1-A2-Z1-S204","uuid":"96a34786-bf05-40e7-bfe6-00becfb5f47c","ip":"10.1.1.236","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:01:00:cc:00:03","dns1":"10.0.32.1","dns2":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1082","isolationUri":"vlan://1082","isSecurityGroupEnabled":"false","name":"cloudbr1"}],"vcpuMaxLimit":"1","vmData":[["metadata","service-offering","Small Instance"],["metadata","availability-zone","ref-trl-9195-k-Mol8-suresh-anaparti"],["metadata","local-hostname","testvm02"],["metadata","local-ipv4","10.1.1.236"],["metadata","public-ipv4","10.1.1.236"],["metadata","public-hostname",],["metadata","instance-id","8bb0d860-5b61-43c2-a128-21d467db0dc3"],["metadata","vm-id","8bb0d860-5b61-43c2-a128-21d467db0dc3"],["metadata","public-keys",],["metadata","cloud-identifier","CloudStack-{d49fa504-d0fe-410b-8e67-ab94e1a7a75b}"],["metadata","hypervisor-host-name",]],"configDriveLabel":"config-2","configDriveLocation":"SECONDARY","guestOsDetails":{},"extraConfig":{},"networkIdToNetworkNameMap":{"204":"D1-A2-Z1-S204"}},"executeInSequence":"false","migrateDiskInfoList":[],"dpdkInterfaceMapping":{},"newVmCpuShares":"250","vlanToPersistenceMap":{"1082":"true"},"wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-11 06:01:11,002 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:[]) (logid:) Seq 1-6100688645226770330: Processing:  { Ans: , MgmtId: 32985852150177, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException: Cannot invoke "com.cloud.storage.Storage$StoragePoolType.toString()" because "type" is null
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:736)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:393)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:339)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.attachOrDetachISO(LibvirtComputingResource.java:3513)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.detachAndAttachConfigDriveISO(LibvirtComputingResource.java:3492)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:232)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:88)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1958)
	at com.cloud.agent.Agent.processRequest(Agent.java:779)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1194)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
","wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-11 06:01:11,003 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:[ctx-9716983a, job-46/job-47, ctx-0e302ca4]) (logid:115dd2bf) Seq 1-6100688645226770330: Received:  { Ans: , MgmtId: 32985852150177, via: 1(ref-trl-9195-k-Mol8-suresh-anaparti-kvm1), Ver: v1, Flags: 10, { Answer } }
2025-08-11 06:01:11,003 INFO  [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-8:[ctx-9716983a, job-46/job-47, ctx-0e302ca4]) (logid:115dd2bf) Migration was unsuccessful.  Cleaning up: VM instance {"id":5,"instanceName":"i-2-5-VM","state":"Migrating","type":"User","uuid":"8bb0d860-5b61-43c2-a128-21d467db0dc3"}
2025-08-11 06:01:11,013 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-8:[ctx-9716983a, job-46/job-47, ctx-0e302ca4]) (logid:115dd2bf) Releasing [1] volumes for VM [VM instance {"id":5,"instanceName":"i-2-5-VM","state":"Migrating","type":"User","uuid":"8bb0d860-5b61-43c2-a128-21d467db0dc3"}] from host [Host {"id":2,"name":"ref-trl-9195-k-Mol8-suresh-anaparti-kvm2","type":"Routing","uuid":"fa74f4d2-e5c6-4382-8d42-e42c195095f5"}].

mysql> SELECT name, value FROM cloud.user_vm_details WHERE vm_id = 5;
+---------------------+-----------+
| name                | value     |
+---------------------+-----------+
| configDriveLocation | SECONDARY |
| cpuOvercommitRatio  | 2.0       |
+---------------------+-----------+
2 rows in set (0.00 sec)

mysql> 

Agent log:

2025-08-11 06:01:10,997 DEBUG [kvm.resource.LibvirtComputingResource] (AgentRequest-Handler-1:[]) (logid:) Attaching ConfigDrive ISO of the VM i-2-5-VM, at path /mnt/3e0fe9c4-bcad-3a7d-9350-44694b760e74/i-2-5-VM.iso
2025-08-11 06:01:10,997 DEBUG [kvm.storage.KVMStoragePoolManager] (AgentRequest-Handler-1:[]) (logid:) Get storage pool by uri: /mnt/3e0fe9c4-bcad-3a7d-9350-44694b760e74
2025-08-11 06:01:10,999 WARN  [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:) Caught: java.lang.NullPointerException: Cannot invoke "com.cloud.storage.Storage$StoragePoolType.toString()" because "type" is null
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:736)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:393)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:339)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.attachOrDetachISO(LibvirtComputingResource.java:3513)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.detachAndAttachConfigDriveISO(LibvirtComputingResource.java:3492)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:232)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:88)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1958)
	at com.cloud.agent.Agent.processRequest(Agent.java:779)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1194)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

2025-08-11 06:01:11,003 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:) Seq 1-6100688645226770330:  { Ans: , MgmtId: 32985852150177, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException: Cannot invoke "com.cloud.storage.Storage$StoragePoolType.toString()" because "type" is null
	at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:736)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:393)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:339)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.attachOrDetachISO(LibvirtComputingResource.java:3513)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.detachAndAttachConfigDriveISO(LibvirtComputingResource.java:3492)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:232)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtMigrateCommandWrapper.execute(LibvirtMigrateCommandWrapper.java:88)
	at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1958)
	at com.cloud.agent.Agent.processRequest(Agent.java:779)
	at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1194)
	at com.cloud.utils.nio.Task.call(Task.java:83)
	at com.cloud.utils.nio.Task.call(Task.java:29)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
","wait":"0","bypassHostMaintenance":"false"}}] }

Before migrate (first attempt):

[root@kvm1 ~]# virsh dumpxml i-2-5-VM | grep iso
      <entry name='product'>CloudStack KVM Hypervisor</entry>
    <feature policy='require' name='hypervisor'/>
      <source file='/mnt/3e0fe9c4-bcad-3a7d-9350-44694b760e74/i-2-5-VM.iso' index='1'/>
[root@kvm1 ~]#

After migrate (first attempt failed):

[root@kvm1 ~]# virsh dumpxml i-2-5-VM | grep iso
      <entry name='product'>CloudStack KVM Hypervisor</entry>
    <feature policy='require' name='hypervisor'/>
[root@kvm1 ~]#

After migrate (second attempt success):

[root@kvm2 ~]# virsh dumpxml i-2-5-VM | grep iso
      <entry name='product'>CloudStack KVM Hypervisor</entry>
    <feature policy='require' name='hypervisor'/>
[root@kvm2 ~]#

versions

ACS 4.20.1 / KVM hypervisor / NFS as primary storage

The steps to reproduce the bug

1. Create a network offering with config drive for user data service
2. Create a network with the above offering
3. Deploy a VM on the above network.
4. Live migrate the VM to another KVM host

What to do about it?

The Live migration should succeed, along with config drive to the destination host.

[wes-dimi]

We faced the same issue. In our deployment Primary storage is Ceph RBD and secondary storage is NFS. CSM and KVMs are Ubuntu 24.04, 20.04. Stopping and starting VM containig config drive on another KVM host is working. We also tryed editing VM .xml by removing extra hdd cdrom device - also not work. This issue prevent us to put hosts in maintanance mode also. (we wanted to upgrade KVM hosts OS). VMs without config Drive, but containing user data (isolated network) live migration working.