[Devel] [PATCH hci-8.1 1/2] vfio: move an open device's kvm reference when its group gets a new kvm

[Andrey Drobyshev]

A device opened through its group takes a reference to the group's kvm
and drops it only at the last close.  QEMU's cpr-exec and cpr-transfer
keep the device open while the VM is re-created: the new QEMU just adds
the group to its new kvm-vfio device.  That updates group->kvm, but the
open device keeps pinning the old kvm until the VM is shut down.  Seen
as a second KVM instance in debugfs after a live update of a vGPU VM.

Move the reference to the new kvm.  Leave NULL alone: that is the old
kvm-vfio device going away under kvm->lock, and the new kvm, if any,
arrives on its own.

https://virtuozzo.atlassian.net/browse/VSTOR-150597

Signed-off-by: Andrey Drobyshev <andrey.drobyshev at virtuozzo.com>
---
 drivers/vfio/group.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/drivers/vfio/group.c b/drivers/vfio/group.c
index 5f2b2c950bbc..e78c06bb7db3 100644
--- a/drivers/vfio/group.c
+++ b/drivers/vfio/group.c
@@ -892,9 +892,30 @@ bool vfio_group_enforced_coherent(struct vfio_group *group)
 
 void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm)
 {
+	struct vfio_device *device;
+
 	spin_lock(&group->kvm_ref_lock);
 	group->kvm = kvm;
 	spin_unlock(&group->kvm_ref_lock);
+
+	/*
+	 * Open devices hold a reference to the kvm they were opened with.
+	 * Move it to the new kvm, or the old one stays pinned until the
+	 * device is closed.  Nothing to move to on NULL.
+	 */
+	if (!kvm)
+		return;
+
+	mutex_lock(&group->device_lock);
+	list_for_each_entry(device, &group->device_list, group_next) {
+		mutex_lock(&device->dev_set->lock);
+		if (device->kvm && device->kvm != kvm) {
+			vfio_device_put_kvm(device);
+			vfio_device_get_kvm_safe(device, kvm);
+		}
+		mutex_unlock(&device->dev_set->lock);
+	}
+	mutex_unlock(&group->device_lock);
 }
 
 /**
-- 
2.47.1