[PATCH 0/3] Postcopy migration: Add userfaultfd- user-mode-only capability

[PATCH 0/3] Postcopy migration: Add userfaultfd- user-mode-only capability

[Lin Ma]

Since kernel v5.11, Unprivileged user (without SYS_CAP_PTRACE capability)
must pass UFFD_USER_MODE_ONLY to userfaultd in case unprivileged_userfaultfd
sysctl knob is 0.
Please refer to https://lwn.net/Articles/819834/ and the kernel commits:
37cd0575 userfaultfd: add UFFD_USER_MODE_ONLY
d0d4730a userfaultfd: add user-mode only option to unprivileged_userfaultfd sysctl knob

This patch set adds a migration capability to pass UFFD_USER_MODE_ONLY
for postcopy migration.

Lin Ma (3):
  migration: introduce postcopy-uffd-usermode-only capability
  migration: postcopy-uffd-usermode-only documentation
  tests: add postcopy-uffd-usermode-only capability into migration-test

 docs/devel/migration.rst     |  9 +++++++++
 migration/migration.c        |  9 +++++++++
 migration/migration.h        |  1 +
 migration/postcopy-ram.c     | 22 +++++++++++++++++++---
 qapi/migration.json          |  8 +++++++-
 tests/qtest/migration-test.c | 11 +++++++++--
 6 files changed, 54 insertions(+), 6 deletions(-)

-- 
2.26.2

[PATCH 1/3] migration: introduce postcopy-uffd-usermode-only capability

[Lin Ma]

The default value of unprivileged_userfaultfd sysctl knob was changed to
0 since kernel v5.11 by commit d0d4730a:
userfaultfd: add user-mode only option to unprivileged_userfaultfd sysctl knob.

In this mode, An unprivileged user (without SYS_CAP_PTRACE capability) must
pass UFFD_USER_MODE_ONLY to userfaultd or the API will fail with EPERM.

So add a capability to pass UFFD_USER_MODE_ONLY to support it.

Signed-off-by: Lin Ma <lma@suse.com>
---
 migration/migration.c    |  9 +++++++++
 migration/migration.h    |  1 +
 migration/postcopy-ram.c | 22 +++++++++++++++++++---
 qapi/migration.json      |  8 +++++++-
 4 files changed, 36 insertions(+), 4 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 6ac807ef3d..86212dcb70 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2380,6 +2380,15 @@ bool migrate_postcopy_blocktime(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_POSTCOPY_BLOCKTIME];
 }
 
+bool migrate_postcopy_uffd_usermode_only(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_POSTCOPY_UFFD_USERMODE_ONLY];
+}
+
 bool migrate_use_compression(void)
 {
     MigrationState *s;
diff --git a/migration/migration.h b/migration/migration.h
index 7a5aa8c2fd..a516d7f59f 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -358,6 +358,7 @@ int migrate_decompress_threads(void);
 bool migrate_use_events(void);
 bool migrate_postcopy_blocktime(void);
 bool migrate_background_snapshot(void);
+bool migrate_postcopy_uffd_usermode_only(void);
 
 /* Sending on the return path - generic and then for each message type */
 void migrate_send_rp_shut(MigrationIncomingState *mis,
diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 2e9697bdd2..078c558626 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -206,9 +206,14 @@ static bool receive_ufd_features(uint64_t *features)
     struct uffdio_api api_struct = {0};
     int ufd;
     bool ret = true;
+    int flags;
+
+    flags = O_CLOEXEC;
+    if (migrate_postcopy_uffd_usermode_only())
+        flags |= UFFD_USER_MODE_ONLY;
 
     /* if we are here __NR_userfaultfd should exists */
-    ufd = syscall(__NR_userfaultfd, O_CLOEXEC);
+    ufd = syscall(__NR_userfaultfd, flags);
     if (ufd == -1) {
         error_report("%s: syscall __NR_userfaultfd failed: %s", __func__,
                      strerror(errno));
@@ -352,13 +357,18 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis)
     struct uffdio_range range_struct;
     uint64_t feature_mask;
     Error *local_err = NULL;
+    int flags;
 
     if (qemu_target_page_size() > pagesize) {
         error_report("Target page size bigger than host page size");
         goto out;
     }
 
-    ufd = syscall(__NR_userfaultfd, O_CLOEXEC);
+    flags = O_CLOEXEC;
+    if (migrate_postcopy_uffd_usermode_only())
+        flags |= UFFD_USER_MODE_ONLY;
+
+    ufd = syscall(__NR_userfaultfd, flags);
     if (ufd == -1) {
         error_report("%s: userfaultfd not available: %s", __func__,
                      strerror(errno));
@@ -1064,8 +1074,14 @@ retry:
 
 int postcopy_ram_incoming_setup(MigrationIncomingState *mis)
 {
+    int flags;
+
+    flags = O_CLOEXEC | O_NONBLOCK;
+    if (migrate_postcopy_uffd_usermode_only())
+        flags |= UFFD_USER_MODE_ONLY;
+
     /* Open the fd for the kernel to give us userfaults */
-    mis->userfault_fd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+    mis->userfault_fd = syscall(__NR_userfaultfd, flags);
     if (mis->userfault_fd == -1) {
         error_report("%s: Failed to open userfault fd: %s", __func__,
                      strerror(errno));
diff --git a/qapi/migration.json b/qapi/migration.json
index 88f07baedd..3af1ec4cec 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -452,6 +452,11 @@
 #                       procedure starts. The VM RAM is saved with running VM.
 #                       (since 6.0)
 #
+# @postcopy-uffd-usermode-only: If enabled, It allows unprivileged users to use
+#                               userfaultfd but with the restriction that page
+#                               faults from only user mode can be handled.
+#                               (since 6.2.0)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
@@ -459,7 +464,8 @@
            'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram',
            'block', 'return-path', 'pause-before-switchover', 'multifd',
            'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate',
-           'x-ignore-shared', 'validate-uuid', 'background-snapshot'] }
+           'x-ignore-shared', 'validate-uuid', 'background-snapshot',
+           'postcopy-uffd-usermode-only'] }
 
 ##
 # @MigrationCapabilityStatus:
-- 
2.26.2

[PATCH 2/3] migration: postcopy-uffd-usermode-only documentation

[Lin Ma]

Signed-off-by: Lin Ma <lma@suse.com>
---
 docs/devel/migration.rst | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/docs/devel/migration.rst b/docs/devel/migration.rst
index 2401253482..dfdd3f20b4 100644
--- a/docs/devel/migration.rst
+++ b/docs/devel/migration.rst
@@ -639,6 +639,15 @@ postcopy-blocktime value of qmp command will show overlapped blocking
 time for all vCPU, postcopy-vcpu-blocktime will show list of blocking
 time per vCPU.
 
+Since kernel v5.11, Unprivileged user (without SYS_CAP_PTRACE capability)
+must pass UFFD_USER_MODE_ONLY to userfaultd if the unprivileged_userfaultfd
+sysctl knob is 0.
+
+To allow unprivileged user postcopy, Issue this command on destination
+monitor prior to turning on postcopy-ram:
+
+``migrate_set_capability postcopy-uffd-usermode-only on``
+
 .. note::
   During the postcopy phase, the bandwidth limits set using
   ``migrate_set_parameter`` is ignored (to avoid delaying requested pages that
-- 
2.26.2

[PATCH 3/3] tests: add postcopy-uffd-usermode-only capability into migration-test

[Lin Ma]

Signed-off-by: Lin Ma <lma@suse.com>
---
 tests/qtest/migration-test.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index cc5e83d98a..0cd4f49bed 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -38,6 +38,7 @@
 unsigned start_address;
 unsigned end_address;
 static bool uffd_feature_thread_id;
+static bool uffd_usermode_only;
 
 /* A downtime where the test really should converge */
 #define CONVERGE_DOWNTIME 1000
@@ -60,8 +61,12 @@ static bool ufd_version_check(void)
     int ufd = syscall(__NR_userfaultfd, O_CLOEXEC);
 
     if (ufd == -1) {
-        g_test_message("Skipping test: userfaultfd not available");
-        return false;
+        ufd = syscall(__NR_userfaultfd, O_CLOEXEC | UFFD_USER_MODE_ONLY);
+        if (ufd == -1) {
+	    g_test_message("Skipping test: userfaultfd not available");
+            return false;
+	} else
+            uffd_usermode_only = true;
     }
 
     api_struct.api = UFFD_API;
@@ -670,6 +675,8 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
     }
 
     migrate_set_capability(from, "postcopy-ram", true);
+    if (uffd_usermode_only)
+        migrate_set_capability(to, "postcopy-uffd-usermode-only", true);
     migrate_set_capability(to, "postcopy-ram", true);
     migrate_set_capability(to, "postcopy-blocktime", true);
 
-- 
2.26.2

Re: [PATCH 0/3] Postcopy migration: Add userfaultfd- user-mode-only capability

[Peter Xu]

On Thu, Oct 14, 2021 at 05:15:48PM +0800, Lin Ma wrote:
> Since kernel v5.11, Unprivileged user (without SYS_CAP_PTRACE capability)
> must pass UFFD_USER_MODE_ONLY to userfaultd in case unprivileged_userfaultfd
> sysctl knob is 0.
> Please refer to https://lwn.net/Articles/819834/ and the kernel commits:
> 37cd0575 userfaultfd: add UFFD_USER_MODE_ONLY
> d0d4730a userfaultfd: add user-mode only option to unprivileged_userfaultfd sysctl knob
> 
> This patch set adds a migration capability to pass UFFD_USER_MODE_ONLY
> for postcopy migration.

Then it's at least no KVM, no vhost, am I right?  Could I ask is there a real
user behind this?  Thanks,

-- 
Peter Xu

Re: [PATCH 0/3] Postcopy migration: Add userfaultfd- user-mode-only capability

[lma]

在 2021-10-15 07:43，Peter Xu 写道：
> On Thu, Oct 14, 2021 at 05:15:48PM +0800, Lin Ma wrote:
>> Since kernel v5.11, Unprivileged user (without SYS_CAP_PTRACE 
>> capability)
>> must pass UFFD_USER_MODE_ONLY to userfaultd in case 
>> unprivileged_userfaultfd
>> sysctl knob is 0.
>> Please refer to https://lwn.net/Articles/819834/ and the kernel 
>> commits:
>> 37cd0575 userfaultfd: add UFFD_USER_MODE_ONLY
>> d0d4730a userfaultfd: add user-mode only option to 
>> unprivileged_userfaultfd sysctl knob
>> 
>> This patch set adds a migration capability to pass UFFD_USER_MODE_ONLY
>> for postcopy migration.
> 
> Then it's at least no KVM, no vhost, am I right?  Could I ask is there 
> a real
> user behind this?  Thanks,

Well, The "user-mode-only" has nothing to do with qemu's user-mode 
emulation.

The unprivileged_userfaultfd sysctl knob controls whether unprivileged 
users can use the userfaultfd system calls.
  set it to 1 to allow unprivileged users to use the userfaultfd system 
calls.
  set it to 0 to restrict userfaultfd to only privileged users (with 
SYS_CAP_PTRACE capability).

If host's unprivileged_userfaultfd sysctl knob is 0(The default value of 
this knob is changed to 0 since host kernel v5.11):
Qemu must pass the UFFD_USER_MODE_ONLY flag when creating userfaultfd 
object for postcopy migration in case qemu runs as unprivileged user.

Before host kernel v5.11, If host's unprivileged_userfaultfd sysctl knob 
is 0, Then postcopy migration is not allowed in case qemu runs as 
unprivileged user.

Thanks,
Lin

Re: [PATCH 0/3] Postcopy migration: Add userfaultfd- user-mode-only capability

[Peter Xu]

On Fri, Oct 15, 2021 at 01:38:06PM +0800, lma wrote:
> 在 2021-10-15 07:43，Peter Xu 写道：
> > On Thu, Oct 14, 2021 at 05:15:48PM +0800, Lin Ma wrote:
> > > Since kernel v5.11, Unprivileged user (without SYS_CAP_PTRACE
> > > capability)
> > > must pass UFFD_USER_MODE_ONLY to userfaultd in case
> > > unprivileged_userfaultfd
> > > sysctl knob is 0.
> > > Please refer to https://lwn.net/Articles/819834/ and the kernel
> > > commits:
> > > 37cd0575 userfaultfd: add UFFD_USER_MODE_ONLY
> > > d0d4730a userfaultfd: add user-mode only option to
> > > unprivileged_userfaultfd sysctl knob
> > > 
> > > This patch set adds a migration capability to pass UFFD_USER_MODE_ONLY
> > > for postcopy migration.
> > 
> > Then it's at least no KVM, no vhost, am I right?  Could I ask is there a
> > real
> > user behind this?  Thanks,
> 
> Well, The "user-mode-only" has nothing to do with qemu's user-mode
> emulation.

I didn't follow why you thought my question was about "user-mode emulation"..

To ask in another way: after this new cap set, qemu will get a SIGBUS and VM
will crash during postcopy migrating as long as either KVM or vhost-kernel
faulted on any of the missing pages, am I right?

Thanks,

-- 
Peter Xu

Re: [PATCH 0/3] Postcopy migration: Add userfaultfd- user-mode-only capability

[lma]

在 2021-10-15 14:12，Peter Xu 写道：
> On Fri, Oct 15, 2021 at 01:38:06PM +0800, lma wrote:
>> 在 2021-10-15 07:43，Peter Xu 写道：
>> > On Thu, Oct 14, 2021 at 05:15:48PM +0800, Lin Ma wrote:
>> > > Since kernel v5.11, Unprivileged user (without SYS_CAP_PTRACE
>> > > capability)
>> > > must pass UFFD_USER_MODE_ONLY to userfaultd in case
>> > > unprivileged_userfaultfd
>> > > sysctl knob is 0.
>> > > Please refer to https://lwn.net/Articles/819834/ and the kernel
>> > > commits:
>> > > 37cd0575 userfaultfd: add UFFD_USER_MODE_ONLY
>> > > d0d4730a userfaultfd: add user-mode only option to
>> > > unprivileged_userfaultfd sysctl knob
>> > >
>> > > This patch set adds a migration capability to pass UFFD_USER_MODE_ONLY
>> > > for postcopy migration.
>> >
>> > Then it's at least no KVM, no vhost, am I right?  Could I ask is there a
>> > real
>> > user behind this?  Thanks,
>> 
>> Well, The "user-mode-only" has nothing to do with qemu's user-mode
>> emulation.
> 
> I didn't follow why you thought my question was about "user-mode 
> emulation"..
Sorry about the misunderstanding.

> To ask in another way: after this new cap set, qemu will get a SIGBUS 
> and VM
> will crash during postcopy migrating as long as either KVM or 
> vhost-kernel
> faulted on any of the missing pages, am I right?

Oops...Yes, you're right. It indeed casues qemu crash on destination due 
to
fault on missing pages.
This patch set and my thought about introducing this cap to qemu are 
wrong.

Thanks,
Lin

Re: [PATCH 0/3] Postcopy migration: Add userfaultfd- user-mode-only capability

[Peter Xu]

On Fri, Oct 15, 2021 at 04:16:15PM +0800, lma wrote:
> 在 2021-10-15 14:12，Peter Xu 写道：
> > On Fri, Oct 15, 2021 at 01:38:06PM +0800, lma wrote:
> > > 在 2021-10-15 07:43，Peter Xu 写道：
> > > > On Thu, Oct 14, 2021 at 05:15:48PM +0800, Lin Ma wrote:
> > > > > Since kernel v5.11, Unprivileged user (without SYS_CAP_PTRACE
> > > > > capability)
> > > > > must pass UFFD_USER_MODE_ONLY to userfaultd in case
> > > > > unprivileged_userfaultfd
> > > > > sysctl knob is 0.
> > > > > Please refer to https://lwn.net/Articles/819834/ and the kernel
> > > > > commits:
> > > > > 37cd0575 userfaultfd: add UFFD_USER_MODE_ONLY
> > > > > d0d4730a userfaultfd: add user-mode only option to
> > > > > unprivileged_userfaultfd sysctl knob
> > > > >
> > > > > This patch set adds a migration capability to pass UFFD_USER_MODE_ONLY
> > > > > for postcopy migration.
> > > >
> > > > Then it's at least no KVM, no vhost, am I right?  Could I ask is there a
> > > > real
> > > > user behind this?  Thanks,
> > > 
> > > Well, The "user-mode-only" has nothing to do with qemu's user-mode
> > > emulation.
> > 
> > I didn't follow why you thought my question was about "user-mode
> > emulation"..
> Sorry about the misunderstanding.

No worry. :)

> 
> > To ask in another way: after this new cap set, qemu will get a SIGBUS
> > and VM
> > will crash during postcopy migrating as long as either KVM or
> > vhost-kernel
> > faulted on any of the missing pages, am I right?
> 
> Oops...Yes, you're right. It indeed casues qemu crash on destination due to
> fault on missing pages.
> This patch set and my thought about introducing this cap to qemu are wrong.

I can't say it's wrong, it's just that it may need some more thoughts on how to
make it applicable.

We'll need to make sure no kernel module will access guest pages, however I
think it'll be so hard to guarantee.  For example, there can be some read()
syscall from qemu initiated with guest pages passed in as the buffer (so the
kernel will fill up the buffer when syscall returns), then if that page is
missing on dst then that'll also trigger a kernel page fault and it'll crash
qemu too even if no kvm/vhost-kernel is used.  We'll need to dig out everything
like that.

The other thing is about my original question on whether it'll be useful in any
way, and I just worry it won't help anyone, because afaiu any real user of
migration (I believe it's majorly public/private cloud) will definitely at
least be kvm based as tcg could be too slow.  Then they'll simply enable the
unprivileged uffd on the hosts, since even if it's unsafe it'll be at least as
unsafe as before unprivileged_userfaultfd is introduced.

Thanks,

-- 
Peter Xu

Re: [PATCH 0/3] Postcopy migration: Add userfaultfd- user-mode-only capability

[lma]

在 2021-10-15 16:28，Peter Xu 写道：
> On Fri, Oct 15, 2021 at 04:16:15PM +0800, lma wrote:
>> 在 2021-10-15 14:12，Peter Xu 写道：
>> > On Fri, Oct 15, 2021 at 01:38:06PM +0800, lma wrote:
>> > > 在 2021-10-15 07:43，Peter Xu 写道：
>> > > > On Thu, Oct 14, 2021 at 05:15:48PM +0800, Lin Ma wrote:
>> > > > > Since kernel v5.11, Unprivileged user (without SYS_CAP_PTRACE
>> > > > > capability)
>> > > > > must pass UFFD_USER_MODE_ONLY to userfaultd in case
>> > > > > unprivileged_userfaultfd
>> > > > > sysctl knob is 0.
>> > > > > Please refer to https://lwn.net/Articles/819834/ and the kernel
>> > > > > commits:
>> > > > > 37cd0575 userfaultfd: add UFFD_USER_MODE_ONLY
>> > > > > d0d4730a userfaultfd: add user-mode only option to
>> > > > > unprivileged_userfaultfd sysctl knob
>> > > > >
>> > > > > This patch set adds a migration capability to pass UFFD_USER_MODE_ONLY
>> > > > > for postcopy migration.
>> > > >
>> > > > Then it's at least no KVM, no vhost, am I right?  Could I ask is there a
>> > > > real
>> > > > user behind this?  Thanks,
>> > >
>> > > Well, The "user-mode-only" has nothing to do with qemu's user-mode
>> > > emulation.
>> >
>> > I didn't follow why you thought my question was about "user-mode
>> > emulation"..
>> Sorry about the misunderstanding.
> 
> No worry. :)
> 
>> 
>> > To ask in another way: after this new cap set, qemu will get a SIGBUS
>> > and VM
>> > will crash during postcopy migrating as long as either KVM or
>> > vhost-kernel
>> > faulted on any of the missing pages, am I right?
>> 
>> Oops...Yes, you're right. It indeed casues qemu crash on destination 
>> due to
>> fault on missing pages.
>> This patch set and my thought about introducing this cap to qemu are 
>> wrong.
> 
> I can't say it's wrong, it's just that it may need some more thoughts 
> on how to
> make it applicable.
> 
> We'll need to make sure no kernel module will access guest pages, 
> however I
> think it'll be so hard to guarantee.  For example, there can be some 
> read()
> syscall from qemu initiated with guest pages passed in as the buffer 
> (so the
> kernel will fill up the buffer when syscall returns), then if that page 
> is
> missing on dst then that'll also trigger a kernel page fault and it'll 
> crash
> qemu too even if no kvm/vhost-kernel is used.  We'll need to dig out 
> everything
> like that.

Yeah, It's hard to avoid pf in kernel completely.

> The other thing is about my original question on whether it'll be 
> useful in any
> way, and I just worry it won't help anyone, because afaiu any real user 
> of
> migration (I believe it's majorly public/private cloud) will definitely 
> at
> least be kvm based as tcg could be too slow.  Then they'll simply 
> enable the
> unprivileged uffd on the hosts, since even if it's unsafe it'll be at 
> least as
> unsafe as before unprivileged_userfaultfd is introduced.

It seems that this capability is useless for qemu/kvm so far :-)

Thanks for your information!

Lin