* [RFC] vhost: vmap virtio descriptor table kernel/kvm
@ 2023-10-24 3:17 ` Liang Chen
0 siblings, 0 replies; 6+ messages in thread
From: Liang Chen @ 2023-10-24 3:17 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang; +Cc: virtualization, kvm
The current vhost code uses 'copy_from_user' to copy descriptors from
userspace to vhost. We attempted to 'vmap' the descriptor table to
reduce the overhead associated with 'copy_from_user' during descriptor
access, because it can be accessed quite frequently. This change
resulted in a moderate performance improvement (approximately 3%) in
our pktgen test, as shown below. Additionally, the latency in the
'vhost_get_vq_desc' function shows a noticeable decrease.
current code:
IFACE rxpck/s txpck/s rxkB/s txkB/s
rxcmp/s txcmp/s rxmcst/s %ifutil
Average: vnet0 0.31 1330807.03 0.02 77976.98
0.00 0.00 0.00 6.39
# /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
avg = 145 nsecs, total: 1455980341 nsecs, count: 9985224
vmap:
IFACE rxpck/s txpck/s rxkB/s txkB/s
rxcmp/s txcmp/s rxmcst/s %ifutil
Average: vnet0 0.07 1371870.49 0.00 80383.04
0.00 0.00 0.00 6.58
# /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
avg = 122 nsecs, total: 1286983929 nsecs, count: 10478134
We are uncertain if there are any aspects we may have overlooked and
would appreciate any advice before we submit an actual patch.
Thanks,
Liang
^ permalink raw reply [flat|nested] 6+ messages in thread
* [RFC] vhost: vmap virtio descriptor table kernel/kvm
@ 2023-10-24 3:17 ` Liang Chen
0 siblings, 0 replies; 6+ messages in thread
From: Liang Chen @ 2023-10-24 3:17 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang; +Cc: kvm, virtualization
The current vhost code uses 'copy_from_user' to copy descriptors from
userspace to vhost. We attempted to 'vmap' the descriptor table to
reduce the overhead associated with 'copy_from_user' during descriptor
access, because it can be accessed quite frequently. This change
resulted in a moderate performance improvement (approximately 3%) in
our pktgen test, as shown below. Additionally, the latency in the
'vhost_get_vq_desc' function shows a noticeable decrease.
current code:
IFACE rxpck/s txpck/s rxkB/s txkB/s
rxcmp/s txcmp/s rxmcst/s %ifutil
Average: vnet0 0.31 1330807.03 0.02 77976.98
0.00 0.00 0.00 6.39
# /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
avg = 145 nsecs, total: 1455980341 nsecs, count: 9985224
vmap:
IFACE rxpck/s txpck/s rxkB/s txkB/s
rxcmp/s txcmp/s rxmcst/s %ifutil
Average: vnet0 0.07 1371870.49 0.00 80383.04
0.00 0.00 0.00 6.58
# /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
avg = 122 nsecs, total: 1286983929 nsecs, count: 10478134
We are uncertain if there are any aspects we may have overlooked and
would appreciate any advice before we submit an actual patch.
Thanks,
Liang
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] vhost: vmap virtio descriptor table kernel/kvm
2023-10-24 3:17 ` Liang Chen
@ 2023-10-24 4:44 ` Jason Wang
-1 siblings, 0 replies; 6+ messages in thread
From: Jason Wang @ 2023-10-24 4:44 UTC (permalink / raw)
To: Liang Chen; +Cc: virtualization, kvm, Michael S. Tsirkin
On Tue, Oct 24, 2023 at 11:17 AM Liang Chen <liangchen.linux@gmail.com> wrote:
>
> The current vhost code uses 'copy_from_user' to copy descriptors from
> userspace to vhost. We attempted to 'vmap' the descriptor table to
> reduce the overhead associated with 'copy_from_user' during descriptor
> access, because it can be accessed quite frequently. This change
> resulted in a moderate performance improvement (approximately 3%) in
> our pktgen test, as shown below. Additionally, the latency in the
> 'vhost_get_vq_desc' function shows a noticeable decrease.
>
> current code:
> IFACE rxpck/s txpck/s rxkB/s txkB/s
> rxcmp/s txcmp/s rxmcst/s %ifutil
> Average: vnet0 0.31 1330807.03 0.02 77976.98
> 0.00 0.00 0.00 6.39
> # /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
> avg = 145 nsecs, total: 1455980341 nsecs, count: 9985224
>
> vmap:
> IFACE rxpck/s txpck/s rxkB/s txkB/s
> rxcmp/s txcmp/s rxmcst/s %ifutil
> Average: vnet0 0.07 1371870.49 0.00 80383.04
> 0.00 0.00 0.00 6.58
> # /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
> avg = 122 nsecs, total: 1286983929 nsecs, count: 10478134
>
> We are uncertain if there are any aspects we may have overlooked and
> would appreciate any advice before we submit an actual patch.
So the idea is to use a shadow page table instead of the userspace one
to avoid things like spec barriers or SMAP.
I've tried this in the past:
commit 7f466032dc9e5a61217f22ea34b2df932786bbfc (HEAD)
Author: Jason Wang <jasowang@redhat.com>
Date: Fri May 24 04:12:18 2019 -0400
vhost: access vq metadata through kernel virtual address
It was noticed that the copy_to/from_user() friends that was used to
access virtqueue metdata tends to be very expensive for dataplane
implementation like vhost since it involves lots of software checks,
speculation barriers, hardware feature toggling (e.g SMAP). The
extra cost will be more obvious when transferring small packets since
the time spent on metadata accessing become more significant.
...
Note that it tries to use a direct map instead of a VMAP as Andrea
suggests. But it led to several fallouts which were tricky to be
fixed[1] (like the use of MMU notifiers to do synchronization). So it
is reverted finally.
I'm not saying it's a dead end. But we need to find a way to solve the
issues or use something different. I'm happy to offer help.
1) Avoid using SMAP for vhost kthread, for example using shed
notifier, I'm not sure this is possible or not
2) A new iov iterator that doesn't do SMAP at all, this looks
dangerous and Al might not like it
3) (Re)using HMM
...
You may want to see archives for more information. We've had a lot of
discussions.
Thanks
[1] https://lore.kernel.org/lkml/20190731084655.7024-1-jasowang@redhat.com/
>
>
> Thanks,
> Liang
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] vhost: vmap virtio descriptor table kernel/kvm
@ 2023-10-24 4:44 ` Jason Wang
0 siblings, 0 replies; 6+ messages in thread
From: Jason Wang @ 2023-10-24 4:44 UTC (permalink / raw)
To: Liang Chen; +Cc: Michael S. Tsirkin, virtualization, kvm
On Tue, Oct 24, 2023 at 11:17 AM Liang Chen <liangchen.linux@gmail.com> wrote:
>
> The current vhost code uses 'copy_from_user' to copy descriptors from
> userspace to vhost. We attempted to 'vmap' the descriptor table to
> reduce the overhead associated with 'copy_from_user' during descriptor
> access, because it can be accessed quite frequently. This change
> resulted in a moderate performance improvement (approximately 3%) in
> our pktgen test, as shown below. Additionally, the latency in the
> 'vhost_get_vq_desc' function shows a noticeable decrease.
>
> current code:
> IFACE rxpck/s txpck/s rxkB/s txkB/s
> rxcmp/s txcmp/s rxmcst/s %ifutil
> Average: vnet0 0.31 1330807.03 0.02 77976.98
> 0.00 0.00 0.00 6.39
> # /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
> avg = 145 nsecs, total: 1455980341 nsecs, count: 9985224
>
> vmap:
> IFACE rxpck/s txpck/s rxkB/s txkB/s
> rxcmp/s txcmp/s rxmcst/s %ifutil
> Average: vnet0 0.07 1371870.49 0.00 80383.04
> 0.00 0.00 0.00 6.58
> # /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
> avg = 122 nsecs, total: 1286983929 nsecs, count: 10478134
>
> We are uncertain if there are any aspects we may have overlooked and
> would appreciate any advice before we submit an actual patch.
So the idea is to use a shadow page table instead of the userspace one
to avoid things like spec barriers or SMAP.
I've tried this in the past:
commit 7f466032dc9e5a61217f22ea34b2df932786bbfc (HEAD)
Author: Jason Wang <jasowang@redhat.com>
Date: Fri May 24 04:12:18 2019 -0400
vhost: access vq metadata through kernel virtual address
It was noticed that the copy_to/from_user() friends that was used to
access virtqueue metdata tends to be very expensive for dataplane
implementation like vhost since it involves lots of software checks,
speculation barriers, hardware feature toggling (e.g SMAP). The
extra cost will be more obvious when transferring small packets since
the time spent on metadata accessing become more significant.
...
Note that it tries to use a direct map instead of a VMAP as Andrea
suggests. But it led to several fallouts which were tricky to be
fixed[1] (like the use of MMU notifiers to do synchronization). So it
is reverted finally.
I'm not saying it's a dead end. But we need to find a way to solve the
issues or use something different. I'm happy to offer help.
1) Avoid using SMAP for vhost kthread, for example using shed
notifier, I'm not sure this is possible or not
2) A new iov iterator that doesn't do SMAP at all, this looks
dangerous and Al might not like it
3) (Re)using HMM
...
You may want to see archives for more information. We've had a lot of
discussions.
Thanks
[1] https://lore.kernel.org/lkml/20190731084655.7024-1-jasowang@redhat.com/
>
>
> Thanks,
> Liang
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] vhost: vmap virtio descriptor table kernel/kvm
2023-10-24 4:44 ` Jason Wang
@ 2023-10-24 7:35 ` Liang Chen
-1 siblings, 0 replies; 6+ messages in thread
From: Liang Chen @ 2023-10-24 7:35 UTC (permalink / raw)
To: Jason Wang; +Cc: Michael S. Tsirkin, virtualization, kvm
On Tue, Oct 24, 2023 at 12:45 PM Jason Wang <jasowang@redhat.com> wrote:
>
> On Tue, Oct 24, 2023 at 11:17 AM Liang Chen <liangchen.linux@gmail.com> wrote:
> >
> > The current vhost code uses 'copy_from_user' to copy descriptors from
> > userspace to vhost. We attempted to 'vmap' the descriptor table to
> > reduce the overhead associated with 'copy_from_user' during descriptor
> > access, because it can be accessed quite frequently. This change
> > resulted in a moderate performance improvement (approximately 3%) in
> > our pktgen test, as shown below. Additionally, the latency in the
> > 'vhost_get_vq_desc' function shows a noticeable decrease.
> >
> > current code:
> > IFACE rxpck/s txpck/s rxkB/s txkB/s
> > rxcmp/s txcmp/s rxmcst/s %ifutil
> > Average: vnet0 0.31 1330807.03 0.02 77976.98
> > 0.00 0.00 0.00 6.39
> > # /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
> > avg = 145 nsecs, total: 1455980341 nsecs, count: 9985224
> >
> > vmap:
> > IFACE rxpck/s txpck/s rxkB/s txkB/s
> > rxcmp/s txcmp/s rxmcst/s %ifutil
> > Average: vnet0 0.07 1371870.49 0.00 80383.04
> > 0.00 0.00 0.00 6.58
> > # /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
> > avg = 122 nsecs, total: 1286983929 nsecs, count: 10478134
> >
> > We are uncertain if there are any aspects we may have overlooked and
> > would appreciate any advice before we submit an actual patch.
>
> So the idea is to use a shadow page table instead of the userspace one
> to avoid things like spec barriers or SMAP.
>
> I've tried this in the past:
>
> commit 7f466032dc9e5a61217f22ea34b2df932786bbfc (HEAD)
> Author: Jason Wang <jasowang@redhat.com>
> Date: Fri May 24 04:12:18 2019 -0400
>
> vhost: access vq metadata through kernel virtual address
>
> It was noticed that the copy_to/from_user() friends that was used to
> access virtqueue metdata tends to be very expensive for dataplane
> implementation like vhost since it involves lots of software checks,
> speculation barriers, hardware feature toggling (e.g SMAP). The
> extra cost will be more obvious when transferring small packets since
> the time spent on metadata accessing become more significant.
> ...
>
> Note that it tries to use a direct map instead of a VMAP as Andrea
> suggests. But it led to several fallouts which were tricky to be
> fixed[1] (like the use of MMU notifiers to do synchronization). So it
> is reverted finally.
>
> I'm not saying it's a dead end. But we need to find a way to solve the
> issues or use something different. I'm happy to offer help.
>
> 1) Avoid using SMAP for vhost kthread, for example using shed
> notifier, I'm not sure this is possible or not
> 2) A new iov iterator that doesn't do SMAP at all, this looks
> dangerous and Al might not like it
> 3) (Re)using HMM
> ...
>
> You may want to see archives for more information. We've had a lot of
> discussions.
>
> Thanks
>
> [1] https://lore.kernel.org/lkml/20190731084655.7024-1-jasowang@redhat.com/
>
Thanks a lot for the help. We will take a deeper look and reach out if
we have any questions or once we've finalized the patch.
Thanks,
Liang
> >
> >
> > Thanks,
> > Liang
> >
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC] vhost: vmap virtio descriptor table kernel/kvm
@ 2023-10-24 7:35 ` Liang Chen
0 siblings, 0 replies; 6+ messages in thread
From: Liang Chen @ 2023-10-24 7:35 UTC (permalink / raw)
To: Jason Wang; +Cc: virtualization, kvm, Michael S. Tsirkin
On Tue, Oct 24, 2023 at 12:45 PM Jason Wang <jasowang@redhat.com> wrote:
>
> On Tue, Oct 24, 2023 at 11:17 AM Liang Chen <liangchen.linux@gmail.com> wrote:
> >
> > The current vhost code uses 'copy_from_user' to copy descriptors from
> > userspace to vhost. We attempted to 'vmap' the descriptor table to
> > reduce the overhead associated with 'copy_from_user' during descriptor
> > access, because it can be accessed quite frequently. This change
> > resulted in a moderate performance improvement (approximately 3%) in
> > our pktgen test, as shown below. Additionally, the latency in the
> > 'vhost_get_vq_desc' function shows a noticeable decrease.
> >
> > current code:
> > IFACE rxpck/s txpck/s rxkB/s txkB/s
> > rxcmp/s txcmp/s rxmcst/s %ifutil
> > Average: vnet0 0.31 1330807.03 0.02 77976.98
> > 0.00 0.00 0.00 6.39
> > # /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
> > avg = 145 nsecs, total: 1455980341 nsecs, count: 9985224
> >
> > vmap:
> > IFACE rxpck/s txpck/s rxkB/s txkB/s
> > rxcmp/s txcmp/s rxmcst/s %ifutil
> > Average: vnet0 0.07 1371870.49 0.00 80383.04
> > 0.00 0.00 0.00 6.58
> > # /usr/share/bcc/tools/funclatency -d 10 vhost_get_vq_desc
> > avg = 122 nsecs, total: 1286983929 nsecs, count: 10478134
> >
> > We are uncertain if there are any aspects we may have overlooked and
> > would appreciate any advice before we submit an actual patch.
>
> So the idea is to use a shadow page table instead of the userspace one
> to avoid things like spec barriers or SMAP.
>
> I've tried this in the past:
>
> commit 7f466032dc9e5a61217f22ea34b2df932786bbfc (HEAD)
> Author: Jason Wang <jasowang@redhat.com>
> Date: Fri May 24 04:12:18 2019 -0400
>
> vhost: access vq metadata through kernel virtual address
>
> It was noticed that the copy_to/from_user() friends that was used to
> access virtqueue metdata tends to be very expensive for dataplane
> implementation like vhost since it involves lots of software checks,
> speculation barriers, hardware feature toggling (e.g SMAP). The
> extra cost will be more obvious when transferring small packets since
> the time spent on metadata accessing become more significant.
> ...
>
> Note that it tries to use a direct map instead of a VMAP as Andrea
> suggests. But it led to several fallouts which were tricky to be
> fixed[1] (like the use of MMU notifiers to do synchronization). So it
> is reverted finally.
>
> I'm not saying it's a dead end. But we need to find a way to solve the
> issues or use something different. I'm happy to offer help.
>
> 1) Avoid using SMAP for vhost kthread, for example using shed
> notifier, I'm not sure this is possible or not
> 2) A new iov iterator that doesn't do SMAP at all, this looks
> dangerous and Al might not like it
> 3) (Re)using HMM
> ...
>
> You may want to see archives for more information. We've had a lot of
> discussions.
>
> Thanks
>
> [1] https://lore.kernel.org/lkml/20190731084655.7024-1-jasowang@redhat.com/
>
Thanks a lot for the help. We will take a deeper look and reach out if
we have any questions or once we've finalized the patch.
Thanks,
Liang
> >
> >
> > Thanks,
> > Liang
> >
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-10-24 7:35 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-10-24 3:17 [RFC] vhost: vmap virtio descriptor table kernel/kvm Liang Chen
2023-10-24 3:17 ` Liang Chen
2023-10-24 4:44 ` Jason Wang
2023-10-24 4:44 ` Jason Wang
2023-10-24 7:35 ` Liang Chen
2023-10-24 7:35 ` Liang Chen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.