From: Jason Wang <jasowang@redhat.com> To: Stefano Garzarella <sgarzare@redhat.com> Cc: virtualization@lists.linux-foundation.org, Stefan Hajnoczi <stefanha@redhat.com>, Laurent Vivier <lvivier@redhat.com>, linux-kernel@vger.kernel.org, Eli Cohen <elic@nvidia.com>, "Michael S. Tsirkin" <mst@redhat.com>, Max Gurtovoy <mgurtovoy@nvidia.com> Subject: Re: [PATCH RFC 00/12] vdpa: generalize vdpa simulator and add block device Date: Mon, 21 Dec 2020 11:16:54 +0800 [thread overview] Message-ID: <7dd3ed02-36c3-fcfd-0a1d-9c31af6f473e@redhat.com> (raw) In-Reply-To: <20201218113816.zcyeyqipux4ao4cp@steredhat> On 2020/12/18 下午7:38, Stefano Garzarella wrote: > On Mon, Nov 16, 2020 at 11:37:48AM +0800, Jason Wang wrote: >> >> On 2020/11/13 下午9:47, Stefano Garzarella wrote: >>> Thanks to Max that started this work! >>> I took his patches, and extended the block simulator a bit. >>> >>> This series moves the network device simulator in a new module >>> (vdpa_sim_net) and leaves the generic functions in the vdpa_sim core >>> module, allowing the possibility to add new vDPA device simulators. >>> Then we added a new vdpa_sim_blk module to simulate a block device. >>> >>> I'm not sure about patch 11 ("vringh: allow vringh_iov_xfer() to skip >>> bytes when ptr is NULL"), maybe we can add a new functions instead of >>> modify vringh_iov_xfer(). >>> >>> As Max reported, I'm also seeing errors with vdpa_sim_blk related to >>> iotlb and vringh when there is high load, these are some of the error >>> messages I can see randomly: >>> >>> vringh: Failed to access avail idx at 00000000e8deb2cc >>> vringh: Failed to read head: idx 6289 address 00000000e1ad1d50 >>> vringh: Failed to get flags at 000000006635d7a3 >>> >>> virtio_vdpa vdpa0: vringh_iov_push_iotlb() error: -14 offset: >>> 0x2840000 len: 0x20000 >>> virtio_vdpa vdpa0: vringh_iov_pull_iotlb() error: -14 offset: >>> 0x58ee000 len: 0x3000 >>> >>> These errors should all be related to the fact that iotlb_translate() >>> fails with -EINVAL, so it seems that we miss some mapping. >> >> >> Is this only reproducible when there's multiple co-current accessing >> of IOTLB? If yes, it's probably a hint that some kind of >> synchronization is still missed somewhere. >> >> It might be useful to log the dma_map/unmp in both virtio_ring and >> vringh to see who is missing the map. >> > > Just an update about these issues with vdpa-sim-blk. > I've been focusing a little bit on these failures over the last few > days and have found two issues related to the IOTLB/IOMMU: > > 1. Some requests coming from the block layer fills the SG list with > multiple buffers that had the same physical address. This happens for > example while using 'mkfs', at some points multiple sectors are zeroed > so multiple SG elements point to the same physical page that is zeroed. > Since we are using vhost_iotlb_del_range() in the > vdpasim_unmap_page(), this removes all the overlapped ranges. I fixed > removing a single map in vdpasim_unmap_page(), but has an alternative > we can implement some kind of reference counts. I think we need to do what hardware do. So using refcount is probably not a good ida. > > 2. There was a race between dma_map/unmap and the worker thread, since > both are accessing the IOMMU. Taking the iommu_lock while using > vhost_iotlb_* API in the worker thread fixes the "vringh: Failed to *" > issues. > > Whit these issues fixed the vdpa-blk simulator seems to work well. > I'll send the patches next week or after the break. Good to know this. Thanks > > Thanks, > Stefano >
WARNING: multiple messages have this Message-ID (diff)
From: Jason Wang <jasowang@redhat.com> To: Stefano Garzarella <sgarzare@redhat.com> Cc: Laurent Vivier <lvivier@redhat.com>, Max Gurtovoy <mgurtovoy@nvidia.com>, "Michael S. Tsirkin" <mst@redhat.com>, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Stefan Hajnoczi <stefanha@redhat.com>, Eli Cohen <elic@nvidia.com> Subject: Re: [PATCH RFC 00/12] vdpa: generalize vdpa simulator and add block device Date: Mon, 21 Dec 2020 11:16:54 +0800 [thread overview] Message-ID: <7dd3ed02-36c3-fcfd-0a1d-9c31af6f473e@redhat.com> (raw) In-Reply-To: <20201218113816.zcyeyqipux4ao4cp@steredhat> On 2020/12/18 下午7:38, Stefano Garzarella wrote: > On Mon, Nov 16, 2020 at 11:37:48AM +0800, Jason Wang wrote: >> >> On 2020/11/13 下午9:47, Stefano Garzarella wrote: >>> Thanks to Max that started this work! >>> I took his patches, and extended the block simulator a bit. >>> >>> This series moves the network device simulator in a new module >>> (vdpa_sim_net) and leaves the generic functions in the vdpa_sim core >>> module, allowing the possibility to add new vDPA device simulators. >>> Then we added a new vdpa_sim_blk module to simulate a block device. >>> >>> I'm not sure about patch 11 ("vringh: allow vringh_iov_xfer() to skip >>> bytes when ptr is NULL"), maybe we can add a new functions instead of >>> modify vringh_iov_xfer(). >>> >>> As Max reported, I'm also seeing errors with vdpa_sim_blk related to >>> iotlb and vringh when there is high load, these are some of the error >>> messages I can see randomly: >>> >>> vringh: Failed to access avail idx at 00000000e8deb2cc >>> vringh: Failed to read head: idx 6289 address 00000000e1ad1d50 >>> vringh: Failed to get flags at 000000006635d7a3 >>> >>> virtio_vdpa vdpa0: vringh_iov_push_iotlb() error: -14 offset: >>> 0x2840000 len: 0x20000 >>> virtio_vdpa vdpa0: vringh_iov_pull_iotlb() error: -14 offset: >>> 0x58ee000 len: 0x3000 >>> >>> These errors should all be related to the fact that iotlb_translate() >>> fails with -EINVAL, so it seems that we miss some mapping. >> >> >> Is this only reproducible when there's multiple co-current accessing >> of IOTLB? If yes, it's probably a hint that some kind of >> synchronization is still missed somewhere. >> >> It might be useful to log the dma_map/unmp in both virtio_ring and >> vringh to see who is missing the map. >> > > Just an update about these issues with vdpa-sim-blk. > I've been focusing a little bit on these failures over the last few > days and have found two issues related to the IOTLB/IOMMU: > > 1. Some requests coming from the block layer fills the SG list with > multiple buffers that had the same physical address. This happens for > example while using 'mkfs', at some points multiple sectors are zeroed > so multiple SG elements point to the same physical page that is zeroed. > Since we are using vhost_iotlb_del_range() in the > vdpasim_unmap_page(), this removes all the overlapped ranges. I fixed > removing a single map in vdpasim_unmap_page(), but has an alternative > we can implement some kind of reference counts. I think we need to do what hardware do. So using refcount is probably not a good ida. > > 2. There was a race between dma_map/unmap and the worker thread, since > both are accessing the IOMMU. Taking the iommu_lock while using > vhost_iotlb_* API in the worker thread fixes the "vringh: Failed to *" > issues. > > Whit these issues fixed the vdpa-blk simulator seems to work well. > I'll send the patches next week or after the break. Good to know this. Thanks > > Thanks, > Stefano > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2020-12-21 3:18 UTC|newest] Thread overview: 131+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-11-13 13:47 [PATCH RFC 00/12] vdpa: generalize vdpa simulator and add block device Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-13 13:47 ` [PATCH RFC 01/12] vhost-vdpa: add support for vDPA blk devices Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-17 10:57 ` Stefan Hajnoczi 2020-11-17 10:57 ` Stefan Hajnoczi 2020-11-17 15:05 ` Stefano Garzarella 2020-11-17 15:05 ` Stefano Garzarella 2020-11-13 13:47 ` [PATCH RFC 02/12] vdpa: split vdpasim to core and net modules Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-13 18:32 ` kernel test robot 2020-11-16 4:00 ` Jason Wang 2020-11-16 4:00 ` Jason Wang 2020-11-16 9:39 ` Stefano Garzarella 2020-11-16 9:39 ` Stefano Garzarella 2020-11-18 13:14 ` Stefano Garzarella 2020-11-18 13:14 ` Stefano Garzarella 2020-11-19 6:16 ` Jason Wang 2020-11-19 6:16 ` Jason Wang 2020-11-16 9:06 ` Dan Carpenter 2020-11-16 9:06 ` Dan Carpenter 2020-11-16 9:12 ` Stefano Garzarella 2020-11-16 10:47 ` Dan Carpenter 2020-11-16 10:47 ` Dan Carpenter 2020-11-16 11:16 ` Stefano Garzarella 2020-11-16 11:58 ` Dan Carpenter 2020-11-16 11:58 ` Dan Carpenter 2020-11-16 12:41 ` Stefano Garzarella 2020-11-16 12:54 ` Dan Carpenter 2020-11-16 12:54 ` Dan Carpenter 2020-11-13 13:47 ` [PATCH RFC 03/12] vdpa_sim: remove hard-coded virtq count Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:02 ` Jason Wang 2020-11-16 4:02 ` Jason Wang 2020-11-13 13:47 ` [PATCH RFC 04/12] vdpa: add vdpa simulator for block device Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:10 ` Jason Wang 2020-11-16 4:10 ` Jason Wang 2020-11-16 10:17 ` Stefano Garzarella 2020-11-16 10:17 ` Stefano Garzarella 2020-11-17 11:11 ` Stefan Hajnoczi 2020-11-17 11:11 ` Stefan Hajnoczi 2020-11-17 14:16 ` Stefano Garzarella 2020-11-17 14:16 ` Stefano Garzarella 2020-11-17 16:43 ` Stefan Hajnoczi 2020-11-17 16:43 ` Stefan Hajnoczi 2020-11-17 17:38 ` Stefano Garzarella 2020-11-17 17:38 ` Stefano Garzarella 2020-11-18 11:23 ` Stefan Hajnoczi 2020-11-18 11:23 ` Stefan Hajnoczi 2020-11-18 11:39 ` Stefano Garzarella 2020-11-18 11:39 ` Stefano Garzarella 2020-11-13 13:47 ` [PATCH RFC 05/12] vdpa_sim: remove the limit of IOTLB entries Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:12 ` Jason Wang 2020-11-16 4:12 ` Jason Wang 2020-11-16 10:22 ` Stefano Garzarella 2020-11-16 10:22 ` Stefano Garzarella 2020-11-13 13:47 ` [PATCH RFC 06/12] vdpa_sim: add struct vdpasim_device to store device properties Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:14 ` Jason Wang 2020-11-16 4:14 ` Jason Wang 2020-11-16 10:30 ` Stefano Garzarella 2020-11-16 10:30 ` Stefano Garzarella 2020-11-17 11:23 ` Stefan Hajnoczi 2020-11-17 11:23 ` Stefan Hajnoczi 2020-11-17 14:24 ` Stefano Garzarella 2020-11-17 14:24 ` Stefano Garzarella 2020-11-13 13:47 ` [PATCH RFC 07/12] vdpa_sim: move config management outside of the core Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:18 ` Jason Wang 2020-11-16 4:18 ` Jason Wang 2020-11-16 10:42 ` Stefano Garzarella 2020-11-16 10:42 ` Stefano Garzarella 2020-11-13 13:47 ` [PATCH RFC 08/12] vdpa_sim: use kvmalloc to allocate vdpasim->buffer Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:19 ` Jason Wang 2020-11-16 4:19 ` Jason Wang 2020-11-13 13:47 ` [PATCH RFC 09/12] vdpa_sim: make vdpasim->buffer size configurable Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:19 ` Jason Wang 2020-11-16 4:19 ` Jason Wang 2020-11-13 13:47 ` [PATCH RFC 10/12] vdpa_sim: split vdpasim_virtqueue's iov field in riov and wiov Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:21 ` Jason Wang 2020-11-16 4:21 ` Jason Wang 2020-11-17 11:27 ` Stefan Hajnoczi 2020-11-17 11:27 ` Stefan Hajnoczi 2020-11-20 14:41 ` Stefano Garzarella 2020-11-20 14:41 ` Stefano Garzarella 2020-11-13 13:47 ` [PATCH RFC 11/12] vringh: allow vringh_iov_xfer() to skip bytes when ptr is NULL Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-16 4:32 ` Jason Wang 2020-11-16 4:32 ` Jason Wang 2020-11-16 11:48 ` Stefano Garzarella 2020-11-16 11:48 ` Stefano Garzarella 2020-11-13 13:47 ` [PATCH RFC 12/12] vdpa_sim_blk: implement ramdisk behaviour Stefano Garzarella 2020-11-13 13:47 ` Stefano Garzarella 2020-11-13 15:15 ` kernel test robot 2020-11-16 5:25 ` Jason Wang 2020-11-16 5:25 ` Jason Wang 2020-11-16 11:08 ` Stefano Garzarella 2020-11-16 11:08 ` Stefano Garzarella 2020-11-16 9:50 ` Michael S. Tsirkin 2020-11-16 9:50 ` Michael S. Tsirkin 2020-11-16 11:23 ` Stefano Garzarella 2020-11-16 11:23 ` Stefano Garzarella 2020-11-17 11:36 ` Stefan Hajnoczi 2020-11-17 11:36 ` Stefan Hajnoczi 2020-11-17 14:32 ` Stefano Garzarella 2020-11-17 14:32 ` Stefano Garzarella 2020-11-16 3:37 ` [PATCH RFC 00/12] vdpa: generalize vdpa simulator and add block device Jason Wang 2020-11-16 3:37 ` Jason Wang 2020-11-16 8:53 ` Stefano Garzarella 2020-11-16 8:53 ` Stefano Garzarella 2020-12-18 11:38 ` Stefano Garzarella 2020-12-18 11:38 ` Stefano Garzarella 2020-12-21 3:16 ` Jason Wang [this message] 2020-12-21 3:16 ` Jason Wang 2020-12-21 11:14 ` Stefano Garzarella 2020-12-21 11:14 ` Stefano Garzarella 2020-12-22 2:44 ` Jason Wang 2020-12-22 2:44 ` Jason Wang 2020-12-22 10:57 ` Stefano Garzarella 2020-12-22 10:57 ` Stefano Garzarella 2020-12-22 12:29 ` Jason Wang 2020-12-22 12:29 ` Jason Wang 2020-12-22 12:56 ` Jason Wang 2020-12-22 12:56 ` Jason Wang 2020-12-22 13:23 ` Stefano Garzarella 2020-12-22 13:23 ` Stefano Garzarella
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=7dd3ed02-36c3-fcfd-0a1d-9c31af6f473e@redhat.com \ --to=jasowang@redhat.com \ --cc=elic@nvidia.com \ --cc=linux-kernel@vger.kernel.org \ --cc=lvivier@redhat.com \ --cc=mgurtovoy@nvidia.com \ --cc=mst@redhat.com \ --cc=sgarzare@redhat.com \ --cc=stefanha@redhat.com \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.