From: Liang Chen <liangchen.linux@gmail.com> To: jasowang@redhat.com, mst@redhat.com Cc: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, xuanzhuo@linux.alibaba.com, kuba@kernel.org, edumazet@google.com, davem@davemloft.net, pabeni@redhat.com, alexander.duyck@gmail.com, Liang Chen <liangchen.linux@gmail.com> Subject: [PATCH net-next 5/5] virtio_net: Implement DMA pre-handler Date: Fri, 26 May 2023 13:46:21 +0800 [thread overview] Message-ID: <20230526054621.18371-5-liangchen.linux@gmail.com> (raw) In-Reply-To: <20230526054621.18371-1-liangchen.linux@gmail.com> Adding a DMA pre-handler that utilizes page pool for managing DMA mappings. When IOMMU is enabled, turning on the page_pool_dma_map module parameter to select page pool for DMA mapping management gives a significant reduction in the overhead caused by DMA mappings. In testing environments with a single core vm and qemu emulated IOMMU, significant performance improvements can be observed: Upstream codebase: 1.76 Gbits/sec Upstream codebase with page pool fragmentation support: 1.81 Gbits/sec Upstream codebase with page pool fragmentation and DMA support: 19.3 Gbits/sec Signed-off-by: Liang Chen <liangchen.linux@gmail.com> --- drivers/net/virtio_net.c | 55 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index ac40b8c66c59..73cc4f9fe4fa 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -22,6 +22,7 @@ #include <net/route.h> #include <net/xdp.h> #include <net/net_failover.h> +#include <linux/iommu.h> static int napi_weight = NAPI_POLL_WEIGHT; module_param(napi_weight, int, 0444); @@ -33,8 +34,10 @@ module_param(napi_tx, bool, 0644); static bool page_pool_enabled; static bool page_pool_frag; +static bool page_pool_dma_map; module_param(page_pool_enabled, bool, 0400); module_param(page_pool_frag, bool, 0400); +module_param(page_pool_dma_map, bool, 0400); /* FIXME: MTU in config. */ #define GOOD_PACKET_LEN (ETH_HLEN + VLAN_HLEN + ETH_DATA_LEN) @@ -3830,6 +3833,49 @@ static void virtnet_del_vqs(struct virtnet_info *vi) virtnet_free_queues(vi); } +static dma_addr_t virtnet_pp_dma_map_page(struct device *dev, struct page *page, + unsigned long offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + struct page *head_page; + + if (dir != DMA_FROM_DEVICE) + return 0; + + head_page = compound_head(page); + return page_pool_get_dma_addr(head_page) + + (page - head_page) * PAGE_SIZE + + offset; +} + +static bool virtnet_pp_dma_unmap_page(struct device *dev, dma_addr_t dma_handle, + size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + phys_addr_t phys; + + /* Handle only the RX direction, and sync the DMA memory only if it's not + * a DMA coherent architecture. + */ + if (dir != DMA_FROM_DEVICE) + return false; + + if (dev_is_dma_coherent(dev)) + return true; + + phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); + if (WARN_ON(!phys)) + return false; + + arch_sync_dma_for_cpu(phys, size, dir); + return true; +} + +static struct virtqueue_pre_dma_ops virtnet_pp_pre_dma_ops = { + .map_page = virtnet_pp_dma_map_page, + .unmap_page = virtnet_pp_dma_unmap_page, +}; + static void virtnet_alloc_page_pool(struct receive_queue *rq) { struct virtio_device *vdev = rq->vq->vdev; @@ -3845,6 +3891,15 @@ static void virtnet_alloc_page_pool(struct receive_queue *rq) if (page_pool_frag) pp_params.flags |= PP_FLAG_PAGE_FRAG; + /* Consider using page pool DMA support only when DMA API is used. */ + if (virtio_has_feature(vdev, VIRTIO_F_ACCESS_PLATFORM) && + page_pool_dma_map) { + pp_params.flags |= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV; + pp_params.dma_dir = DMA_FROM_DEVICE; + pp_params.max_len = PAGE_SIZE << pp_params.order; + virtqueue_register_pre_dma_ops(rq->vq, &virtnet_pp_pre_dma_ops); + } + rq->page_pool = page_pool_create(&pp_params); if (IS_ERR(rq->page_pool)) { dev_warn(&vdev->dev, "page pool creation failed: %ld\n", -- 2.31.1
WARNING: multiple messages have this Message-ID (diff)
From: Liang Chen <liangchen.linux@gmail.com> To: jasowang@redhat.com, mst@redhat.com Cc: xuanzhuo@linux.alibaba.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, virtualization@lists.linux-foundation.org, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, davem@davemloft.net Subject: [PATCH net-next 5/5] virtio_net: Implement DMA pre-handler Date: Fri, 26 May 2023 13:46:21 +0800 [thread overview] Message-ID: <20230526054621.18371-5-liangchen.linux@gmail.com> (raw) In-Reply-To: <20230526054621.18371-1-liangchen.linux@gmail.com> Adding a DMA pre-handler that utilizes page pool for managing DMA mappings. When IOMMU is enabled, turning on the page_pool_dma_map module parameter to select page pool for DMA mapping management gives a significant reduction in the overhead caused by DMA mappings. In testing environments with a single core vm and qemu emulated IOMMU, significant performance improvements can be observed: Upstream codebase: 1.76 Gbits/sec Upstream codebase with page pool fragmentation support: 1.81 Gbits/sec Upstream codebase with page pool fragmentation and DMA support: 19.3 Gbits/sec Signed-off-by: Liang Chen <liangchen.linux@gmail.com> --- drivers/net/virtio_net.c | 55 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index ac40b8c66c59..73cc4f9fe4fa 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -22,6 +22,7 @@ #include <net/route.h> #include <net/xdp.h> #include <net/net_failover.h> +#include <linux/iommu.h> static int napi_weight = NAPI_POLL_WEIGHT; module_param(napi_weight, int, 0444); @@ -33,8 +34,10 @@ module_param(napi_tx, bool, 0644); static bool page_pool_enabled; static bool page_pool_frag; +static bool page_pool_dma_map; module_param(page_pool_enabled, bool, 0400); module_param(page_pool_frag, bool, 0400); +module_param(page_pool_dma_map, bool, 0400); /* FIXME: MTU in config. */ #define GOOD_PACKET_LEN (ETH_HLEN + VLAN_HLEN + ETH_DATA_LEN) @@ -3830,6 +3833,49 @@ static void virtnet_del_vqs(struct virtnet_info *vi) virtnet_free_queues(vi); } +static dma_addr_t virtnet_pp_dma_map_page(struct device *dev, struct page *page, + unsigned long offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + struct page *head_page; + + if (dir != DMA_FROM_DEVICE) + return 0; + + head_page = compound_head(page); + return page_pool_get_dma_addr(head_page) + + (page - head_page) * PAGE_SIZE + + offset; +} + +static bool virtnet_pp_dma_unmap_page(struct device *dev, dma_addr_t dma_handle, + size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + phys_addr_t phys; + + /* Handle only the RX direction, and sync the DMA memory only if it's not + * a DMA coherent architecture. + */ + if (dir != DMA_FROM_DEVICE) + return false; + + if (dev_is_dma_coherent(dev)) + return true; + + phys = iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); + if (WARN_ON(!phys)) + return false; + + arch_sync_dma_for_cpu(phys, size, dir); + return true; +} + +static struct virtqueue_pre_dma_ops virtnet_pp_pre_dma_ops = { + .map_page = virtnet_pp_dma_map_page, + .unmap_page = virtnet_pp_dma_unmap_page, +}; + static void virtnet_alloc_page_pool(struct receive_queue *rq) { struct virtio_device *vdev = rq->vq->vdev; @@ -3845,6 +3891,15 @@ static void virtnet_alloc_page_pool(struct receive_queue *rq) if (page_pool_frag) pp_params.flags |= PP_FLAG_PAGE_FRAG; + /* Consider using page pool DMA support only when DMA API is used. */ + if (virtio_has_feature(vdev, VIRTIO_F_ACCESS_PLATFORM) && + page_pool_dma_map) { + pp_params.flags |= PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV; + pp_params.dma_dir = DMA_FROM_DEVICE; + pp_params.max_len = PAGE_SIZE << pp_params.order; + virtqueue_register_pre_dma_ops(rq->vq, &virtnet_pp_pre_dma_ops); + } + rq->page_pool = page_pool_create(&pp_params); if (IS_ERR(rq->page_pool)) { dev_warn(&vdev->dev, "page pool creation failed: %ld\n", -- 2.31.1 _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
next prev parent reply other threads:[~2023-05-26 5:47 UTC|newest] Thread overview: 101+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-05-26 5:46 [PATCH net-next 1/5] virtio_net: Fix an unsafe reference to the page chain Liang Chen 2023-05-26 5:46 ` Liang Chen 2023-05-26 5:46 ` [PATCH net-next 2/5] virtio_net: Add page_pool support to improve performance Liang Chen 2023-05-26 5:46 ` Liang Chen 2023-05-26 6:50 ` Jason Wang 2023-05-26 6:50 ` Jason Wang 2023-05-27 12:35 ` Liang Chen 2023-05-27 12:35 ` Liang Chen 2023-05-28 6:40 ` Michael S. Tsirkin 2023-05-28 6:40 ` Michael S. Tsirkin 2023-05-29 7:28 ` Liang Chen 2023-05-29 7:28 ` Liang Chen 2023-05-31 3:10 ` Xuan Zhuo 2023-05-31 3:10 ` Xuan Zhuo 2023-06-07 9:11 ` Liang Chen 2023-06-07 9:11 ` Liang Chen 2023-06-07 9:33 ` Xuan Zhuo 2023-06-07 9:33 ` Xuan Zhuo 2023-05-26 16:11 ` kernel test robot 2023-05-26 16:11 ` kernel test robot 2023-05-28 6:27 ` Michael S. Tsirkin 2023-05-28 6:27 ` Michael S. Tsirkin 2023-05-29 7:28 ` Liang Chen 2023-05-29 7:28 ` Liang Chen 2023-05-28 6:20 ` Michael S. Tsirkin 2023-05-28 6:20 ` Michael S. Tsirkin 2023-05-29 7:27 ` Liang Chen 2023-05-29 7:27 ` Liang Chen 2023-05-29 9:55 ` Michael S. Tsirkin 2023-05-29 9:55 ` Michael S. Tsirkin 2023-05-30 1:19 ` Liang Chen 2023-05-30 1:19 ` Liang Chen 2023-06-07 9:08 ` Liang Chen 2023-06-07 9:08 ` Liang Chen 2023-06-07 9:35 ` Xuan Zhuo 2023-06-07 9:35 ` Xuan Zhuo 2023-06-07 13:58 ` Liang Chen 2023-06-07 13:58 ` Liang Chen 2023-06-07 20:17 ` Michael S. Tsirkin 2023-06-07 20:17 ` Michael S. Tsirkin 2023-06-08 0:38 ` Jason Wang 2023-06-08 0:38 ` Jason Wang 2023-06-08 3:54 ` Xuan Zhuo 2023-06-08 3:54 ` Xuan Zhuo 2023-06-09 2:57 ` Liang Chen 2023-06-09 2:57 ` Liang Chen 2023-07-05 5:41 ` Liang Chen 2023-07-05 5:41 ` Liang Chen 2023-07-05 6:04 ` Jason Wang 2023-07-05 6:04 ` Jason Wang 2023-07-06 1:01 ` Liang Chen 2023-07-06 1:01 ` Liang Chen 2023-06-09 2:57 ` Liang Chen 2023-06-09 2:57 ` Liang Chen 2023-11-29 14:50 ` Zhu Yanjun 2023-11-29 14:59 ` Michael S. Tsirkin 2023-11-29 15:22 ` Zhu Yanjun 2023-11-29 15:29 ` Zhu Yanjun 2023-11-30 2:34 ` Xuan Zhuo 2023-11-30 5:30 ` Zhu Yanjun 2023-12-01 1:38 ` Xuan Zhuo 2023-12-04 5:24 ` Zhu Yanjun 2023-11-30 7:17 ` Zhu Yanjun 2023-05-26 5:46 ` [PATCH net-next 3/5] virtio_net: Add page pool fragmentation support Liang Chen 2023-05-26 5:46 ` Liang Chen 2023-05-26 8:29 ` Horatiu Vultur 2023-05-27 12:36 ` Liang Chen 2023-05-27 12:36 ` Liang Chen 2023-05-26 17:44 ` kernel test robot 2023-05-26 17:44 ` kernel test robot 2023-05-28 6:25 ` Michael S. Tsirkin 2023-05-28 6:25 ` Michael S. Tsirkin 2023-05-29 7:29 ` Liang Chen 2023-05-29 7:29 ` Liang Chen 2023-05-29 1:33 ` Yunsheng Lin 2023-05-29 7:30 ` Liang Chen 2023-05-29 7:30 ` Liang Chen 2023-05-26 5:46 ` [PATCH net-next 4/5] virtio_ring: Introduce DMA pre-handler Liang Chen 2023-05-26 5:46 ` Liang Chen 2023-05-26 6:57 ` Jason Wang 2023-05-26 6:57 ` Jason Wang 2023-05-26 5:46 ` Liang Chen [this message] 2023-05-26 5:46 ` [PATCH net-next 5/5] virtio_net: Implement " Liang Chen 2023-05-26 7:06 ` Jason Wang 2023-05-26 7:06 ` Jason Wang 2023-05-27 12:35 ` Liang Chen 2023-05-27 12:35 ` Liang Chen 2023-05-26 17:34 ` kernel test robot 2023-05-26 17:34 ` kernel test robot 2023-05-26 6:38 ` [PATCH net-next 1/5] virtio_net: Fix an unsafe reference to the page chain Jason Wang 2023-05-26 6:38 ` Jason Wang 2023-05-27 12:33 ` Liang Chen 2023-05-27 12:33 ` Liang Chen 2023-05-28 6:29 ` Michael S. Tsirkin 2023-05-28 6:29 ` Michael S. Tsirkin 2023-05-29 7:25 ` Liang Chen 2023-05-29 7:25 ` Liang Chen 2023-05-28 6:16 ` Michael S. Tsirkin 2023-05-28 6:16 ` Michael S. Tsirkin 2023-05-29 7:25 ` Liang Chen 2023-05-29 7:25 ` Liang Chen
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20230526054621.18371-5-liangchen.linux@gmail.com \ --to=liangchen.linux@gmail.com \ --cc=alexander.duyck@gmail.com \ --cc=davem@davemloft.net \ --cc=edumazet@google.com \ --cc=jasowang@redhat.com \ --cc=kuba@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mst@redhat.com \ --cc=netdev@vger.kernel.org \ --cc=pabeni@redhat.com \ --cc=virtualization@lists.linux-foundation.org \ --cc=xuanzhuo@linux.alibaba.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.