From: Yi Zhang <yizhan-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> To: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> Cc: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Christoph Hellwig <hch-jcswGhMUV9g@public.gmane.org>, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org Subject: Re: mlx4_core 0000:07:00.0: swiotlb buffer is full and OOM observed during stress test on reset_controller Date: Sat, 18 Mar 2017 07:51:56 -0400 (EDT) [thread overview] Message-ID: <1768681609.3995777.1489837916289.JavaMail.zimbra@redhat.com> (raw) In-Reply-To: <31678a43-f76c-a921-e40c-470b0de1a86c-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> Hi Sagi With this path, the OOM cannot be reproduced now. But there is another problem, the reset operation[1] failed at iteration 1007. [1] echo 1 >/sys/block/nvme0n1/device/reset_controller Execution log: -------------------------------1007 reset.sh: line 8: echo: write error: Device or resource busy Server log: Client side log: [ 55.712617] virbr0: port 1(virbr0-nic) entered listening state [ 55.880978] virbr0: port 1(virbr0-nic) entered disabled state [ 269.995587] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.2.3:1023 [ 270.178461] nvme nvme0: creating 16 I/O queues. [ 270.624840] nvme nvme0: new ctrl: NQN "nvme-subsystem-name", addr 172.31.2.3:1023 [ 1221.955386] nvme nvme0: rdma_resolve_addr wait failed (-110). [ 1221.987117] nvme nvme0: failed to initialize i/o queue: -110 [ 1222.013938] nvme nvme0: Removing after reset failure Server side log: [ 1211.370445] nvmet: creating controller 1 for subsystem nvme-subsystem-name for NQN nqn.2014-08.org.nvmexpress:NVMf:uuid:6ed0e109-0b81-4bda-9950-786d67c91b5d. [ 1211.471407] nvmet: adding queue 1 to ctrl 1. [ 1211.490980] nvmet: adding queue 2 to ctrl 1. [ 1211.511142] nvmet: adding queue 3 to ctrl 1. [ 1211.530775] nvmet: adding queue 4 to ctrl 1. [ 1211.550138] nvmet: adding queue 5 to ctrl 1. [ 1211.569147] nvmet: adding queue 6 to ctrl 1. [ 1211.588649] nvmet: adding queue 7 to ctrl 1. [ 1211.608043] nvmet: adding queue 8 to ctrl 1. [ 1211.626965] nvmet: adding queue 9 to ctrl 1. [ 1211.646310] nvmet: adding queue 10 to ctrl 1. [ 1211.666774] nvmet: adding queue 11 to ctrl 1. [ 1211.686848] nvmet: adding queue 12 to ctrl 1. [ 1211.706654] nvmet: adding queue 13 to ctrl 1. [ 1211.726504] nvmet: adding queue 14 to ctrl 1. [ 1211.747046] nvmet: adding queue 15 to ctrl 1. [ 1211.767842] nvmet: adding queue 16 to ctrl 1. [ 1211.822222] nvmet_rdma: freeing queue 0 [ 1211.840225] nvmet_rdma: freeing queue 1 [ 1211.840301] nvmet_rdma: freeing queue 12 [ 1211.841740] nvmet_rdma: freeing queue 13 [ 1211.843222] nvmet_rdma: freeing queue 14 [ 1211.844511] nvmet_rdma: freeing queue 15 [ 1211.846102] nvmet_rdma: freeing queue 16 [ 1211.946919] nvmet_rdma: freeing queue 2 [ 1211.964700] nvmet_rdma: freeing queue 3 [ 1211.982548] nvmet_rdma: freeing queue 4 [ 1212.001528] nvmet_rdma: freeing queue 5 [ 1212.020271] nvmet_rdma: freeing queue 6 [ 1212.038598] nvmet_rdma: freeing queue 7 [ 1212.048886] nvmet: creating controller 2 for subsystem nvme-subsystem-name for NQN nqn.2014-08.org.nvmexpress:NVMf:uuid:6ed0e109-0b81-4bda-9950-786d67c91b5d. [ 1212.120320] nvmet_rdma: freeing queue 8 [ 1212.860605] nvmet_rdma: freeing queue 9 [ 1214.039350] nvmet_rdma: freeing queue 10 [ 1215.244894] nvmet_rdma: freeing queue 11 [ 1216.235774] nvmet_rdma: failed to connect queue 0 [ 1216.256877] nvmet_rdma: freeing queue 0 [ 1217.356506] nvmet_rdma: freeing queue 17 Best Regards, Yi Zhang ----- Original Message ----- From: "Sagi Grimberg" <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> To: "Yi Zhang" <yizhan-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Max Gurtovoy" <maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Leon Romanovsky" <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, "Christoph Hellwig" <hch-jcswGhMUV9g@public.gmane.org>, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org Sent: Friday, March 17, 2017 12:51:16 AM Subject: Re: mlx4_core 0000:07:00.0: swiotlb buffer is full and OOM observed during stress test on reset_controller >>>>> Sagi, >>>>> The release function is placed in global workqueue. I'm not familiar >>>>> with NVMe design and I don't know all the details, but maybe the >>>>> proper way will >>>>> be to create special workqueue with MEM_RECLAIM flag to ensure the >>>>> progress? Leon, the release work makes progress, but it is inherently slower than the establishment work and when we are bombarded with establishments we have no backpressure... > I tried with 4.11.0-rc2, and still can reproduced it with less than 2000 > times. Yi, Can you try the below (untested) patch: I'm not at all convinced this is the way to go because it will slow down all the connect requests, but I'm curious to know if it'll make the issue go away. -- diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index ecc4fe862561..f15fa6e6b640 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -1199,6 +1199,9 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, } queue->port = cm_id->context; + /* Let inflight queue teardown complete */ + flush_scheduled_work(); + ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn); if (ret) goto release_queue; -- Any other good ideas are welcome... _______________________________________________ Linux-nvme mailing list Linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org http://lists.infradead.org/mailman/listinfo/linux-nvme -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: yizhan@redhat.com (Yi Zhang) Subject: mlx4_core 0000:07:00.0: swiotlb buffer is full and OOM observed during stress test on reset_controller Date: Sat, 18 Mar 2017 07:51:56 -0400 (EDT) [thread overview] Message-ID: <1768681609.3995777.1489837916289.JavaMail.zimbra@redhat.com> (raw) In-Reply-To: <31678a43-f76c-a921-e40c-470b0de1a86c@grimberg.me> Hi Sagi With this path, the OOM cannot be reproduced now. But there is another problem, the reset operation[1] failed at iteration 1007. [1] echo 1 >/sys/block/nvme0n1/device/reset_controller Execution log: -------------------------------1007 reset.sh: line 8: echo: write error: Device or resource busy Server log: Client side log: [ 55.712617] virbr0: port 1(virbr0-nic) entered listening state [ 55.880978] virbr0: port 1(virbr0-nic) entered disabled state [ 269.995587] nvme nvme0: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.2.3:1023 [ 270.178461] nvme nvme0: creating 16 I/O queues. [ 270.624840] nvme nvme0: new ctrl: NQN "nvme-subsystem-name", addr 172.31.2.3:1023 [ 1221.955386] nvme nvme0: rdma_resolve_addr wait failed (-110). [ 1221.987117] nvme nvme0: failed to initialize i/o queue: -110 [ 1222.013938] nvme nvme0: Removing after reset failure Server side log: [ 1211.370445] nvmet: creating controller 1 for subsystem nvme-subsystem-name for NQN nqn.2014-08.org.nvmexpress:NVMf:uuid:6ed0e109-0b81-4bda-9950-786d67c91b5d. [ 1211.471407] nvmet: adding queue 1 to ctrl 1. [ 1211.490980] nvmet: adding queue 2 to ctrl 1. [ 1211.511142] nvmet: adding queue 3 to ctrl 1. [ 1211.530775] nvmet: adding queue 4 to ctrl 1. [ 1211.550138] nvmet: adding queue 5 to ctrl 1. [ 1211.569147] nvmet: adding queue 6 to ctrl 1. [ 1211.588649] nvmet: adding queue 7 to ctrl 1. [ 1211.608043] nvmet: adding queue 8 to ctrl 1. [ 1211.626965] nvmet: adding queue 9 to ctrl 1. [ 1211.646310] nvmet: adding queue 10 to ctrl 1. [ 1211.666774] nvmet: adding queue 11 to ctrl 1. [ 1211.686848] nvmet: adding queue 12 to ctrl 1. [ 1211.706654] nvmet: adding queue 13 to ctrl 1. [ 1211.726504] nvmet: adding queue 14 to ctrl 1. [ 1211.747046] nvmet: adding queue 15 to ctrl 1. [ 1211.767842] nvmet: adding queue 16 to ctrl 1. [ 1211.822222] nvmet_rdma: freeing queue 0 [ 1211.840225] nvmet_rdma: freeing queue 1 [ 1211.840301] nvmet_rdma: freeing queue 12 [ 1211.841740] nvmet_rdma: freeing queue 13 [ 1211.843222] nvmet_rdma: freeing queue 14 [ 1211.844511] nvmet_rdma: freeing queue 15 [ 1211.846102] nvmet_rdma: freeing queue 16 [ 1211.946919] nvmet_rdma: freeing queue 2 [ 1211.964700] nvmet_rdma: freeing queue 3 [ 1211.982548] nvmet_rdma: freeing queue 4 [ 1212.001528] nvmet_rdma: freeing queue 5 [ 1212.020271] nvmet_rdma: freeing queue 6 [ 1212.038598] nvmet_rdma: freeing queue 7 [ 1212.048886] nvmet: creating controller 2 for subsystem nvme-subsystem-name for NQN nqn.2014-08.org.nvmexpress:NVMf:uuid:6ed0e109-0b81-4bda-9950-786d67c91b5d. [ 1212.120320] nvmet_rdma: freeing queue 8 [ 1212.860605] nvmet_rdma: freeing queue 9 [ 1214.039350] nvmet_rdma: freeing queue 10 [ 1215.244894] nvmet_rdma: freeing queue 11 [ 1216.235774] nvmet_rdma: failed to connect queue 0 [ 1216.256877] nvmet_rdma: freeing queue 0 [ 1217.356506] nvmet_rdma: freeing queue 17 Best Regards, Yi Zhang ----- Original Message ----- From: "Sagi Grimberg" <sagi@grimberg.me> To: "Yi Zhang" <yizhan at redhat.com>, "Max Gurtovoy" <maxg at mellanox.com>, "Leon Romanovsky" <leon at kernel.org> Cc: linux-rdma at vger.kernel.org, "Christoph Hellwig" <hch at lst.de>, linux-nvme at lists.infradead.org Sent: Friday, March 17, 2017 12:51:16 AM Subject: Re: mlx4_core 0000:07:00.0: swiotlb buffer is full and OOM observed during stress test on reset_controller >>>>> Sagi, >>>>> The release function is placed in global workqueue. I'm not familiar >>>>> with NVMe design and I don't know all the details, but maybe the >>>>> proper way will >>>>> be to create special workqueue with MEM_RECLAIM flag to ensure the >>>>> progress? Leon, the release work makes progress, but it is inherently slower than the establishment work and when we are bombarded with establishments we have no backpressure... > I tried with 4.11.0-rc2, and still can reproduced it with less than 2000 > times. Yi, Can you try the below (untested) patch: I'm not at all convinced this is the way to go because it will slow down all the connect requests, but I'm curious to know if it'll make the issue go away. -- diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index ecc4fe862561..f15fa6e6b640 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -1199,6 +1199,9 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, } queue->port = cm_id->context; + /* Let inflight queue teardown complete */ + flush_scheduled_work(); + ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn); if (ret) goto release_queue; -- Any other good ideas are welcome... _______________________________________________ Linux-nvme mailing list Linux-nvme at lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2017-03-18 11:51 UTC|newest] Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top [not found] <1908657724.31179983.1488539944957.JavaMail.zimbra@redhat.com> [not found] ` <1908657724.31179983.1488539944957.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-03-03 11:55 ` mlx4_core 0000:07:00.0: swiotlb buffer is full and OOM observed during stress test on reset_controller Yi Zhang 2017-03-03 11:55 ` Yi Zhang [not found] ` <2013049462.31187009.1488542111040.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-03-05 8:12 ` Leon Romanovsky 2017-03-05 8:12 ` Leon Romanovsky [not found] ` <20170305081206.GI14379-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-03-08 15:48 ` Christoph Hellwig 2017-03-08 15:48 ` Christoph Hellwig [not found] ` <20170308154815.GB24437-jcswGhMUV9g@public.gmane.org> 2017-03-09 8:42 ` Leon Romanovsky 2017-03-09 8:42 ` Leon Romanovsky 2017-03-09 8:46 ` Leon Romanovsky 2017-03-09 8:46 ` Leon Romanovsky [not found] ` <20170309084641.GY14379-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-03-09 10:33 ` Yi Zhang 2017-03-09 10:33 ` Yi Zhang 2017-03-06 11:23 ` Sagi Grimberg 2017-03-06 11:23 ` Sagi Grimberg [not found] ` <95e045a8-ace0-6a9a-b9a9-555cb2670572-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-03-09 4:20 ` Yi Zhang 2017-03-09 4:20 ` Yi Zhang 2017-03-09 11:42 ` Max Gurtovoy 2017-03-10 8:12 ` Yi Zhang [not found] ` <d21c5571-78fd-7882-b4cc-c24f76f6ff47-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-03-10 16:52 ` Leon Romanovsky 2017-03-10 16:52 ` Leon Romanovsky [not found] ` <20170310165214.GC14379-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-03-12 18:16 ` Max Gurtovoy 2017-03-12 18:16 ` Max Gurtovoy [not found] ` <56e8ccd3-8116-89a1-2f65-eb61a91c5f84-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-03-14 13:35 ` Yi Zhang 2017-03-14 13:35 ` Yi Zhang [not found] ` <860db62d-ae93-d94c-e5fb-88e7b643f737-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-03-14 16:52 ` Max Gurtovoy 2017-03-14 16:52 ` Max Gurtovoy [not found] ` <0a825b18-df06-9a6d-38c9-402f4ee121f7-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-03-15 7:48 ` Yi Zhang 2017-03-15 7:48 ` Yi Zhang [not found] ` <7496c68a-15f3-d8cb-b17f-20f5a59a24d2-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-03-16 16:51 ` Sagi Grimberg 2017-03-16 16:51 ` Sagi Grimberg [not found] ` <31678a43-f76c-a921-e40c-470b0de1a86c-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-03-18 11:51 ` Yi Zhang [this message] 2017-03-18 11:51 ` Yi Zhang [not found] ` <1768681609.3995777.1489837916289.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-03-18 17:50 ` Sagi Grimberg 2017-03-18 17:50 ` Sagi Grimberg 2017-03-19 7:01 ` Leon Romanovsky 2017-03-19 7:01 ` Leon Romanovsky [not found] ` <20170319070115.GP2079-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-18 17:01 ` Yi Zhang 2017-05-18 17:01 ` Yi Zhang [not found] ` <136275928.8307994.1495126919829.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-05-19 16:17 ` Yi Zhang 2017-05-19 16:17 ` Yi Zhang [not found] ` <358169046.8629042.1495210672801.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-06-04 15:49 ` Sagi Grimberg 2017-06-04 15:49 ` Sagi Grimberg [not found] ` <6bf26cbc-71e4-a030-628b-a2ee1d1de94b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-15 8:45 ` Yi Zhang 2017-06-15 8:45 ` Yi Zhang
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1768681609.3995777.1489837916289.JavaMail.zimbra@redhat.com \ --to=yizhan-h+wxahxf7alqt0dzr+alfa@public.gmane.org \ --cc=hch-jcswGhMUV9g@public.gmane.org \ --cc=leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \ --cc=linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=maxg-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org \ --cc=sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.