From: Sagi Grimberg <sagi-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> To: "Gruher, Joseph R" <joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>, "shahar.salzman" <shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, Laurence Oberman <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Riches Jr, Robert M" <robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> Cc: "linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>, "linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org" <linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org> Subject: Re: Unexpected issues with 2 NVME initiators using the same target Date: Mon, 27 Feb 2017 22:33:16 +0200 [thread overview] Message-ID: <3eb5814f-14cb-2b94-adf8-335d4b2eb7e9@grimberg.me> (raw) In-Reply-To: <DE927C68B458BE418D582EC97927A92854655137-8oqHQFITsIFcIJlls4ac1rfspsVTdybXVpNB7YpNyf8@public.gmane.org> Hey Joseph, > In our lab we are dealing with an issue which has some of the same symptoms. Wanted to add to the thread in case it is useful here. We have a target system with 16 Intel P3520 disks and a Mellanox CX4 50Gb NIC directly connected (no switch) to a single initiator system with a matching Mellanox CX4 50Gb NIC. We are running Ubuntu 16.10 with 4.10-RC8 mainline kernel. All drivers are kernel default drivers. I've attached our nvmetcli json, and FIO workload, and dmesg from both systems. > > We are able to provoke this problem with a variety of workloads but a high bandwidth read operation seems to cause it the most reliably, harder to produce with smaller block sizes. For some reason the problem seems produced when we stop and restart IO - I can run the FIO workload on the initiator system for 1-2 hours without any new events in dmesg, pushing about 5500MB/sec the whole time, then kill it and wait 10 seconds and restart it, and the errors and reconnect events happen reliably at that point. Working to characterize further this week and also to see if we can produce on a smaller configuration. Happy to provide any additional details that would be useful or try any fixes! > > On the initiator we see events like this: > > [51390.065641] mlx5_0:dump_cqe:262:(pid 0): dump error cqe > [51390.065644] 00000000 00000000 00000000 00000000 > [51390.065645] 00000000 00000000 00000000 00000000 > [51390.065646] 00000000 00000000 00000000 00000000 > [51390.065648] 00000000 08007806 250003ab 02b9dcd2 > [51390.065666] nvme nvme3: MEMREG for CQE 0xffff9fc845039410 failed with status memory management operation error (6) > [51390.079156] nvme nvme3: reconnecting in 10 seconds > [51400.432782] nvme nvme3: Successfully reconnected Seems to me this is a CX4 FW issue. Mellanox can elaborate on these vendor specific syndromes on this output. > On the target we see events like this: > > [51370.394694] mlx5_0:dump_cqe:262:(pid 6623): dump error cqe > [51370.394696] 00000000 00000000 00000000 00000000 > [51370.394697] 00000000 00000000 00000000 00000000 > [51370.394699] 00000000 00000000 00000000 00000000 > [51370.394701] 00000000 00008813 080003ea 00c3b1d2 If the host is failing on memory mapping while the target is initiating rdma access it makes sense that it will see errors. > > Sometimes, but less frequently, we also will see events on the target like this as part of the problem: > > [21322.678571] nvmet: ctrl 1 fatal error occurred! Again, also makes sense because for nvmet this is a fatal error and we need to teardown the controller. You can try out this patch to see if it makes the memreg issues to go away: -- diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index ad8a2638e339..0f9a12570262 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -3893,7 +3893,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, goto out; case IB_WR_LOCAL_INV: - next_fence = MLX5_FENCE_MODE_INITIATOR_SMALL; + next_fence = MLX5_FENCE_MODE_STRONG_ORDERING; qp->sq.wr_data[idx] = IB_WR_LOCAL_INV; ctrl->imm = cpu_to_be32(wr->ex.invalidate_rkey); set_linv_wr(qp, &seg, &size); @@ -3901,7 +3901,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, break; case IB_WR_REG_MR: - next_fence = MLX5_FENCE_MODE_INITIATOR_SMALL; + next_fence = MLX5_FENCE_MODE_STRONG_ORDERING; qp->sq.wr_data[idx] = IB_WR_REG_MR; ctrl->imm = cpu_to_be32(reg_wr(wr)->key); err = set_reg_wr(qp, reg_wr(wr), &seg, &size); -- Note that this will have a big performance (negative) impact on small read workloads. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: sagi@grimberg.me (Sagi Grimberg) Subject: Unexpected issues with 2 NVME initiators using the same target Date: Mon, 27 Feb 2017 22:33:16 +0200 [thread overview] Message-ID: <3eb5814f-14cb-2b94-adf8-335d4b2eb7e9@grimberg.me> (raw) In-Reply-To: <DE927C68B458BE418D582EC97927A92854655137@ORSMSX112.amr.corp.intel.com> Hey Joseph, > In our lab we are dealing with an issue which has some of the same symptoms. Wanted to add to the thread in case it is useful here. We have a target system with 16 Intel P3520 disks and a Mellanox CX4 50Gb NIC directly connected (no switch) to a single initiator system with a matching Mellanox CX4 50Gb NIC. We are running Ubuntu 16.10 with 4.10-RC8 mainline kernel. All drivers are kernel default drivers. I've attached our nvmetcli json, and FIO workload, and dmesg from both systems. > > We are able to provoke this problem with a variety of workloads but a high bandwidth read operation seems to cause it the most reliably, harder to produce with smaller block sizes. For some reason the problem seems produced when we stop and restart IO - I can run the FIO workload on the initiator system for 1-2 hours without any new events in dmesg, pushing about 5500MB/sec the whole time, then kill it and wait 10 seconds and restart it, and the errors and reconnect events happen reliably at that point. Working to characterize further this week and also to see if we can produce on a smaller configuration. Happy to provide any additional details that would be useful or try any fixes! > > On the initiator we see events like this: > > [51390.065641] mlx5_0:dump_cqe:262:(pid 0): dump error cqe > [51390.065644] 00000000 00000000 00000000 00000000 > [51390.065645] 00000000 00000000 00000000 00000000 > [51390.065646] 00000000 00000000 00000000 00000000 > [51390.065648] 00000000 08007806 250003ab 02b9dcd2 > [51390.065666] nvme nvme3: MEMREG for CQE 0xffff9fc845039410 failed with status memory management operation error (6) > [51390.079156] nvme nvme3: reconnecting in 10 seconds > [51400.432782] nvme nvme3: Successfully reconnected Seems to me this is a CX4 FW issue. Mellanox can elaborate on these vendor specific syndromes on this output. > On the target we see events like this: > > [51370.394694] mlx5_0:dump_cqe:262:(pid 6623): dump error cqe > [51370.394696] 00000000 00000000 00000000 00000000 > [51370.394697] 00000000 00000000 00000000 00000000 > [51370.394699] 00000000 00000000 00000000 00000000 > [51370.394701] 00000000 00008813 080003ea 00c3b1d2 If the host is failing on memory mapping while the target is initiating rdma access it makes sense that it will see errors. > > Sometimes, but less frequently, we also will see events on the target like this as part of the problem: > > [21322.678571] nvmet: ctrl 1 fatal error occurred! Again, also makes sense because for nvmet this is a fatal error and we need to teardown the controller. You can try out this patch to see if it makes the memreg issues to go away: -- diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index ad8a2638e339..0f9a12570262 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -3893,7 +3893,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, goto out; case IB_WR_LOCAL_INV: - next_fence = MLX5_FENCE_MODE_INITIATOR_SMALL; + next_fence = MLX5_FENCE_MODE_STRONG_ORDERING; qp->sq.wr_data[idx] = IB_WR_LOCAL_INV; ctrl->imm = cpu_to_be32(wr->ex.invalidate_rkey); set_linv_wr(qp, &seg, &size); @@ -3901,7 +3901,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, break; case IB_WR_REG_MR: - next_fence = MLX5_FENCE_MODE_INITIATOR_SMALL; + next_fence = MLX5_FENCE_MODE_STRONG_ORDERING; qp->sq.wr_data[idx] = IB_WR_REG_MR; ctrl->imm = cpu_to_be32(reg_wr(wr)->key); err = set_reg_wr(qp, reg_wr(wr), &seg, &size); -- Note that this will have a big performance (negative) impact on small read workloads.
next prev parent reply other threads:[~2017-02-27 20:33 UTC|newest] Thread overview: 171+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-02-21 19:38 Unexpected issues with 2 NVME initiators using the same target shahar.salzman [not found] ` <08131a05-1f56-ef61-990a-7fff04eea095-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2017-02-21 22:50 ` Sagi Grimberg 2017-02-21 22:50 ` Sagi Grimberg [not found] ` <de1a559a-bf24-0d73-5fc7-148d6cd4d4e0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-02-22 16:52 ` Laurence Oberman 2017-02-22 16:52 ` Laurence Oberman [not found] ` <1848296658.37025722.1487782361271.JavaMail.zimbra-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-02-22 19:39 ` Sagi Grimberg 2017-02-22 19:39 ` Sagi Grimberg 2017-02-26 8:03 ` shahar.salzman 2017-02-26 8:03 ` shahar.salzman 2017-02-26 17:58 ` Gruher, Joseph R [not found] ` <DE927C68B458BE418D582EC97927A92854655137-8oqHQFITsIFcIJlls4ac1rfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-02-27 20:33 ` Sagi Grimberg [this message] 2017-02-27 20:33 ` Sagi Grimberg 2017-02-27 20:57 ` Gruher, Joseph R 2017-03-05 18:23 ` Leon Romanovsky 2017-03-06 0:07 ` Max Gurtovoy [not found] ` <26912d0c-578f-26e9-490d-94fc95bdf259-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-03-06 11:28 ` Sagi Grimberg 2017-03-06 11:28 ` Sagi Grimberg 2017-03-07 9:27 ` Max Gurtovoy [not found] ` <fbd647dd-3a16-8155-107d-f98e8326cc63-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-03-07 13:41 ` Sagi Grimberg 2017-03-07 13:41 ` Sagi Grimberg [not found] ` <c5a6f55b-a633-53ca-4378-7a7eaf7f77bd-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-03-09 12:18 ` shahar.salzman 2017-03-09 12:18 ` shahar.salzman 2017-03-12 12:33 ` Vladimir Neyelov [not found] ` <AM4PR0501MB278621363209E177A738D75FCB220-dp/nxUn679hhbxXPg6FtWcDSnupUy6xnnBOFsp37pqbUKgpGm//BTAC/G2K4zDHf@public.gmane.org> 2017-03-13 9:43 ` Sagi Grimberg 2017-03-13 9:43 ` Sagi Grimberg 2017-03-14 8:55 ` Max Gurtovoy 2017-03-14 19:57 ` Gruher, Joseph R 2017-03-14 23:42 ` Gruher, Joseph R 2017-03-16 0:03 ` Gruher, Joseph R 2017-03-17 18:37 ` Gruher, Joseph R 2017-03-17 19:49 ` Max Gurtovoy [not found] ` <DE927C68B458BE418D582EC97927A928550391C2@ORSMSX113.amr.corp.intel.com> 2017-03-24 18:30 ` Gruher, Joseph R 2017-03-27 14:17 ` Max Gurtovoy 2017-03-27 15:39 ` Gruher, Joseph R 2017-03-28 8:38 ` Max Gurtovoy 2017-03-28 10:21 ` shahar.salzman [not found] ` <DE927C68B458BE418D582EC97927A928550419FA-8oqHQFITsIFQxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-03-28 11:34 ` Sagi Grimberg 2017-03-28 11:34 ` Sagi Grimberg [not found] ` <809f87ab-b787-9d40-5840-07500d12e81a-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-10 11:40 ` Marta Rybczynska 2017-04-10 11:40 ` Marta Rybczynska 2017-04-10 14:09 ` Max Gurtovoy [not found] ` <33e2cc35-f147-d4a4-9a42-8f1245e35842-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-11 12:47 ` Marta Rybczynska 2017-04-11 12:47 ` Marta Rybczynska 2017-04-20 10:18 ` Sagi Grimberg 2017-04-20 10:18 ` Sagi Grimberg 2017-04-26 11:56 ` Max Gurtovoy [not found] ` <af9044c2-fb7c-e8b3-d8fc-4874cfd1bb67-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-04-26 14:45 ` Sagi Grimberg 2017-04-26 14:45 ` Sagi Grimberg 2017-05-12 19:20 ` Gruher, Joseph R [not found] ` <DE927C68B458BE418D582EC97927A92855088C6F-8oqHQFITsIFQxe9IK+vIArfspsVTdybXVpNB7YpNyf8@public.gmane.org> 2017-05-15 12:00 ` Sagi Grimberg 2017-05-15 12:00 ` Sagi Grimberg [not found] ` <82dd5b24-5657-ae5e-8a33-646fddd8b75b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-05-15 13:31 ` Leon Romanovsky 2017-05-15 13:31 ` Leon Romanovsky [not found] ` <20170515133122.GG3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-15 13:43 ` Sagi Grimberg 2017-05-15 13:43 ` Sagi Grimberg [not found] ` <9465cd0c-83db-b058-7615-5626ef60dbb0-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-05-15 14:36 ` Leon Romanovsky 2017-05-15 14:36 ` Leon Romanovsky [not found] ` <20170515143632.GH3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-15 14:59 ` Christoph Hellwig 2017-05-15 14:59 ` Christoph Hellwig [not found] ` <20170515145952.GA7871-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> 2017-05-15 17:05 ` Leon Romanovsky 2017-05-15 17:05 ` Leon Romanovsky [not found] ` <20170515170506.GK3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-05-17 12:56 ` Marta Rybczynska 2017-05-17 12:56 ` Marta Rybczynska [not found] ` <779753075.36035391.1495025796237.JavaMail.zimbra-FNhOzJFKnXGHXe+LvDLADg@public.gmane.org> 2017-05-18 13:34 ` Leon Romanovsky 2017-05-18 13:34 ` Leon Romanovsky [not found] ` <20170518133439.GD3616-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-19 17:21 ` Robert LeBlanc 2017-06-19 17:21 ` Robert LeBlanc [not found] ` <CAANLjFrCLpX3nb3q7LpFPpLJKciU+1Hvmt_hxyTovQJM2-zQmg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-20 6:39 ` Sagi Grimberg 2017-06-20 6:39 ` Sagi Grimberg [not found] ` <6073e553-e8c2-6d14-ba5d-c2bd5aff15eb-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 7:46 ` Leon Romanovsky 2017-06-20 7:46 ` Leon Romanovsky [not found] ` <20170620074639.GP17846-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-20 7:58 ` Sagi Grimberg 2017-06-20 7:58 ` Sagi Grimberg [not found] ` <1c706958-992e-b104-6bae-4a6616c0a9f9-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 8:33 ` Leon Romanovsky 2017-06-20 8:33 ` Leon Romanovsky [not found] ` <20170620083309.GQ17846-U/DQcQFIOTAAJjI8aNfphQ@public.gmane.org> 2017-06-20 9:33 ` Sagi Grimberg 2017-06-20 9:33 ` Sagi Grimberg 2017-06-20 10:31 ` Max Gurtovoy [not found] ` <78b2c1db-6ece-0274-c4c9-5ee1f7c88469-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> 2017-06-20 22:58 ` Robert LeBlanc 2017-06-20 22:58 ` Robert LeBlanc 2017-06-27 7:16 ` Sagi Grimberg 2017-06-27 7:16 ` Sagi Grimberg [not found] ` <bd0b986f-9bed-3dfa-7454-0661559a527b-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 12:02 ` Sagi Grimberg 2017-06-20 12:02 ` Sagi Grimberg 2017-06-20 13:28 ` Max Gurtovoy [not found] ` <614481c7-22dd-d93b-e97e-52f868727ec3-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 17:01 ` Chuck Lever 2017-06-20 17:01 ` Chuck Lever [not found] ` <59FF0C04-2BFB-4F66-81BA-A598A9A087FC-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 17:12 ` Sagi Grimberg 2017-06-20 17:12 ` Sagi Grimberg 2017-06-20 17:35 ` Jason Gunthorpe 2017-06-20 17:35 ` Jason Gunthorpe [not found] ` <20170620173532.GA827-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-20 18:17 ` Chuck Lever 2017-06-20 18:17 ` Chuck Lever [not found] ` <D3DC49A2-FFC9-4F62-8876-3E6AD5167DE5-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 19:27 ` Jason Gunthorpe 2017-06-20 19:27 ` Jason Gunthorpe [not found] ` <20170620192742.GB827-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-20 20:56 ` Chuck Lever 2017-06-20 20:56 ` Chuck Lever [not found] ` <C14B071E-F1B2-466A-82CF-4E20BFAD9DC1-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-20 21:19 ` Jason Gunthorpe 2017-06-20 21:19 ` Jason Gunthorpe [not found] ` <20170620211958.GA5574-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-27 7:37 ` Sagi Grimberg 2017-06-27 7:37 ` Sagi Grimberg [not found] ` <4f0812f1-0067-4e63-e383-b913ee1f319d-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-27 14:42 ` Chuck Lever 2017-06-27 14:42 ` Chuck Lever [not found] ` <28F6F58E-B6F4-4114-8DFF-B72353CE814B-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-27 16:07 ` Sagi Grimberg 2017-06-27 16:07 ` Sagi Grimberg [not found] ` <52ad3547-efcf-f428-6b39-117efda3379f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-27 16:28 ` Jason Gunthorpe 2017-06-27 16:28 ` Jason Gunthorpe [not found] ` <20170627162800.GA22592-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-06-28 7:03 ` Sagi Grimberg 2017-06-28 7:03 ` Sagi Grimberg 2017-06-27 16:28 ` Chuck Lever 2017-06-27 16:28 ` Chuck Lever [not found] ` <9990B5CB-E0FF-481E-9F34-21EACF0E796E-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-28 7:08 ` Sagi Grimberg 2017-06-28 7:08 ` Sagi Grimberg [not found] ` <f1f1a68c-90db-e6bf-e35e-55c4b469c339-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-28 16:11 ` Chuck Lever 2017-06-28 16:11 ` Chuck Lever [not found] ` <7D1C540B-FEA0-4101-8B58-87BCB7DB5492-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-06-29 5:35 ` Sagi Grimberg 2017-06-29 5:35 ` Sagi Grimberg [not found] ` <66b1b8be-e506-50b8-c01f-fa0e3cea98a4-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-29 14:55 ` Chuck Lever 2017-06-29 14:55 ` Chuck Lever [not found] ` <9D8C7BC8-7E18-405A-9017-9DB23A6B5C15-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-02 9:45 ` Sagi Grimberg 2017-07-02 9:45 ` Sagi Grimberg [not found] ` <11aa1a24-9f0b-dbb8-18eb-ad357c7727b2-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-07-02 18:17 ` Chuck Lever 2017-07-02 18:17 ` Chuck Lever [not found] ` <9E30754F-464A-4B62-ADE7-F6B2F6D95763-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-09 16:47 ` Jason Gunthorpe 2017-07-09 16:47 ` Jason Gunthorpe [not found] ` <20170709164755.GB3058-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 19:03 ` Chuck Lever 2017-07-10 19:03 ` Chuck Lever [not found] ` <A7C8C159-E916-4060-9FD1-8726D816B3C0-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 20:05 ` Jason Gunthorpe 2017-07-10 20:05 ` Jason Gunthorpe [not found] ` <20170710200522.GA19293-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 20:51 ` Chuck Lever 2017-07-10 20:51 ` Chuck Lever [not found] ` <C04891DF-5B3B-4156-9E04-9E18B238864A-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 21:14 ` Jason Gunthorpe 2017-07-10 21:14 ` Jason Gunthorpe 2017-07-10 21:24 ` Jason Gunthorpe 2017-07-10 21:24 ` Jason Gunthorpe [not found] ` <20170710212430.GA21721-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 21:29 ` Chuck Lever 2017-07-10 21:29 ` Chuck Lever [not found] ` <C142385D-3A54-44C9-BA3D-0AABBC5E9E7B-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 21:32 ` Jason Gunthorpe 2017-07-10 21:32 ` Jason Gunthorpe [not found] ` <20170710213251.GA21908-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-10 22:04 ` Chuck Lever 2017-07-10 22:04 ` Chuck Lever [not found] ` <A342254B-1ECB-4644-8D68-328A52940C52-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-10 22:09 ` Jason Gunthorpe 2017-07-10 22:09 ` Jason Gunthorpe [not found] ` <20170710220905.GA22589-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> 2017-07-11 3:57 ` Chuck Lever 2017-07-11 3:57 ` Chuck Lever [not found] ` <718A099D-3597-4262-9A33-0BA7EFE5461F-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2017-07-11 13:23 ` Tom Talpey 2017-07-11 13:23 ` Tom Talpey [not found] ` <8dd77b19-3846-96ea-f50f-22182989e941-CLs1Zie5N5HQT0dZR+AlfA@public.gmane.org> 2017-07-11 14:55 ` Chuck Lever 2017-07-11 14:55 ` Chuck Lever 2017-06-27 18:08 ` Bart Van Assche 2017-06-27 18:08 ` Bart Van Assche [not found] ` <1498586933.14963.1.camel-Sjgp3cTcYWE@public.gmane.org> 2017-06-27 18:14 ` Jason Gunthorpe 2017-06-27 18:14 ` Jason Gunthorpe 2017-06-28 7:16 ` Sagi Grimberg 2017-06-28 7:16 ` Sagi Grimberg [not found] ` <47bbf598-6c82-610f-dc1d-706a1d869b8d-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-28 9:43 ` Bart Van Assche 2017-06-28 9:43 ` Bart Van Assche 2017-06-20 17:08 ` Robert LeBlanc 2017-06-20 17:08 ` Robert LeBlanc [not found] ` <CAANLjFrpHzapvqgBajUu7QpgNNPvNjThMZrsXGcKt58E+6siMA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-20 17:19 ` Sagi Grimberg 2017-06-20 17:19 ` Sagi Grimberg [not found] ` <3f3830eb-68c5-f862-58c7-7021e6462f6f-NQWnxTmZq1alnMjI0IkVqw@public.gmane.org> 2017-06-20 17:28 ` Robert LeBlanc 2017-06-20 17:28 ` Robert LeBlanc [not found] ` <CAANLjFo_1E5nPkXMmG0VxtXpQ2m6=WNd0OtvG4rd-__1TD0-VQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-06-27 7:22 ` Sagi Grimberg 2017-06-27 7:22 ` Sagi Grimberg 2017-06-20 14:43 ` Robert LeBlanc 2017-06-20 14:43 ` Robert LeBlanc 2017-06-20 14:41 ` Robert LeBlanc 2017-06-20 14:41 ` Robert LeBlanc [not found] ` <1554c1d1-6bf4-9ca2-12d4-a0125d8c5715-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2017-02-27 20:13 ` Sagi Grimberg 2017-02-27 20:13 ` Sagi Grimberg
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=3eb5814f-14cb-2b94-adf8-335d4b2eb7e9@grimberg.me \ --to=sagi-nqwnxtmzq1alnmji0ikvqw@public.gmane.org \ --cc=joseph.r.gruher-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \ --cc=linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org \ --cc=linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org \ --cc=robert.m.riches.jr-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \ --cc=shahar.salzman-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.