From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sagi Grimberg Subject: Re: Unexpected issues with 2 NVME initiators using the same target Date: Sun, 2 Jul 2017 12:45:26 +0300 Message-ID: <11aa1a24-9f0b-dbb8-18eb-ad357c7727b2@grimberg.me> References: <20170620074639.GP17846@mtr-leonro.local> <1c706958-992e-b104-6bae-4a6616c0a9f9@grimberg.me> <20170620083309.GQ17846@mtr-leonro.local> <614481c7-22dd-d93b-e97e-52f868727ec3@grimberg.me> <59FF0C04-2BFB-4F66-81BA-A598A9A087FC@oracle.com> <20170620173532.GA827@obsidianresearch.com> <20170620192742.GB827@obsidianresearch.com> <20170620211958.GA5574@obsidianresearch.com> <4f0812f1-0067-4e63-e383-b913ee1f319d@grimberg.me> <28F6F58E-B6F4-4114-8DFF-B72353CE814B@oracle.com> <52ad3547-efcf-f428-6b39-117efda3379f@grimberg.me> <9990B5CB-E0FF-481E-9F34-21EACF0E796E@oracle.com> <7D1C540B-FEA0-4101-8B58-87BCB7DB5492@oracle.com> <66b1b8be-e506-50b8-c01f-fa0e3cea98a4@grimberg.me> <9D8C7BC8-7E18-405A-9017-9DB23A6B5C15@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <9D8C7BC8-7E18-405A-9017-9DB23A6B5C15-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> Content-Language: en-US Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Chuck Lever Cc: Jason Gunthorpe , Leon Romanovsky , Robert LeBlanc , Marta Rybczynska , Max Gurtovoy , Christoph Hellwig , "Gruher, Joseph R" , "shahar.salzman" , Laurence Oberman , "Riches Jr, Robert M" , linux-rdma , linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, Liran Liss , Bart Van Assche List-Id: linux-rdma@vger.kernel.org >> Or wait for the send completion before completing the I/O? > > In the normal case, that works. > > If a POSIX signal occurs (^C, RPC timeout), the RPC exits immediately > and recovers all resources. The Send can still be running at that > point, and it can't be stopped (without transitioning the QP to > error state, I guess). In that case we can't complete the I/O either (or move the QP into error state), we need to defer/sleep on send completion. > The alternative is reference-counting the data structure that has > the ib_cqe and the SGE array. That adds one or more atomic_t > operations per I/O that I'd like to avoid. Why atomics? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: sagi@grimberg.me (Sagi Grimberg) Date: Sun, 2 Jul 2017 12:45:26 +0300 Subject: Unexpected issues with 2 NVME initiators using the same target In-Reply-To: <9D8C7BC8-7E18-405A-9017-9DB23A6B5C15@oracle.com> References: <20170620074639.GP17846@mtr-leonro.local> <1c706958-992e-b104-6bae-4a6616c0a9f9@grimberg.me> <20170620083309.GQ17846@mtr-leonro.local> <614481c7-22dd-d93b-e97e-52f868727ec3@grimberg.me> <59FF0C04-2BFB-4F66-81BA-A598A9A087FC@oracle.com> <20170620173532.GA827@obsidianresearch.com> <20170620192742.GB827@obsidianresearch.com> <20170620211958.GA5574@obsidianresearch.com> <4f0812f1-0067-4e63-e383-b913ee1f319d@grimberg.me> <28F6F58E-B6F4-4114-8DFF-B72353CE814B@oracle.com> <52ad3547-efcf-f428-6b39-117efda3379f@grimberg.me> <9990B5CB-E0FF-481E-9F34-21EACF0E796E@oracle.com> <7D1C540B-FEA0-4101-8B58-87BCB7DB5492@oracle.com> <66b1b8be-e506-50b8-c01f-fa0e3cea98a4@grimberg.me> <9D8C7BC8-7E18-405A-9017-9DB23A6B5C15@oracle.com> Message-ID: <11aa1a24-9f0b-dbb8-18eb-ad357c7727b2@grimberg.me> >> Or wait for the send completion before completing the I/O? > > In the normal case, that works. > > If a POSIX signal occurs (^C, RPC timeout), the RPC exits immediately > and recovers all resources. The Send can still be running at that > point, and it can't be stopped (without transitioning the QP to > error state, I guess). In that case we can't complete the I/O either (or move the QP into error state), we need to defer/sleep on send completion. > The alternative is reference-counting the data structure that has > the ib_cqe and the SGE array. That adds one or more atomic_t > operations per I/O that I'd like to avoid. Why atomics?