From: Gal Pressman <galpress@amazon.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Michal Kalderon <mkalderon@marvell.com>,
Ariel Elior <aelior@marvell.com>,
"dledford@redhat.com" <dledford@redhat.com>,
"bmt@zurich.ibm.com" <bmt@zurich.ibm.com>,
"Leybovich, Yossi" <sleybo@amazon.com>,
"leon@kernel.org" <leon@kernel.org>,
"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH v11 rdma-next 5/7] RDMA/qedr: Use the common mmap API
Date: Wed, 23 Oct 2019 09:40:44 +0300 [thread overview]
Message-ID: <215079fa-03bc-1b5b-dfbe-561f6072de94@amazon.com> (raw)
In-Reply-To: <20191021173349.GH25178@ziepe.ca>
On 21/10/2019 20:33, Jason Gunthorpe wrote:
> On Sun, Oct 20, 2019 at 10:19:34AM +0300, Gal Pressman wrote:
>> On 24/09/2019 12:31, Michal Kalderon wrote:
>>>> From: Pressman, Gal <galpress@amazon.com>
>>>> Sent: Tuesday, September 24, 2019 11:50 AM
>>>>
>>>>
>>>>> On 23 Sep 2019, at 18:22, Michal Kalderon <mkalderon@marvell.com>
>>>> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> From: linux-rdma-owner@vger.kernel.org <linux-rdma-
>>>>>> owner@vger.kernel.org> On Behalf Of Gal Pressman
>>>>>>
>>>>>>> On 19/09/2019 20:55, Jason Gunthorpe wrote:
>>>>>>> Huh. If you recall we did all this work with the XA and the free
>>>>>>> callback because you said qedr was mmaping BAR pages that had some
>>>>>>> HW lifetime associated with them, and the HW resource was not to be
>>>>>>> reallocated until all users were gone.
>>>>>>>
>>>>>>> I think it would be a better example of this API if you pulled the
>>>>>>>
>>>>>>> dev->ops->rdma_remove_user(dev->rdma_ctx, ctx->dpi);
>>>>>>>
>>>>>>> Into qedr_mmap_free().
>>>>>>>
>>>>>>> Then the rdma_user_mmap_entry_remove() will call it naturally as it
>>>>>>> does entry_put() and if we are destroying the ucontext we already
>>>>>>> know the mmaps are destroyed.
>>>>>>>
>>>>>>> Maybe the same basic comment for EFA, not sure. Gal?
>>>>>>
>>>>>> That's what EFA already does in this series, no?
>>>>>> We no longer remove entries on dealloc_ucontext, only when the entry
>>>>>> is freed.
>>>>>
>>>>> Actually, I think most of the discussions you had on the topic were
>>>>> with Gal, but Some apply to qedr as well, however, for qedr, the only
>>>>> hw resource we allocate (bar) is on alloc_ucontext , therefore we were
>>>>> safe to free it on dealloc_ucontext as all mappings were already
>>>>> zapped. Making the mmap_free a bit redundant for qedr except for the
>>>> need to free the entry.
>>>>>
>>>>> For EFA, it seemed the only operation delayed was freeing memory - I
>>>>> didn't see hw resources being freed... Gal?
>>>>
>>>> What do you mean by hw resources being freed? The BAR mappings are
>>>> under the device’s control and are associated to the lifetime of the UAR.
>>> The bar offset you get is from the device -> don't you release it back to the device
>>> So it can be allocated to a different application ?
>>> In efa_com_create_qp -> you get offsets from the device that you use for mapping
>>> The bar -> are these unique for every call ? are they released during destroy_qp ?
>>> Before this patch series mmap_entries_remove_free only freed the DMA pages, but
>>> Following this thread, it seemed the initial intention was that only the hw resources would
>>> Be delayed as the DMA pages are ref counted anyway. I didn't see any delay to returning
>>> The bar offsets to the device. Thanks.
>> The BAR pages are being "freed" back to the device once the UAR is freed.
>> These pages' lifetime is under the control of the device so there's nothing the
>> driver needs to do, just make sure no one else is using them.
>
> What frees the UAR?
>
> In the mlx drivers this was done during destruction of the ucontext,
> but with this new mmap stuff it could be moved to the mmap_free..
Dealloc UAR is currently being called during dealloc_ucontext.
The mmap_free callback is per entry, how can dealloc_uar be moved there?
next prev parent reply other threads:[~2019-10-23 6:41 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-05 10:01 [PATCH v11 rdma-next 0/7] RDMA/qedr: Use the doorbell overflow recovery mechanism for RDMA Michal Kalderon
2019-09-05 10:01 ` [PATCH v11 rdma-next 1/7] RDMA/core: Move core content from ib_uverbs to ib_core Michal Kalderon
2019-09-05 10:01 ` [PATCH v11 rdma-next 2/7] RDMA/core: Create mmap database and cookie helper functions Michal Kalderon
2019-09-19 17:14 ` Jason Gunthorpe
2019-09-19 17:48 ` Jason Gunthorpe
2019-09-19 18:07 ` Jason Gunthorpe
2019-09-23 9:36 ` [EXT] " Michal Kalderon
2019-09-23 13:30 ` Jason Gunthorpe
2019-09-24 8:31 ` Michal Kalderon
2019-09-24 12:37 ` Jason Gunthorpe
2019-09-05 10:01 ` [PATCH v11 rdma-next 3/7] RDMA/efa: Use the common mmap_xa helpers Michal Kalderon
2019-09-19 17:37 ` Jason Gunthorpe
2019-09-23 9:15 ` Michal Kalderon
2019-09-23 13:28 ` Jason Gunthorpe
2019-09-05 10:01 ` [PATCH v11 rdma-next 4/7] RDMA/siw: " Michal Kalderon
2019-09-05 10:01 ` [PATCH v11 rdma-next 5/7] RDMA/qedr: Use the common mmap API Michal Kalderon
2019-09-19 17:55 ` Jason Gunthorpe
2019-09-20 13:39 ` Gal Pressman
2019-09-23 9:21 ` Michal Kalderon
2019-09-23 13:29 ` Jason Gunthorpe
2019-09-24 8:25 ` [EXT] " Michal Kalderon
2019-09-24 8:49 ` Pressman, Gal
2019-09-24 9:31 ` Michal Kalderon
2019-10-20 7:19 ` Gal Pressman
2019-10-21 17:33 ` Jason Gunthorpe
2019-10-23 6:40 ` Gal Pressman [this message]
2019-10-23 14:41 ` Jason Gunthorpe
2019-10-24 8:06 ` Gal Pressman
2019-09-05 10:01 ` [PATCH v11 rdma-next 6/7] RDMA/qedr: Add doorbell overflow recovery support Michal Kalderon
2019-09-19 18:02 ` Jason Gunthorpe
2019-09-20 13:30 ` Gal Pressman
2019-09-20 13:38 ` Jason Gunthorpe
2019-09-20 14:00 ` Gal Pressman
2019-09-23 9:37 ` Michal Kalderon
2019-09-25 19:16 ` [EXT] " Michal Kalderon
2019-09-25 19:21 ` Jason Gunthorpe
2019-09-25 19:37 ` Michal Kalderon
2019-09-26 19:10 ` Jason Gunthorpe
2019-09-23 9:30 ` Michal Kalderon
2019-09-23 13:26 ` Jason Gunthorpe
2019-09-05 10:01 ` [PATCH v11 rdma-next 7/7] RDMA/qedr: Add iWARP doorbell " Michal Kalderon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=215079fa-03bc-1b5b-dfbe-561f6072de94@amazon.com \
--to=galpress@amazon.com \
--cc=aelior@marvell.com \
--cc=bmt@zurich.ibm.com \
--cc=dledford@redhat.com \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mkalderon@marvell.com \
--cc=sleybo@amazon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).