Re: [PATCH for-next v2 2/2] RDMA/erdma: Support non-4K page size in doorbell allocation

[Leon Romanovsky <leon@kernel.org>]

On Tue, Mar 14, 2023 at 07:50:19PM +0800, Cheng Xu wrote:
> 
> 
> On 3/14/23 6:23 PM, Leon Romanovsky wrote:
> > On Tue, Mar 07, 2023 at 06:29:24PM +0800, Cheng Xu wrote:
> >> Doorbell resources are exposed to userspace by mmap. The size unit of mmap
> >> is PAGE_SIZE, previous implementation can not work correctly if PAGE_SIZE
> >> is not 4K. We support non-4K page size in this commit.
> > 
> > Why do you need this information in rdma-core?
> > Can you use sysconf(_SC_PAGESIZE) there to understand the page size like
> > other providers?
> > 
> 
> I don't expose PAGE_SIZE to userspace in this patchset, but the *offset* in
> PAGE of the DBs:
> 
> diff --git a/include/uapi/rdma/erdma-abi.h b/include/uapi/rdma/erdma-abi.h
> index b7a0222f978f..57f8942a3c56 100644
> --- a/include/uapi/rdma/erdma-abi.h
> +++ b/include/uapi/rdma/erdma-abi.h
> @@ -40,10 +40,13 @@ struct erdma_uresp_alloc_ctx {
>  	__u32 dev_id;
>  	__u32 pad;
>  	__u32 sdb_type;
> -	__u32 sdb_offset;
> +	__u32 sdb_entid;
>  	__aligned_u64 sdb;
>  	__aligned_u64 rdb;
>  	__aligned_u64 cdb;
> +	__u32 sdb_off;
> +	__u32 rdb_off;
> +	__u32 cdb_off;
>  };
> 
> Our doorbell space is aligned to 4096, this works fine when PAGE_SIZE is
> also 4096, and the doorbell space starts from the mapped page. When
> PAGE_SIZE is not 4096, the doorbell space may starts from the middle of
> the mapped page.
> 
> For example, our SQ doorbell starts from the offset 4096 in PCIe bar 0.
> When we map the first SQ doorbell to userspace when PAGE_SIZE is 64K,
> the doorbell space starts from the offset 4096 in mmap returned address.
> 
> So the userspace needs to know the doorbell space offset in mmaped page.

And can't you preserve same alignment in the kernel for doorbells for every page size?
Just always start from 0.

> 
> Thanks,
> Cheng Xu
> 
> 
> 
> > Thanks