Re: [PATCH rdma-next 1/2] IB/core: Add scatter end padding flags for WQ and QP

[Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 4749 bytes --]

On Wed, Oct 25, 2017 at 01:58:59PM -0400, Doug Ledford wrote:
> On Tue, 2017-10-17 at 18:18 +0300, Leon Romanovsky wrote:
> > From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> >
> > There are root complexes that are able to optimize their
> > performance when incoming data is multiple full cache lines.
> >
> > Scatter end padding is the device's ability to pad the ending of
> > incoming packets (scatter)
>
> I hate this naming.  I'm sure people inside of Mellanox have gotten
> used to it, but this feature really has no bearing on scatter/gather at
> all.  This is merely final write padding.  You might have a
> scatter/gather list, you might have a single buffer.  Either way, the
> PCI root complex couldn't care less about scatter/gather or not, it's
> all a byte stream to it.  I would be much happier with a name that
> reflected what this really does.

Scatter has more broad meaning than "scatter/gather list". The name
here came from statistical meaning of "scatter", I'll try to come into
something more descriptive.

>
>
> >  to full cache line such that the last
> > upstream write generated by an incoming packet will be a full cache
> > line.
> >
> > Add a relevant entry to ib_device_cap_flags to report scatter end
> > padding capability of an RDMA device.
> >
> > Add the QP and WQ create flags with an entry for scatter end padding:
> >  * A QP/WQ created with a scatter end padding flag will cause
> >    HW to pad the last upstream write generated by a packet to cache
> > line.
> >
> > User should consider several factors before activating this feature:
> > - In case of high CPU memory load (which may cause PCI back pressure
> > in
> >   turn), if a large percent of the writes are partial cache line,
> > this
> >   feature should be checked as an optional solution.
> > - This feature might reduce performance if most packets are between
> > one
> >   and two cache lines and PCIe throughput has reached its maximum
> >   capacity. E.g. 65B packet from the network port will lead to 128B
> >   write on PCIe, which may cause traffic on PCIe to reach high
> >   throughput.
> >
> > Signed-off-by: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Reviewed-by: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> > ---
> >  drivers/infiniband/core/uverbs_cmd.c | 3 ++-
> >  include/rdma/ib_verbs.h              | 4 ++++
> >  2 files changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/infiniband/core/uverbs_cmd.c
> > b/drivers/infiniband/core/uverbs_cmd.c
> > index d31e4bc58e9a..ab29a0327831 100644
> > --- a/drivers/infiniband/core/uverbs_cmd.c
> > +++ b/drivers/infiniband/core/uverbs_cmd.c
> > @@ -1491,7 +1491,8 @@ static int create_qp(struct ib_uverbs_file
> > *file,
> >  				IB_QP_CREATE_MANAGED_RECV |
> >  				IB_QP_CREATE_SCATTER_FCS |
> >  				IB_QP_CREATE_CVLAN_STRIPPING |
> > -				IB_QP_CREATE_SOURCE_QPN)) {
> > +				IB_QP_CREATE_SOURCE_QPN |
> > +				IB_QP_CREATE_SCATTER_END_PADDING)) {
>
> Maybe IB_QP_CREATE_PCI_WRITE_PAD?
>
> >  		ret = -EINVAL;
> >  		goto err_put;
> >  	}
> > diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> > index 9810e4568635..4c0a539cd2a2 100644
> > --- a/include/rdma/ib_verbs.h
> > +++ b/include/rdma/ib_verbs.h
> > @@ -229,6 +229,8 @@ enum ib_device_cap_flags {
> >  	/* Deprecated. Please use IB_RAW_PACKET_CAP_SCATTER_FCS. */
> >  	IB_DEVICE_RAW_SCATTER_FCS		= (1ULL << 34),
> >  	IB_DEVICE_RDMA_NETDEV_OPA_VNIC		= (1ULL <<
> > 35),
> > +	/* The device supports padding incoming writes to cacheline.
> > */
> /* The device supports padding the final write of a PCI write
> transaction to a cacheline boundry so that the PCI root complex can
> optimize its memory accesses */
> > +	IB_DEVICE_SCATTER_END_PADDING		= (1ULL << 36),
> >  };
> >
> >  enum ib_signature_prot_cap {
> > @@ -1098,6 +1100,7 @@ enum ib_qp_create_flags {
> >  	IB_QP_CREATE_SCATTER_FCS		= 1 << 8,
> >  	IB_QP_CREATE_CVLAN_STRIPPING		= 1 << 9,
> >  	IB_QP_CREATE_SOURCE_QPN			= 1 << 10,
> > +	IB_QP_CREATE_SCATTER_END_PADDING	= 1 << 11,
> >  	/* reserve bits 26-31 for low level drivers' internal use */
> >  	IB_QP_CREATE_RESERVED_START		= 1 << 26,
> >  	IB_QP_CREATE_RESERVED_END		= 1 << 31,
> > @@ -1621,6 +1624,7 @@ enum ib_wq_flags {
> >  	IB_WQ_FLAGS_CVLAN_STRIPPING	= 1 << 0,
> >  	IB_WQ_FLAGS_SCATTER_FCS		= 1 << 1,
> >  	IB_WQ_FLAGS_DELAY_DROP		= 1 << 2,
> > +	IB_WQ_FLAGS_SCATTER_END_PADDING = 1 << 3,
> >  };
> >
> >  struct ib_wq_init_attr {
> --
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>     GPG KeyID: B826A3330E572FDD
>     Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]