Re: [PATCH rdma-next v1 10/15] RDMA/cm: Use an attribute_group on the ib_port_attribute intead of kobj's

From: Leon Romanovsky <leon@kernel.org>
To: Greg KH <gregkh@linuxfoundation.org>
Cc: Doug Ledford <dledford@redhat.com>,
	Jason Gunthorpe <jgg@nvidia.com>,
	Kees Cook <keescook@chromium.org>,
	Nathan Chancellor <nathan@kernel.org>,
	Adit Ranadive <aditr@vmware.com>,
	Ariel Elior <aelior@marvell.com>,
	Christian Benvenuti <benve@cisco.com>,
	clang-built-linux@googlegroups.com,
	Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>,
	Devesh Sharma <devesh.sharma@broadcom.com>,
	Gal Pressman <galpress@amazon.com>,
	linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org,
	Michal Kalderon <mkalderon@marvell.com>,
	Mike Marciniszyn <mike.marciniszyn@cornelisnetworks.com>,
	Mustafa Ismail <mustafa.ismail@intel.com>,
	Naresh Kumar PBS <nareshkumar.pbs@broadcom.com>,
	Nelson Escobar <neescoba@cisco.com>,
	Nick Desaulniers <ndesaulniers@google.com>,
	Potnuri Bharat Teja <bharat@chelsio.com>,
	Selvin Xavier <selvin.xavier@broadcom.com>,
	Shiraz Saleem <shiraz.saleem@intel.com>,
	VMware PV-Drivers <pv-drivers@vmware.com>,
	Yishai Hadas <yishaih@nvidia.com>,
	Zhu Yanjun <zyjzyj2000@gmail.com>
Subject: Re: [PATCH rdma-next v1 10/15] RDMA/cm: Use an attribute_group on the ib_port_attribute intead of kobj's
Date: Mon, 7 Jun 2021 15:41:37 +0300	[thread overview]
Message-ID: <YL4UAarihtbwyzoB@unreal> (raw)
In-Reply-To: <YL4MOyxQi1O3dog5@kroah.com>

On Mon, Jun 07, 2021 at 02:08:27PM +0200, Greg KH wrote:
> On Mon, Jun 07, 2021 at 02:37:16PM +0300, Leon Romanovsky wrote:
> > On Mon, Jun 07, 2021 at 01:22:26PM +0200, Greg KH wrote:
> > > On Mon, Jun 07, 2021 at 01:51:36PM +0300, Leon Romanovsky wrote:
> > > > On Mon, Jun 07, 2021 at 12:25:03PM +0200, Greg KH wrote:
> > > > > On Mon, Jun 07, 2021 at 11:17:35AM +0300, Leon Romanovsky wrote:
> > > > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > > > 
> > > > > > This code is trying to attach a list of counters grouped into 4 groups to
> > > > > > the ib_port sysfs. Instead of creating a bunch of kobjects simply express
> > > > > > everything naturally as an ib_port_attribute and add a single
> > > > > > attribute_groups list.
> > > > > > 
> > > > > > Remove all the naked kobject manipulations.
> > > > > 
> > > > > Much nicer.
> > > > > 
> > > > > But why do you need your counters to be atomic in the first place?  What
> > > > > are they counting that requires this?  Given that they are just a
> > > > > statistic for userspace, making them be a u64 should work just the same,
> > > > > right?
> > > > 
> > > > The statistic counters are per-port, while the cm.c flows run in
> > > > asynchronically in parallel for every CM connection.
> > > > 
> > > > We need atomic variable to ensure that "write to u64" is not
> > > > interrupted.
> > > 
> > > On what system is "write to u64" interruptable? 
> > 
> > On 32 bits, and yes, we have a customer who still uses such system.
> 
> So you will see what, a "tear"?  Or a stale value?

Does it really make any difference for the users? The end result is that
value is incorrect.

> 
> > > As these are per-port, do multiple threads try to increment these at
> > > the same time?  
> > 
> > Yes, CM connection can be seen as thread. Bottom line everything in parallel.
> > 
> > > And even if they do, what happens if one is 'dropped' somehow because of this?
> > 
> > Probably nothing, we increment the statistics only.
> 
> So you are hitting cache lines for no good reason, probably not a good
> idea, you are wasting cpu cycles for nothing :(

I prefer correctness over performance optimization in paths that far
from being hot.

The thing is that we have such correctness for free without any extra
code complexity by simply using proper data structure. Why shouldn't
we use atomics?

Also, we aren't talking about some driver code, but about core infrastructure
for whole RDMA subsystem.

Thanks

> 
> thanks,
> 
> greg k-h