RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support

From: "Steve Wise" <swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org>
To: 'Jason Gunthorpe'
	<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>
Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	'Doug Ledford' <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
	'Ram Amrani' <Ram.Amrani-YGCgFSpz5w/QT0dZR+AlfA@public.gmane.org>,
	'Ira Weiny' <ira.weiny-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
	'Benjamin Drung'
	<benjamin.drung-EIkl63zCoXaH+58JC4qpiA@public.gmane.org>,
	'Jarod Wilson' <jarod-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>,
	Chien Tin Tung
	<chien.tin.tung-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>,
	mustafa.ismail-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org
Subject: RE: [PATCH rdma-core 0/5] Common systemd/udev based boot support
Date: Wed, 26 Jul 2017 11:29:15 -0500	[thread overview]
Message-ID: <00e101d3062c$52bcd0f0$f83672d0$@opengridcomputing.com> (raw)
In-Reply-To: <20170726162419.GC20499-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>

> On Wed, Jul 26, 2017 at 09:05:03AM -0500, Steve Wise wrote:
> 
> > > I think you should change this to create the RDMA device when the
> > > module is installed and the hardware is present..
> > >
> >
> > Not gonna happen.  cxgb4 doesn't setup the queues, rss, irq mappings, etc,
until
> > an interface is brought up.  So iw_cxgb4 cannot initialize and register with
the
> > rdma core until that happens.
> 
> But it doesn't tear all that stuff down on ifdown?

hmm.  I need to relook at this.  I'm very sure the intent was offload
connections do not get torn down if the link goes down.

> 
> > > Basically, it is very hard to start a RDMA daemon and not have it race
> > > with something and randomly fail to start properly the more hotpluggy
> > > things are.
> >
> > I think these races exist today, no?  Or is this patch series introducing
the
> > races?   The iwpmd does not need the providers registered at the time it
starts.
> > It will discover new iwarp providers as they initialize.
> 
> There are races today, particularly with systemd, but I suspect they
> are slightly different.. That doesn't mean we should ignore them :)
> 
> In particular the rdma.target approach loaded all the modules, even if
> there wasn't hardware support, so when the cxgb4 RDMA device is
> created everything is already to go. We still have races, but they are
> probably smaller
> 
> Now, we don't even plug the modules until the RDMA device appears, so
> we have broader possibilities for racing..
> 
> Overall it makes it hard to fit into the ideal system configuration
> where RDMA sets up before network-pre.target when a RDMA device
> doesn't even plug in until after network.target.
> 
> > Can you think of other ways to address your concerns?
> 
> Maybe we should have udev trigger loading iwarp services on the cxgb4
> ethernet device? At least the races then become more like what we had
> before. Are other roce/iwarp devices like this?
> 

I'm not sure.  I would be interested to hear what Intel does. +Chien + Mustafa

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html