From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933246AbeEIARj (ORCPT ); Tue, 8 May 2018 20:17:39 -0400 Received: from mx1.redhat.com ([209.132.183.28]:53170 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932475AbeEIARh (ORCPT ); Tue, 8 May 2018 20:17:37 -0400 Date: Tue, 8 May 2018 18:17:33 -0600 From: Alex Williamson To: Logan Gunthorpe Cc: Stephen Bates , Christian =?UTF-8?B?S8O2bmln?= , Bjorn Helgaas , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "linux-block@vger.kernel.org" , Christoph Hellwig , Jens Axboe , Keith Busch , Sagi Grimberg , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy , Dan Williams , =?UTF-8?B?SsOpcsO0bWU=?= Glisse , Benjamin Herrenschmidt Subject: Re: [PATCH v4 04/14] PCI/P2PDMA: Clear ACS P2P flags for all devices behind switches Message-ID: <20180508181733.13cbee8b@w520.home> In-Reply-To: <0a0a1782-04b2-c571-63c7-bfff68312946@deltatee.com> References: <20180423233046.21476-1-logang@deltatee.com> <20180423233046.21476-5-logang@deltatee.com> <20180507231306.GG161390@bhelgaas-glaptop.roam.corp.google.com> <0b4183ef-e720-204b-9e85-b9eaf7a4136a@deltatee.com> <3584a6ac-95c7-5d23-1859-aee30605776e@deltatee.com> <20180508133407.57a46902@w520.home> <5fc9b1c1-9208-06cc-0ec5-1f54c2520494@deltatee.com> <20180508141331.7cd737cb@w520.home> <20180508144341.0441b676@w520.home> <20180508152631.50fd583c@w520.home> <354F7407-0DC7-470C-B9AA-74FDF9C46B08@raithlin.com> <20180508160336.0935ddde@w520.home> <20905682-9440-7d4b-0260-99d3dc794c3d@deltatee.com> <20180508171150.0e8fd291@w520.home> <0a0a1782-04b2-c571-63c7-bfff68312946@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 8 May 2018 17:31:48 -0600 Logan Gunthorpe wrote: > On 08/05/18 05:11 PM, Alex Williamson wrote: > > A runtime, sysfs approach has some benefits here, > > especially in identifying the device assuming we're ok with leaving > > the persistence problem to userspace tools. I'm still a little fond of > > the idea of exposing an acs_flags attribute for devices in sysfs where > > a write would do a soft unplug and re-add of all affected devices to > > automatically recreate the proper grouping. Any dynamic change in > > routing and grouping would require all DMA be re-established anyway and > > a soft hotplug seems like an elegant way of handling it. Thanks, > > This approach sounds like it has a lot more issues to contend with: > > For starters, a soft unplug/re-add of all the devices behind a switch is > going to be difficult if a lot of those devices have had drivers > installed and their respective resources are now mounted or otherwise in > use. > > Then, do we have to redo a the soft-replace every time we change the ACS > bit for every downstream port? That could mean you have to do dozens > soft-replaces before you have all the ACS bits set which means you have > a storm of drivers being added and removed. True, anything requiring tweaking multiple downstream ports would induce a hot-unplug/replug for each. A better sysfs interface would allow multiple downstream ports to be updated in a single shot. > This would require some kind of fancy custom setup software that runs at > just the right time in the boot sequence or a lot of work on the users > part to unbind all the resources, setup the ACS bits and then rebind > everything (assuming the soft re-add doesn't rebind it every time you > adjust one ACS bit). Ugly. > > IMO, if we need to do the sysfs approach then we need to be able to > adjust the groups dynamically in a sensible way and not through the > large hammer that is soft-replaces. I think this would be great but I > don't think we will be tackling that with this patch set. OTOH, I think the only sensible way to dynamically adjust groups is through hotplug, we cannot have running drivers attached to downstream endpoints as we're adjusting the routing. Thanks, Alex