Re: [RFC][PATCH 2/5] pci: allow shared device instances.

[Jerin Jacob <jerin.jacob@caviumnetworks.com>]

-----Original Message-----
> Date: Wed, 10 May 2017 12:31:48 +0100
> From: Declan Doherty <declan.doherty@intel.com>
> To: Jerin Jacob <jerin.jacob@caviumnetworks.com>
> CC: Thomas Monjalon <thomas@monjalon.net>, Radu Nicolau
>  <radu.nicolau@intel.com>, dev@dpdk.org
> Subject: Re: [dpdk-dev] [RFC][PATCH 2/5] pci: allow shared device instances.
> User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
>  Thunderbird/45.8.0
> 
> On 10/05/2017 12:08 PM, Jerin Jacob wrote:
> > -----Original Message-----
> > > Date: Wed, 10 May 2017 11:52:45 +0100
> > > From: Declan Doherty <declan.doherty@intel.com>
> > > To: Thomas Monjalon <thomas@monjalon.net>, Radu Nicolau
> > >  <radu.nicolau@intel.com>
> > > CC: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] [RFC][PATCH 2/5] pci: allow shared device instances.
> > > User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
> > >  Thunderbird/45.8.0
> > > 
> > > Hey Thomas, I've been working on this with Radu, so see my take below
> > > 
> > > On 10/05/2017 11:28 AM, Thomas Monjalon wrote:
> > > > 10/05/2017 12:11, Radu Nicolau:
> > > > > Hi
> > > > > 
> > > > > 
> > > > > On 5/10/2017 10:09 AM, Thomas Monjalon wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > 09/05/2017 16:57, Radu Nicolau:
> > > > > > > Updated PCI initialization code to allow devices to be shared across multiple PMDs.
> > > > > > > 
> > > > > > > Signed-off-by: Radu Nicolau <radu.nicolau@intel.com>
> > > > > > I was waiting the day when we have a device shared
> > > > > > by two different interfaces.
> > > > > > Note that some Mellanox and Chelsio devices already instantiate
> > > > > > two ethdev ports per PCI device.
> > > > > > 
> > > > > > Please explain your idea behind this "shared" flag.
> > > > > > What is your exact need?
> > > > > 
> > > > > Currently for each pci device a look-up into a list of PMDs is
> > > > > performed, and when a match is found the system moves to the next
> > > > > device. Having this flag will allow a PMD to inform the system that
> > > > > there may be more matches, more PMDs that can be used for this
> > > > > particular device.
> > > > > There is a difference when comparing to the devices you mentioned above,
> > > > > in this case the PMDs are totally different types, one network and one
> > > > > cryptodev PMD for each IXGBE network card.
> > > > 
> > > > Yes I know it is a lack in DPDK.
> > > > Linux introduced MultiFunction Device in 2005:
> > > > 	http://events.linuxfoundation.org/sites/events/files/slides/belloni-mfd-regmap-syscon_0.pdf
> > > > 
> > > 
> > > So at the most basic level the intention is to allow more than one device of
> > > different types, in our case a net PMD and a crypto PMD, to be instantiated
> > > on a single PCI bar, in essence to share the bar. I'm not familiar with the
> > > approaches taken in the Mellanox and Chelsio devices but I assume they are
> > > handled with the driver probe/create  functions independently from the EAL
> > > infrastructure?
> > > 
> > > For the initial proto-typing of this RFC we only implemented the
> > > multi-device creation but I envisage that there will be a requirement for
> > > sharing state between drivers, or at a minimum implementing locking around
> > > shared resources, registers etc. And I would like to see this done in a
> > > generic fashion that can me leverage by any driver and not have each driver
> > > having to solve this independently.
> > 
> > Cavium's next generation PCI based NW devices has similar scheme where we
> > need to share the same BAR with multiple DPDK subsystems(ethdev,
> > eventdev etc) unlike current generation(OcteonTX).
> > 
> 
> Have you done investigation into how you would like to support this, and are
> you trending to any particular approach. The rte_bus approach as you outline
> below does sound like it would suit this multi-function device.

Not much investigation has been done as its for next generation.
It is no PCIe multi function device.

There will be a lot shared functions between these shared DPDK devices and there
should be place holder for this in code.I thought driver/bus/foo may a
option. In additional to this, If we expose new function pointer based
interfaces in bus for the shared device register access and other shared resource
alloc/free between these two DPDK devices, it can be centralized to one
place(driver/bus/foo) and generalized.

Just 2c. We haven't done any prototype.

> 
> > I think, Another possible way to handle this in generic way is to:
> > Register a new rte_bus for the shared PCI access which sits on top PCIe bus.
> > With new bus's scan and probe scheme, it can probe the two devices.
> > 
> > 
> 
> Yes, this would work and I think it makes a lot of sense in the case where
> you have logically independent hardware functional blocks on a shared bus.
> In our particular case, we only have a single physical device, which we are
> presenting as 2 logical devices purely to improve the sw model through DPDK
> existing infrastructure. We may also need to implement some shared context
> for protecting access to shared resources such as register and to
> synchronized exposure of capabilities. In the case of the IXBGE family of
> devices they can support MACsec or IPsec functionality but not both at the
> same time, so some mechanism of passing this state between the net and
> crypto PMDs will be required. I guess it should be possible to do this
> through the bus model as well but we'll need to have another look, although
> my initial feeling is they are slightly different problems.