RE: [PATCH 3/5] vDPA: introduce vDPA bus

From: Shahaf Shuler <shahafs@mellanox.com>
To: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Gunthorpe <jgg@mellanox.com>,
	Jason Wang <jasowang@redhat.com>,
	Rob Miller <rob.miller@broadcom.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"virtualization@lists.linux-foundation.org" 
	<virtualization@lists.linux-foundation.org>,
	Netdev <netdev@vger.kernel.org>,
	"Bie, Tiwei" <tiwei.bie@intel.com>,
	"maxime.coquelin@redhat.com" <maxime.coquelin@redhat.com>,
	"Liang, Cunming" <cunming.liang@intel.com>,
	"Wang, Zhihong" <zhihong.wang@intel.com>,
	"Wang, Xiao W" <xiao.w.wang@intel.com>,
	"haotian.wang@sifive.com" <haotian.wang@sifive.com>,
	"Zhu, Lingshan" <lingshan.zhu@intel.com>,
	"eperezma@redhat.com" <eperezma@redhat.com>,
	"lulu@redhat.com" <lulu@redhat.com>,
	Parav Pandit <parav@mellanox.com>,
	"Tian, Kevin" <kevin.tian@intel.com>,
	"stefanha@redhat.com" <stefanha@redhat.com>,
	"rdunlap@infradead.org" <rdunlap@infradead.org>,
	"hch@infradead.org" <hch@infradead.org>,
	Ariel Adam <aadam@redhat.com>,
	"jakub.kicinski@netronome.com" <jakub.kicinski@netronome.com>,
	Jiri Pirko <jiri@mellanox.com>,
	"hanand@xilinx.com" <hanand@xilinx.com>,
	"mhabets@solarflare.com" <mhabets@solarflare.com>
Subject: RE: [PATCH 3/5] vDPA: introduce vDPA bus
Date: Tue, 21 Jan 2020 06:01:35 +0000	[thread overview]
Message-ID: <AM0PR0502MB3795940D8E0A269D4A32FF8FC30D0@AM0PR0502MB3795.eurprd05.prod.outlook.com> (raw)
In-Reply-To: <20200120165640-mutt-send-email-mst@kernel.org>

Tuesday, January 21, 2020 12:00 AM, Michael S. Tsirkin:
> Subject: Re: [PATCH 3/5] vDPA: introduce vDPA bus
> 
> On Mon, Jan 20, 2020 at 09:47:18PM +0000, Shahaf Shuler wrote:
> > Monday, January 20, 2020 11:25 PM, Michael S. Tsirkin:
> > > Subject: Re: [PATCH 3/5] vDPA: introduce vDPA bus
> > >
> > > On Mon, Jan 20, 2020 at 08:51:43PM +0000, Shahaf Shuler wrote:
> > > > Monday, January 20, 2020 7:50 PM, Jason Gunthorpe:
> > > > > Subject: Re: [PATCH 3/5] vDPA: introduce vDPA bus
> > > > >
> > > > > On Mon, Jan 20, 2020 at 04:43:53PM +0800, Jason Wang wrote:
> > > > > > This is similar to the design of platform IOMMU part of
> > > > > > vhost-vdpa. We decide to send diffs to platform IOMMU there.
> > > > > > If it's ok to do that in driver, we can replace set_map with
> > > > > > incremental API
> > > like map()/unmap().
> > > > > >
> > > > > > Then driver need to maintain rbtree itself.
> > > > >
> > > > > I think we really need to see two modes, one where there is a
> > > > > fixed translation without dynamic vIOMMU driven changes and one
> > > > > that supports vIOMMU.
> > > > >
> > > > > There are different optimization goals in the drivers for these
> > > > > two configurations.
> > > >
> > > > +1.
> > > > It will be best to have one API for static config (i.e. mapping
> > > > can be set only before virtio device gets active), and one API for
> > > > dynamic changes that can be set after the virtio device is active.
> > >
> > > Frankly I don't see when we'd use the static one.
> > > Memory hotplug is enabled for most guests...
> >
> > The fact memory hotplug is enabled doesn't necessarily means there is not
> cold-plugged memory on the hot plugged slots.
> > So your claim is majority of guests are deployed w/o any cold-plugged
> memory?
> 
> Sorry for not being clear. I was merely saying that dynamic one can't be
> optional, and static one can. So how about we start just with the dynamic
> one, then add the static one as a later optimization?

Since we have the use case (cold plugged memory to guest, e.g. when populated w/ hugepages) I think we should start w/ both. The static one can be optional for drivers. 

Moreover am not yet clear about the suggested API for dynamic, can you share the prototype you have in mind?
Also will it be :
1. multiple add_map and then flag the driver to set
Or
2. each add_map should be set by the driver as stand alone. 