On Tue, Dec 18, 2018 at 06:25:27PM +0100, David Hildenbrand wrote:
> On 18.12.18 18:13, Cornelia Huck wrote:
> > On Mon, 17 Dec 2018 14:56:38 +0000
> > Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > 
> >> On Mon, Dec 17, 2018 at 11:53:46AM +0100, David Hildenbrand wrote:
> >>> On 14.12.18 14:44, Stefan Hajnoczi wrote:  
> >>>> On Thu, Dec 13, 2018 at 01:38:23PM +0100, Cornelia Huck wrote:  
> >>>>> On Thu, 13 Dec 2018 13:24:31 +0100
> >>>>> David Hildenbrand <david@redhat.com> wrote:
> > 
> >>>>>> As s390x does not have the concept of memory mapped io (RAM is RAM,
> >>>>>> nothing else), this is not architectured. vitio-ccw can therefore not
> >>>>>> define anything similar like that. However, in virtual environments we
> >>>>>> can do whatever we want on top of the pure transport (e.g. on the virtio
> >>>>>> layer).
> >>>>>>
> >>>>>> Conny can correct me if I am wrong.  
> >>>>>
> >>>>> I don't think you're wrong, but I haven't read the code yet and I'm
> >>>>> therefore not aware of the purpose of this BAR.
> >>>>>
> >>>>> Generally, if there is a memory location shared between host and guest,
> >>>>> we need a way to communicate its location, which will likely differ
> >>>>> between transports. For ccw, I could imagine a new channel command
> >>>>> dedicated to exchanging configuration information (similar to what
> >>>>> exists today to communicate the locations of virtqueues), but I'd
> >>>>> rather not go down this path.
> >>>>>
> >>>>> Without reading the code/design further, can we use one of the
> >>>>> following instead of a BAR:
> >>>>> - a virtqueue;
> >>>>> - something in config space?
> >>>>> That would be implementable by any virtio transport.  
> >>>>
> >>>> The way I think about this is that we wish to extend the VIRTIO device
> >>>> model with the concept of shared memory.  virtio-fs, virtio-gpu, and
> >>>> virtio-vhost-user all have requirements for shared memory.
> >>>>
> >>>> This seems like a transport-level issue to me.  PCI supports
> >>>> memory-mapped I/O and that's the right place to do it.  If you try to
> >>>> put it into config space or the virtqueue, you'll end up with something
> >>>> that cannot be realized as a PCI device because it bypasses PCI bus
> >>>> address translation.
> >>>>
> >>>> If CCW needs a side-channel, that's fine.  But that side-channel is a
> >>>> CCW-specific mechanism and probably doesn't apply to all other
> >>>> transports.
> >>>>
> >>>> Stefan
> >>>>   
> >>>
> >>> I think the problem is more fundamental. There is no iommu. Whatever
> >>> shared region you want to indicate, you want it to be assigned a memory
> >>> region in guest physical memory. Like a DIMM/NVDIMM. And this should be
> >>> different to the concept of a BAR. Or am I missing something?  
> >>
> >> If you implement a physical virtio PCI adapter then there is bus
> >> addressing and an IOMMU and VIRTIO has support for that.  I'm not sure I
> >> understand what you mean by "there is no iommu"?
> > 
> > For ccw, there is no iommu; channel-program translation is doing
> > similar things. (I hope that is what David meant :)
> > 
> >>
> >>> I am ok with using whatever other channel to transport such information.
> >>> But I believe this is different to a typical BAR. (I wish I knew more
> >>> about PCI internals ;) ).
> >>>
> >>> I would also like to know how shared memory works as of now for e.g.
> >>> virtio-gpu.  
> >>
> >> virtio-gpu currently does not use shared memory, it needs it for future
> >> features.
> > 
> > OK, that all sounds like we need to define a generic, per transport,
> > device agnostic way to specify shared memory.
> > 
> > Where is that memory situated? Is it something in guest memory (like
> > virtqueues)? If it is something provided by the device, things will get
> > tricky for ccw (remember that there's no mmio on s390; pci on s390 uses
> > special instructions for that.)
> > 
> 
> I am just very very confused right now. What I am struggling with right
> now (Stefan, hope you can clarify it for me):
> 
> We need some place where this shared memory is located in the guest
> physical memory. On x86 - if I am not wrong - this BAR is placed into
> the reserved memory area between 3 and 4 GB.

Right, the shared memory is provided by the device and does not live in
guest RAM.

> There is no such thing on
> s390x. Because we don't have IO via memory (yet). All we have is one or
> two KVM memory slots filled with all memory.
> 
> So what we will need on s390x is on the QEMU side such a reserved memory
> region where devices like virtio-fs can reserve a region for shared memory.
> 
> So it is something like a dimm/nvdimm except that it is smaller and not
> visible to the user directly (via memory backends).

I see.  That makes sense.

Stefan