Re: [Qemu-devel] [GSoC] Help needed in implementing live migration

Re: [Qemu-devel] [GSoC] Help needed in implementing live migration

[Sukrit Bhatnagar]

On Tue, 25 Jun 2019 at 00:11, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
>
> * Sukrit Bhatnagar (skrtbhtngr@gmail.com) wrote:
> > Hi David,
> >
> > I am Sukrit, GSoC participant working on PVRDMA live migration.
> > We had a short chat about vmxnet3 migration about a week ago
> > on the IRC channel.
> >
> > I am facing an issue while doing migration of the pvrdma device.
> > While loading the device state, we need to perform a few dma
> > mappings on the destination. But on the destination, the migration
> > fails due a BounceBuffer being locked (in_use). This global
> > BounceBuffer is used in address_space_map/unmap functions
> > which the rdma_pci_dma_map/unmap calls.
> > Essentially, we need a way to remap guest physical address on
> > the destination after migration.
> >
> > I had posted an RFC a while ago on the list:
> > https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg04924.html
> > https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg04923.html
> >
> > My mentors (Marcel and Yuval) told me to ask you for help
> > regarding this. It would be really great if you can guide me in
> > finding a workaround for this.
>
> Hi,
>   I'll have a look; I need to get some other things finished first.

Adding cc: qemu-devel, sorry for the private email.

> Dave
> > Thanks,
> > Sukrit
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [GSoC] Help needed in implementing live migration

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

On Thu, Jun 27, 2019 at 05:02:33AM +0530, Sukrit Bhatnagar wrote:
> On Tue, 25 Jun 2019 at 00:11, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> >
> > * Sukrit Bhatnagar (skrtbhtngr@gmail.com) wrote:
> > > Hi David,
> > >
> > > I am Sukrit, GSoC participant working on PVRDMA live migration.
> > > We had a short chat about vmxnet3 migration about a week ago
> > > on the IRC channel.
> > >
> > > I am facing an issue while doing migration of the pvrdma device.
> > > While loading the device state, we need to perform a few dma
> > > mappings on the destination. But on the destination, the migration
> > > fails due a BounceBuffer being locked (in_use). This global
> > > BounceBuffer is used in address_space_map/unmap functions
> > > which the rdma_pci_dma_map/unmap calls.
> > > Essentially, we need a way to remap guest physical address on
> > > the destination after migration.
> > >
> > > I had posted an RFC a while ago on the list:
> > > https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg04924.html
> > > https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg04923.html
> > >
> > > My mentors (Marcel and Yuval) told me to ask you for help
> > > regarding this. It would be really great if you can guide me in
> > > finding a workaround for this.
> >
> > Hi,
> >   I'll have a look; I need to get some other things finished first.
> 
> Adding cc: qemu-devel, sorry for the private email.

I haven't looked deeply but it's surprising that you're hitting
BounceBuffer.  My understanding is that's an old mechanism for
supporting exotic things like DMAing to/from device MMIO registers.

Modern machines and guest software usually don't do this.  I wonder why
you're hitting this case.

If you look at the BounceBuffer code there's an API to register a
callback (cpu_register_map_client()).  That's how the case of multiple
BounceBuffers is supposed to be handled.

Can you double-check your code and figure out how it got here?  I don't
think it should be taking this path.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [GSoC] Help needed in implementing live migration

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@gmail.com) wrote:
> On Thu, Jun 27, 2019 at 05:02:33AM +0530, Sukrit Bhatnagar wrote:
> > On Tue, 25 Jun 2019 at 00:11, Dr. David Alan Gilbert
> > <dgilbert@redhat.com> wrote:
> > >
> > > * Sukrit Bhatnagar (skrtbhtngr@gmail.com) wrote:
> > > > Hi David,
> > > >
> > > > I am Sukrit, GSoC participant working on PVRDMA live migration.
> > > > We had a short chat about vmxnet3 migration about a week ago
> > > > on the IRC channel.
> > > >
> > > > I am facing an issue while doing migration of the pvrdma device.
> > > > While loading the device state, we need to perform a few dma
> > > > mappings on the destination. But on the destination, the migration
> > > > fails due a BounceBuffer being locked (in_use). This global
> > > > BounceBuffer is used in address_space_map/unmap functions
> > > > which the rdma_pci_dma_map/unmap calls.
> > > > Essentially, we need a way to remap guest physical address on
> > > > the destination after migration.
> > > >
> > > > I had posted an RFC a while ago on the list:
> > > > https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg04924.html
> > > > https://lists.gnu.org/archive/html/qemu-devel/2019-06/msg04923.html
> > > >
> > > > My mentors (Marcel and Yuval) told me to ask you for help
> > > > regarding this. It would be really great if you can guide me in
> > > > finding a workaround for this.
> > >
> > > Hi,
> > >   I'll have a look; I need to get some other things finished first.
> > 
> > Adding cc: qemu-devel, sorry for the private email.
> 
> I haven't looked deeply but it's surprising that you're hitting
> BounceBuffer.  My understanding is that's an old mechanism for
> supporting exotic things like DMAing to/from device MMIO registers.
> 
> Modern machines and guest software usually don't do this.  I wonder why
> you're hitting this case.
> 
> If you look at the BounceBuffer code there's an API to register a
> callback (cpu_register_map_client()).  That's how the case of multiple
> BounceBuffers is supposed to be handled.
> 
> Can you double-check your code and figure out how it got here?  I don't
> think it should be taking this path.

Looking at address_space_map I see it uses bounce buffers if
!memory_access_is_direct, and that seems to be a check to see if
the memory region is actually a normal block of RAM.

Dave

> Stefan


--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK