nec-usb-xhci migration breakage

nec-usb-xhci migration breakage

[Dr. David Alan Gilbert]

Hi,
  Somewhere between 5.1.0 and 5.2.0-rc1 there's a migration breakage
on the nec-usb-xhci device:

[dgilbert@dgilbert-t580 try]$ ./x86_64-softmmu/qemu-system-x86_64 -M pc-q35-5.1 -device nec-usb-xhci -nographic -incoming tcp::4444
qemu-system-x86_64: get_pci_config_device: Bad config data: i=0x72 read: 88 device: 80 cmask: ff wmask: 71 w1cmask:0
qemu-system-x86_64: Failed to load PCIDevice:config
qemu-system-x86_64: Failed to load xhci:parent_obj
qemu-system-x86_64: error while loading state for instance 0x0 of device '0000:00:03.0/xhci'
qemu-system-x86_64: load of migration failed: Invalid argument

lspci is showing the MSI count ifferent, but I'm not sure if that's the
only problem or not:

5.1.0 showing:
[root@localhost ~]# lspci -v -s 00:03.0 -v
00:03.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 23
        Region 0: Memory at febd4000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [90] MSI-X: Enable+ Count=16 Masked-
                Vector table: BAR=0 offset=00003000
                PBA: BAR=0 offset=00003800
        Capabilities: [a0] Express (v2) Root Complex Integrated Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
        Capabilities: [70] MSI: Enable- Count=1/16 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Kernel driver in use: xhci_hcd
 
5.2.0rc1 showing
[root@localhost ~]# lspci -v -s 00:03.0 -v
00:03.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])
        Subsystem: Red Hat, Inc. QEMU Virtual Machine
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 23
        Region 0: Memory at febd4000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [90] MSI-X: Enable+ Count=1 Masked-
                Vector table: BAR=0 offset=00003000
                PBA: BAR=0 offset=00003800
        Capabilities: [a0] Express (v2) Root Complex Integrated Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
        Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Kernel driver in use: xhci_hcd

I see there's quite a few differences in the output of info qtree for
the device.

Dave

-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

RE: nec-usb-xhci migration breakage

[Sai Pavan Boddu]

+Gerd
Hi David,

Sorry for the delayed response. I was able to reproduce your issue with nec-usb-xhci,  in my previous testing we have tested with qemu-xhci model which was good.
I would look further into this. 

@Gerd: Do you have any high level comments on this ?

Regards,
Sai Pavan

> -----Original Message-----
> From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Sent: Wednesday, November 11, 2020 10:46 PM
> To: kraxel@redhat.com; Sai Pavan Boddu <saipava@xilinx.com>;
> marcandre.lureau@redhat.com
> Cc: qemu-devel@nongnu.org
> Subject: nec-usb-xhci migration breakage
> 
> Hi,
>   Somewhere between 5.1.0 and 5.2.0-rc1 there's a migration breakage on the
> nec-usb-xhci device:
> 
> [dgilbert@dgilbert-t580 try]$ ./x86_64-softmmu/qemu-system-x86_64 -M pc-
> q35-5.1 -device nec-usb-xhci -nographic -incoming tcp::4444
> qemu-system-x86_64: get_pci_config_device: Bad config data: i=0x72 read: 88
> device: 80 cmask: ff wmask: 71 w1cmask:0
> qemu-system-x86_64: Failed to load PCIDevice:config
> qemu-system-x86_64: Failed to load xhci:parent_obj
> qemu-system-x86_64: error while loading state for instance 0x0 of device
> '0000:00:03.0/xhci'
> qemu-system-x86_64: load of migration failed: Invalid argument
> 
> lspci is showing the MSI count ifferent, but I'm not sure if that's the only
> problem or not:
> 
> 5.1.0 showing:
> [root@localhost ~]# lspci -v -s 00:03.0 -v
> 00:03.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller
> (rev 03) (prog-if 30 [XHCI])
>         Subsystem: Red Hat, Inc. QEMU Virtual Machine
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 23
>         Region 0: Memory at febd4000 (64-bit, non-prefetchable) [size=16K]
>         Capabilities: [90] MSI-X: Enable+ Count=16 Masked-
>                 Vector table: BAR=0 offset=00003000
>                 PBA: BAR=0 offset=00003800
>         Capabilities: [a0] Express (v2) Root Complex Integrated Endpoint, MSI 00
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0
>                         ExtTag- RBE+
>                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
>                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 128 bytes, MaxReadReq 128 bytes
>                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr-
> TransPend-
>                 DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-,
> OBFF Not Supported
>                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF
> Disabled
>                          AtomicOpsCtl: ReqEn-
>         Capabilities: [70] MSI: Enable- Count=1/16 Maskable- 64bit+
>                 Address: 0000000000000000  Data: 0000
>         Kernel driver in use: xhci_hcd
> 
> 5.2.0rc1 showing
> [root@localhost ~]# lspci -v -s 00:03.0 -v
> 00:03.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller
> (rev 03) (prog-if 30 [XHCI])
>         Subsystem: Red Hat, Inc. QEMU Virtual Machine
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
> <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 23
>         Region 0: Memory at febd4000 (64-bit, non-prefetchable) [size=16K]
>         Capabilities: [90] MSI-X: Enable+ Count=1 Masked-
>                 Vector table: BAR=0 offset=00003000
>                 PBA: BAR=0 offset=00003800
>         Capabilities: [a0] Express (v2) Root Complex Integrated Endpoint, MSI 00
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0
>                         ExtTag- RBE+
>                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
>                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 128 bytes, MaxReadReq 128 bytes
>                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr-
> TransPend-
>                 DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-,
> OBFF Not Supported
>                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF
> Disabled
>                          AtomicOpsCtl: ReqEn-
>         Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit+
>                 Address: 0000000000000000  Data: 0000
>         Kernel driver in use: xhci_hcd
> 
> I see there's quite a few differences in the output of info qtree for the device.
> 
> Dave
> 
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: nec-usb-xhci migration breakage

[kraxel]

On Mon, Nov 16, 2020 at 05:18:22AM +0000, Sai Pavan Boddu wrote:
> +Gerd
> Hi David,
> 
> Sorry for the delayed response. I was able to reproduce your issue with nec-usb-xhci,  in my previous testing we have tested with qemu-xhci model which was good.
> I would look further into this. 
> 
> @Gerd: Do you have any high level comments on this ?

Already fixed in master, see commit 172bc8520db1cb98d09b367360068a675fbc9413

take care,
  Gerd

RE: nec-usb-xhci migration breakage

[Sai Pavan Boddu]

Thanks, I did not observe that.

Regards,
Sai Pavan

> -----Original Message-----
> From: kraxel@redhat.com <kraxel@redhat.com>
> Sent: Monday, November 16, 2020 12:34 PM
> To: Sai Pavan Boddu <saipava@xilinx.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>;
> marcandre.lureau@redhat.com; qemu-devel@nongnu.org
> Subject: Re: nec-usb-xhci migration breakage
> 
> On Mon, Nov 16, 2020 at 05:18:22AM +0000, Sai Pavan Boddu wrote:
> > +Gerd
> > Hi David,
> >
> > Sorry for the delayed response. I was able to reproduce your issue with nec-
> usb-xhci,  in my previous testing we have tested with qemu-xhci model which
> was good.
> > I would look further into this.
> >
> > @Gerd: Do you have any high level comments on this ?
> 
> Already fixed in master, see commit
> 172bc8520db1cb98d09b367360068a675fbc9413
> 
> take care,
>   Gerd

RE: nec-usb-xhci migration breakage

[Sai Pavan Boddu]

HI,

> -----Original Message-----
> From: kraxel@redhat.com <kraxel@redhat.com>
> Sent: Monday, November 16, 2020 12:34 PM
> To: Sai Pavan Boddu <saipava@xilinx.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>;
> marcandre.lureau@redhat.com; qemu-devel@nongnu.org
> Subject: Re: nec-usb-xhci migration breakage
> 
> On Mon, Nov 16, 2020 at 05:18:22AM +0000, Sai Pavan Boddu wrote:
> > +Gerd
> > Hi David,
> >
> > Sorry for the delayed response. I was able to reproduce your issue with nec-
> usb-xhci,  in my previous testing we have tested with qemu-xhci model which
> was good.
> > I would look further into this.
> >
> > @Gerd: Do you have any high level comments on this ?
> 
> Already fixed in master, see commit
> 172bc8520db1cb98d09b367360068a675fbc9413
[Sai Pavan Boddu] I went ahead and test this. Works fine!

Thanks,
Sai Pavan
> 
> take care,
>   Gerd