PCI passthrough issue

PCI passthrough issue

[Jean Baptiste Favre]

Hello,
I'm facing a strange issue with network card PCI passthrough on my
openwrt test domU.

- With network PCI passthrough, DNS lookup failed for some domains
(exemple, google.com) but not for other (free.fr my ISP, or my domain
jbfavre.org). I can ping an IP address without any problem.

- Starting domU as a "normal" (ie without PCI passthrough), no problem.


As far as I can say, domU is not the root cause. I really think this is
related to PCI passsthrough. This seems to be related to packet length.

Did not see anything strange in dom0 logs.

Is there any incompatibility between 2.6.32 dom0 kernel with Xen 4.0.1
and 2.6.37 domU kernel ?

Regards,
JB

Dom0: Debian Squeeze
# uname -a
Linux remus 2.6.32-5-xen-amd64
# dpkg -l|grep hyper
ii  xen-hypervisor-4.0-amd64                           4.0.1-1

DomU: Openwrt backfire
# uname -a
Linux OpenWrt 2.6.37
# lsmod
Module                  Size  Used by    Not tainted
sky2                   35744  0
All Xen related stuff is integrated into the kernel, not compiled as module.

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote:
> Hello,
> I'm facing a strange issue with network card PCI passthrough on my
> openwrt test domU.
> 
> - With network PCI passthrough, DNS lookup failed for some domains
> (exemple, google.com) but not for other (free.fr my ISP, or my domain
> jbfavre.org). I can ping an IP address without any problem.

Do you have "both" (so PCI passthrough and the Xen network driver)
in the guest? If so, have you tried eliminating the xen network driver
to see if it is just a routing issue?

What does your routing table look like? Your IP table?

> 
> - Starting domU as a "normal" (ie without PCI passthrough), no problem.
> 
> 
> As far as I can say, domU is not the root cause. I really think this is
> related to PCI passsthrough. This seems to be related to packet length.

Then that would imply the MTU is not set right. 
> 
> Did not see anything strange in dom0 logs.
> 
> Is there any incompatibility between 2.6.32 dom0 kernel with Xen 4.0.1
> and 2.6.37 domU kernel ?

No.
> 
> Regards,
> JB
> 
> Dom0: Debian Squeeze
> # uname -a
> Linux remus 2.6.32-5-xen-amd64
> # dpkg -l|grep hyper
> ii  xen-hypervisor-4.0-amd64                           4.0.1-1
> 
> DomU: Openwrt backfire
> # uname -a
> Linux OpenWrt 2.6.37
> # lsmod
> Module                  Size  Used by    Not tainted
> sky2                   35744  0
> All Xen related stuff is integrated into the kernel, not compiled as module.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello Konrad,

Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit :
> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote:
>> Hello,
>> I'm facing a strange issue with network card PCI passthrough on my
>> openwrt test domU.
>>
>> - With network PCI passthrough, DNS lookup failed for some domains
>> (exemple, google.com) but not for other (free.fr my ISP, or my domain
>> jbfavre.org). I can ping an IP address without any problem.
> 
> Do you have "both" (so PCI passthrough and the Xen network driver)
> in the guest? If so, have you tried eliminating the xen network driver
> to see if it is just a routing issue?
Have not tried to eliminate xen network driver. Think I have both drivers.

My kernel .config looks like:
$ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config
CONFIG_XEN=y
# CONFIG_XEN_PRIVILEGED_GUEST is not set
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=128
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
# CONFIG_XEN_PLATFORM_PCI is not set
CONFIG_SWIOTLB_XEN=y

So, I should remove CONFIG_XEN_NETDEV_FRONTEND ?

> What does your routing table look like? Your IP table?
My routing table is pretty clean, nothing strange here
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use
Iface
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0
br-wan
0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0
br-wan

>> - Starting domU as a "normal" (ie without PCI passthrough), no problem.
>>
>>
>> As far as I can say, domU is not the root cause. I really think this is
>> related to PCI passsthrough. This seems to be related to packet length.
> 
> Then that would imply the MTU is not set right. 
Already checked it: 1500 :)

>> Did not see anything strange in dom0 logs.
>>
>> Is there any incompatibility between 2.6.32 dom0 kernel with Xen 4.0.1
>> and 2.6.37 domU kernel ?
> 
> No.
So, will try to remove Xen Network driver and see what happen.

Regards,
JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello again,
Just tried without Xen network driver and got the same behaviour.
Would say hopefully because removing xen network driver prevent me to
use my domU as firewall for dom0 as I intented to do.

BTW, have no idea left :-/
Regards,
JB


Le 12/01/2011 16:53, Jean Baptiste Favre a écrit :
> Hello Konrad,
> 
> Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit :
>> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote:
>>> Hello,
>>> I'm facing a strange issue with network card PCI passthrough on my
>>> openwrt test domU.
>>>
>>> - With network PCI passthrough, DNS lookup failed for some domains
>>> (exemple, google.com) but not for other (free.fr my ISP, or my domain
>>> jbfavre.org). I can ping an IP address without any problem.
>>
>> Do you have "both" (so PCI passthrough and the Xen network driver)
>> in the guest? If so, have you tried eliminating the xen network driver
>> to see if it is just a routing issue?
> Have not tried to eliminate xen network driver. Think I have both drivers.
> 
> My kernel .config looks like:
> $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config
> CONFIG_XEN=y
> # CONFIG_XEN_PRIVILEGED_GUEST is not set
> CONFIG_XEN_PVHVM=y
> CONFIG_XEN_MAX_DOMAIN_MEMORY=128
> CONFIG_XEN_SAVE_RESTORE=y
> CONFIG_PCI_XEN=y
> CONFIG_XEN_PCIDEV_FRONTEND=y
> CONFIG_XEN_BLKDEV_FRONTEND=y
> CONFIG_XEN_NETDEV_FRONTEND=y
> CONFIG_HVC_XEN=y
> CONFIG_XEN_BALLOON=y
> CONFIG_XEN_SCRUB_PAGES=y
> CONFIG_XEN_DEV_EVTCHN=y
> CONFIG_XENFS=y
> CONFIG_XEN_COMPAT_XENFS=y
> CONFIG_XEN_SYS_HYPERVISOR=y
> # CONFIG_XEN_PLATFORM_PCI is not set
> CONFIG_SWIOTLB_XEN=y
> 
> So, I should remove CONFIG_XEN_NETDEV_FRONTEND ?
> 
>> What does your routing table look like? Your IP table?
> My routing table is pretty clean, nothing strange here
> # route -n
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use
> Iface
> 10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0
> br-wan
> 0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0
> br-wan
> 
>>> - Starting domU as a "normal" (ie without PCI passthrough), no problem.
>>>
>>>
>>> As far as I can say, domU is not the root cause. I really think this is
>>> related to PCI passsthrough. This seems to be related to packet length.
>>
>> Then that would imply the MTU is not set right. 
> Already checked it: 1500 :)
> 
>>> Did not see anything strange in dom0 logs.
>>>
>>> Is there any incompatibility between 2.6.32 dom0 kernel with Xen 4.0.1
>>> and 2.6.37 domU kernel ?
>>
>> No.
> So, will try to remove Xen Network driver and see what happen.
> 
> Regards,
> JB
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Wed, Jan 12, 2011 at 04:53:51PM +0100, Jean Baptiste Favre wrote:
> Hello Konrad,
> 
> Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit :
> > On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote:
> >> Hello,
> >> I'm facing a strange issue with network card PCI passthrough on my
> >> openwrt test domU.
> >>
> >> - With network PCI passthrough, DNS lookup failed for some domains
> >> (exemple, google.com) but not for other (free.fr my ISP, or my domain
> >> jbfavre.org). I can ping an IP address without any problem.
> > 
> > Do you have "both" (so PCI passthrough and the Xen network driver)
> > in the guest? If so, have you tried eliminating the xen network driver
> > to see if it is just a routing issue?
> Have not tried to eliminate xen network driver. Think I have both drivers.
> 
> My kernel .config looks like:
> $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config
> CONFIG_XEN=y
> # CONFIG_XEN_PRIVILEGED_GUEST is not set
> CONFIG_XEN_PVHVM=y
> CONFIG_XEN_MAX_DOMAIN_MEMORY=128
> CONFIG_XEN_SAVE_RESTORE=y
> CONFIG_PCI_XEN=y
> CONFIG_XEN_PCIDEV_FRONTEND=y
> CONFIG_XEN_BLKDEV_FRONTEND=y
> CONFIG_XEN_NETDEV_FRONTEND=y
> CONFIG_HVC_XEN=y
> CONFIG_XEN_BALLOON=y
> CONFIG_XEN_SCRUB_PAGES=y
> CONFIG_XEN_DEV_EVTCHN=y
> CONFIG_XENFS=y
> CONFIG_XEN_COMPAT_XENFS=y
> CONFIG_XEN_SYS_HYPERVISOR=y
> # CONFIG_XEN_PLATFORM_PCI is not set
> CONFIG_SWIOTLB_XEN=y
> 
> So, I should remove CONFIG_XEN_NETDEV_FRONTEND ?

No. You can just do 'ifconfig <x> down' whatever your Xen netfront
NIC is. .. but.
> 
> > What does your routing table look like? Your IP table?
> My routing table is pretty clean, nothing strange here
> # route -n
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use
> Iface
> 10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0
> br-wan
> 0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0
> br-wan

Ok, then the idea that the Xen networking driver and the PCI passthrough send
packets over is not the way..

So ignore about the Xen networking part.
> 
> >> - Starting domU as a "normal" (ie without PCI passthrough), no problem.
> >>
> >>
> >> As far as I can say, domU is not the root cause. I really think this is
> >> related to PCI passsthrough. This seems to be related to packet length.
> > 
> > Then that would imply the MTU is not set right. 
> Already checked it: 1500 :)
Ok.

Next thing, did you try to disable the rx/tx checksumming?

If you connect the Ethernet cable for this 'br-wan' device to another machine
(so you could set it up as bridge and just let it pass through packets and sniff
the data) what do the packets look like?

What happens if the PCI passthrough device is not under the ownership of a bridge?
What then? You wouldn't have any bridge firewall code in?

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 12/01/2011 17:36, Konrad Rzeszutek Wilk a écrit :
> On Wed, Jan 12, 2011 at 04:53:51PM +0100, Jean Baptiste Favre wrote:
>> Hello Konrad,
>>
>> Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit :
>>> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote:
>>>> Hello,
>>>> I'm facing a strange issue with network card PCI passthrough on my
>>>> openwrt test domU.
>>>>
>>>> - With network PCI passthrough, DNS lookup failed for some domains
>>>> (exemple, google.com) but not for other (free.fr my ISP, or my domain
>>>> jbfavre.org). I can ping an IP address without any problem.
>>>
>>> Do you have "both" (so PCI passthrough and the Xen network driver)
>>> in the guest? If so, have you tried eliminating the xen network driver
>>> to see if it is just a routing issue?
>> Have not tried to eliminate xen network driver. Think I have both drivers.
>>
>> My kernel .config looks like:
>> $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config
>> CONFIG_XEN=y
>> # CONFIG_XEN_PRIVILEGED_GUEST is not set
>> CONFIG_XEN_PVHVM=y
>> CONFIG_XEN_MAX_DOMAIN_MEMORY=128
>> CONFIG_XEN_SAVE_RESTORE=y
>> CONFIG_PCI_XEN=y
>> CONFIG_XEN_PCIDEV_FRONTEND=y
>> CONFIG_XEN_BLKDEV_FRONTEND=y
>> CONFIG_XEN_NETDEV_FRONTEND=y
>> CONFIG_HVC_XEN=y
>> CONFIG_XEN_BALLOON=y
>> CONFIG_XEN_SCRUB_PAGES=y
>> CONFIG_XEN_DEV_EVTCHN=y
>> CONFIG_XENFS=y
>> CONFIG_XEN_COMPAT_XENFS=y
>> CONFIG_XEN_SYS_HYPERVISOR=y
>> # CONFIG_XEN_PLATFORM_PCI is not set
>> CONFIG_SWIOTLB_XEN=y
>>
>> So, I should remove CONFIG_XEN_NETDEV_FRONTEND ?
> 
> No. You can just do 'ifconfig <x> down' whatever your Xen netfront
> NIC is. .. but.
>>
>>> What does your routing table look like? Your IP table?
>> My routing table is pretty clean, nothing strange here
>> # route -n
>> Kernel IP routing table
>> Destination     Gateway         Genmask         Flags Metric Ref    Use
>> Iface
>> 10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0
>> br-wan
>> 0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0
>> br-wan
> 
> Ok, then the idea that the Xen networking driver and the PCI passthrough send
> packets over is not the way..
> 
> So ignore about the Xen networking part.

OK.

>>
>>>> - Starting domU as a "normal" (ie without PCI passthrough), no problem.
>>>>
>>>>
>>>> As far as I can say, domU is not the root cause. I really think this is
>>>> related to PCI passsthrough. This seems to be related to packet length.
>>>
>>> Then that would imply the MTU is not set right. 
>> Already checked it: 1500 :)
> Ok.
> 
> Next thing, did you try to disable the rx/tx checksumming?

No. Not sure I even know how to do it, but will have a look on that.

> If you connect the Ethernet cable for this 'br-wan' device to another machine
> (so you could set it up as bridge and just let it pass through packets and sniff
> the data) what do the packets look like?

I can see packets on my gateway (which acts as DNS as well). They looks
good. I can see DNS answers as well leaving the gateway, but not
reaching my domU. Have only basic switches between my gateway and my domU.

> What happens if the PCI passthrough device is not under the ownership of a bridge?
> What then? You wouldn't have any bridge firewall code in?

Removed bridge + configure eth: no change.
Checked ebtables/iptables (all tables: net filter and mangle) rules:
empty, Policy to ACCEPT.
Tried to enable/disable ip_forwarding: no change
Also disabled IPV6 support (I saw some DNS answers as AAAA but no IPv6
available at home for now): no change

Regards,
JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Wed, Jan 12, 2011 at 05:56:45PM +0100, Jean Baptiste Favre wrote:
> Le 12/01/2011 17:36, Konrad Rzeszutek Wilk a écrit :
> > On Wed, Jan 12, 2011 at 04:53:51PM +0100, Jean Baptiste Favre wrote:
> >> Hello Konrad,
> >>
> >> Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit :
> >>> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote:
> >>>> Hello,
> >>>> I'm facing a strange issue with network card PCI passthrough on my
> >>>> openwrt test domU.
> >>>>
> >>>> - With network PCI passthrough, DNS lookup failed for some domains
> >>>> (exemple, google.com) but not for other (free.fr my ISP, or my domain
> >>>> jbfavre.org). I can ping an IP address without any problem.
> >>>
> >>> Do you have "both" (so PCI passthrough and the Xen network driver)
> >>> in the guest? If so, have you tried eliminating the xen network driver
> >>> to see if it is just a routing issue?
> >> Have not tried to eliminate xen network driver. Think I have both drivers.
> >>
> >> My kernel .config looks like:
> >> $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config
> >> CONFIG_XEN=y
> >> # CONFIG_XEN_PRIVILEGED_GUEST is not set
> >> CONFIG_XEN_PVHVM=y
> >> CONFIG_XEN_MAX_DOMAIN_MEMORY=128
> >> CONFIG_XEN_SAVE_RESTORE=y
> >> CONFIG_PCI_XEN=y
> >> CONFIG_XEN_PCIDEV_FRONTEND=y
> >> CONFIG_XEN_BLKDEV_FRONTEND=y
> >> CONFIG_XEN_NETDEV_FRONTEND=y
> >> CONFIG_HVC_XEN=y
> >> CONFIG_XEN_BALLOON=y
> >> CONFIG_XEN_SCRUB_PAGES=y
> >> CONFIG_XEN_DEV_EVTCHN=y
> >> CONFIG_XENFS=y
> >> CONFIG_XEN_COMPAT_XENFS=y
> >> CONFIG_XEN_SYS_HYPERVISOR=y
> >> # CONFIG_XEN_PLATFORM_PCI is not set
> >> CONFIG_SWIOTLB_XEN=y
> >>
> >> So, I should remove CONFIG_XEN_NETDEV_FRONTEND ?
> > 
> > No. You can just do 'ifconfig <x> down' whatever your Xen netfront
> > NIC is. .. but.
> >>
> >>> What does your routing table look like? Your IP table?
> >> My routing table is pretty clean, nothing strange here
> >> # route -n
> >> Kernel IP routing table
> >> Destination     Gateway         Genmask         Flags Metric Ref    Use
> >> Iface
> >> 10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0
> >> br-wan
> >> 0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0
> >> br-wan
> > 
> > Ok, then the idea that the Xen networking driver and the PCI passthrough send
> > packets over is not the way..
> > 
> > So ignore about the Xen networking part.
> 
> OK.
> 
> >>
> >>>> - Starting domU as a "normal" (ie without PCI passthrough), no problem.
> >>>>
> >>>>
> >>>> As far as I can say, domU is not the root cause. I really think this is
> >>>> related to PCI passsthrough. This seems to be related to packet length.
> >>>
> >>> Then that would imply the MTU is not set right. 
> >> Already checked it: 1500 :)
> > Ok.
> > 
> > Next thing, did you try to disable the rx/tx checksumming?
> 
> No. Not sure I even know how to do it, but will have a look on that.
> 
> > If you connect the Ethernet cable for this 'br-wan' device to another machine
> > (so you could set it up as bridge and just let it pass through packets and sniff
> > the data) what do the packets look like?
> 
> I can see packets on my gateway (which acts as DNS as well). They looks
> good. I can see DNS answers as well leaving the gateway, but not
> reaching my domU. Have only basic switches between my gateway and my domU.

Wait a minute. How are your domU's connected? You did turn on
forwarding in your guest with the PCI passthrough right?

How is your domain (with the PCI passthrought) connected to the
other domains?
> 
> > What happens if the PCI passthrough device is not under the ownership of a bridge?
> > What then? You wouldn't have any bridge firewall code in?
> 
> Removed bridge + configure eth: no change.
> Checked ebtables/iptables (all tables: net filter and mangle) rules:
> empty, Policy to ACCEPT.
> Tried to enable/disable ip_forwarding: no change
> Also disabled IPV6 support (I saw some DNS answers as AAAA but no IPv6
> available at home for now): no change
> 
> Regards,
> JB
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 12/01/2011 18:26, Konrad Rzeszutek Wilk a écrit :
> On Wed, Jan 12, 2011 at 05:56:45PM +0100, Jean Baptiste Favre wrote:
>> Le 12/01/2011 17:36, Konrad Rzeszutek Wilk a écrit :
>>> On Wed, Jan 12, 2011 at 04:53:51PM +0100, Jean Baptiste Favre wrote:
>>>> Hello Konrad,
>>>>
>>>> Le 12/01/2011 16:43, Konrad Rzeszutek Wilk a écrit :
>>>>> On Wed, Jan 12, 2011 at 04:38:49PM +0100, Jean Baptiste Favre wrote:
>>>>>> Hello,
>>>>>> I'm facing a strange issue with network card PCI passthrough on my
>>>>>> openwrt test domU.
>>>>>>
>>>>>> - With network PCI passthrough, DNS lookup failed for some domains
>>>>>> (exemple, google.com) but not for other (free.fr my ISP, or my domain
>>>>>> jbfavre.org). I can ping an IP address without any problem.
>>>>>
>>>>> Do you have "both" (so PCI passthrough and the Xen network driver)
>>>>> in the guest? If so, have you tried eliminating the xen network driver
>>>>> to see if it is just a routing issue?
>>>> Have not tried to eliminate xen network driver. Think I have both drivers.
>>>>
>>>> My kernel .config looks like:
>>>> $ grep XEN build_dir/linux-x86_xen_domu/linux-2.6.37/.config
>>>> CONFIG_XEN=y
>>>> # CONFIG_XEN_PRIVILEGED_GUEST is not set
>>>> CONFIG_XEN_PVHVM=y
>>>> CONFIG_XEN_MAX_DOMAIN_MEMORY=128
>>>> CONFIG_XEN_SAVE_RESTORE=y
>>>> CONFIG_PCI_XEN=y
>>>> CONFIG_XEN_PCIDEV_FRONTEND=y
>>>> CONFIG_XEN_BLKDEV_FRONTEND=y
>>>> CONFIG_XEN_NETDEV_FRONTEND=y
>>>> CONFIG_HVC_XEN=y
>>>> CONFIG_XEN_BALLOON=y
>>>> CONFIG_XEN_SCRUB_PAGES=y
>>>> CONFIG_XEN_DEV_EVTCHN=y
>>>> CONFIG_XENFS=y
>>>> CONFIG_XEN_COMPAT_XENFS=y
>>>> CONFIG_XEN_SYS_HYPERVISOR=y
>>>> # CONFIG_XEN_PLATFORM_PCI is not set
>>>> CONFIG_SWIOTLB_XEN=y
>>>>
>>>> So, I should remove CONFIG_XEN_NETDEV_FRONTEND ?
>>>
>>> No. You can just do 'ifconfig <x> down' whatever your Xen netfront
>>> NIC is. .. but.
>>>>
>>>>> What does your routing table look like? Your IP table?
>>>> My routing table is pretty clean, nothing strange here
>>>> # route -n
>>>> Kernel IP routing table
>>>> Destination     Gateway         Genmask         Flags Metric Ref    Use
>>>> Iface
>>>> 10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0
>>>> br-wan
>>>> 0.0.0.0         10.0.0.1        0.0.0.0         UG    0      0        0
>>>> br-wan
>>>
>>> Ok, then the idea that the Xen networking driver and the PCI passthrough send
>>> packets over is not the way..
>>>
>>> So ignore about the Xen networking part.
>>
>> OK.
>>
>>>>
>>>>>> - Starting domU as a "normal" (ie without PCI passthrough), no problem.
>>>>>>
>>>>>>
>>>>>> As far as I can say, domU is not the root cause. I really think this is
>>>>>> related to PCI passsthrough. This seems to be related to packet length.
>>>>>
>>>>> Then that would imply the MTU is not set right. 
>>>> Already checked it: 1500 :)
>>> Ok.
>>>
>>> Next thing, did you try to disable the rx/tx checksumming?
>>
>> No. Not sure I even know how to do it, but will have a look on that.
>>
>>> If you connect the Ethernet cable for this 'br-wan' device to another machine
>>> (so you could set it up as bridge and just let it pass through packets and sniff
>>> the data) what do the packets look like?
>>
>> I can see packets on my gateway (which acts as DNS as well). They looks
>> good. I can see DNS answers as well leaving the gateway, but not
>> reaching my domU. Have only basic switches between my gateway and my domU.
> 
> Wait a minute. How are your domU's connected? You did turn on
> forwarding in your guest with the PCI passthrough right?

My OpenWRT domU with PCI passthrough is directly connected to my LAN
through network card

> How is your domain (with the PCI passthrought) connected to the
> other domains?

For the moment, other domU are not connected in any way to my OpenWRT
domU. I want first to make sur PCI passthrough is working before trying
more complex architecture.

My OpenWRT domU network setup is:

dom0           domU            LAN

vif ----- eth0      eth1 ----- SW ----- GW ----- NET
            |__ br __|

All tests are done from OpenWRT domU for now (but I guess this won't
work either from dom0).

I also tried following setup:
domU            LAN

    eth1 ----- SW ----- GW ----- NET
 br __|

That means no vif from dom0. But with PCI passthrough, I should reach
network directly, right ?

Finally, I think that my setup shall not be responsible for the fact
that ping works, small DNS answers are received but not bigger ones
(limit seems to be around 100 bytes).

>>> What happens if the PCI passthrough device is not under the ownership of a bridge?
>>> What then? You wouldn't have any bridge firewall code in?
>>
>> Removed bridge + configure eth: no change.
>> Checked ebtables/iptables (all tables: net filter and mangle) rules:
>> empty, Policy to ACCEPT.
>> Tried to enable/disable ip_forwarding: no change
>> Also disabled IPV6 support (I saw some DNS answers as AAAA but no IPv6
>> available at home for now): no change
>>
>> Regards,
>> JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

> 
> Finally, I think that my setup shall not be responsible for the fact
> that ping works, small DNS answers are received but not bigger ones

Aah, somehow I missed that fact.
> (limit seems to be around 100 bytes).

What does wireshark or tcpdump tell you when you send the larger packets?

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit :
>>
>> Finally, I think that my setup shall not be responsible for the fact
>> that ping works, small DNS answers are received but not bigger ones
> 
> Aah, somehow I missed that fact.

Seems yes :)

>> (limit seems to be around 100 bytes).
> 
> What does wireshark or tcpdump tell you when you send the larger packets?

That's one of my problem: nothing special :(

I can only run tcpdump on my gateway (no SSH available on my OpenWRT
domU and only xm console access is available).
I see DNS requests comming in, I see DNS answers going out.
ARP requests are fine.

As an example:
dig free.fr works
dig jbfavre.org works
dig amazon.fr works
dig amazon.com works (quite long, but works)
dig google.fr fails
dig google.com fails

>From another machine in the same LAN, all requests work fine.

Thought this was due to some ARP restriction setted up by default in
OpenWRT, but in this case, I can not explain why some DNS lookup do work
and some don't.

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Wed, Jan 12, 2011 at 09:07:30PM +0100, Jean Baptiste Favre wrote:
> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit :
> >>
> >> Finally, I think that my setup shall not be responsible for the fact
> >> that ping works, small DNS answers are received but not bigger ones
> > 
> > Aah, somehow I missed that fact.
> 
> Seems yes :)
> 
> >> (limit seems to be around 100 bytes).
> > 
> > What does wireshark or tcpdump tell you when you send the larger packets?
> 
> That's one of my problem: nothing special :(
> 
> I can only run tcpdump on my gateway (no SSH available on my OpenWRT
> domU and only xm console access is available).

You can install (or just compile it statically) tcpdump on your OpenWRT,
or just hook it up to a beffier guest..
> I see DNS requests comming in, I see DNS answers going out.
> ARP requests are fine.
> 
> As an example:
> dig free.fr works
> dig jbfavre.org works
> dig amazon.fr works
> dig amazon.com works (quite long, but works)
> dig google.fr fails
> dig google.com fails

Fails as in response or fails halfway?
> 
> >From another machine in the same LAN, all requests work fine.
> 
> Thought this was due to some ARP restriction setted up by default in
> OpenWRT, but in this case, I can not explain why some DNS lookup do work
> and some don't.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Jean Baptiste Favre]

Strange,
Just noticed some message I missed before:
When starting domain:
[175004.836398] pciback 0000:04:00.0: device has been assigned to
another domain! Over-writting the ownership, but beware.

and
(XEN) traps.c:2302:d44 Domain attempted WRMSR 00000000c0010004 from
0000c827:4e390408 to 00000000:0000abcd.


The first one is very strange as I'm absolutely sure device is NOT
assigned to any domain (the only domU I have on this dom0 is my OpenWRT).
The second one... well, I don't know what it's talking about :-/

Regards,
JB


Le 12/01/2011 21:07, Jean Baptiste Favre a écrit :
> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit :
>>>
>>> Finally, I think that my setup shall not be responsible for the fact
>>> that ping works, small DNS answers are received but not bigger ones
>>
>> Aah, somehow I missed that fact.
> 
> Seems yes :)
> 
>>> (limit seems to be around 100 bytes).
>>
>> What does wireshark or tcpdump tell you when you send the larger packets?
> 
> That's one of my problem: nothing special :(
> 
> I can only run tcpdump on my gateway (no SSH available on my OpenWRT
> domU and only xm console access is available).
> I see DNS requests comming in, I see DNS answers going out.
> ARP requests are fine.
> 
> As an example:
> dig free.fr works
> dig jbfavre.org works
> dig amazon.fr works
> dig amazon.com works (quite long, but works)
> dig google.fr fails
> dig google.com fails
> 
>>From another machine in the same LAN, all requests work fine.
> 
> Thought this was due to some ARP restriction setted up by default in
> OpenWRT, but in this case, I can not explain why some DNS lookup do work
> and some don't.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 12/01/2011 22:40, Konrad Rzeszutek Wilk a écrit :
> On Wed, Jan 12, 2011 at 09:07:30PM +0100, Jean Baptiste Favre wrote:
>> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit :
>>>>
>>>> Finally, I think that my setup shall not be responsible for the fact
>>>> that ping works, small DNS answers are received but not bigger ones
>>>
>>> Aah, somehow I missed that fact.
>>
>> Seems yes :)
>>
>>>> (limit seems to be around 100 bytes).
>>>
>>> What does wireshark or tcpdump tell you when you send the larger packets?
>>
>> That's one of my problem: nothing special :(
>>
>> I can only run tcpdump on my gateway (no SSH available on my OpenWRT
>> domU and only xm console access is available).
> 
> You can install (or just compile it statically) tcpdump on your OpenWRT,
> or just hook it up to a beffier guest..

tcpdump is available on my OpenWRT domU, but I can not perform tcpdump
and DNS lookup with only console access and AFAIK screen is not
available on OpenWRT

>> I see DNS requests comming in, I see DNS answers going out.
>> ARP requests are fine.
>>
>> As an example:
>> dig free.fr works
>> dig jbfavre.org works
>> dig amazon.fr works
>> dig amazon.com works (quite long, but works)
>> dig google.fr fails
>> dig google.com fails
> 
> Fails as in response or fails halfway?

Fails as in:
# dig google.com

; <<>> DiG 9.7.2-P3 <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached

>>
>> >From another machine in the same LAN, all requests work fine.
>>
>> Thought this was due to some ARP restriction setted up by default in
>> OpenWRT, but in this case, I can not explain why some DNS lookup do work
>> and some don't.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Ok, my bad. Screen IS available on OpenWRT.
I've added it to my disk image and now have a tcpdump capture of network
traffic of my DomU

Have the same network capture from my GW so I 'll be able to investigate
deeper.

Let me know if you have specific points you want me to look at.

Regards,
JB


Le 12/01/2011 22:46, Jean Baptiste Favre a écrit :
> Le 12/01/2011 22:40, Konrad Rzeszutek Wilk a écrit :
>> On Wed, Jan 12, 2011 at 09:07:30PM +0100, Jean Baptiste Favre wrote:
>>> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit :
>>>>>
>>>>> Finally, I think that my setup shall not be responsible for the fact
>>>>> that ping works, small DNS answers are received but not bigger ones
>>>>
>>>> Aah, somehow I missed that fact.
>>>
>>> Seems yes :)
>>>
>>>>> (limit seems to be around 100 bytes).
>>>>
>>>> What does wireshark or tcpdump tell you when you send the larger packets?
>>>
>>> That's one of my problem: nothing special :(
>>>
>>> I can only run tcpdump on my gateway (no SSH available on my OpenWRT
>>> domU and only xm console access is available).
>>
>> You can install (or just compile it statically) tcpdump on your OpenWRT,
>> or just hook it up to a beffier guest..
> 
> tcpdump is available on my OpenWRT domU, but I can not perform tcpdump
> and DNS lookup with only console access and AFAIK screen is not
> available on OpenWRT
> 
>>> I see DNS requests comming in, I see DNS answers going out.
>>> ARP requests are fine.
>>>
>>> As an example:
>>> dig free.fr works
>>> dig jbfavre.org works
>>> dig amazon.fr works
>>> dig amazon.com works (quite long, but works)
>>> dig google.fr fails
>>> dig google.com fails
>>
>> Fails as in response or fails halfway?
> 
> Fails as in:
> # dig google.com
> 
> ; <<>> DiG 9.7.2-P3 <<>> google.com
> ;; global options: +cmd
> ;; connection timed out; no servers could be reached
> 
>>>
>>> >From another machine in the same LAN, all requests work fine.
>>>
>>> Thought this was due to some ARP restriction setted up by default in
>>> OpenWRT, but in this case, I can not explain why some DNS lookup do work
>>> and some don't.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Jean Baptiste Favre]

Have had a look on both tcpdump captures.
1. From the OpenWRT domU: nothing special. Lot of STP traffic due to my
network architecture. Don't thing this could be the cause of my problem
since I'm always able to reproduce the failure whatever could be the
order of DNS requests. Don't see "big" DNS answers coming in, see
multiple requests before timeout.

2. From my gateway: nothing special here too. Same STP traffic (have
checked GW network setup as well). See each DNS requests my domU sent
and can see as well answers, which are not received by domU. No strange
ARP traffic.

I wanted to change the network, thinking about a bogus driver (never
know). My Dom0 has crashed during reboot and I've no way to remotely
reboot it (no remote power control and it seems that Magic SysRQ are not
transfered through my Putty/screen connection).

Will reboot it when at home and continue investigation with another
network card and maybe another operating system to check if bad
behaviour is still there.

Regards,
JB


Le 12/01/2011 23:18, Jean Baptiste Favre a écrit :
> Ok, my bad. Screen IS available on OpenWRT.
> I've added it to my disk image and now have a tcpdump capture of network
> traffic of my DomU
> 
> Have the same network capture from my GW so I 'll be able to investigate
> deeper.
> 
> Let me know if you have specific points you want me to look at.
> 
> Regards,
> JB
> 
> 
> Le 12/01/2011 22:46, Jean Baptiste Favre a écrit :
>> Le 12/01/2011 22:40, Konrad Rzeszutek Wilk a écrit :
>>> On Wed, Jan 12, 2011 at 09:07:30PM +0100, Jean Baptiste Favre wrote:
>>>> Le 12/01/2011 19:32, Konrad Rzeszutek Wilk a écrit :
>>>>>>
>>>>>> Finally, I think that my setup shall not be responsible for the fact
>>>>>> that ping works, small DNS answers are received but not bigger ones
>>>>>
>>>>> Aah, somehow I missed that fact.
>>>>
>>>> Seems yes :)
>>>>
>>>>>> (limit seems to be around 100 bytes).
>>>>>
>>>>> What does wireshark or tcpdump tell you when you send the larger packets?
>>>>
>>>> That's one of my problem: nothing special :(
>>>>
>>>> I can only run tcpdump on my gateway (no SSH available on my OpenWRT
>>>> domU and only xm console access is available).
>>>
>>> You can install (or just compile it statically) tcpdump on your OpenWRT,
>>> or just hook it up to a beffier guest..
>>
>> tcpdump is available on my OpenWRT domU, but I can not perform tcpdump
>> and DNS lookup with only console access and AFAIK screen is not
>> available on OpenWRT
>>
>>>> I see DNS requests comming in, I see DNS answers going out.
>>>> ARP requests are fine.
>>>>
>>>> As an example:
>>>> dig free.fr works
>>>> dig jbfavre.org works
>>>> dig amazon.fr works
>>>> dig amazon.com works (quite long, but works)
>>>> dig google.fr fails
>>>> dig google.com fails
>>>
>>> Fails as in response or fails halfway?
>>
>> Fails as in:
>> # dig google.com
>>
>> ; <<>> DiG 9.7.2-P3 <<>> google.com
>> ;; global options: +cmd
>> ;; connection timed out; no servers could be reached
>>
>>>>
>>>> >From another machine in the same LAN, all requests work fine.
>>>>
>>>> Thought this was due to some ARP restriction setted up by default in
>>>> OpenWRT, but in this case, I can not explain why some DNS lookup do work
>>>> and some don't.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
My dom0 is back and I performed some more tests.

I told in my first mail that ping works. Indeed it works, but not always:
# ping -c2 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: seq=0 ttl=64 time=0.846 ms
64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.824 ms

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.824/0.835/0.846 ms
# ping -c2 -s60 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 60 data bytes
68 bytes from 10.0.0.1: seq=0 ttl=64 time=0.819 ms
68 bytes from 10.0.0.1: seq=1 ttl=64 time=0.807 ms

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.807/0.813/0.819 ms

Increasing packet size is ok until this one:
# ping -c2 -s85 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 85 data bytes
93 bytes from 10.0.0.1: seq=0 ttl=64 time=0.823 ms
93 bytes from 10.0.0.1: seq=1 ttl=64 time=0.816 ms

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.816/0.819/0.823 ms
root@OpenWrt:/# ping -c2 -s86 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 86 data bytes

--- 10.0.0.1 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

As you see, packet size seems to be limited in a way. From another
machine on the same LAN I can do ping -s1500 without any problem.

So I think I hit a bug. Either it's Xen related, or Debian (through
debian kernel version). Now the question is: how can I determine which
part is responsible ?

Regards,
JB


Le 13/01/2011 12:28, Jean Baptiste Favre a écrit :
> Have had a look on both tcpdump captures.
> 1. From the OpenWRT domU: nothing special. Lot of STP traffic due to my
> network architecture. Don't thing this could be the cause of my problem
> since I'm always able to reproduce the failure whatever could be the
> order of DNS requests. Don't see "big" DNS answers coming in, see
> multiple requests before timeout.
> 
> 2. From my gateway: nothing special here too. Same STP traffic (have
> checked GW network setup as well). See each DNS requests my domU sent
> and can see as well answers, which are not received by domU. No strange
> ARP traffic.
> 
> I wanted to change the network, thinking about a bogus driver (never
> know). My Dom0 has crashed during reboot and I've no way to remotely
> reboot it (no remote power control and it seems that Magic SysRQ are not
> transfered through my Putty/screen connection).
> 
> Will reboot it when at home and continue investigation with another
> network card and maybe another operating system to check if bad
> behaviour is still there.
> 
> Regards,
> JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Thu, Jan 13, 2011 at 08:18:33PM +0100, Jean Baptiste Favre wrote:
> Hello,
> My dom0 is back and I performed some more tests.
> 
> I told in my first mail that ping works. Indeed it works, but not always:
> # ping -c2 10.0.0.1
> PING 10.0.0.1 (10.0.0.1): 56 data bytes
> 64 bytes from 10.0.0.1: seq=0 ttl=64 time=0.846 ms
> 64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.824 ms
> 
> --- 10.0.0.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max = 0.824/0.835/0.846 ms
> # ping -c2 -s60 10.0.0.1
> PING 10.0.0.1 (10.0.0.1): 60 data bytes
> 68 bytes from 10.0.0.1: seq=0 ttl=64 time=0.819 ms
> 68 bytes from 10.0.0.1: seq=1 ttl=64 time=0.807 ms
> 
> --- 10.0.0.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max = 0.807/0.813/0.819 ms
> 
> Increasing packet size is ok until this one:
> # ping -c2 -s85 10.0.0.1
> PING 10.0.0.1 (10.0.0.1): 85 data bytes
> 93 bytes from 10.0.0.1: seq=0 ttl=64 time=0.823 ms
> 93 bytes from 10.0.0.1: seq=1 ttl=64 time=0.816 ms
> 
> --- 10.0.0.1 ping statistics ---
> 2 packets transmitted, 2 packets received, 0% packet loss
> round-trip min/avg/max = 0.816/0.819/0.823 ms
> root@OpenWrt:/# ping -c2 -s86 10.0.0.1
> PING 10.0.0.1 (10.0.0.1): 86 data bytes
> 
> --- 10.0.0.1 ping statistics ---
> 2 packets transmitted, 0 packets received, 100% packet loss
> 
> As you see, packet size seems to be limited in a way. From another
> machine on the same LAN I can do ping -s1500 without any problem.

One thing that I just thought (which I keep on forgetting to do).
You did set 'iommu=soft' in your Linux guest, right?

> 
> So I think I hit a bug. Either it's Xen related, or Debian (through
> debian kernel version). Now the question is: how can I determine which
> part is responsible ?

What does tcpdump tell you when you try to send it at -s86?

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 13/01/2011 21:19, Konrad Rzeszutek Wilk a écrit :
> On Thu, Jan 13, 2011 at 08:18:33PM +0100, Jean Baptiste Favre wrote:
>> Hello,
>> My dom0 is back and I performed some more tests.
>>
>> I told in my first mail that ping works. Indeed it works, but not always:
>> # ping -c2 10.0.0.1
>> PING 10.0.0.1 (10.0.0.1): 56 data bytes
>> 64 bytes from 10.0.0.1: seq=0 ttl=64 time=0.846 ms
>> 64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.824 ms
>>
>> --- 10.0.0.1 ping statistics ---
>> 2 packets transmitted, 2 packets received, 0% packet loss
>> round-trip min/avg/max = 0.824/0.835/0.846 ms
>> # ping -c2 -s60 10.0.0.1
>> PING 10.0.0.1 (10.0.0.1): 60 data bytes
>> 68 bytes from 10.0.0.1: seq=0 ttl=64 time=0.819 ms
>> 68 bytes from 10.0.0.1: seq=1 ttl=64 time=0.807 ms
>>
>> --- 10.0.0.1 ping statistics ---
>> 2 packets transmitted, 2 packets received, 0% packet loss
>> round-trip min/avg/max = 0.807/0.813/0.819 ms
>>
>> Increasing packet size is ok until this one:
>> # ping -c2 -s85 10.0.0.1
>> PING 10.0.0.1 (10.0.0.1): 85 data bytes
>> 93 bytes from 10.0.0.1: seq=0 ttl=64 time=0.823 ms
>> 93 bytes from 10.0.0.1: seq=1 ttl=64 time=0.816 ms
>>
>> --- 10.0.0.1 ping statistics ---
>> 2 packets transmitted, 2 packets received, 0% packet loss
>> round-trip min/avg/max = 0.816/0.819/0.823 ms
>> root@OpenWrt:/# ping -c2 -s86 10.0.0.1
>> PING 10.0.0.1 (10.0.0.1): 86 data bytes
>>
>> --- 10.0.0.1 ping statistics ---
>> 2 packets transmitted, 0 packets received, 100% packet loss
>>
>> As you see, packet size seems to be limited in a way. From another
>> machine on the same LAN I can do ping -s1500 without any problem.
> 
> One thing that I just thought (which I keep on forgetting to do).
> You did set 'iommu=soft' in your Linux guest, right?

Tought I told it in my previou smails. Sorry to missed it:

On my dom0:
$ cat /proc/cmdline
placeholder root=/dev/mapper/system-root ro console=hvc0 earlyprintk=xen
nomodeset xen-pciback.permissive xen-pciback.hide=(04:00.0)
pci=resource_alignment=04:00.0 quiet

Xen hypervisor options are:
# xm dmesg
(XEN) Xen version 4.0.1 (Debian 4.0.1-1) (waldi@debian.org) (gcc version
4.4.5 20100824 (prerelease) (Debian 4.4.4-11) ) Fri Sep  3 15:38:12 UTC 2010
(XEN) Bootloader: GRUB 1.98+20100804-11
(XEN) Command line: placeholder dom0_mem=256M dom0_max_vcpus=1
dom0_vcpus_pin loglvl=all guest_loglvl=all com1=115200,8n1 console=com1

OpenWRT domU config file is:
$ cat /etc/xen/auto/openwrt.cfg

kernel       = '/home/domU/wrt/openwrt-x86-xen_domu-vmlinuz'
root         = '/dev/xvda2 rw'
memory       = '256'
vcpus        = '1'
cpus         = '1'
localtime    = 0
serial       = 'pty'
disk = ['file:/home/domU/wrt/openwrt-x86-xen_domu-combined-ext4.img,xvda,w']

#vif = [ 'bridge=br-wan, mac=00:16:3E:01:00:64, vifname=wrt-100.eth1' ]

on_poweroff  = 'destroy'
on_reboot    = 'restart'
on_crash     = 'restart'

extra = "iommu=soft swiotlb=force console=hvc0 xencons=tty"
pci = [ '04:00.0' ]

name         = 'openwrt'
hostname     = 'openwrt.clichy.jbfavre.org'


>> So I think I hit a bug. Either it's Xen related, or Debian (through
>> debian kernel version). Now the question is: how can I determine which
>> part is responsible ?
> 
> What does tcpdump tell you when you try to send it at -s86?

I can see echo requests coming in on my gateway, replies going back but
replies are never received on the domU.

Regards,
JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

> Tought I told it in my previou smails. Sorry to missed it:

You probably did and I missed it too. Good that you have those options, now:

.. snip ..
> > What does tcpdump tell you when you try to send it at -s86?
> 
> I can see echo requests coming in on my gateway, replies going back but
> replies are never received on the domU.

What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like?
Can you send me the full domU output? What does the Xen hypervisor mapping look like
(xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have?
Did you look up to see if there are any errate for the motherboard or the NIC you are using?
Do you have the latest firmware for the NIC and the latest BIOS for your motherboard?
Is the OpenWRT kernel you are using one that you built yourself or do they package it
for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect).
Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be
able to run a 64-bit kernel alongside 32-bit userspace).

Re: PCI passthrough issue

[Jean Baptiste Favre]

[-- Attachment #1: Type: text/plain, Size: 3296 bytes --]

Hello,

Le 14/01/2011 15:53, Konrad Rzeszutek Wilk a écrit :
>> Tought I told it in my previou smails. Sorry to missed it:
> 
> You probably did and I missed it too. Good that you have those options, now:
> 
> .. snip ..
>>> What does tcpdump tell you when you try to send it at -s86?
>>
>> I can see echo requests coming in on my gateway, replies going back but
>> replies are never received on the domU.
> 
> What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like?

The NIC I use is "Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit
Ethernet Controller", not Intel, with sky2 driver.

# cat /proc/interrupts
           CPU0
 55:       1393  xen-pirq-pcifront-msi  sky2@pci:0000:00:00.0
246:        202   xen-dyn-event     blkif
247:        103   xen-dyn-event     hvc_console
248:        603   xen-dyn-event     pcifront
249:        259   xen-dyn-event     xenbus
250:          0  xen-percpu-ipi       callfuncsingle0
251:          0  xen-percpu-virq      debug0
252:          0  xen-percpu-ipi       callfunc0
253:          0  xen-percpu-ipi       resched0
254:          0  xen-percpu-ipi       spinlock0
255:     246058  xen-percpu-virq      timer0
NMI:          0   Non-maskable interrupts
LOC:          0   Local timer interrupts
SPU:          0   Spurious interrupts
PMI:          0   Performance monitoring interrupts
IWI:          0   IRQ work interrupts
RES:          0   Rescheduling interrupts
CAL:          0   Function call interrupts
TLB:          0   TLB shootdowns
TRM:          0   Thermal event interrupts
THR:          0   Threshold APIC interrupts
MCE:          0   Machine check exceptions
MCP:          0   Machine check polls
ERR:          0
MIS:          0


> Can you send me the full domU output? What does the Xen hypervisor mapping look like
Output are attached to this mail.

> (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have?
dmidecode
...
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: ASUSTeK Computer INC.
        Product Name: M3N18L T-M3N8200
        Version: Rev x.xx
        Serial Number: MS1C85B07000633
        Asset Tag: To Be Filled By O.E.M.
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis: To Be Filled By O.E.M.
        Chassis Handle: 0x0003
        Type: Motherboard
        Contained Object Handles: 0
...

> Did you look up to see if there are any errate for the motherboard or the NIC you are using?
No, since everything was working fine (before I decided to test PCI
passthrough).

> Do you have the latest firmware for the NIC and the latest BIOS for your motherboard?
Not sure. Will check

> Is the OpenWRT kernel you are using one that you built yourself or do they package it
> for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect).
It is packaged by openWRT Builroot. Just have to set the kernel version
I want to use.
Will try a vanilla kernel as soon as possible.

> Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be
> able to run a 64-bit kernel alongside 32-bit userspace).
OK, will try it as well.

Regards,
JB

[-- Attachment #2: xen_debug_keys_i.txt --]
[-- Type: text/plain, Size: 8332 bytes --]

(XEN) Guest interrupt information:
(XEN)    IRQ:   0 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:f0 type=IO-APIC-edge    status=00000000 mapped, unbound
(XEN)    IRQ:   1 affinity:00000000,00000000,00000000,00000001 vec:28 type=IO-APIC-edge    status=00000010 in-flight=0 domain-list=0:  1(----),
(XEN)    IRQ:   2 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:e2 type=XT-PIC          status=00000000 mapped, unbound
(XEN)    IRQ:   3 affinity:00000000,00000000,00000000,00000001 vec:30 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   4 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:f1 type=IO-APIC-edge    status=00000000 mapped, unbound
(XEN)    IRQ:   5 affinity:00000000,00000000,00000000,00000001 vec:38 type=IO-APIC-edge    status=00000006 mapped, unbound
(XEN)    IRQ:   6 affinity:00000000,00000000,00000000,00000001 vec:40 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   7 affinity:00000000,00000000,00000000,00000001 vec:48 type=IO-APIC-edge    status=00000006 mapped, unbound
(XEN)    IRQ:   8 affinity:00000000,00000000,00000000,00000001 vec:50 type=IO-APIC-edge    status=00000010 in-flight=0 domain-list=0:  8(----),
(XEN)    IRQ:   9 affinity:00000000,00000000,00000000,00000001 vec:58 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0:  9(----),
(XEN)    IRQ:  10 affinity:00000000,00000000,00000000,00000001 vec:60 type=IO-APIC-edge    status=00000006 mapped, unbound
(XEN)    IRQ:  11 affinity:00000000,00000000,00000000,00000001 vec:68 type=IO-APIC-edge    status=00000006 mapped, unbound
(XEN)    IRQ:  12 affinity:00000000,00000000,00000000,00000001 vec:70 type=IO-APIC-edge    status=00000010 in-flight=0 domain-list=0: 12(----),
(XEN)    IRQ:  13 affinity:00000000,00000000,00000000,0000000f vec:78 type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  14 affinity:00000000,00000000,00000000,00000001 vec:88 type=IO-APIC-edge    status=00000010 in-flight=0 domain-list=0: 14(----),
(XEN)    IRQ:  15 affinity:00000000,00000000,00000000,00000001 vec:90 type=IO-APIC-edge    status=00000010 in-flight=0 domain-list=0: 15(----),
(XEN)    IRQ:  16 affinity:00000000,00000000,00000000,0000000f vec:b0 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  17 affinity:00000000,00000000,00000000,0000000f vec:a8 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  18 affinity:00000000,00000000,00000000,00000001 vec:a0 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 18(----),
(XEN)    IRQ:  19 affinity:00000000,00000000,00000000,00000001 vec:98 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 19(----),
(XEN)    IRQ:  20 affinity:00000000,00000000,00000000,00000001 vec:41 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 20(----),
(XEN)    IRQ:  21 affinity:00000000,00000000,00000000,00000001 vec:39 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 21(----),
(XEN)    IRQ:  22 affinity:00000000,00000000,00000000,0000000f vec:29 type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  23 affinity:00000000,00000000,00000000,00000001 vec:d8 type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0: 23(----),
(XEN)    IRQ:  24 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b8 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  25 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c0 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  26 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c8 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  27 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:d0 type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  28 affinity:00000000,00000000,00000000,00000001 vec:59 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:275(----),
(XEN)    IRQ:  29 affinity:00000000,00000000,00000000,00000001 vec:31 type=PCI-MSI         status=00000014 in-flight=0 domain-list=0:274(----),
(XEN)    IRQ:  30 affinity:00000000,00000000,00000000,00000001 vec:69 type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:273(----),
(XEN)    IRQ:  31 affinity:00000000,00000000,00000000,00000002 vec:d9 type=PCI-MSI         status=00000010 in-flight=0 domain-list=21: 55(----),
(XEN) IO-APIC interrupt information:
(XEN)     IRQ  0 Vec240:
(XEN)       Apic 0x00, Pin  2: vector=240, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:255
(XEN)     IRQ  1 Vec 40:
(XEN)       Apic 0x00, Pin  1: vector=40, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  3 Vec 48:
(XEN)       Apic 0x00, Pin  3: vector=48, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  4 Vec241:
(XEN)       Apic 0x00, Pin  4: vector=241, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:255
(XEN)     IRQ  5 Vec 56:
(XEN)       Apic 0x00, Pin  5: vector=56, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  6 Vec 64:
(XEN)       Apic 0x00, Pin  6: vector=64, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  7 Vec 72:
(XEN)       Apic 0x00, Pin  7: vector=72, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  8 Vec 80:
(XEN)       Apic 0x00, Pin  8: vector=80, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ  9 Vec 88:
(XEN)       Apic 0x00, Pin  9: vector=88, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 10 Vec 96:
(XEN)       Apic 0x00, Pin 10: vector=96, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 11 Vec104:
(XEN)       Apic 0x00, Pin 11: vector=104, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 12 Vec112:
(XEN)       Apic 0x00, Pin 12: vector=112, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 13 Vec120:
(XEN)       Apic 0x00, Pin 13: vector=120, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=1, dest_id:15
(XEN)     IRQ 14 Vec136:
(XEN)       Apic 0x00, Pin 14: vector=136, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 15 Vec144:
(XEN)       Apic 0x00, Pin 15: vector=144, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=0, irr=0, trigger=edge, mask=0, dest_id:1
(XEN)     IRQ 16 Vec176:
(XEN)       Apic 0x00, Pin 16: vector=176, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:15
(XEN)     IRQ 17 Vec168:
(XEN)       Apic 0x00, Pin 17: vector=168, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:15
(XEN)     IRQ 18 Vec160:
(XEN)       Apic 0x00, Pin 18: vector=160, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 19 Vec152:
(XEN)       Apic 0x00, Pin 19: vector=152, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 20 Vec 65:
(XEN)       Apic 0x00, Pin 20: vector=65, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 21 Vec 57:
(XEN)       Apic 0x00, Pin 21: vector=57, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1
(XEN)     IRQ 22 Vec 41:
(XEN)       Apic 0x00, Pin 22: vector=41, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=1, dest_id:15
(XEN)     IRQ 23 Vec216:
(XEN)       Apic 0x00, Pin 23: vector=216, delivery_mode=1, dest_mode=logical, delivery_status=0, polarity=1, irr=0, trigger=level, mask=0, dest_id:1

[-- Attachment #3: xen_debug_keys_q.txt --]
[-- Type: text/plain, Size: 2312 bytes --]

(XEN) 'q' pressed -> dumping domain info (now=0x5F00:7A49A1F4)
(XEN) General information for domain 0:
(XEN)     refcnt=3 dying=0 nr_pages=64576 xenheap_pages=5 dirty_cpus={0} max_pages=4294967295
(XEN)     handle=00000000-0000-0000-0000-000000000000 vm_assist=0000000d
(XEN) Rangesets belonging to domain 0:
(XEN)     I/O Ports  { 0-1f, 22-3f, 44-60, 62-9f, a2-3f7, 400-507, 50c-cfb, d00-ffff }
(XEN)     Interrupts { 0-279 }
(XEN)     I/O Memory { 0-febff, fec01-fedff, fee01-ffffffffffffffff }
(XEN) Memory pages belonging to domain 0:
(XEN)     DomPage list too long to display
(XEN)     XenPage 000000000023fcda: caf=c000000000000002, taf=7400000000000002
(XEN)     XenPage 000000000023fcd9: caf=c000000000000001, taf=7400000000000001
(XEN)     XenPage 000000000023fcd8: caf=c000000000000001, taf=7400000000000001
(XEN)     XenPage 000000000023fcd7: caf=c000000000000001, taf=7400000000000001
(XEN)     XenPage 00000000000b7afe: caf=c000000000000002, taf=7400000000000002
(XEN) VCPU information and callbacks for domain 0:
(XEN)     VCPU0: CPU0 [has=T] flags=0 poll=0 upcall_pend = 00, upcall_mask = 00 dirty_cpus={0} cpu_affinity={0}
(XEN)     No periodic timer
(XEN) General information for domain 21:
(XEN)     refcnt=3 dying=0 nr_pages=65536 xenheap_pages=5 dirty_cpus={1} max_pages=65536
(XEN)     handle=a9e1d4f5-b66f-c785-aaf0-48186e5e1e20 vm_assist=0000000d
(XEN) Rangesets belonging to domain 21:
(XEN)     I/O Ports  { e800-e8ff }
(XEN)     Interrupts { 18, 55 }
(XEN)     I/O Memory { 10100-1011f, feb00-feb03 }
(XEN) Memory pages belonging to domain 21:
(XEN)     DomPage list too long to display
(XEN)     XenPage 000000000022b8e1: caf=c000000000000002, taf=7400000000000002
(XEN)     XenPage 000000000022bd6b: caf=c000000000000001, taf=7400000000000001
(XEN)     XenPage 0000000000236521: caf=c000000000000001, taf=7400000000000001
(XEN)     XenPage 000000000022bbbc: caf=c000000000000001, taf=7400000000000001
(XEN)     XenPage 00000000000b7cbf: caf=c000000000000002, taf=7400000000000002
(XEN) VCPU information and callbacks for domain 21:
(XEN)     VCPU0: CPU1 [has=F] flags=1 poll=0 upcall_pend = 00, upcall_mask = 00 dirty_cpus={1} cpu_affinity={1}
(XEN)     No periodic timer
(XEN) Notifying guest 0:0 (virq 1, port 4, stat 0/0/0)
(XEN) Notifying guest 21:0 (virq 1, port 7, stat 0/0/0)

[-- Attachment #4: xen_debug_keys_Q.txt --]
[-- Type: text/plain, Size: 1032 bytes --]

(XEN) ==== PCI devices ====
(XEN) 04:00.0 - dom 0   - MSIs < 31 >
(XEN) 03:00.1 - dom 0   - MSIs < 30 >
(XEN) 03:00.0 - dom 0   - MSIs < 28 >
(XEN) 02:00.0 - dom 0   - MSIs < >
(XEN) 01:05.0 - dom 0   - MSIs < >
(XEN) 00:18.4 - dom 0   - MSIs < >
(XEN) 00:18.3 - dom 0   - MSIs < >
(XEN) 00:18.2 - dom 0   - MSIs < >
(XEN) 00:18.1 - dom 0   - MSIs < >
(XEN) 00:18.0 - dom 0   - MSIs < >
(XEN) 00:14.0 - dom 0   - MSIs < 27 >
(XEN) 00:13.0 - dom 0   - MSIs < 26 >
(XEN) 00:12.0 - dom 0   - MSIs < 25 >
(XEN) 00:10.0 - dom 0   - MSIs < 24 >
(XEN) 00:0b.0 - dom 0   - MSIs < >
(XEN) 00:09.0 - dom 0   - MSIs < 29 >
(XEN) 00:08.0 - dom 0   - MSIs < >
(XEN) 00:06.0 - dom 0   - MSIs < >
(XEN) 00:04.1 - dom 0   - MSIs < >
(XEN) 00:04.0 - dom 0   - MSIs < >
(XEN) 00:02.1 - dom 0   - MSIs < >
(XEN) 00:02.0 - dom 0   - MSIs < >
(XEN) 00:01.4 - dom 0   - MSIs < >
(XEN) 00:01.3 - dom 0   - MSIs < >
(XEN) 00:01.2 - dom 0   - MSIs < >
(XEN) 00:01.1 - dom 0   - MSIs < >
(XEN) 00:01.0 - dom 0   - MSIs < >
(XEN) 00:00.0 - dom 0   - MSIs < >

[-- Attachment #5: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
Has made little tests this week-end.
- Can not use full vanilla kernel with OpenWRT. Must apply some patch to
make OpenWRT boot and don't had enough time to figure out which one
exactly. Will work on it this week.
- Have tested a Debian PV 64bits domU with 2.6.37 kernel from Debian
experimental repository. Network is fully working. Will try a 32bits
domU this week

So, at this point, either because of 64bits, either because of "better"
kernel, network with PCI passthrough is working.

Still have to try 32bits kernel, as well as filtering OpenWRT patches.

Regards,
JB


Le 15/01/2011 00:29, Jean Baptiste Favre a écrit :
> Hello,
> 
> Le 14/01/2011 15:53, Konrad Rzeszutek Wilk a écrit :
>>> Tought I told it in my previou smails. Sorry to missed it:
>>
>> You probably did and I missed it too. Good that you have those options, now:
>>
>> .. snip ..
>>>> What does tcpdump tell you when you try to send it at -s86?
>>>
>>> I can see echo requests coming in on my gateway, replies going back but
>>> replies are never received on the domU.
>>
>> What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like?
> 
> The NIC I use is "Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit
> Ethernet Controller", not Intel, with sky2 driver.
> 
> # cat /proc/interrupts
>            CPU0
>  55:       1393  xen-pirq-pcifront-msi  sky2@pci:0000:00:00.0
> 246:        202   xen-dyn-event     blkif
> 247:        103   xen-dyn-event     hvc_console
> 248:        603   xen-dyn-event     pcifront
> 249:        259   xen-dyn-event     xenbus
> 250:          0  xen-percpu-ipi       callfuncsingle0
> 251:          0  xen-percpu-virq      debug0
> 252:          0  xen-percpu-ipi       callfunc0
> 253:          0  xen-percpu-ipi       resched0
> 254:          0  xen-percpu-ipi       spinlock0
> 255:     246058  xen-percpu-virq      timer0
> NMI:          0   Non-maskable interrupts
> LOC:          0   Local timer interrupts
> SPU:          0   Spurious interrupts
> PMI:          0   Performance monitoring interrupts
> IWI:          0   IRQ work interrupts
> RES:          0   Rescheduling interrupts
> CAL:          0   Function call interrupts
> TLB:          0   TLB shootdowns
> TRM:          0   Thermal event interrupts
> THR:          0   Threshold APIC interrupts
> MCE:          0   Machine check exceptions
> MCP:          0   Machine check polls
> ERR:          0
> MIS:          0
> 
> 
>> Can you send me the full domU output? What does the Xen hypervisor mapping look like
> Output are attached to this mail.
> 
>> (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have?
> dmidecode
> ...
> Handle 0x0002, DMI type 2, 15 bytes
> Base Board Information
>         Manufacturer: ASUSTeK Computer INC.
>         Product Name: M3N18L T-M3N8200
>         Version: Rev x.xx
>         Serial Number: MS1C85B07000633
>         Asset Tag: To Be Filled By O.E.M.
>         Features:
>                 Board is a hosting board
>                 Board is replaceable
>         Location In Chassis: To Be Filled By O.E.M.
>         Chassis Handle: 0x0003
>         Type: Motherboard
>         Contained Object Handles: 0
> ...
> 
>> Did you look up to see if there are any errate for the motherboard or the NIC you are using?
> No, since everything was working fine (before I decided to test PCI
> passthrough).
> 
>> Do you have the latest firmware for the NIC and the latest BIOS for your motherboard?
> Not sure. Will check
> 
>> Is the OpenWRT kernel you are using one that you built yourself or do they package it
>> for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect).
> It is packaged by openWRT Builroot. Just have to set the kernel version
> I want to use.
> Will try a vanilla kernel as soon as possible.
> 
>> Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be
>> able to run a 64-bit kernel alongside 32-bit userspace).
> OK, will try it as well.
> 
> Regards,
> JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello Konrad,
Have tested with Debian 32bits domU. Once domU is started, log in via xm
console and exec "ping 10.0.0.1" and "ping -s86 10.0.0.1"
Here are the results:

- 32bits kernel + 32bits Debian Squeeze => ping -s86 FAILS
- 64bits kernel + 32bits Debian Squeeze => WORKS
- 64bits kernel + 64bits Debian Squeeze => WORKS

Used kernels are:
32bit: linux-image-2.6.37-trunk-686-bigmem
64bit: linux-image-2.6.37-trunk-amd64

Will now try to compile openwrt kernel as 64bits and update you.

Regards,
JB

Le 17/01/2011 09:59, Jean Baptiste Favre a écrit :
> Hello,
> Has made little tests this week-end.
> - Can not use full vanilla kernel with OpenWRT. Must apply some patch to
> make OpenWRT boot and don't had enough time to figure out which one
> exactly. Will work on it this week.
> - Have tested a Debian PV 64bits domU with 2.6.37 kernel from Debian
> experimental repository. Network is fully working. Will try a 32bits
> domU this week
> 
> So, at this point, either because of 64bits, either because of "better"
> kernel, network with PCI passthrough is working.
> 
> Still have to try 32bits kernel, as well as filtering OpenWRT patches.
> 
> Regards,
> JB
> 
> 
> Le 15/01/2011 00:29, Jean Baptiste Favre a écrit :
>> Hello,
>>
>> Le 14/01/2011 15:53, Konrad Rzeszutek Wilk a écrit :
>>>> Tought I told it in my previou smails. Sorry to missed it:
>>>
>>> You probably did and I missed it too. Good that you have those options, now:
>>>
>>> .. snip ..
>>>>> What does tcpdump tell you when you try to send it at -s86?
>>>>
>>>> I can see echo requests coming in on my gateway, replies going back but
>>>> replies are never received on the domU.
>>>
>>> What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like?
>>
>> The NIC I use is "Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit
>> Ethernet Controller", not Intel, with sky2 driver.
>>
>> # cat /proc/interrupts
>>            CPU0
>>  55:       1393  xen-pirq-pcifront-msi  sky2@pci:0000:00:00.0
>> 246:        202   xen-dyn-event     blkif
>> 247:        103   xen-dyn-event     hvc_console
>> 248:        603   xen-dyn-event     pcifront
>> 249:        259   xen-dyn-event     xenbus
>> 250:          0  xen-percpu-ipi       callfuncsingle0
>> 251:          0  xen-percpu-virq      debug0
>> 252:          0  xen-percpu-ipi       callfunc0
>> 253:          0  xen-percpu-ipi       resched0
>> 254:          0  xen-percpu-ipi       spinlock0
>> 255:     246058  xen-percpu-virq      timer0
>> NMI:          0   Non-maskable interrupts
>> LOC:          0   Local timer interrupts
>> SPU:          0   Spurious interrupts
>> PMI:          0   Performance monitoring interrupts
>> IWI:          0   IRQ work interrupts
>> RES:          0   Rescheduling interrupts
>> CAL:          0   Function call interrupts
>> TLB:          0   TLB shootdowns
>> TRM:          0   Thermal event interrupts
>> THR:          0   Threshold APIC interrupts
>> MCE:          0   Machine check exceptions
>> MCP:          0   Machine check polls
>> ERR:          0
>> MIS:          0
>>
>>
>>> Can you send me the full domU output? What does the Xen hypervisor mapping look like
>> Output are attached to this mail.
>>
>>> (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have?
>> dmidecode
>> ...
>> Handle 0x0002, DMI type 2, 15 bytes
>> Base Board Information
>>         Manufacturer: ASUSTeK Computer INC.
>>         Product Name: M3N18L T-M3N8200
>>         Version: Rev x.xx
>>         Serial Number: MS1C85B07000633
>>         Asset Tag: To Be Filled By O.E.M.
>>         Features:
>>                 Board is a hosting board
>>                 Board is replaceable
>>         Location In Chassis: To Be Filled By O.E.M.
>>         Chassis Handle: 0x0003
>>         Type: Motherboard
>>         Contained Object Handles: 0
>> ...
>>
>>> Did you look up to see if there are any errate for the motherboard or the NIC you are using?
>> No, since everything was working fine (before I decided to test PCI
>> passthrough).
>>
>>> Do you have the latest firmware for the NIC and the latest BIOS for your motherboard?
>> Not sure. Will check
>>
>>> Is the OpenWRT kernel you are using one that you built yourself or do they package it
>>> for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect).
>> It is packaged by openWRT Builroot. Just have to set the kernel version
>> I want to use.
>> Will try a vanilla kernel as soon as possible.
>>
>>> Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be
>>> able to run a 64-bit kernel alongside 32-bit userspace).
>> OK, will try it as well.
>>
>> Regards,
>> JB
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
Last investigations show that I've the latest BIOS version for my
motherboard.
Do you need more tests, if yes which ones ?

This problem annoys me because I did not manage to get OpenWRT working
with 64bits kernel.

Regards,
JB


Le 17/01/2011 14:58, Jean Baptiste Favre a écrit :
> Hello Konrad,
> Have tested with Debian 32bits domU. Once domU is started, log in via xm
> console and exec "ping 10.0.0.1" and "ping -s86 10.0.0.1"
> Here are the results:
> 
> - 32bits kernel + 32bits Debian Squeeze => ping -s86 FAILS
> - 64bits kernel + 32bits Debian Squeeze => WORKS
> - 64bits kernel + 64bits Debian Squeeze => WORKS
> 
> Used kernels are:
> 32bit: linux-image-2.6.37-trunk-686-bigmem
> 64bit: linux-image-2.6.37-trunk-amd64
> 
> Will now try to compile openwrt kernel as 64bits and update you.
> 
> Regards,
> JB
> 
> Le 17/01/2011 09:59, Jean Baptiste Favre a écrit :
>> Hello,
>> Has made little tests this week-end.
>> - Can not use full vanilla kernel with OpenWRT. Must apply some patch to
>> make OpenWRT boot and don't had enough time to figure out which one
>> exactly. Will work on it this week.
>> - Have tested a Debian PV 64bits domU with 2.6.37 kernel from Debian
>> experimental repository. Network is fully working. Will try a 32bits
>> domU this week
>>
>> So, at this point, either because of 64bits, either because of "better"
>> kernel, network with PCI passthrough is working.
>>
>> Still have to try 32bits kernel, as well as filtering OpenWRT patches.
>>
>> Regards,
>> JB
>>
>>
>> Le 15/01/2011 00:29, Jean Baptiste Favre a écrit :
>>> Hello,
>>>
>>> Le 14/01/2011 15:53, Konrad Rzeszutek Wilk a écrit :
>>>>> Tought I told it in my previou smails. Sorry to missed it:
>>>>
>>>> You probably did and I missed it too. Good that you have those options, now:
>>>>
>>>> .. snip ..
>>>>>> What does tcpdump tell you when you try to send it at -s86?
>>>>>
>>>>> I can see echo requests coming in on my gateway, replies going back but
>>>>> replies are never received on the domU.
>>>>
>>>> What is the NIC you are using? Is it a Intel one? What does /proc/interrupts look like?
>>>
>>> The NIC I use is "Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit
>>> Ethernet Controller", not Intel, with sky2 driver.
>>>
>>> # cat /proc/interrupts
>>>            CPU0
>>>  55:       1393  xen-pirq-pcifront-msi  sky2@pci:0000:00:00.0
>>> 246:        202   xen-dyn-event     blkif
>>> 247:        103   xen-dyn-event     hvc_console
>>> 248:        603   xen-dyn-event     pcifront
>>> 249:        259   xen-dyn-event     xenbus
>>> 250:          0  xen-percpu-ipi       callfuncsingle0
>>> 251:          0  xen-percpu-virq      debug0
>>> 252:          0  xen-percpu-ipi       callfunc0
>>> 253:          0  xen-percpu-ipi       resched0
>>> 254:          0  xen-percpu-ipi       spinlock0
>>> 255:     246058  xen-percpu-virq      timer0
>>> NMI:          0   Non-maskable interrupts
>>> LOC:          0   Local timer interrupts
>>> SPU:          0   Spurious interrupts
>>> PMI:          0   Performance monitoring interrupts
>>> IWI:          0   IRQ work interrupts
>>> RES:          0   Rescheduling interrupts
>>> CAL:          0   Function call interrupts
>>> TLB:          0   TLB shootdowns
>>> TRM:          0   Thermal event interrupts
>>> THR:          0   Threshold APIC interrupts
>>> MCE:          0   Machine check exceptions
>>> MCP:          0   Machine check polls
>>> ERR:          0
>>> MIS:          0
>>>
>>>
>>>> Can you send me the full domU output? What does the Xen hypervisor mapping look like
>>> Output are attached to this mail.
>>>
>>>> (xm debug-keys q, xm debug-keys Q, xm debug-keys i)? What is the motherboard you have?
>>> dmidecode
>>> ...
>>> Handle 0x0002, DMI type 2, 15 bytes
>>> Base Board Information
>>>         Manufacturer: ASUSTeK Computer INC.
>>>         Product Name: M3N18L T-M3N8200
>>>         Version: Rev x.xx
>>>         Serial Number: MS1C85B07000633
>>>         Asset Tag: To Be Filled By O.E.M.
>>>         Features:
>>>                 Board is a hosting board
>>>                 Board is replaceable
>>>         Location In Chassis: To Be Filled By O.E.M.
>>>         Chassis Handle: 0x0003
>>>         Type: Motherboard
>>>         Contained Object Handles: 0
>>> ...
>>>
>>>> Did you look up to see if there are any errate for the motherboard or the NIC you are using?
>>> No, since everything was working fine (before I decided to test PCI
>>> passthrough).
>>>
>>>> Do you have the latest firmware for the NIC and the latest BIOS for your motherboard?
>>> Not sure. Will check
>>>
>>>> Is the OpenWRT kernel you are using one that you built yourself or do they package it
>>>> for you? Can you try using a vaniall built one (2.6.37 vanilla is perfect).
>>> It is packaged by openWRT Builroot. Just have to set the kernel version
>>> I want to use.
>>> Will try a vanilla kernel as soon as possible.
>>>
>>>> Is this a 64-bit kernel or 32-bit? Can you try using a 64-bit one (you should be
>>>> able to run a 64-bit kernel alongside 32-bit userspace).
>>> OK, will try it as well.
>>>
>>> Regards,
>>> JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Sat, Jan 22, 2011 at 11:22:59AM +0100, Jean Baptiste Favre wrote:
> Hello,
> Last investigations show that I've the latest BIOS version for my
> motherboard.
> Do you need more tests, if yes which ones ?

I tried it on my 2.6.32.27 (32-bit and 64-bit) and I am not seeing
the failures you have. These are the devices I passed in:

00:00.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
00:00.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
00:00.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
00:00.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
00:01.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
00:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
00:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
00:03.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)

Granted the kernel I am using a jeremy/xen/stable-2.6.32.x so not sure
how divergent from Debian or Debian Squeeze it is.

How are you launching your guest? Do you something like this:

kernel="/mnt/lab/2.6.32.27/vmlinuz"
ramdisk="/mnt/lab/2.6.32.27/initramfs.cpio.gz"
extra="console=hvc0 debug test=crashme iommu=soft"
memory=1024
maxmem=2048
vcpus=4
on_crash="preserve"
pci= ["00:1d.0","00:1d.1","00:1d.2","00:1d.7","0a:00.1","0000:06:01.1","0000:06:01.0", "09:00.0"]
vif = [ 'mac=00:0f:4b:00:00:68, bridge=switch' ]

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello Konrad,

Le 27/01/2011 21:27, Konrad Rzeszutek Wilk a écrit :
> On Sat, Jan 22, 2011 at 11:22:59AM +0100, Jean Baptiste Favre wrote:
>> Hello,
>> Last investigations show that I've the latest BIOS version for my
>> motherboard.
>> Do you need more tests, if yes which ones ?
> 
> I tried it on my 2.6.32.27 (32-bit and 64-bit) and I am not seeing
> the failures you have. These are the devices I passed in:
> 
> 00:00.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
> 00:00.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
> 00:00.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
> 00:00.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
> 00:01.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
> 00:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
> 00:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
> 00:03.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
> 
> Granted the kernel I am using a jeremy/xen/stable-2.6.32.x so not sure
> how divergent from Debian or Debian Squeeze it is.
> 
> How are you launching your guest? Do you something like this:
> 
> kernel="/mnt/lab/2.6.32.27/vmlinuz"
> ramdisk="/mnt/lab/2.6.32.27/initramfs.cpio.gz"
> extra="console=hvc0 debug test=crashme iommu=soft"
> memory=1024
> maxmem=2048
> vcpus=4
> on_crash="preserve"
> pci= ["00:1d.0","00:1d.1","00:1d.2","00:1d.7","0a:00.1","0000:06:01.1","0000:06:01.0", "09:00.0"]
> vif = [ 'mac=00:0f:4b:00:00:68, bridge=switch' ]

Here is my domU config file:
####################
kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
builder      = 'linux'

memory       = '256'
memory       = '512'
vcpus        = '1'
cpus         = '2'
localtime    = 0
serial       = 'pty'

disk         = [ 'drbd:xps-106,xvda,w' ]

on_poweroff  = 'destroy'
on_reboot    = 'restart'
on_crash     = 'restart'

extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
console=hvc0 xencons=tty"

pci = [ '04:00.0' ]

name         = 'xps-106'
hostname     = 'xps-106.clichy.jbfavre.org'
####################

As I privately told you, I made the tests you suggested and the result
is that problem occurs with 256mb of memory, but is solved with 512mb or
more.

Basically, what I find surprising is that with 256mb of memory, the max
size for incoming packets to be blocked is 128 bytes.
Makes me think about an unsigned integer or something like that, but I
don't have enough kernel knowledge to be more precise.

Regards,
JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
I made some more tests today, still with 2.6.37 32bits kernel from
Debian experimental, with various memory allocation value.

For each test, I make ping on my gateway with various packet size:
ping -s15 10.0.0.1
ping -s85 10.0.0.1
ping -s86 10.0.0.1
ping -s150 10.0.0.1

Results bellow:

- less than 256mb: works
- between 256 and 512mb: ping greater than 85 bytes does not work
- more than 512mb: works

I'm lost...
Regards,
JB


Le 27/01/2011 22:47, Jean Baptiste Favre a écrit :
> Hello Konrad,
> 
> Le 27/01/2011 21:27, Konrad Rzeszutek Wilk a écrit :
>> On Sat, Jan 22, 2011 at 11:22:59AM +0100, Jean Baptiste Favre wrote:
>>> Hello,
>>> Last investigations show that I've the latest BIOS version for my
>>> motherboard.
>>> Do you need more tests, if yes which ones ?
>>
>> I tried it on my 2.6.32.27 (32-bit and 64-bit) and I am not seeing
>> the failures you have. These are the devices I passed in:
>>
>> 00:00.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
>> 00:00.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
>> 00:00.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
>> 00:00.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
>> 00:01.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
>> 00:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
>> 00:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
>> 00:03.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
>>
>> Granted the kernel I am using a jeremy/xen/stable-2.6.32.x so not sure
>> how divergent from Debian or Debian Squeeze it is.
>>
>> How are you launching your guest? Do you something like this:
>>
>> kernel="/mnt/lab/2.6.32.27/vmlinuz"
>> ramdisk="/mnt/lab/2.6.32.27/initramfs.cpio.gz"
>> extra="console=hvc0 debug test=crashme iommu=soft"
>> memory=1024
>> maxmem=2048
>> vcpus=4
>> on_crash="preserve"
>> pci= ["00:1d.0","00:1d.1","00:1d.2","00:1d.7","0a:00.1","0000:06:01.1","0000:06:01.0", "09:00.0"]
>> vif = [ 'mac=00:0f:4b:00:00:68, bridge=switch' ]
> 
> Here is my domU config file:
> ####################
> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
> builder      = 'linux'
> 
> memory       = '256'
> memory       = '512'
> vcpus        = '1'
> cpus         = '2'
> localtime    = 0
> serial       = 'pty'
> 
> disk         = [ 'drbd:xps-106,xvda,w' ]
> 
> on_poweroff  = 'destroy'
> on_reboot    = 'restart'
> on_crash     = 'restart'
> 
> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
> console=hvc0 xencons=tty"
> 
> pci = [ '04:00.0' ]
> 
> name         = 'xps-106'
> hostname     = 'xps-106.clichy.jbfavre.org'
> ####################
> 
> As I privately told you, I made the tests you suggested and the result
> is that problem occurs with 256mb of memory, but is solved with 512mb or
> more.
> 
> Basically, what I find surprising is that with 256mb of memory, the max
> size for incoming packets to be blocked is 128 bytes.
> Makes me think about an unsigned integer or something like that, but I
> don't have enough kernel knowledge to be more precise.
> 
> Regards,
> JB
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Ian Campbell]

On Fri, 2011-01-28 at 15:47 +0000, Jean Baptiste Favre wrote:
> Hello,
> I made some more tests today, still with 2.6.37 32bits kernel from
> Debian experimental, with various memory allocation value.
> 
> For each test, I make ping on my gateway with various packet size:
> ping -s15 10.0.0.1
> ping -s85 10.0.0.1
> ping -s86 10.0.0.1
> ping -s150 10.0.0.1
> 
> Results bellow:
> 
> - less than 256mb: works
> - between 256 and 512mb: ping greater than 85 bytes does not work
> - more than 512mb: works
> 
> I'm lost...

Me too, this really is the most inexplicable set of symptoms...

Does it work correctly with any other guest kernel, e.g. the
xen/stable-2.6.32.x branch from xen.git or maybe one of the old-style
Xen kernels?

The network device in use is one of the Intel NICs below? Any luck just
passing through that one device without all the others?

Previously you mentioned using a Marvell NIC, so I guess the failure is
independent of the NIC type?

Your userspace is still OpenWRT in these most recent tests? Is that some
sort of busybox based thing? Can you try with e.g. a regular Debian
guest userspace to rule out any funnyness from that end?

If you restrict dom0 to >256MB but <512MB (using dom0_mem= on hypervisor
command line) does the NIC work correctly in non-passedthrough form?
Similarly does the kernel running native with mem= cause the failure?

Bit of a long shot but are you able to try a 4.0.2-rc hypervisor+tools
and/or a 4.1.0-rc setup (not branched yet so still in xen-unstable.hg)?

Is the 10.0.0.1 address you are testing against a VM on the same host or
some sort of external entity?

Ian.

> Regards,
> JB
> 
> 
> Le 27/01/2011 22:47, Jean Baptiste Favre a écrit :
> > Hello Konrad,
> > 
> > Le 27/01/2011 21:27, Konrad Rzeszutek Wilk a écrit :
> >> On Sat, Jan 22, 2011 at 11:22:59AM +0100, Jean Baptiste Favre wrote:
> >>> Hello,
> >>> Last investigations show that I've the latest BIOS version for my
> >>> motherboard.
> >>> Do you need more tests, if yes which ones ?
> >>
> >> I tried it on my 2.6.32.27 (32-bit and 64-bit) and I am not seeing
> >> the failures you have. These are the devices I passed in:
> >>
> >> 00:00.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
> >> 00:00.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
> >> 00:00.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
> >> 00:00.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
> >> 00:01.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
> >> 00:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
> >> 00:02.1 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
> >> 00:03.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
> >>
> >> Granted the kernel I am using a jeremy/xen/stable-2.6.32.x so not sure
> >> how divergent from Debian or Debian Squeeze it is.
> >>
> >> How are you launching your guest? Do you something like this:
> >>
> >> kernel="/mnt/lab/2.6.32.27/vmlinuz"
> >> ramdisk="/mnt/lab/2.6.32.27/initramfs.cpio.gz"
> >> extra="console=hvc0 debug test=crashme iommu=soft"
> >> memory=1024
> >> maxmem=2048
> >> vcpus=4
> >> on_crash="preserve"
> >> pci= ["00:1d.0","00:1d.1","00:1d.2","00:1d.7","0a:00.1","0000:06:01.1","0000:06:01.0", "09:00.0"]
> >> vif = [ 'mac=00:0f:4b:00:00:68, bridge=switch' ]
> > 
> > Here is my domU config file:
> > ####################
> > kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
> > ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
> > builder      = 'linux'
> > 
> > memory       = '256'
> > memory       = '512'
> > vcpus        = '1'
> > cpus         = '2'
> > localtime    = 0
> > serial       = 'pty'
> > 
> > disk         = [ 'drbd:xps-106,xvda,w' ]
> > 
> > on_poweroff  = 'destroy'
> > on_reboot    = 'restart'
> > on_crash     = 'restart'
> > 
> > extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
> > console=hvc0 xencons=tty"
> > 
> > pci = [ '04:00.0' ]
> > 
> > name         = 'xps-106'
> > hostname     = 'xps-106.clichy.jbfavre.org'
> > ####################
> > 
> > As I privately told you, I made the tests you suggested and the result
> > is that problem occurs with 256mb of memory, but is solved with 512mb or
> > more.
> > 
> > Basically, what I find surprising is that with 256mb of memory, the max
> > size for incoming packets to be blocked is 128 bytes.
> > Makes me think about an unsigned integer or something like that, but I
> > don't have enough kernel knowledge to be more precise.
> > 
> > Regards,
> > JB
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> > 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello Ian,

Le 01/02/2011 12:34, Ian Campbell a écrit :
> On Fri, 2011-01-28 at 15:47 +0000, Jean Baptiste Favre wrote:
> > Hello,
> > I made some more tests today, still with 2.6.37 32bits kernel from
> > Debian experimental, with various memory allocation value.
> >
> > For each test, I make ping on my gateway with various packet size:
> > ping -s15 10.0.0.1
> > ping -s85 10.0.0.1
> > ping -s86 10.0.0.1
> > ping -s150 10.0.0.1
> >
> > Results bellow:
> >
> > - less than 256mb: works
> > - between 256 and 512mb: ping greater than 85 bytes does not work
> > - more than 512mb: works
> >
> > I'm lost...
>
> Me too, this really is the most inexplicable set of symptoms...
>
> Does it work correctly with any other guest kernel, e.g. the
> xen/stable-2.6.32.x branch from xen.git or maybe one of the old-style
> Xen kernels?
I'm compiling 2.6.32 kernel from Jeremy's GIT repos to check that.

> The network device in use is one of the Intel NICs below? Any luck just
> passing through that one device without all the others?
>
> Previously you mentioned using a Marvell NIC, so I guess the failure is
> independent of the NIC type?
I made all tests with Marvell NIC (driver sky2).
I don't know if the same behaviour occurs with another NIC type. I'll
try with an Intel one (I've a dual port Intel NIC, so I could
passthrough only one port)

> Your userspace is still OpenWRT in these most recent tests? Is that some
> sort of busybox based thing? Can you try with e.g. a regular Debian
> guest userspace to rule out any funnyness from that end?
I made tests with both OpenWRT and Debian Squeeze.
Had problems to compile OpenWRT kernel in 64bits :)
I tested Debian Squeeze with 2.6.37 kernel from experimental (because of
Xen PCI Frontend integration. Not sure it has been backported into 2.6.32).
I will test with Jeremy's 2.6.32 kernel (see above).
As a conclusion, last results are from Debian Squeeze with 2.6.37 and
that's why I wrote on debian-kernel maillist :)

> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on hypervisor
> command line) does the NIC work correctly in non-passedthrough form?
My Xen hypervisor commandline is as follow:
placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all
guest_loglvl=all com1=115200,8n1 console=com1

Everything works great without passthrough, but my dom0 is 64bits which
may explain that (I do have this strange behaviour only with 32bits
kernels).

I did not tried changing dom0_mem param.

> Similarly does the kernel running native with mem= cause the failure?
Not sure I understand what you mean here.
BTW, I'm preparing a set of automatic tests with different memory
values. That will be:
* loop for each mem value
- set memory in domU configfile
- starting domU with 128Mb memory
- rc.local will ping my gateway with different packet size, store result
in file
- halt domU
* end of loop
* Check results :-/

> Bit of a long shot but are you able to try a 4.0.2-rc hypervisor+tools
> and/or a 4.1.0-rc setup (not branched yet so still in xen-unstable.hg)?
I can eventually try it, but after my looong test list :)

> Is the 10.0.0.1 address you are testing against a VM on the same host or
> some sort of external entity?
It's my gateway (for the history, WRT54GL with OpenWRT).

Regards,
JB

Re: PCI passthrough issue

[Ian Campbell]

On Tue, 2011-02-01 at 12:17 +0000, Jean Baptiste Favre wrote:
> > The network device in use is one of the Intel NICs below? Any luck just
> > passing through that one device without all the others?
> >
> > Previously you mentioned using a Marvell NIC, so I guess the failure is
> > independent of the NIC type?
> I made all tests with Marvell NIC (driver sky2).
> I don't know if the same behaviour occurs with another NIC type. I'll
> try with an Intel one (I've a dual port Intel NIC, so I could
> passthrough only one port)

I was confused -- the quoted list of passthrough devices including Intel
NICs I saw was from Konrad not you. It would still be interesting to
confirm if the issue is specific to the particular NIC or not.

> > Your userspace is still OpenWRT in these most recent tests? Is that some
> > sort of busybox based thing? Can you try with e.g. a regular Debian
> > guest userspace to rule out any funnyness from that end?
> I made tests with both OpenWRT and Debian Squeeze.
> Had problems to compile OpenWRT kernel in 64bits :)
> I tested Debian Squeeze with 2.6.37 kernel from experimental (because of
> Xen PCI Frontend integration. Not sure it has been backported into 2.6.32).

The PCI frontend is in the xen.git xen/stable-2.6.32.x branch but not in
mainline 2.6.32.y stable branches.

> > If you restrict dom0 to >256MB but <512MB (using dom0_mem= on hypervisor
> > command line) does the NIC work correctly in non-passedthrough form?
> My Xen hypervisor commandline is as follow:
> placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all
> guest_loglvl=all com1=115200,8n1 console=com1
> 
> Everything works great without passthrough, but my dom0 is 64bits which
> may explain that (I do have this strange behaviour only with 32bits
> kernels).

I don't suppose you could try installing a 32 bit OS in dom0? (e.g. as a
skanky hack you might fit something into your existing swap
partition ;-))

> I did not tried changing dom0_mem param.
> 
> > Similarly does the kernel running native with mem= cause the failure?
> Not sure I understand what you mean here.

I meant to run the system as a native (non-Xen) system and use the
kernels mem= command line parameter to restrict the system to the
problematic amounts of memory. Really just trying to verify if something
is up specifically due to Xen or not. Probably needs a 32 bit
user/kernel to be a useful experiment.

What do "ifconfig" or "ethtool -S" show for the device after some
failures. Do any of the numbers increment inline with the dropped
frames?

Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header
sizes is something around 128 bytes (depending on options etc). This is
pure numerology but I notice that sky2 has a copybreak parameter
("Receive copy threshold") which defaults to 128. I think it would be
worth trying 64 and 256.

Are you able to see any traffic with tcpdump, either in guest and/or
from another host (may require switch configuration to allow it to see
the traffic). e.g. do you see the ICMP Echo Request but not the Echo
Response or do you see nothing at all etc.

Ian.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,


Le 01/02/2011 14:20, Ian Campbell a écrit :
> On Tue, 2011-02-01 at 12:17 +0000, Jean Baptiste Favre wrote:
> >> The network device in use is one of the Intel NICs below? Any luck just
> >> passing through that one device without all the others?
> >>
> >> Previously you mentioned using a Marvell NIC, so I guess the failure is
> >> independent of the NIC type?
> > I made all tests with Marvell NIC (driver sky2).
> > I don't know if the same behaviour occurs with another NIC type. I'll
> > try with an Intel one (I've a dual port Intel NIC, so I could
> > passthrough only one port)
>
> I was confused -- the quoted list of passthrough devices including Intel
> NICs I saw was from Konrad not you. It would still be interesting to
> confirm if the issue is specific to the particular NIC or not.
>
> >> Your userspace is still OpenWRT in these most recent tests? Is that
> some
> >> sort of busybox based thing? Can you try with e.g. a regular Debian
> >> guest userspace to rule out any funnyness from that end?
> > I made tests with both OpenWRT and Debian Squeeze.
> > Had problems to compile OpenWRT kernel in 64bits :)
> > I tested Debian Squeeze with 2.6.37 kernel from experimental (because of
> > Xen PCI Frontend integration. Not sure it has been backported into
> 2.6.32).
>
> The PCI frontend is in the xen.git xen/stable-2.6.32.x branch but not in
> mainline 2.6.32.y stable branches.
Yes, that's why I tried with 2.6.37 from experimental, after Pasi's
answer to one of my question on the maillist one month ago or so.

> >> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on
> hypervisor
> >> command line) does the NIC work correctly in non-passedthrough form?
> > My Xen hypervisor commandline is as follow:
> > placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all
> > guest_loglvl=all com1=115200,8n1 console=com1
> >
> > Everything works great without passthrough, but my dom0 is 64bits which
> > may explain that (I do have this strange behaviour only with 32bits
> > kernels).
>
> I don't suppose you could try installing a 32 bit OS in dom0? (e.g. as a
> skanky hack you might fit something into your existing swap
> partition ;-))
I could try, but that would means breaking my test platform. Let's say I
prefer complete other tests before :)

> > I did not tried changing dom0_mem param.
> >
> >> Similarly does the kernel running native with mem= cause the failure?
> > Not sure I understand what you mean here.
>
> I meant to run the system as a native (non-Xen) system and use the
> kernels mem= command line parameter to restrict the system to the
> problematic amounts of memory. Really just trying to verify if something
> is up specifically due to Xen or not. Probably needs a 32 bit
> user/kernel to be a useful experiment.
Ah ok. But then, running 32bits kernel with 64 bits OS will work ? (I
didn't know 64bits kernel worked with 32bits OS before Konrad told me)

> What do "ifconfig" or "ethtool -S" show for the device after some
> failures. Do any of the numbers increment inline with the dropped
> frames?
>
> Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header
> sizes is something around 128 bytes (depending on options etc). This is
> pure numerology but I notice that sky2 has a copybreak parameter
> ("Receive copy threshold") which defaults to 128. I think it would be
> worth trying 64 and 256.
That's exactly the problem I faced with 256mb memory for my domU.
Outgoing packets reached my gateway (tcpdump saw them on it, as well as
replies) but incoming packets greater than 128bits were blocked
somewhere between Xen hypervisor and domU user space (where I was
listening traffic with tcpdump).

I didn't notice the copybreak from sky2. Where can I change it ? It
could be exactly what we're looking for from the beginning, couldn't it ?
How can I change it ?

> Are you able to see any traffic with tcpdump, either in guest and/or
> from another host (may require switch configuration to allow it to see
> the traffic). e.g. do you see the ICMP Echo Request but not the Echo
> Response or do you see nothing at all etc.
tcpdump tests have been done from my gateway and from domU.
On the GW: can see all incoming packets (whatever could be the size) and
send all replies.
On the DomU: can see all outgoing packets but only incoming ones shorter
than 128bytes (global size, means "ping -s85" and less)

Regards,
JB

Re: PCI passthrough issue

[Ian Campbell]

On Tue, 2011-02-01 at 14:12 +0000, Jean Baptiste Favre wrote:

> Le 01/02/2011 14:20, Ian Campbell a écrit :

> > >> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on
> > hypervisor
> > >> command line) does the NIC work correctly in non-passedthrough form?
> > > My Xen hypervisor commandline is as follow:
> > > placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all
> > > guest_loglvl=all com1=115200,8n1 console=com1
> > >
> > > Everything works great without passthrough, but my dom0 is 64bits which
> > > may explain that (I do have this strange behaviour only with 32bits
> > > kernels).
> >
> > I don't suppose you could try installing a 32 bit OS in dom0? (e.g. as a
> > skanky hack you might fit something into your existing swap
> > partition ;-))
> I could try, but that would means breaking my test platform. Let's say I
> prefer complete other tests before :)

Sure, no problem.

> > > I did not tried changing dom0_mem param.
> > >
> > >> Similarly does the kernel running native with mem= cause the failure?
> > > Not sure I understand what you mean here.
> >
> > I meant to run the system as a native (non-Xen) system and use the
> > kernels mem= command line parameter to restrict the system to the
> > problematic amounts of memory. Really just trying to verify if something
> > is up specifically due to Xen or not. Probably needs a 32 bit
> > user/kernel to be a useful experiment.
> Ah ok. But then, running 32bits kernel with 64 bits OS will work ? (I
> didn't know 64bits kernel worked with 32bits OS before Konrad told me)

32 user on 64 kernel works, 64 user on 32 kernel does not so this will
tie in with the 32 bit test above too.

> > What do "ifconfig" or "ethtool -S" show for the device after some
> > failures. Do any of the numbers increment inline with the dropped
> > frames?
> >
> > Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header
> > sizes is something around 128 bytes (depending on options etc). This is
> > pure numerology but I notice that sky2 has a copybreak parameter
> > ("Receive copy threshold") which defaults to 128. I think it would be
> > worth trying 64 and 256.
> That's exactly the problem I faced with 256mb memory for my domU.
> Outgoing packets reached my gateway (tcpdump saw them on it, as well as
> replies) but incoming packets greater than 128bits were blocked
> somewhere between Xen hypervisor and domU user space (where I was
> listening traffic with tcpdump).
> 
> I didn't notice the copybreak from sky2. Where can I change it ? It
> could be exactly what we're looking for from the beginning, couldn't it ?
> How can I change it ?

Assuming the driver is modular:
        "modprobe sky2 copybreak=<N>"

Depending on your distro there will be somewhere in /etc you can add
this. e.g. on Debian you can create a file in /etc/modprobe.d/
containing "option sky copybreak=<N>" other distros
use /etc/modprobe.conf etc.

> > Are you able to see any traffic with tcpdump, either in guest and/or
> > from another host (may require switch configuration to allow it to see
> > the traffic). e.g. do you see the ICMP Echo Request but not the Echo
> > Response or do you see nothing at all etc.
> tcpdump tests have been done from my gateway and from domU.
> On the GW: can see all incoming packets (whatever could be the size) and
> send all replies.
> On the DomU: can see all outgoing packets but only incoming ones shorter
> than 128bytes (global size, means "ping -s85" and less)

OK, so it is the sky2 receive path which is at fault, that's very useful
info. It corresponds with the copybreak theory too.

Ian.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 01/02/2011 15:18, Ian Campbell a écrit :
> On Tue, 2011-02-01 at 14:12 +0000, Jean Baptiste Favre wrote:
>
> > Le 01/02/2011 14:20, Ian Campbell a écrit :
>
> >>>> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on
> >> hypervisor
> >>>> command line) does the NIC work correctly in non-passedthrough form?
> >>> My Xen hypervisor commandline is as follow:
> >>> placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all
> >>> guest_loglvl=all com1=115200,8n1 console=com1
> >>>
> >>> Everything works great without passthrough, but my dom0 is 64bits
> which
> >>> may explain that (I do have this strange behaviour only with 32bits
> >>> kernels).
> >>
> >> I don't suppose you could try installing a 32 bit OS in dom0? (e.g.
> as a
> >> skanky hack you might fit something into your existing swap
> >> partition ;-))
> > I could try, but that would means breaking my test platform. Let's say I
> > prefer complete other tests before :)
>
> Sure, no problem.
>
> >>> I did not tried changing dom0_mem param.
> >>>
> >>>> Similarly does the kernel running native with mem= cause the failure?
> >>> Not sure I understand what you mean here.
> >>
> >> I meant to run the system as a native (non-Xen) system and use the
> >> kernels mem= command line parameter to restrict the system to the
> >> problematic amounts of memory. Really just trying to verify if
> something
> >> is up specifically due to Xen or not. Probably needs a 32 bit
> >> user/kernel to be a useful experiment.
> > Ah ok. But then, running 32bits kernel with 64 bits OS will work ? (I
> > didn't know 64bits kernel worked with 32bits OS before Konrad told me)
>
> 32 user on 64 kernel works, 64 user on 32 kernel does not so this will
> tie in with the 32 bit test above too.
>
> >> What do "ifconfig" or "ethtool -S" show for the device after some
> >> failures. Do any of the numbers increment inline with the dropped
> >> frames?
> >>
> >> Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header
> >> sizes is something around 128 bytes (depending on options etc). This is
> >> pure numerology but I notice that sky2 has a copybreak parameter
> >> ("Receive copy threshold") which defaults to 128. I think it would be
> >> worth trying 64 and 256.
> > That's exactly the problem I faced with 256mb memory for my domU.
> > Outgoing packets reached my gateway (tcpdump saw them on it, as well as
> > replies) but incoming packets greater than 128bits were blocked
> > somewhere between Xen hypervisor and domU user space (where I was
> > listening traffic with tcpdump).
> >
> > I didn't notice the copybreak from sky2. Where can I change it ? It
> > could be exactly what we're looking for from the beginning, couldn't
> it ?
> > How can I change it ?
>
> Assuming the driver is modular:
>         "modprobe sky2 copybreak=<N>"
>
> Depending on your distro there will be somewhere in /etc you can add
> this. e.g. on Debian you can create a file in /etc/modprobe.d/
> containing "option sky copybreak=<N>" other distros
> use /etc/modprobe.conf etc.
OK I see but it doesn't seems to have any effect.
I tried "option sky copybreak=0" to get all packet copied with no change.

But I have to say that I'm a bit confused: as I run a PV domU, kernel
and initrd are provided by dom0.
So basically, I had no module related binaries installed. After
installation, I tried to remove module and reload it with different
configuration without changes.
Is there any way to provide this sort of option in kernel commandline so
that it 'll be taken into account even in initrd ?

> >> Are you able to see any traffic with tcpdump, either in guest and/or
> >> from another host (may require switch configuration to allow it to see
> >> the traffic). e.g. do you see the ICMP Echo Request but not the Echo
> >> Response or do you see nothing at all etc.
> > tcpdump tests have been done from my gateway and from domU.
> > On the GW: can see all incoming packets (whatever could be the size) and
> > send all replies.
> > On the DomU: can see all outgoing packets but only incoming ones shorter
> > than 128bytes (global size, means "ping -s85" and less)
>
> OK, so it is the sky2 receive path which is at fault, that's very useful
> info. It corresponds with the copybreak theory too.
And that makes me happy, because I began to desperate :)

JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
I think I'll have to read another time "man modprobe" :)
See bellow, I've good news

Le 01/02/2011 16:14, Jean Baptiste Favre a écrit :
> Le 01/02/2011 15:18, Ian Campbell a écrit :
> > On Tue, 2011-02-01 at 14:12 +0000, Jean Baptiste Favre wrote:
> >
> >> Le 01/02/2011 14:20, Ian Campbell a écrit :
> >
> >>>>> If you restrict dom0 to >256MB but <512MB (using dom0_mem= on
> >>> hypervisor
> >>>>> command line) does the NIC work correctly in non-passedthrough form?
> >>>> My Xen hypervisor commandline is as follow:
> >>>> placeholder dom0_mem=256M dom0_max_vcpus=1 dom0_vcpus_pin loglvl=all
> >>>> guest_loglvl=all com1=115200,8n1 console=com1
> >>>>
> >>>> Everything works great without passthrough, but my dom0 is 64bits
> > which
> >>>> may explain that (I do have this strange behaviour only with 32bits
> >>>> kernels).
> >>>
> >>> I don't suppose you could try installing a 32 bit OS in dom0? (e.g.
> > as a
> >>> skanky hack you might fit something into your existing swap
> >>> partition ;-))
> >> I could try, but that would means breaking my test platform. Let's
> say I
> >> prefer complete other tests before :)
> >
> > Sure, no problem.
> >
> >>>> I did not tried changing dom0_mem param.
> >>>>
> >>>>> Similarly does the kernel running native with mem= cause the
> failure?
> >>>> Not sure I understand what you mean here.
> >>>
> >>> I meant to run the system as a native (non-Xen) system and use the
> >>> kernels mem= command line parameter to restrict the system to the
> >>> problematic amounts of memory. Really just trying to verify if
> > something
> >>> is up specifically due to Xen or not. Probably needs a 32 bit
> >>> user/kernel to be a useful experiment.
> >> Ah ok. But then, running 32bits kernel with 64 bits OS will work ? (I
> >> didn't know 64bits kernel worked with 32bits OS before Konrad told me)
> >
> > 32 user on 64 kernel works, 64 user on 32 kernel does not so this will
> > tie in with the 32 bit test above too.
> >
> >>> What do "ifconfig" or "ethtool -S" show for the device after some
> >>> failures. Do any of the numbers increment inline with the dropped
> >>> frames?
> >>>
> >>> Perhaps interestingly 85 bytes, plus the ICMP, IP and Ethernet header
> >>> sizes is something around 128 bytes (depending on options etc).
> This is
> >>> pure numerology but I notice that sky2 has a copybreak parameter
> >>> ("Receive copy threshold") which defaults to 128. I think it would be
> >>> worth trying 64 and 256.
> >> That's exactly the problem I faced with 256mb memory for my domU.
> >> Outgoing packets reached my gateway (tcpdump saw them on it, as well as
> >> replies) but incoming packets greater than 128bits were blocked
> >> somewhere between Xen hypervisor and domU user space (where I was
> >> listening traffic with tcpdump).
> >>
> >> I didn't notice the copybreak from sky2. Where can I change it ? It
> >> could be exactly what we're looking for from the beginning, couldn't
> > it ?
> >> How can I change it ?
> >
> > Assuming the driver is modular:
> >         "modprobe sky2 copybreak=<N>"
> >
> > Depending on your distro there will be somewhere in /etc you can add
> > this. e.g. on Debian you can create a file in /etc/modprobe.d/
> > containing "option sky copybreak=<N>" other distros
> > use /etc/modprobe.conf etc.
> OK I see but it doesn't seems to have any effect.
> I tried "option sky copybreak=0" to get all packet copied with no change.
>
> But I have to say that I'm a bit confused: as I run a PV domU, kernel
> and initrd are provided by dom0.
> So basically, I had no module related binaries installed. After
> installation, I tried to remove module and reload it with different
> configuration without changes.
> Is there any way to provide this sort of option in kernel commandline so
> that it 'll be taken into account even in initrd ?
>
> >>> Are you able to see any traffic with tcpdump, either in guest and/or
> >>> from another host (may require switch configuration to allow it to see
> >>> the traffic). e.g. do you see the ICMP Echo Request but not the Echo
> >>> Response or do you see nothing at all etc.
> >> tcpdump tests have been done from my gateway and from domU.
> >> On the GW: can see all incoming packets (whatever could be the
> size) and
> >> send all replies.
> >> On the DomU: can see all outgoing packets but only incoming ones
> shorter
> >> than 128bytes (global size, means "ping -s85" and less)
> >
> > OK, so it is the sky2 receive path which is at fault, that's very useful
> > info. It corresponds with the copybreak theory too.
OK, just found it:
after domU boot:
- log in
- ping -s 86 10.0.0.1 (fails)
- rmmod sky2
- modprobe sky2 copybreak=0 (no packet copied)
- ping -s 86 10.0.0.1 (works)

So it's clearly related to that option.
Now the question: what am I supposed to do ?
It seems that adding this options in /etc/modprobe.d/sky2.conf does not
work, neither using /etc/modules

Regards,
JB

Re: PCI passthrough issue

[Ian Campbell]

On Tue, 2011-02-01 at 15:14 +0000, Jean Baptiste Favre wrote:
> Le 01/02/2011 15:18, Ian Campbell a écrit :

> > Assuming the driver is modular:
> >         "modprobe sky2 copybreak=<N>"
> >
> > Depending on your distro there will be somewhere in /etc you can add
> > this. e.g. on Debian you can create a file in /etc/modprobe.d/
> > containing "option sky copybreak=<N>" other distros
> > use /etc/modprobe.conf etc.
> OK I see but it doesn't seems to have any effect.
> I tried "option sky copybreak=0" to get all packet copied with no change.

The driver is called sky2 not sky so this won't have done anything. I
typo'd it above, sorry.

> But I have to say that I'm a bit confused: as I run a PV domU, kernel
> and initrd are provided by dom0.
> So basically, I had no module related binaries installed. After
> installation, I tried to remove module and reload it with different
> configuration without changes.
> Is there any way to provide this sort of option in kernel commandline so
> that it 'll be taken into account even in initrd ?

It depends on your distro and/or initrd tool, I don't know a generic
answer. 

If the driver is statically compiled and not modular at all you can do
<module>.<param>=<value> on the kernel command line, e.g.
"sky2.copybreak=256".

There doesn't appear to be any way to find out what copybreak the driver
actually uses, short of hacking something into the driver itself. e.g.

diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..786b8c6 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -87,7 +87,7 @@ module_param(debug, int, 0);
 MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all)");
 
 static int copybreak __read_mostly = 128;
-module_param(copybreak, int, 0);
+module_param(copybreak, int, 0444);
 MODULE_PARM_DESC(copybreak, "Receive copy threshold");
 
 static int disable_msi = 0;

Allow you to see the current active value
in /sys/module/sky2/parameters/copybreak. If you are going to do that
you might as well just change the constant above though ;-)

Ian.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 01/02/2011 16:38, Ian Campbell a écrit :
> On Tue, 2011-02-01 at 15:14 +0000, Jean Baptiste Favre wrote:
> > Le 01/02/2011 15:18, Ian Campbell a écrit :
>
> >> Assuming the driver is modular:
> >>         "modprobe sky2 copybreak=<N>"
> >>
> >> Depending on your distro there will be somewhere in /etc you can add
> >> this. e.g. on Debian you can create a file in /etc/modprobe.d/
> >> containing "option sky copybreak=<N>" other distros
> >> use /etc/modprobe.conf etc.
> > OK I see but it doesn't seems to have any effect.
> > I tried "option sky copybreak=0" to get all packet copied with no
> change.
>
> The driver is called sky2 not sky so this won't have done anything. I
> typo'd it above, sorry.
My bad, I fixed your typo during tests, but I just copied it while
writing the mail. So basically, using:
# cat /etc/modprobe.d/sky2.conf
option sky2 copybreak=0

does not help.
I have to remove module and load it with right option.
BTW it's not a real issue for me, I can add it in init script :)

> > But I have to say that I'm a bit confused: as I run a PV domU, kernel
> > and initrd are provided by dom0.
> > So basically, I had no module related binaries installed. After
> > installation, I tried to remove module and reload it with different
> > configuration without changes.
> > Is there any way to provide this sort of option in kernel commandline so
> > that it 'll be taken into account even in initrd ?
>
> It depends on your distro and/or initrd tool, I don't know a generic
> answer.
Both are Debian ones.
I've tried to compile kernel from Jeremy's git repo, but face some
problems: it's been a long time since my last kernel compilation and I
did not find all configuration options I need :-/

> If the driver is statically compiled and not modular at all you can do
> <module>.<param>=<value> on the kernel command line, e.g.
> "sky2.copybreak=256".
I may use this way because, as a firewall, I prefer using static kernels :)

> There doesn't appear to be any way to find out what copybreak the driver
> actually uses, short of hacking something into the driver itself. e.g.
>
> diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
> index 7d85a38..786b8c6 100644
> --- a/drivers/net/sky2.c
> +++ b/drivers/net/sky2.c
> @@ -87,7 +87,7 @@ module_param(debug, int, 0);
>  MODULE_PARM_DESC(debug, "Debug level (0=none,...,16=all)");
>  
>  static int copybreak __read_mostly = 128;
> -module_param(copybreak, int, 0);
> +module_param(copybreak, int, 0444);
>  MODULE_PARM_DESC(copybreak, "Receive copy threshold");
>  
>  static int disable_msi = 0;
>
> Allow you to see the current active value
> in /sys/module/sky2/parameters/copybreak. If you are going to do that
> you might as well just change the constant above though ;-)
Will try this as well

Regards,
JB

Re: PCI passthrough issue

[Ian Campbell]

On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote:
> OK, just found it:
> after domU boot:
> - log in
> - ping -s 86 10.0.0.1 (fails)
> - rmmod sky2
> - modprobe sky2 copybreak=0 (no packet copied)
> - ping -s 86 10.0.0.1 (works)
> 
> So it's clearly related to that option.

Awesome!

> Now the question: what am I supposed to do ?

I think the next step is to try and reproduce on native 32 bit, with RAM
artificially limited via the mem= kernel command in option, this will
let us determine if this is a generic issue or is somehow Xen specific.

The main difference caused by the copybreak option is that for larger
frames (i.e. always when copybreak==0) it hits a code path which uses
pci_map_single and pci_map_page to access the received data. When len <
copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it
seems like the later path is broken somehow. If the issue does turn out
to be related to Xen then I think that points to the swiotlb code.

I assume you are not seeing "rx mapping error" in your domU dmesg? Did
you post a full guest console log at some point? Comparing the logs for
the 256MB, 398MB and 512MB guest RAM case might be useful.

Konrad, is there some way to force swiotlb use even for native to make
the native test cases more relevant? Or is there some other direction we
should try first?

> It seems that adding this options in /etc/modprobe.d/sky2.conf does not
> work, neither using /etc/modules

That will be a userspace issue in your distro, perhaps with openwrt they
are not expected to work that way.

Ian.

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote:
> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote:
> > OK, just found it:
> > after domU boot:
> > - log in
> > - ping -s 86 10.0.0.1 (fails)
> > - rmmod sky2
> > - modprobe sky2 copybreak=0 (no packet copied)
> > - ping -s 86 10.0.0.1 (works)
> > 
> > So it's clearly related to that option.
> 
> Awesome!
> 
> > Now the question: what am I supposed to do ?
> 
> I think the next step is to try and reproduce on native 32 bit, with RAM
> artificially limited via the mem= kernel command in option, this will
> let us determine if this is a generic issue or is somehow Xen specific.
> 
> The main difference caused by the copybreak option is that for larger
> frames (i.e. always when copybreak==0) it hits a code path which uses
> pci_map_single and pci_map_page to access the received data. When len <
> copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it
> seems like the later path is broken somehow. If the issue does turn out

It could also have gotten the direction reverted (the 3c5XX code had it
wrong at some point so..). Might make sense to compile the kernel with
CONFIG_DMA_API_DEBUG  which is good at detecting these issues.

> to be related to Xen then I think that points to the swiotlb code.

> 
> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> you post a full guest console log at some point? Comparing the logs for
> the 256MB, 398MB and 512MB guest RAM case might be useful.
> 
> Konrad, is there some way to force swiotlb use even for native to make
> the native test cases more relevant? Or is there some other direction we

swiotlb=force will do it.

> should try first?

<shakes his head> I think we have a good lead.

> 
> > It seems that adding this options in /etc/modprobe.d/sky2.conf does not
> > work, neither using /etc/modules

You could just pass as Linux kernel parameter 'sky2.copybreak=0'. But I am
not sure if that stays if the 'sky2' driver is compiled as module.

> 
> That will be a userspace issue in your distro, perhaps with openwrt they
> are not expected to work that way.
> 
> Ian.
> 

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 01/02/2011 17:23, Ian Campbell a écrit :
> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote:
>> OK, just found it:
>> after domU boot:
>> - log in
>> - ping -s 86 10.0.0.1 (fails)
>> - rmmod sky2
>> - modprobe sky2 copybreak=0 (no packet copied)
>> - ping -s 86 10.0.0.1 (works)
>>
>> So it's clearly related to that option.
> 
> Awesome!
> 
>> Now the question: what am I supposed to do ?
> 
> I think the next step is to try and reproduce on native 32 bit, with RAM
> artificially limited via the mem= kernel command in option, this will
> let us determine if this is a generic issue or is somehow Xen specific.
OK, so I'll install a 32bits Debian Squeeze with 2.6.37 32bits kernel to
check it.

> The main difference caused by the copybreak option is that for larger
> frames (i.e. always when copybreak==0) it hits a code path which uses
> pci_map_single and pci_map_page to access the received data. When len <
> copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it
> seems like the later path is broken somehow. If the issue does turn out
> to be related to Xen then I think that points to the swiotlb code.
> 
> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> you post a full guest console log at some point? Comparing the logs for
> the 256MB, 398MB and 512MB guest RAM case might be useful.
No sure I've ever posted that logs. But I can redo my tests :)

> Konrad, is there some way to force swiotlb use even for native to make
> the native test cases more relevant? Or is there some other direction we
> should try first?
> 
>> It seems that adding this options in /etc/modprobe.d/sky2.conf does not
>> work, neither using /etc/modules
> 
> That will be a userspace issue in your distro, perhaps with openwrt they
> are not expected to work that way.
All tests were done with Debian. As the module is included in initramfs
and not on domU, since I've no kernel installed in the domU, module is
loaded before system FS, so /etc/modprobe.d/sky2.conf is not taken into
account.
I think that the best solution here is to compile kernel with  static
sky2 driver inclusion and use kernel commandline to pass sky2 copybreak
option.

Regards,
JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 01/02/2011 20:37, Konrad Rzeszutek Wilk a écrit :
> On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote:
>> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote:
>>> OK, just found it:
>>> after domU boot:
>>> - log in
>>> - ping -s 86 10.0.0.1 (fails)
>>> - rmmod sky2
>>> - modprobe sky2 copybreak=0 (no packet copied)
>>> - ping -s 86 10.0.0.1 (works)
>>>
>>> So it's clearly related to that option.
>>
>> Awesome!
>>
>>> Now the question: what am I supposed to do ?
>>
>> I think the next step is to try and reproduce on native 32 bit, with RAM
>> artificially limited via the mem= kernel command in option, this will
>> let us determine if this is a generic issue or is somehow Xen specific.
>>
>> The main difference caused by the copybreak option is that for larger
>> frames (i.e. always when copybreak==0) it hits a code path which uses
>> pci_map_single and pci_map_page to access the received data. When len <
>> copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it
>> seems like the later path is broken somehow. If the issue does turn out
> 
> It could also have gotten the direction reverted (the 3c5XX code had it
> wrong at some point so..). Might make sense to compile the kernel with
> CONFIG_DMA_API_DEBUG  which is good at detecting these issues.
> 
>> to be related to Xen then I think that points to the swiotlb code.
> 
>>
>> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
>> you post a full guest console log at some point? Comparing the logs for
>> the 256MB, 398MB and 512MB guest RAM case might be useful.
>>
>> Konrad, is there some way to force swiotlb use even for native to make
>> the native test cases more relevant? Or is there some other direction we
> 
> swiotlb=force will do it.
I already use this option as specified on wiki:
http://wiki.xen.org/xenwiki/XenPCIpassthrough

>> should try first?
> 
> <shakes his head> I think we have a good lead.
So let's go for testing that way :)

>>> It seems that adding this options in /etc/modprobe.d/sky2.conf does not
>>> work, neither using /etc/modules
> 
> You could just pass as Linux kernel parameter 'sky2.copybreak=0'. But I am
> not sure if that stays if the 'sky2' driver is compiled as module.
According to Ian, this will work only with statically compiled driver.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 01/02/2011 20:37, Konrad Rzeszutek Wilk a écrit :
> On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote:
>> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote:
>>> OK, just found it:
>>> after domU boot:
>>> - log in
>>> - ping -s 86 10.0.0.1 (fails)
>>> - rmmod sky2
>>> - modprobe sky2 copybreak=0 (no packet copied)
>>> - ping -s 86 10.0.0.1 (works)
>>>
>>> So it's clearly related to that option.
>>
>> Awesome!
>>
>>> Now the question: what am I supposed to do ?
>>
>> I think the next step is to try and reproduce on native 32 bit, with RAM
>> artificially limited via the mem= kernel command in option, this will
>> let us determine if this is a generic issue or is somehow Xen specific.
>>
>> The main difference caused by the copybreak option is that for larger
>> frames (i.e. always when copybreak==0) it hits a code path which uses
>> pci_map_single and pci_map_page to access the received data. When len <
>> copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it
>> seems like the later path is broken somehow. If the issue does turn out
> 
> It could also have gotten the direction reverted (the 3c5XX code had it
> wrong at some point so..). Might make sense to compile the kernel with
> CONFIG_DMA_API_DEBUG  which is good at detecting these issues.
> 
>> to be related to Xen then I think that points to the swiotlb code.
> 
>>
>> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
>> you post a full guest console log at some point? Comparing the logs for
>> the 256MB, 398MB and 512MB guest RAM case might be useful.
>>
>> Konrad, is there some way to force swiotlb use even for native to make
>> the native test cases more relevant? Or is there some other direction we
> 
> swiotlb=force will do it.
OK, just performed native test.
Installed 32bits Debian Squeeze, add 2.6.37 32bits kernel from
experimental, setup grub with following options:
"mem=256M swiotlb=force"

Ping tests work whatever can be packet size.

If I understood well what you explain to me, it's now clear that the
problem is somewhat Xen related, isn't it ?

I'll be able to continue any tests you want tomorrow.
Do you still need me to compile domU kernel with "CONFIG_DMA_API_DEBUG"
enabled ?
If so, what tools will I need to get debug informations ?

Now, it's time to sleep here in France :) (or I'll get killed by my girl
friend :D )

Regards,
JB

Re: PCI passthrough issue

[Ian Campbell]

On Tue, 2011-02-01 at 22:06 +0000, Jean Baptiste Favre wrote:
> Le 01/02/2011 20:37, Konrad Rzeszutek Wilk a écrit :
> > On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote:

> >> Konrad, is there some way to force swiotlb use even for native to make
> >> the native test cases more relevant? Or is there some other direction we
> > 
> > swiotlb=force will do it.
> I already use this option as specified on wiki:
> http://wiki.xen.org/xenwiki/XenPCIpassthrough

Yes, we were just confirming that you should also use this option for
your native tests.

> 
> >> should try first?
> > 
> > <shakes his head> I think we have a good lead.
> So let's go for testing that way :)
> 
> >>> It seems that adding this options in /etc/modprobe.d/sky2.conf does not
> >>> work, neither using /etc/modules
> > 
> > You could just pass as Linux kernel parameter 'sky2.copybreak=0'. But I am
> > not sure if that stays if the 'sky2' driver is compiled as module.
> According to Ian, this will work only with statically compiled driver.

Well, I'm only sure it works for a statically compiled driver. I don't
know about modules -- depends on whether modprobe groks /proc/cmdline or
something, I suspect it does not though.

Ian.

Re: PCI passthrough issue

[Ian Campbell]

On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:
> Le 01/02/2011 17:23, Ian Campbell a écrit :

> > I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> > you post a full guest console log at some point? Comparing the logs for
> > the 256MB, 398MB and 512MB guest RAM case might be useful.
> No sure I've ever posted that logs. But I can redo my tests :)

yes, please do that.

Please can you also collect and post the information from ifconfig and
ethtool -S which I asked for earlier.

Can you confirm that on the domU tcpdump shows no incoming frames at
all, including no corrupted or strange frames. It has been suggested
that the frames could be being received ok but contain garbage (e.g. due
to swiotlb not syncing the memory properly) and hence are not ICMP
Responses but appear as some random frame type.

Please can you use "tcpdump -w" on both the gateway/peer side and the
affected domU to capture a short session while the failing ping is
running and post (or link to if large) the resulting pcap files. On the
peer side you can use "ether host <domU-MAC>" (plus optionally "or ether
broadcast") on the tcpdump command line to limit the capture to just the
one peer and reduce the size of the capture.

Ian.

Re: PCI passthrough issue

[Ian Campbell]

On Tue, 2011-02-01 at 23:01 +0000, Jean Baptiste Favre wrote:
> Le 01/02/2011 20:37, Konrad Rzeszutek Wilk a écrit :
> > On Tue, Feb 01, 2011 at 04:23:09PM +0000, Ian Campbell wrote:
> >> On Tue, 2011-02-01 at 15:38 +0000, Jean Baptiste Favre wrote:
> >>> OK, just found it:
> >>> after domU boot:
> >>> - log in
> >>> - ping -s 86 10.0.0.1 (fails)
> >>> - rmmod sky2
> >>> - modprobe sky2 copybreak=0 (no packet copied)
> >>> - ping -s 86 10.0.0.1 (works)
> >>>
> >>> So it's clearly related to that option.
> >>
> >> Awesome!
> >>
> >>> Now the question: what am I supposed to do ?
> >>
> >> I think the next step is to try and reproduce on native 32 bit, with RAM
> >> artificially limited via the mem= kernel command in option, this will
> >> let us determine if this is a generic issue or is somehow Xen specific.
> >>
> >> The main difference caused by the copybreak option is that for larger
> >> frames (i.e. always when copybreak==0) it hits a code path which uses
> >> pci_map_single and pci_map_page to access the received data. When len <
> >> copybreak it takes a path which uses pci_dma_sync_single_for_cpu, so it
> >> seems like the later path is broken somehow. If the issue does turn out
> > 
> > It could also have gotten the direction reverted (the 3c5XX code had it
> > wrong at some point so..). Might make sense to compile the kernel with
> > CONFIG_DMA_API_DEBUG  which is good at detecting these issues.
> > 
> >> to be related to Xen then I think that points to the swiotlb code.
> > 
> >>
> >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> >> you post a full guest console log at some point? Comparing the logs for
> >> the 256MB, 398MB and 512MB guest RAM case might be useful.
> >>
> >> Konrad, is there some way to force swiotlb use even for native to make
> >> the native test cases more relevant? Or is there some other direction we
> > 
> > swiotlb=force will do it.
> OK, just performed native test.
> Installed 32bits Debian Squeeze, add 2.6.37 32bits kernel from
> experimental, setup grub with following options:
> "mem=256M swiotlb=force"
> 
> Ping tests work whatever can be packet size.
> 
> If I understood well what you explain to me, it's now clear that the
> problem is somewhat Xen related, isn't it ?

It certainly seems that way. I'm not 100% sure that swiotlb=force will
have actually made the driver use the swiotlb, it may just have forced
the swiotlb to be allocated. Konrad?

> I'll be able to continue any tests you want tomorrow.
> Do you still need me to compile domU kernel with "CONFIG_DMA_API_DEBUG"
> enabled ?

Yes please.

> If so, what tools will I need to get debug informations ?

Uh, Konrad?

Ian.

Re: PCI passthrough issue

[Jean Baptiste Favre]

[-- Attachment #1: Type: text/plain, Size: 3064 bytes --]

Hello Ian,

My domU config file:

# cat /cluster/xen/xps-106.cfg
kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
builder      = 'linux'
memory       = '398'
vcpus        = '1'
cpus         = '2'
localtime    = 0
serial       = 'pty'
boot         = 'cdn'
disk         = [ 'drbd:xps-106,xvda,w' ]
on_poweroff  = 'destroy'
on_reboot    = 'restart'
on_crash     = 'restart'
extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
console=hvc0 xencons=tty"
pci = [ '04:00.0' ]
name         = 'xps-106'
hostname     = 'xps-106.clichy.jbfavre.org'


Le 02/02/2011 10:27, Ian Campbell a écrit :
> On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:
> > Le 01/02/2011 17:23, Ian Campbell a écrit :
>
> >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> >> you post a full guest console log at some point? Comparing the logs for
> >> the 256MB, 398MB and 512MB guest RAM case might be useful.
> > No sure I've ever posted that logs. But I can redo my tests :)
>
> yes, please do that.
Please find attached both console startup logs with 256M and 512M:
256M_domU_console_logs.txt
512M_domU_console_logs.txt

For 512M, I saw some kernel CallTrace I can not explain. There are not
present with 256M.

For 398M memory, I can't even start domU :
# xm create /cluster/xen/xps-106.cfg -c
Using config file "/cluster/xen/xps-106.cfg".
[215739.007871] pciback 0000:04:00.0: device has been assigned to
another domain! Over-writting the ownership, but beware.
Started domain xps-106 (id=23)
(XEN) mm.c:798:d23 Non-privileged (23) attempt to map I/O space 00000000
(XEN) mm.c:4644:d23 ptwr_emulate: could not get_page_from_l1e()

As I told you, I'm still using Debian 2.6.37 kernel because I've some
problem to compile 2.6.32.27 from Jeremy's git repository.
I hope I can get it compiled today so I'll be able to test with that
kernel as well.

>
> Please can you also collect and post the information from ifconfig and
> ethtool -S which I asked for earlier.
Attached as well:
256_domU_ifconfig_ping_ethtool.txt
512_domU_ifconfig_ping_ethtool.txt

> Can you confirm that on the domU tcpdump shows no incoming frames at
> all, including no corrupted or strange frames. It has been suggested
> that the frames could be being received ok but contain garbage (e.g. due
> to swiotlb not syncing the memory properly) and hence are not ICMP
> Responses but appear as some random frame type.
>
> Please can you use "tcpdump -w" on both the gateway/peer side and the
> affected domU to capture a short session while the failing ping is
> running and post (or link to if large) the resulting pcap files. On the
> peer side you can use "ether host <domU-MAC>" (plus optionally "or ether
> broadcast") on the tcpdump command line to limit the capture to just the
> one peer and reduce the size of the capture.
I'll update you as soon as I have tcpdump captures ready.

Regards,
JB

[-- Attachment #2: 512M_domU_ifconfig_ping_ethtool.txt --]
[-- Type: text/plain, Size: 2936 bytes --]

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1f:c6:eb:71:43
          inet addr:10.0.0.106  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::21f:c6ff:feeb:7143/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:120 (120.0 B)  TX bytes:468 (468.0 B)
          Interrupt:18

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

# ping -c5 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_req=1 ttl=64 time=0.823 ms
64 bytes from 10.0.0.1: icmp_req=2 ttl=64 time=0.856 ms
64 bytes from 10.0.0.1: icmp_req=3 ttl=64 time=0.831 ms
64 bytes from 10.0.0.1: icmp_req=4 ttl=64 time=0.824 ms
64 bytes from 10.0.0.1: icmp_req=5 ttl=64 time=0.830 ms

--- 10.0.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.823/0.832/0.856/0.038 ms

# ping -c5 -s86 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 86(114) bytes of data.
94 bytes from 10.0.0.1: icmp_req=1 ttl=64 time=1.25 ms
94 bytes from 10.0.0.1: icmp_req=2 ttl=64 time=0.861 ms
94 bytes from 10.0.0.1: icmp_req=3 ttl=64 time=0.895 ms
94 bytes from 10.0.0.1: icmp_req=4 ttl=64 time=0.868 ms
94 bytes from 10.0.0.1: icmp_req=5 ttl=64 time=0.864 ms

--- 10.0.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.861/0.949/1.259/0.157 ms
# ethtool -S eth0
NIC statistics:
     tx_bytes: 5651
     rx_bytes: 78580
     tx_broadcast: 1
     rx_broadcast: 5
     tx_multicast: 6
     rx_multicast: 0
     tx_unicast: 60
     rx_unicast: 75
     tx_mac_pause: 0
     rx_mac_pause: 0
     collisions: 0
     late_collision: 0
     aborted: 0
     single_collisions: 0
     multi_collisions: 0
     rx_short: 0
     rx_runt: 0
     rx_64_byte_packets: 9
     rx_65_to_127_byte_packets: 12
     rx_128_to_255_byte_packets: 9
     rx_256_to_511_byte_packets: 0
     rx_512_to_1023_byte_packets: 0
     rx_1024_to_1518_byte_packets: 50
     rx_1518_to_max_byte_packets: 0
     rx_too_long: 0
     rx_fifo_overflow: 0
     rx_jabber: 0
     rx_fcs_error: 0
     tx_64_byte_packets: 4
     tx_65_to_127_byte_packets: 55
     tx_128_to_255_byte_packets: 8
     tx_256_to_511_byte_packets: 0
     tx_512_to_1023_byte_packets: 0
     tx_1024_to_1518_byte_packets: 0
     tx_1519_to_max_byte_packets: 0
     tx_fifo_underrun: 0

[-- Attachment #3: 256M_domU_console_logs.txt --]
[-- Type: text/plain, Size: 12426 bytes --]

# xm create /cluster/xen/xps-106.cfg -c
Using config file "/cluster/xen/xps-106.cfg".
[213362.865921] pciback 0000:04:00.0: device has been assigned to another domain! Over-writting the ownership, but beware.
Started domain xps-106 (id=16)
[    0.000000] Reserving virtual address space above 0xf5800000
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.37-trunk-686-bigmem (Debian 2.6.37-1~experimental.1) (ben@decadent.org.uk) (gcc version 4.4.5 (Debian 4.4.5-10)
 ) #1 SMP Thu Jan 6 15:21:44 UTC 2011
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  (XEN) traps.c:2308:d16 Domain attempted WRMSR 00000000c0010004 from 0000c0b2:cbfa8461 to 00000000:0000abcd.
Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000010800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] last_pfn = 0x10800 max_arch_pfn = 0x1000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000010800000
[    0.000000] RAMDISK: 01793000 - 02ecc000
[    0.000000] 0MB HIGHMEM available.
[    0.000000] 264MB LOWMEM available.
[    0.000000]   mapped low ram: 0 - 10800000
[    0.000000]   low ram: 0 - 10800000
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   Normal   0x00001000 -> 0x00010800
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00010800
[    0.000000] Using APIC driver default
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 10800000 (gap: 10800000:ef800000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.0.1 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Embedded 13 pages/cpu @cfddf000 s30336 r0 d22912 u53248
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 66944
[    0.000000] Kernel command line: root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Initializing CPU#0
[    0.000000] Placing 64MB software IO TLB between cbd7ee40 - cfd7ee40
[    0.000000] software IO TLB at phys 0xbd7ee40 - 0xfd7ee40
[    0.000000] Initializing HighMem for node 0 (00000000:00000000)
[    0.000000] Memory: 163268k/270336k available (2703k kernel code, 106620k reserved, 1410k data, 412k init, 0k highmem)
[    0.000000] virtual kernel memory layout:
[    0.000000]     fixmap  : 0xf5536000 - 0xf57ff000   (2852 kB)
[    0.000000]     pkmap   : 0xf5200000 - 0xf5400000   (2048 kB)
[    0.000000]     vmalloc : 0xd1000000 - 0xf51fe000   ( 577 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xd0800000   ( 264 MB)
[    0.000000]       .init : 0xc1405000 - 0xc146c000   ( 412 kB)
[    0.000000]       .data : 0xc12a3d40 - 0xc14046e0   (1410 kB)
[    0.000000]       .text : 0xc1000000 - 0xc12a3d40   (2703 kB)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:1280
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2506.222 MHz processor.
[    0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 5012.44 BogoMIPS (lpj=10024888)
[    0.004000] pid_max: default: 32768 minimum: 301
[    0.004000] Security Framework initialized
[    0.004000] SELinux:  Disabled at boot.
[    0.004000] Mount-cache hash table entries: 512
[    0.004000] Initializing cgroup subsys ns
[    0.004000] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.004000] Initializing cgroup subsys cpuacct
[    0.004000] Initializing cgroup subsys devices
[    0.004000] Initializing cgroup subsys freezer
[    0.004000] Initializing cgroup subsys net_cls
[    0.004000] Initializing cgroup subsys blkio
[    0.004000] CPU: Physical Processor ID: 0
[    0.004000] CPU: Processor Core ID: 0
[    0.004000] Performance Events:
[    0.004000] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.004000] no hardware sampling interrupt available.
[    0.004000] Broken PMU hardware detected, software events only.
[    0.004000] SMP alternatives: switching to UP code
[    0.008000] Freeing SMP alternatives: 16k freed
[    0.008246] NMI watchdog disabled for cpu0: unable to create perf event: -2
[    0.008354] Brought up 1 CPUs
[    0.008444] devtmpfs: initialized
[    0.008790] Grant table initialized
[    0.008797] regulator: core version 0.5
[    0.008851] regulator: dummy:
[    0.008900] NET: Registered protocol family 16
[    0.009099] PCI: setting up Xen PCI frontend stub
[    0.009099] bio: create slab <bio-0> at 0
[    0.009099] ACPI: Interpreter disabled.
[    0.009099] xen_balloon: Initialising balloon driver.
[    0.009099] last_pfn = 0x10800 max_arch_pfn = 0x1000000
[    0.009099] vgaarb: loaded
[    0.009099] PCI: System does not support PCI
[    0.009099] PCI: System does not support PCI
[    0.009099] Switching to clocksource xen
[    0.009443] pnp: PnP ACPI: disabled
[    0.009452] PnPBIOS: Disabled
[    0.011710] NET: Registered protocol family 2
[    0.011757] IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.011940] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[    0.012014] TCP bind hash table entries: 16384 (order: 5, 131072 bytes)
[    0.012075] TCP: Hash tables configured (established 16384 bind 16384)
[    0.012082] TCP reno registered
[    0.012088] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.012098] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.012164] NET: Registered protocol family 1
[    0.012221] Unpacking initramfs...
[    0.068739] Freeing initrd memory: 23780k freed
[    0.076752] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.076997] audit: initializing netlink socket (disabled)
[    0.077014] type=2000 audit(1296639642.987:1): initialized
[    0.085690] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.087728] VFS: Disk quotas dquot_6.5.2
[    0.087800] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.087899] msgmni has been set to 365
[    0.088100] alg: No test for stdrng (krng)
[    0.088169] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.088178] io scheduler noop registered
[    0.088183] io scheduler deadline registered
[    0.088199] io scheduler cfq registered (default)
[    0.088422] isapnp: Scanning for PnP cards...
[    0.804965] isapnp: No Plug & Play device found
[    0.810882] Linux agpgart interface v0.103
[    0.810978] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.811280] PNP: No PS/2 controller found. Probing ports directly.
[    0.812099] i8042.c: No controller found.
[    0.812196] mice: PS/2 mouse device common for all mice
[    0.812315] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.812339] pcifront pci-0: Installing PCI frontend
[    0.812380] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.812407] cpuidle: using governor ladder
[    0.812413] cpuidle: using governor menu
[    0.812690] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.816145] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.816145] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.818358] TCP cubic registered
[    0.818508] NET: Registered protocol family 10
[    0.818840] lo: Disabled Privacy Extensions
[    0.819043] Mobile IPv6
[    0.819050] NET: Registered protocol family 17
[    0.819058] Registering the dns_resolver key type
[    0.819084] Using IPI No-Shortcut mode
[    0.819211] registered taskstats version 1
[    0.916059] XENBUS: Device with no driver: device/vbd/51712
[    0.916074] XENBUS: Device with no driver: device/console/0
[    0.916101] /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    0.916134] Initalizing network drop monitor service
[    0.916238] Freeing unused kernel memory: 412k freed
[    0.917011] Write protecting the kernel text: 2704k
[    0.917255] Write protecting the kernel read-only data: 1016k
Loading, please wait...
[    0.976345] udev[46]: starting version 164
[    1.105266] sky2: driver version 1.28
[    1.121166] sky2 0000:00:00.0: BAR 0: set to [mem 0xfeb00000-0xfeb03fff 64bit] (PCI address [0xfeb00000-0xfeb03fff])
[    1.121258] sky2 0000:00:00.0: BAR 2: set to [io  0xe800-0xe8ff] (PCI address [0xe800-0xe8ff])
[    1.121330] sky2 0000:00:00.0: enabling device (0000 -> 0003)
[    1.121454] sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
[    1.121649] sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
[    1.122871] sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43
[    1.133172] blkfront: xvda: barriers enabled
[    1.134906]  xvda: xvda1 xvda2
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... [    1.340290] device-mapper: uevent: version 1.0.3
[    1.340867] device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@redhat.com
done.
Begin: Running /scripts/local-premount ... done.
[    1.579694] EXT3-fs: barriers not enabled
[    1.585025] kjournald starting.  Commit interval 5 seconds
[    1.585063] EXT3-fs (dm-0): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd[    2.575753] udev[167]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[    2.904843] input: PC Speaker as /devices/platform/pcspkr/input/input0
done.
Activating swap...done.
[    6.253299] EXT3-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
[    6.284848] EXT3-fs (dm-0): using internal journal
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
Configuring network interfaces...[    6.831911] sky2 0000:00:00.0: eth0: enabling interface
[    6.831997] ADDRCONF(NETDEV_UP): eth0: link is not ready
done.
Cleaning up temporary files....
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 6.0 xps-106 hvc0

xps-106 login: [    9.298080] sky2 0000:00:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow control both
[    9.298150] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

[-- Attachment #4: 256M_domU_ifconfig_ping_ethtool.txt --]
[-- Type: text/plain, Size: 3787 bytes --]

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1f:c6:eb:71:43
          inet addr:10.0.0.106  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::21f:c6ff:feeb:7143/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:468 (468.0 B)
          Interrupt:18

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

# ping -c5 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
From 10.0.0.106 icmp_seq=1 Destination Host Unreachable
From 10.0.0.106 icmp_seq=2 Destination Host Unreachable
From 10.0.0.106 icmp_seq=3 Destination Host Unreachable
From 10.0.0.106 icmp_seq=4 Destination Host Unreachable
From 10.0.0.106 icmp_seq=5 Destination Host Unreachable

--- 10.0.0.1 ping statistics ---
5 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4023ms
pipe 3
# ping -c5 -s86 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 86(114) bytes of data.
From 10.0.0.106 icmp_seq=1 Destination Host Unreachable
From 10.0.0.106 icmp_seq=2 Destination Host Unreachable
From 10.0.0.106 icmp_seq=3 Destination Host Unreachable
From 10.0.0.106 icmp_seq=4 Destination Host Unreachable
From 10.0.0.106 icmp_seq=5 Destination Host Unreachable

--- 10.0.0.1 ping statistics ---
5 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4024ms
pipe 3
# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:1f:c6:eb:71:43
          inet addr:10.0.0.106  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::21f:c6ff:feeb:7143/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:14 errors:0 dropped:14 overruns:0 frame:0
          TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:840 (840.0 B)  TX bytes:972 (972.0 B)
          Interrupt:18

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:10 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1270 (1.2 KiB)  TX bytes:1270 (1.2 KiB)

# ethtool -S eth0
NIC statistics:
     tx_bytes: 1260
     rx_bytes: 896
     tx_broadcast: 12
     rx_broadcast: 2
     tx_multicast: 6
     rx_multicast: 0
     tx_unicast: 0
     rx_unicast: 12
     tx_mac_pause: 0
     rx_mac_pause: 0
     collisions: 0
     late_collision: 0
     aborted: 0
     single_collisions: 0
     multi_collisions: 0
     rx_short: 0
     rx_runt: 0
     rx_64_byte_packets: 14
     rx_65_to_127_byte_packets: 0
     rx_128_to_255_byte_packets: 0
     rx_256_to_511_byte_packets: 0
     rx_512_to_1023_byte_packets: 0
     rx_1024_to_1518_byte_packets: 0
     rx_1518_to_max_byte_packets: 0
     rx_too_long: 0
     rx_fifo_overflow: 0
     rx_jabber: 0
     rx_fcs_error: 0
     tx_64_byte_packets: 12
     tx_65_to_127_byte_packets: 6
     tx_128_to_255_byte_packets: 0
     tx_256_to_511_byte_packets: 0
     tx_512_to_1023_byte_packets: 0
     tx_1024_to_1518_byte_packets: 0
     tx_1519_to_max_byte_packets: 0
     tx_fifo_underrun: 0

[-- Attachment #5: 512M_domU_console_logs.txt --]
[-- Type: text/plain, Size: 16412 bytes --]

# xm create /cluster/xen/xps-106.cfg -c
Using config file "/cluster/xen/xps-106.cfg".
[214817.006300] pciback 0000:04:00.0: device has been assigned to another domain! Over-writting the ownership, but beware.
Started domain xps-106 (id=21)
(XEN) traps.c:2308:d21 Domain attempted WRMSR 00000000c0010004 from 0000c0b2:cbfa8461 to 00000000:0000abcd.
[    0.000000] Reserving virtual address space above 0xf5800000
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.37-trunk-686-bigmem (Debian 2.6.37-1~experimental.1) (ben@decadent.org.uk) (gcc version 4.4.5 (Debian 4.4.5-10) ) #1 SMP Thu Jan 6 15:21:44 UTC 2011
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000020800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] last_pfn = 0x20800 max_arch_pfn = 0x1000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000020800000
[    0.000000] RAMDISK: 01793000 - 02ecc000
[    0.000000] 0MB HIGHMEM available.
[    0.000000] 520MB LOWMEM available.
[    0.000000]   mapped low ram: 0 - 20800000
[    0.000000]   low ram: 0 - 20800000
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   Normal   0x00001000 -> 0x00020800
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00020800
[    0.000000] Using APIC driver default
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 20800000 (gap: 20800000:df800000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.0.1 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Embedded 13 pages/cpu @dfbde000 s30336 r0 d22912 u53248
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 131968
[    0.000000] Kernel command line: root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Initializing CPU#0
[    0.000000] Placing 64MB software IO TLB between dbb1a000 - dfb1a000
[    0.000000] software IO TLB at phys 0x1bb1a000 - 0x1fb1a000
[    0.000000] Initializing HighMem for node 0 (00000000:00000000)
[    0.000000] Memory: 422200k/532480k available (2703k kernel code, 109832k reserved, 1410k data, 412k init, 0k highmem)
[    0.000000] virtual kernel memory layout:
[    0.000000]     fixmap  : 0xf5536000 - 0xf57ff000   (2852 kB)
[    0.000000]     pkmap   : 0xf5200000 - 0xf5400000   (2048 kB)
[    0.000000]     vmalloc : 0xe1000000 - 0xf51fe000   ( 321 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xe0800000   ( 520 MB)
[    0.000000]       .init : 0xc1405000 - 0xc146c000   ( 412 kB)
[    0.000000]       .data : 0xc12a3d40 - 0xc14046e0   (1410 kB)
[    0.000000]       .text : 0xc1000000 - 0xc12a3d40   (2703 kB)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:1280
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2506.222 MHz processor.
[    0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 5012.44 BogoMIPS (lpj=10024888)
[    0.004000] pid_max: default: 32768 minimum: 301
[    0.004000] Security Framework initialized
[    0.004000] SELinux:  Disabled at boot.
[    0.004000] Mount-cache hash table entries: 512
[    0.004000] Initializing cgroup subsys ns
[    0.004000] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.004000] Initializing cgroup subsys cpuacct
[    0.004000] Initializing cgroup subsys devices
[    0.004000] Initializing cgroup subsys freezer
[    0.004000] Initializing cgroup subsys net_cls
[    0.004000] Initializing cgroup subsys blkio
[    0.004000] CPU: Physical Processor ID: 0
[    0.004000] CPU: Processor Core ID: 0
[    0.004000] Performance Events:
[    0.004000] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.004000] no hardware sampling interrupt available.
[    0.004000] Broken PMU hardware detected, software events only.
[    0.004000] SMP alternatives: switching to UP code
[    0.010317] Freeing SMP alternatives: 16k freed
[    0.010560] NMI watchdog disabled for cpu0: unable to create perf event: -2
[    0.010645] Brought up 1 CPUs
[    0.010735] devtmpfs: initialized
[    0.011056] Grant table initialized
[    0.011063] regulator: core version 0.5
[    0.011122] regulator: dummy:
[    0.011169] NET: Registered protocol family 16
[    0.011366] PCI: setting up Xen PCI frontend stub
[    0.012488] bio: create slab <bio-0> at 0
[    0.012557] ACPI: Interpreter disabled.
[    0.012582] xen_balloon: Initialising balloon driver.
[    0.012582] last_pfn = 0x20800 max_arch_pfn = 0x1000000
[    0.012582] vgaarb: loaded
[    0.012582] PCI: System does not support PCI
[    0.012582] PCI: System does not support PCI
[    0.012582] Switching to clocksource xen
[    0.013423] pnp: PnP ACPI: disabled
[    0.013430] PnPBIOS: Disabled
[    0.015641] NET: Registered protocol family 2
[    0.015690] IP route cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.015896] TCP established hash table entries: 32768 (order: 6, 262144 bytes)
[    0.016029] TCP bind hash table entries: 32768 (order: 6, 262144 bytes)
[    0.016148] TCP: Hash tables configured (established 32768 bind 32768)
[    0.016155] TCP reno registered
[    0.016161] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.016176] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.016308] NET: Registered protocol family 1
[    0.016366] Unpacking initramfs...
[    0.072960] Freeing initrd memory: 23780k freed
[    0.080894] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.081131] audit: initializing netlink socket (disabled)
[    0.081148] type=2000 audit(1296641097.317:1): initialized
[    0.089493] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.091528] VFS: Disk quotas dquot_6.5.2
[    0.091595] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.091696] msgmni has been set to 871
[    0.091893] alg: No test for stdrng (krng)
[    0.091961] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.091970] io scheduler noop registered
[    0.091976] io scheduler deadline registered
[    0.091992] io scheduler cfq registered (default)
[    0.152319] isapnp: Scanning for PnP cards...
[    0.856077] isapnp: No Plug & Play device found
[    0.861733] Linux agpgart interface v0.103
[    0.861829] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.862138] PNP: No PS/2 controller found. Probing ports directly.
[    0.862960] i8042.c: No controller found.
[    0.863034] mice: PS/2 mouse device common for all mice
[    0.863161] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.863210] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.863235] cpuidle: using governor ladder
[    0.863242] cpuidle: using governor menu
[    0.863432] pcifront pci-0: Installing PCI frontend
[    0.863498] TCP cubic registered
[    0.863646] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.881473] NET: Registered protocol family 10
[    0.881816] lo: Disabled Privacy Extensions
[    0.882024] Mobile IPv6
[    0.882031] NET: Registered protocol family 17
[    0.882039] Registering the dns_resolver key type
[    0.882058] Using IPI No-Shortcut mode
[    0.882190] registered taskstats version 1
[    0.884859] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.884868] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.980061] XENBUS: Device with no driver: device/vbd/51712
[    0.980075] XENBUS: Device with no driver: device/console/0
[    0.980100] /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    0.980132] Initalizing network drop monitor service
[    0.980232] Freeing unused kernel memory: 412k freed
[    0.981036] Write protecting the kernel text: 2704k
[    0.981303] Write protecting the kernel read-only data: 1016k
Loading, please wait...
[    1.060454] udev[46]: starting version 164
[    1.183225] sky2: driver version 1.28
[    1.183333] sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
[    1.183528] sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
[    1.184784] sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43
[    1.193834] blkfront: xvda: barriers enabled
[    1.195628]  xvda: xvda1 xvda2
[    1.349422] ------------[ cut here ]------------
[    1.349443] WARNING: at /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/block/blk-core.c:337 blk_start_queue+0x1d/0x31()
[    1.349457] Modules linked in: sky2 xen_blkfront
[    1.349471] Pid: 0, comm: swapper Not tainted 2.6.37-trunk-686-bigmem #1
[    1.349478] Call Trace:
[    1.349487]  [<c1036005>] ? warn_slowpath_common+0x6a/0x7b
[    1.349494]  [<c1139fd9>] ? blk_start_queue+0x1d/0x31
[    1.349502]  [<c1036023>] ? warn_slowpath_null+0xd/0x10
[    1.349509]  [<c1139fd9>] ? blk_start_queue+0x1d/0x31
[    1.349519]  [<e102a723>] ? kick_pending_request_queues+0x19/0x27 [xen_blkfront]
[    1.349529]  [<e102a925>] ? blkif_interrupt+0x1c8/0x1e3 [xen_blkfront]
[    1.349542]  [<c10753dc>] ? handle_IRQ_event+0x45/0xef
[    1.349550]  [<c1076c3d>] ? handle_fasteoi_irq+0x80/0xb1
[    1.349559]  [<c11a85d1>] ? __xen_evtchn_do_upcall+0xd0/0x155
[    1.349567]  [<c11a8cce>] ? xen_evtchn_do_upcall+0x18/0x26
[    1.349577]  [<c1009117>] ? xen_do_upcall+0x7/0xc
[    1.349584]  [<c10023a7>] ? hypercall_page+0x3a7/0x1001
[    1.349591]  [<c1006246>] ? xen_safe_halt+0xf/0x1b
[    1.349598]  [<c100e276>] ? default_idle+0x3f/0x5d
[    1.349605]  [<c1007dc4>] ? cpu_idle+0x98/0xdc
[    1.349613]  [<c14058a0>] ? start_kernel+0x320/0x325
[    1.349620]  [<c1407479>] ? xen_start_kernel+0x5a1/0x5a8
[    1.349627]  [<c1409046>] ? trap_init+0x1fe/0x367
[    1.349633] ---[ end trace c57c5a433d71a17e ]---
[    1.349638] ------------[ cut here ]------------
[    1.349645] WARNING: at /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/block/blk-core.c:245 blk_remove_plug+0x1d/0x78()
[    1.349657] Modules linked in: sky2 xen_blkfront
[    1.349667] Pid: 0, comm: swapper Tainted: G        W   2.6.37-trunk-686-bigmem #1
[    1.349674] Call Trace:
[    1.349682]  [<c1036005>] ? warn_slowpath_common+0x6a/0x7b
[    1.349689]  [<c1139e6e>] ? blk_remove_plug+0x1d/0x78
[    1.349696]  [<c1036023>] ? warn_slowpath_null+0xd/0x10
[    1.349702]  [<c1139e6e>] ? blk_remove_plug+0x1d/0x78
[    1.349709]  [<c1139ed1>] ? __blk_run_queue+0x8/0x5b
[    1.349716]  [<e102a723>] ? kick_pending_request_queues+0x19/0x27 [xen_blkfront]
[    1.349725]  [<e102a925>] ? blkif_interrupt+0x1c8/0x1e3 [xen_blkfront]
[    1.349733]  [<c10753dc>] ? handle_IRQ_event+0x45/0xef
[    1.349743]  [<c1076c3d>] ? handle_fasteoi_irq+0x80/0xb1
[    1.349750]  [<c11a85d1>] ? __xen_evtchn_do_upcall+0xd0/0x155
[    1.349758]  [<c11a8cce>] ? xen_evtchn_do_upcall+0x18/0x26
[    1.349764]  [<c1009117>] ? xen_do_upcall+0x7/0xc
[    1.349773]  [<c10023a7>] ? hypercall_page+0x3a7/0x1001
[    1.349780]  [<c1006246>] ? xen_safe_halt+0xf/0x1b
[    1.349788]  [<c100e276>] ? default_idle+0x3f/0x5d
[    1.349795]  [<c1007dc4>] ? cpu_idle+0x98/0xdc
[    1.349801]  [<c14058a0>] ? start_kernel+0x320/0x325
[    1.349809]  [<c1407479>] ? xen_start_kernel+0x5a1/0x5a8
[    1.349816]  [<c1409046>] ? trap_init+0x1fe/0x367
[    1.349824] ---[ end trace c57c5a433d71a17f ]---
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... [    1.507113] device-mapper: uevent: version 1.0.3
[    1.507692] device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@redhat.com
done.
Begin: Running /scripts/local-premount ... done.
[    1.644939] EXT3-fs: barriers not enabled
[    1.657461] kjournald starting.  Commit interval 5 seconds
[    1.657498] EXT3-fs (dm-0): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd[    2.746299] udev[156]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[    3.080564] input: PC Speaker as /devices/platform/pcspkr/input/input0
done.
Activating swap...done.
[    6.062837] EXT3-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
[    6.098388] EXT3-fs (dm-0): using internal journal
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
[    6.652343] sky2 0000:00:00.0: eth0: enabling interface
[    6.652423] ADDRCONF(NETDEV_UP): eth0: link is not ready
Configuring network interfaces...done.
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd[    2.746299] udev[156]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[    3.080564] input: PC Speaker as /devices/platform/pcspkr/input/input0
done.
Activating swap...done.
[    6.062837] EXT3-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
[    6.098388] EXT3-fs (dm-0): using internal journal
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
[    6.652343] sky2 0000:00:00.0: eth0: enabling interface
[    6.652423] ADDRCONF(NETDEV_UP): eth0: link is not ready
Configuring network interfaces...done.
Cleaning up temporary files....
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 6.0 xps-106 hvc0

xps-106 login: [    9.148520] sky2 0000:00:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow control both
[    9.148605] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

[-- Attachment #6: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Ian Campbell]

[-- Attachment #1: Type: text/plain, Size: 5354 bytes --]

On Wed, 2011-02-02 at 10:24 +0000, Jean Baptiste Favre wrote:
> Hello Ian,
> 
> My domU config file:
> 
> # cat /cluster/xen/xps-106.cfg
> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
> builder      = 'linux'
> memory       = '398'
> vcpus        = '1'
> cpus         = '2'
> localtime    = 0
> serial       = 'pty'
> boot         = 'cdn'
> disk         = [ 'drbd:xps-106,xvda,w' ]
> on_poweroff  = 'destroy'
> on_reboot    = 'restart'
> on_crash     = 'restart'
> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
> console=hvc0 xencons=tty"
> pci = [ '04:00.0' ]
> name         = 'xps-106'
> hostname     = 'xps-106.clichy.jbfavre.org'
> 
> 
> Le 02/02/2011 10:27, Ian Campbell a écrit :
> > On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:
> > > Le 01/02/2011 17:23, Ian Campbell a écrit :
> >
> > >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> > >> you post a full guest console log at some point? Comparing the logs for
> > >> the 256MB, 398MB and 512MB guest RAM case might be useful.
> > > No sure I've ever posted that logs. But I can redo my tests :)
> >
> > yes, please do that.
> Please find attached both console startup logs with 256M and 512M:
> 256M_domU_console_logs.txt
> 512M_domU_console_logs.txt

Thanks, I stripped the timestamps and cleaned up some differences due to
missing carriage returns, those versions are attached.

Interesting bits of diff (- == 256, + == 512) are:
        -Allocating PCI resources starting at 10800000 (gap: 10800000:ef800000)
        +Allocating PCI resources starting at 20800000 (gap: 20800000:df800000)

So the PCI resources are higher and smaller in the 512MB case.

        -Placing 64MB software IO TLB between cbd7ee40 - cfd7ee40
        -software IO TLB at phys 0xbd7ee40 - 0xfd7ee40
        +Placing 64MB software IO TLB between dbb1a000 - dfb1a000
        +software IO TLB at phys 0x1bb1a000 - 0x1fb1a000

Fair enough?

         sky2: driver version 1.28
        -sky2 0000:00:00.0: BAR 0: set to [mem 0xfeb00000-0xfeb03fff 64bit] (PCI address [0xfeb00000-0xfeb03fff])
        -sky2 0000:00:00.0: BAR 2: set to [io  0xe800-0xe8ff] (PCI address [0xe800-0xe8ff])
        -sky2 0000:00:00.0: enabling device (0000 -> 0003)
         sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
         sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
         sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43

So there is some sort of remapping going on in the 256MB case? (or the
512MB logs are missing a bit, which can happen). Can you post the
content of domU's /proc/{interrupts,iomem,ioports} for both cases?

> >
> > Please can you also collect and post the information from ifconfig and
> > ethtool -S which I asked for earlier.
> Attached as well:
> 256_domU_ifconfig_ping_ethtool.txt
> 512_domU_ifconfig_ping_ethtool.txt

Thanks. The interesting bit here is that the 256MB case is registering
RX dropped frames. 512_domU_ifconfig_ping_ethtool.txt did not include an
ifconfig after the experiment but I assume the dropped frames are not
present there.

The rx_dropped statistic is only cranked in a small number of places,
once in the driver and a handful in the networking core, only one of the
core cases looks likely to be relevant. The following patch should help
us figure out where the frames are dropped...

Ian.

diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..645d9e9 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2339,6 +2339,7 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2,
 		re->skb->ip_summed = CHECKSUM_NONE;
 		skb_put(skb, length);
 	}
+	WARN(skb == NULL, "sky2: receive_copy failed netdev_alloc_skb_ip_align length %u\n", length);
 	return skb;
 }
 
@@ -2386,10 +2387,15 @@ static struct sk_buff *receive_new(struct sky2_port *sky2,
 
 	nre.skb = sky2_rx_alloc(sky2);
 	if (unlikely(!nre.skb))
+	{
+		WARN(1, "sky2: receive_new failed sky2_rx_alloc\n");
 		goto nobuf;
-
+	}
 	if (sky2_rx_map_skb(sky2->hw->pdev, &nre, hdr_space))
+	{
+		WARN(1, "sky2: receive_new failed sky2_rx_map_skb\n");
 		goto nomap;
+	}
 
 	skb = re->skb;
 	sky2_rx_unmap_skb(sky2->hw->pdev, re);
diff --git a/net/core/dev.c b/net/core/dev.c
index 54277df..9c99612 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1516,6 +1516,7 @@ int dev_forward_skb(struct net_device *dev, struct sk_buff *skb)
 
 	if (unlikely(!(dev->flags & IFF_UP) ||
 		     (skb->len > (dev->mtu + dev->hard_header_len + VLAN_HLEN)))) {
+		printk(KERN_CRIT "dev_forward_skb dropping skb\n");
 		atomic_long_inc(&dev->rx_dropped);
 		kfree_skb(skb);
 		return NET_RX_DROP;
@@ -2700,6 +2701,7 @@ enqueue:
 
 	local_irq_restore(flags);
 
+	printk(KERN_CRIT "enqueue_to_backlog dropping skb\n");
 	atomic_long_inc(&skb->dev->rx_dropped);
 	kfree_skb(skb);
 	return NET_RX_DROP;
@@ -3125,6 +3127,7 @@ ncls:
 	if (pt_prev) {
 		ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
 	} else {
+		printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x\n", type);
 		atomic_long_inc(&skb->dev->rx_dropped);
 		kfree_skb(skb);
 		/* Jamal, now you will not able to escape explaining


[-- Attachment #2: 512M_domU_console_logs.txtX --]
[-- Type: text/plain, Size: 12631 bytes --]

# xm create /cluster/xen/xps-106.cfg -c
Using config file "/cluster/xen/xps-106.cfg".
[214817.006300] pciback 0000:04:00.0: device has been assigned to another domain! Over-writting the ownership, but beware.
Started domain xps-106 (id=21)
(XEN) traps.c:2308:d21 Domain attempted WRMSR 00000000c0010004 from 0000c0b2:cbfa8461 to 00000000:0000abcd.
Reserving virtual address space above 0xf5800000
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.37-trunk-686-bigmem (Debian 2.6.37-1~experimental.1) (ben@decadent.org.uk) (gcc version 4.4.5 (Debian 4.4.5-10) ) #1 SMP Thu Jan 6 15:21:44 UTC 2011
ACPI in unprivileged domain disabled
released 0 pages of unused memory
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000020800000 (usable)
NX (Execute Disable) protection: active
DMI not present or invalid.
last_pfn = 0x20800 max_arch_pfn = 0x1000000
init_memory_mapping: 0000000000000000-0000000020800000
RAMDISK: 01793000 - 02ecc000
0MB HIGHMEM available.
520MB LOWMEM available.
  mapped low ram: 0 - 20800000
  low ram: 0 - 20800000
Zone PFN ranges:
  DMA      0x00000010 -> 0x00001000
  Normal   0x00001000 -> 0x00020800
  HighMem  empty
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000010 -> 0x000000a0
    0: 0x00000100 -> 0x00020800
Using APIC driver default
SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
SMP: Allowing 1 CPUs, 0 hotplug CPUs
Local APIC disabled by BIOS -- you can enable it with "lapic"
APIC: disable apic facility
APIC: switched to apic NOOP
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
Allocating PCI resources starting at 20800000 (gap: 20800000:df800000)
Booting paravirtualized kernel on Xen
Xen version: 4.0.1 (preserve-AD)
setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 13 pages/cpu @dfbde000 s30336 r0 d22912 u53248
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 131968
Kernel command line: root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Initializing CPU#0
Placing 64MB software IO TLB between dbb1a000 - dfb1a000
software IO TLB at phys 0x1bb1a000 - 0x1fb1a000
Initializing HighMem for node 0 (00000000:00000000)
Memory: 422200k/532480k available (2703k kernel code, 109832k reserved, 1410k data, 412k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xf5536000 - 0xf57ff000   (2852 kB)
    pkmap   : 0xf5200000 - 0xf5400000   (2048 kB)
    vmalloc : 0xe1000000 - 0xf51fe000   ( 321 MB)
    lowmem  : 0xc0000000 - 0xe0800000   ( 520 MB)
      .init : 0xc1405000 - 0xc146c000   ( 412 kB)
      .data : 0xc12a3d40 - 0xc14046e0   (1410 kB)
      .text : 0xc1000000 - 0xc12a3d40   (2703 kB)
SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Hierarchical RCU implementation.
 RCU dyntick-idle grace-period acceleration is enabled.
 RCU-based detection of stalled CPUs is disabled.
NR_IRQS:1280
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
installing Xen timer for CPU 0
Detected 2506.222 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5012.44 BogoMIPS (lpj=10024888)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Disabled at boot.
Mount-cache hash table entries: 512
Initializing cgroup subsys ns
ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
Initializing cgroup subsys cpuacct
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Performance Events:
no APIC, boot with the "lapic" boot parameter to force-enable it.
no hardware sampling interrupt available.
Broken PMU hardware detected, software events only.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 16k freed
NMI watchdog disabled for cpu0: unable to create perf event: -2
Brought up 1 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
regulator: dummy:
NET: Registered protocol family 16
PCI: setting up Xen PCI frontend stub
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
last_pfn = 0x20800 max_arch_pfn = 0x1000000
vgaarb: loaded
PCI: System does not support PCI
PCI: System does not support PCI
Switching to clocksource xen
pnp: PnP ACPI: disabled
PnPBIOS: Disabled
NET: Registered protocol family 2
IP route cache hash table entries: 8192 (order: 3, 32768 bytes)
TCP established hash table entries: 32768 (order: 6, 262144 bytes)
TCP bind hash table entries: 32768 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 32768 bind 32768)
TCP reno registered
UDP hash table entries: 512 (order: 2, 16384 bytes)
UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
NET: Registered protocol family 1
Unpacking initramfs...
Freeing initrd memory: 23780k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1296641097.317:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
msgmni has been set to 871
alg: No test for stdrng (krng)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Linux agpgart interface v0.103
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
PNP: No PS/2 controller found. Probing ports directly.
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
rtc_cmos: probe of rtc_cmos failed with error -38
cpuidle: using governor ladder
cpuidle: using governor menu
pcifront pci-0: Installing PCI frontend
TCP cubic registered
pcifront pci-0: Creating PCI Frontend Bus 0000:00
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Mobile IPv6
NET: Registered protocol family 17
Registering the dns_resolver key type
Using IPI No-Shortcut mode
registered taskstats version 1
pcifront pci-0: claiming resource 0000:00:00.0/0
pcifront pci-0: claiming resource 0000:00:00.0/2
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/console/0
/build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 412k freed
Write protecting the kernel text: 2704k
Write protecting the kernel read-only data: 1016k
Loading, please wait...
udev[46]: starting version 164
sky2: driver version 1.28
sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43
blkfront: xvda: barriers enabled
 xvda: xvda1 xvda2
------------[ cut here ]------------
WARNING: at /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/block/blk-core.c:337 blk_start_queue+0x1d/0x31()
Modules linked in: sky2 xen_blkfront
Pid: 0, comm: swapper Not tainted 2.6.37-trunk-686-bigmem #1
Call Trace:
 [<c1036005>] ? warn_slowpath_common+0x6a/0x7b
 [<c1139fd9>] ? blk_start_queue+0x1d/0x31
 [<c1036023>] ? warn_slowpath_null+0xd/0x10
 [<c1139fd9>] ? blk_start_queue+0x1d/0x31
 [<e102a723>] ? kick_pending_request_queues+0x19/0x27 [xen_blkfront]
 [<e102a925>] ? blkif_interrupt+0x1c8/0x1e3 [xen_blkfront]
 [<c10753dc>] ? handle_IRQ_event+0x45/0xef
 [<c1076c3d>] ? handle_fasteoi_irq+0x80/0xb1
 [<c11a85d1>] ? __xen_evtchn_do_upcall+0xd0/0x155
 [<c11a8cce>] ? xen_evtchn_do_upcall+0x18/0x26
 [<c1009117>] ? xen_do_upcall+0x7/0xc
 [<c10023a7>] ? hypercall_page+0x3a7/0x1001
 [<c1006246>] ? xen_safe_halt+0xf/0x1b
 [<c100e276>] ? default_idle+0x3f/0x5d
 [<c1007dc4>] ? cpu_idle+0x98/0xdc
 [<c14058a0>] ? start_kernel+0x320/0x325
 [<c1407479>] ? xen_start_kernel+0x5a1/0x5a8
 [<c1409046>] ? trap_init+0x1fe/0x367
---[ end trace c57c5a433d71a17e ]---
------------[ cut here ]------------
WARNING: at /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/block/blk-core.c:245 blk_remove_plug+0x1d/0x78()
Modules linked in: sky2 xen_blkfront
Pid: 0, comm: swapper Tainted: G        W   2.6.37-trunk-686-bigmem #1
Call Trace:
 [<c1036005>] ? warn_slowpath_common+0x6a/0x7b
 [<c1139e6e>] ? blk_remove_plug+0x1d/0x78
 [<c1036023>] ? warn_slowpath_null+0xd/0x10
 [<c1139e6e>] ? blk_remove_plug+0x1d/0x78
 [<c1139ed1>] ? __blk_run_queue+0x8/0x5b
 [<e102a723>] ? kick_pending_request_queues+0x19/0x27 [xen_blkfront]
 [<e102a925>] ? blkif_interrupt+0x1c8/0x1e3 [xen_blkfront]
 [<c10753dc>] ? handle_IRQ_event+0x45/0xef
 [<c1076c3d>] ? handle_fasteoi_irq+0x80/0xb1
 [<c11a85d1>] ? __xen_evtchn_do_upcall+0xd0/0x155
 [<c11a8cce>] ? xen_evtchn_do_upcall+0x18/0x26
 [<c1009117>] ? xen_do_upcall+0x7/0xc
 [<c10023a7>] ? hypercall_page+0x3a7/0x1001
 [<c1006246>] ? xen_safe_halt+0xf/0x1b
 [<c100e276>] ? default_idle+0x3f/0x5d
 [<c1007dc4>] ? cpu_idle+0x98/0xdc
 [<c14058a0>] ? start_kernel+0x320/0x325
 [<c1407479>] ? xen_start_kernel+0x5a1/0x5a8
 [<c1409046>] ? trap_init+0x1fe/0x367
---[ end trace c57c5a433d71a17f ]---
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ...
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@redhat.com
done.
Begin: Running /scripts/local-premount ... done.
EXT3-fs: barriers not enabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (dm-0): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd
udev[156]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...
input: PC Speaker as /devices/platform/pcspkr/input/input0
done.
Activating swap...done.
EXT3-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs (dm-0): using internal journal
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
sky2 0000:00:00.0: eth0: enabling interface
ADDRCONF(NETDEV_UP): eth0: link is not ready
Configuring network interfaces...done.
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd
udev[156]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...
input: PC Speaker as /devices/platform/pcspkr/input/input0
done.
Activating swap...done.
EXT3-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs (dm-0): using internal journal
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
sky2 0000:00:00.0: eth0: enabling interface
ADDRCONF(NETDEV_UP): eth0: link is not ready
Configuring network interfaces...done.
Cleaning up temporary files....
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 6.0 xps-106 hvc0

xps-106 login: 
sky2 0000:00:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow control both
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

[-- Attachment #3: 256M_domU_console_logs.txtX --]
[-- Type: text/plain, Size: 9495 bytes --]

# xm create /cluster/xen/xps-106.cfg -c
Using config file "/cluster/xen/xps-106.cfg".
[213362.865921] pciback 0000:04:00.0: device has been assigned to another domain! Over-writting the ownership, but beware.
Started domain xps-106 (id=16)
Reserving virtual address space above 0xf5800000
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.37-trunk-686-bigmem (Debian 2.6.37-1~experimental.1) (ben@decadent.org.uk) (gcc version 4.4.5 (Debian 4.4.5-10) ) #1 SMP Thu Jan 6 15:21:44 UTC 2011
ACPI in unprivileged domain disabled
released 0 pages of unused memory
BIOS-provided physical RAM map:
 (XEN) traps.c:2308:d16 Domain attempted WRMSR 00000000c0010004 from 0000c0b2:cbfa8461 to 00000000:0000abcd.
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000010800000 (usable)
NX (Execute Disable) protection: active
DMI not present or invalid.
last_pfn = 0x10800 max_arch_pfn = 0x1000000
init_memory_mapping: 0000000000000000-0000000010800000
RAMDISK: 01793000 - 02ecc000
0MB HIGHMEM available.
264MB LOWMEM available.
  mapped low ram: 0 - 10800000
  low ram: 0 - 10800000
Zone PFN ranges:
  DMA      0x00000010 -> 0x00001000
  Normal   0x00001000 -> 0x00010800
  HighMem  empty
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000010 -> 0x000000a0
    0: 0x00000100 -> 0x00010800
Using APIC driver default
SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
SMP: Allowing 1 CPUs, 0 hotplug CPUs
Local APIC disabled by BIOS -- you can enable it with "lapic"
APIC: disable apic facility
APIC: switched to apic NOOP
PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
Allocating PCI resources starting at 10800000 (gap: 10800000:ef800000)
Booting paravirtualized kernel on Xen
Xen version: 4.0.1 (preserve-AD)
setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 13 pages/cpu @cfddf000 s30336 r0 d22912 u53248
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 66944
Kernel command line: root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty
PID hash table entries: 2048 (order: 1, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Initializing CPU#0
Placing 64MB software IO TLB between cbd7ee40 - cfd7ee40
software IO TLB at phys 0xbd7ee40 - 0xfd7ee40
Initializing HighMem for node 0 (00000000:00000000)
Memory: 163268k/270336k available (2703k kernel code, 106620k reserved, 1410k data, 412k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xf5536000 - 0xf57ff000   (2852 kB)
    pkmap   : 0xf5200000 - 0xf5400000   (2048 kB)
    vmalloc : 0xd1000000 - 0xf51fe000   ( 577 MB)
    lowmem  : 0xc0000000 - 0xd0800000   ( 264 MB)
      .init : 0xc1405000 - 0xc146c000   ( 412 kB)
      .data : 0xc12a3d40 - 0xc14046e0   (1410 kB)
      .text : 0xc1000000 - 0xc12a3d40   (2703 kB)
SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Hierarchical RCU implementation.
 RCU dyntick-idle grace-period acceleration is enabled.
 RCU-based detection of stalled CPUs is disabled.
NR_IRQS:1280
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
installing Xen timer for CPU 0
Detected 2506.222 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 5012.44 BogoMIPS (lpj=10024888)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Disabled at boot.
Mount-cache hash table entries: 512
Initializing cgroup subsys ns
ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
Initializing cgroup subsys cpuacct
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Performance Events:
no APIC, boot with the "lapic" boot parameter to force-enable it.
no hardware sampling interrupt available.
Broken PMU hardware detected, software events only.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 16k freed
NMI watchdog disabled for cpu0: unable to create perf event: -2
Brought up 1 CPUs
devtmpfs: initialized
Grant table initialized
regulator: core version 0.5
regulator: dummy:
NET: Registered protocol family 16
PCI: setting up Xen PCI frontend stub
bio: create slab <bio-0> at 0
ACPI: Interpreter disabled.
xen_balloon: Initialising balloon driver.
last_pfn = 0x10800 max_arch_pfn = 0x1000000
vgaarb: loaded
PCI: System does not support PCI
PCI: System does not support PCI
Switching to clocksource xen
pnp: PnP ACPI: disabled
PnPBIOS: Disabled
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 16384 (order: 5, 131072 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
Unpacking initramfs...
Freeing initrd memory: 23780k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
audit: initializing netlink socket (disabled)
type=2000 audit(1296639642.987:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
msgmni has been set to 365
alg: No test for stdrng (krng)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Linux agpgart interface v0.103
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
PNP: No PS/2 controller found. Probing ports directly.
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
pcifront pci-0: Installing PCI frontend
rtc_cmos: probe of rtc_cmos failed with error -38
cpuidle: using governor ladder
cpuidle: using governor menu
pcifront pci-0: Creating PCI Frontend Bus 0000:00
pcifront pci-0: claiming resource 0000:00:00.0/0
pcifront pci-0: claiming resource 0000:00:00.0/2
TCP cubic registered
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Mobile IPv6
NET: Registered protocol family 17
Registering the dns_resolver key type
Using IPI No-Shortcut mode
registered taskstats version 1
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/console/0
/build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
Initalizing network drop monitor service
Freeing unused kernel memory: 412k freed
Write protecting the kernel text: 2704k
Write protecting the kernel read-only data: 1016k
Loading, please wait...
udev[46]: starting version 164
sky2: driver version 1.28
sky2 0000:00:00.0: BAR 0: set to [mem 0xfeb00000-0xfeb03fff 64bit] (PCI address [0xfeb00000-0xfeb03fff])
sky2 0000:00:00.0: BAR 2: set to [io  0xe800-0xe8ff] (PCI address [0xe800-0xe8ff])
sky2 0000:00:00.0: enabling device (0000 -> 0003)
sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43
blkfront: xvda: barriers enabled
 xvda: xvda1 xvda2
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ...
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@redhat.com
done.
Begin: Running /scripts/local-premount ... done.
EXT3-fs: barriers not enabled
kjournald starting.  Commit interval 5 seconds
EXT3-fs (dm-0): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd
udev[167]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...
input: PC Speaker as /devices/platform/pcspkr/input/input0
done.
Activating swap...done.
EXT3-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
EXT3-fs (dm-0): using internal journal
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
Configuring network interfaces...
sky2 0000:00:00.0: eth0: enabling interface
ADDRCONF(NETDEV_UP): eth0: link is not ready
done.
Cleaning up temporary files....
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 6.0 xps-106 hvc0

xps-106 login: 
sky2 0000:00:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow control both
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Jean Baptiste Favre]

[-- Attachment #1: Type: text/plain, Size: 2831 bytes --]

Hello,

Le 02/02/2011 11:59, Ian Campbell a écrit :
> [...]
> >>> No sure I've ever posted that logs. But I can redo my tests :)
> >>
> >> yes, please do that.
> > Please find attached both console startup logs with 256M and 512M:
> > 256M_domU_console_logs.txt
> > 512M_domU_console_logs.txt
>
> Thanks, I stripped the timestamps and cleaned up some differences due to
> missing carriage returns, those versions are attached.
>
> Interesting bits of diff (- == 256, + == 512) are:
>         -Allocating PCI resources starting at 10800000 (gap:
> 10800000:ef800000)
>         +Allocating PCI resources starting at 20800000 (gap:
> 20800000:df800000)
>
> So the PCI resources are higher and smaller in the 512MB case.
>
>         -Placing 64MB software IO TLB between cbd7ee40 - cfd7ee40
>         -software IO TLB at phys 0xbd7ee40 - 0xfd7ee40
>         +Placing 64MB software IO TLB between dbb1a000 - dfb1a000
>         +software IO TLB at phys 0x1bb1a000 - 0x1fb1a000
>
> Fair enough?
>
>          sky2: driver version 1.28
>         -sky2 0000:00:00.0: BAR 0: set to [mem 0xfeb00000-0xfeb03fff
> 64bit] (PCI address [0xfeb00000-0xfeb03fff])
>         -sky2 0000:00:00.0: BAR 2: set to [io  0xe800-0xe8ff] (PCI
> address [0xe800-0xe8ff])
>         -sky2 0000:00:00.0: enabling device (0000 -> 0003)
>          sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
>          sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
>          sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43
>
> So there is some sort of remapping going on in the 256MB case? (or the
> 512MB logs are missing a bit, which can happen). Can you post the
> content of domU's /proc/{interrupts,iomem,ioports} for both cases?
Please find them attached.
I also join last full console logs for both 256M and 512M: I saw "Call
Trace" for 256M I did not had last time.

> >> Please can you also collect and post the information from ifconfig and
> >> ethtool -S which I asked for earlier.
> > Attached as well:
> > 256_domU_ifconfig_ping_ethtool.txt
> > 512_domU_ifconfig_ping_ethtool.txt
>
> Thanks. The interesting bit here is that the 256MB case is registering
> RX dropped frames. 512_domU_ifconfig_ping_ethtool.txt did not include an
> ifconfig after the experiment but I assume the dropped frames are not
> present there.
I did not include second ifconfig for 512M as ping always works with
512M memory.
As far as I can remember, I did not noticed anything special.

> The rx_dropped statistic is only cranked in a small number of places,
> once in the driver and a handful in the networking core, only one of the
> core cases looks likely to be relevant. The following patch should help
> us figure out where the frames are dropped...
Let's patch Debian 2.6.37 32bits kernel so :)

Regards,
JB

[-- Attachment #2: 256M_domU_proc_outputs.txt --]
[-- Type: text/plain, Size: 1733 bytes --]

# cat /proc/interrupts
            CPU0
  55:          8  xen-pirq-pcifront-msi  sky2@pci:0000:00:00.0
1271:        946   xen-dyn-event     blkif
1272:        133   xen-dyn-event     hvc_console
1273:          0   xen-dyn-event     pcifront
1274:        259   xen-dyn-event     xenbus
1275:          0  xen-percpu-ipi       callfuncsingle0
1276:          0  xen-percpu-virq      debug0
1277:          0  xen-percpu-ipi       callfunc0
1278:          0  xen-percpu-ipi       resched0
1279:        761  xen-percpu-virq      timer0
 NMI:          0   Non-maskable interrupts
 LOC:          0   Local timer interrupts
 SPU:          0   Spurious interrupts
 PMI:          0   Performance monitoring interrupts
 IWI:          0   IRQ work interrupts
 RES:          0   Rescheduling interrupts
 CAL:          0   Function call interrupts
 TLB:          0   TLB shootdowns
 TRM:          0   Thermal event interrupts
 THR:          0   Threshold APIC interrupts
 MCE:          0   Machine check exceptions
 MCP:          0   Machine check polls
 ERR:          0
 MIS:          0

# cat /proc/iomem
00000000-0000ffff : reserved
00010000-0009ffff : System RAM
000a0000-000fffff : reserved
  000f0000-000fffff : System ROM
00100000-107fffff : System RAM
  01000000-012a3d3f : Kernel code
  012a3d40-014046df : Kernel data
  01470000-015215ab : Kernel bss
feb00000-feb03fff : 0000:00:00.0
  feb00000-feb03fff : sky2

# cat /proc/ioports
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard
0070-0071 : rtc_cmos
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
e800-e8ff : 0000:00:00.0
  e800-e8ff : sky2

[-- Attachment #3: 512M_domU_console_logs_2.txt --]
[-- Type: text/plain, Size: 15584 bytes --]

# xm create /cluster/xen/xps-106.cfg -c
Using config file "/cluster/xen/xps-106.cfg".
[219194.082018] pciback 0000:04:00.0: device has been assigned to another domain! Over-writting the ownership, but beware.
Started domain xps-106 (id=25)
(XEN) traps.c:2308:d25 Domain attempted WRMSR 00000000c0010004 from 0000c0b2:cbfa8461 to 00000000:0000abcd.
[    0.000000] Reserving virtual address space above 0xf5800000
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.37-trunk-686-bigmem (Debian 2.6.37-1~experimental.1) (ben@decadent.org.uk) (gcc version 4.4.5 (Debian 4.4.5-10) ) #1 SMP Thu Jan 6 15:21:44 UTC 2011
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000020800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] last_pfn = 0x20800 max_arch_pfn = 0x1000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000020800000
[    0.000000] RAMDISK: 01793000 - 02ecc000
[    0.000000] 0MB HIGHMEM available.
[    0.000000] 520MB LOWMEM available.
[    0.000000]   mapped low ram: 0 - 20800000
[    0.000000]   low ram: 0 - 20800000
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   Normal   0x00001000 -> 0x00020800
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00020800
[    0.000000] Using APIC driver default
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 20800000 (gap: 20800000:df800000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.0.1 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Embedded 13 pages/cpu @dfbde000 s30336 r0 d22912 u53248
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 131968
[    0.000000] Kernel command line: root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Initializing CPU#0
[    0.000000] Placing 64MB software IO TLB between dbb1a000 - dfb1a000
[    0.000000] software IO TLB at phys 0x1bb1a000 - 0x1fb1a000
[    0.000000] Initializing HighMem for node 0 (00000000:00000000)
[    0.000000] Memory: 422200k/532480k available (2703k kernel code, 109832k reserved, 1410k data, 412k init, 0k highmem)
[    0.000000] virtual kernel memory layout:
[    0.000000]     fixmap  : 0xf5536000 - 0xf57ff000   (2852 kB)
[    0.000000]     pkmap   : 0xf5200000 - 0xf5400000   (2048 kB)
[    0.000000]     vmalloc : 0xe1000000 - 0xf51fe000   ( 321 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xe0800000   ( 520 MB)
[    0.000000]       .init : 0xc1405000 - 0xc146c000   ( 412 kB)
[    0.000000]       .data : 0xc12a3d40 - 0xc14046e0   (1410 kB)
[    0.000000]       .text : 0xc1000000 - 0xc12a3d40   (2703 kB)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:1280
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2506.222 MHz processor.
[    0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 5012.44 BogoMIPS (lpj=10024888)
[    0.004000] pid_max: default: 32768 minimum: 301
[    0.004000] Security Framework initialized
[    0.004000] SELinux:  Disabled at boot.
[    0.004000] Mount-cache hash table entries: 512
[    0.004000] Initializing cgroup subsys ns
[    0.004000] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.004000] Initializing cgroup subsys cpuacct
[    0.004000] Initializing cgroup subsys devices
[    0.004000] Initializing cgroup subsys freezer
[    0.004000] Initializing cgroup subsys net_cls
[    0.004000] Initializing cgroup subsys blkio
[    0.004000] CPU: Physical Processor ID: 0
[    0.004000] CPU: Processor Core ID: 0
[    0.004000] Performance Events:
[    0.004000] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.004000] no hardware sampling interrupt available.
[    0.004000] Broken PMU hardware detected, software events only.
[    0.004000] SMP alternatives: switching to UP code
[    0.014434] Freeing SMP alternatives: 16k freed
[    0.014650] NMI watchdog disabled for cpu0: unable to create perf event: -2
[    0.014726] Brought up 1 CPUs
[    0.014806] devtmpfs: initialized
[    0.015100] Grant table initialized
[    0.015107] regulator: core version 0.5
[    0.015161] regulator: dummy:
[    0.015207] NET: Registered protocol family 16
[    0.015410] PCI: setting up Xen PCI frontend stub
[    0.016503] bio: create slab <bio-0> at 0
[    0.016570] ACPI: Interpreter disabled.
[    0.016596] xen_balloon: Initialising balloon driver.
[    0.016596] last_pfn = 0x20800 max_arch_pfn = 0x1000000
[    0.016596] vgaarb: loaded
[    0.016596] PCI: System does not support PCI
[    0.016596] PCI: System does not support PCI
[    0.016596] Switching to clocksource xen
[    0.017420] pnp: PnP ACPI: disabled
[    0.017427] PnPBIOS: Disabled
[    0.019619] NET: Registered protocol family 2
[    0.019667] IP route cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.019847] TCP established hash table entries: 32768 (order: 6, 262144 bytes)
[    0.019964] TCP bind hash table entries: 32768 (order: 6, 262144 bytes)
[    0.020091] TCP: Hash tables configured (established 32768 bind 32768)
[    0.020098] TCP reno registered
[    0.020104] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.020119] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.020217] NET: Registered protocol family 1
[    0.020275] Unpacking initramfs...
[    0.046676] Freeing initrd memory: 23780k freed
[    0.054687] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.054923] audit: initializing netlink socket (disabled)
[    0.054940] type=2000 audit(1296645473.960:1): initialized
[    0.063287] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.065699] VFS: Disk quotas dquot_6.5.2
[    0.065764] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.065859] msgmni has been set to 871
[    0.066056] alg: No test for stdrng (krng)
[    0.066121] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.066130] io scheduler noop registered
[    0.066136] io scheduler deadline registered
[    0.066153] io scheduler cfq registered (default)
[    0.163208] isapnp: Scanning for PnP cards...
[    0.515462] isapnp: No Plug & Play device found
[    0.520929] Linux agpgart interface v0.103
[    0.521028] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.521307] PNP: No PS/2 controller found. Probing ports directly.
[    0.522128] i8042.c: No controller found.
[    0.522198] mice: PS/2 mouse device common for all mice
[    0.522291] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.522342] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.522368] cpuidle: using governor ladder
[    0.522373] cpuidle: using governor menu
[    0.522627] pcifront pci-0: Installing PCI frontend
[    0.522639] TCP cubic registered
[    0.522804] NET: Registered protocol family 10
[    0.522843] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.527895] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.527904] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.528255] lo: Disabled Privacy Extensions
[    0.528454] Mobile IPv6
[    0.528460] NET: Registered protocol family 17
[    0.528468] Registering the dns_resolver key type
[    0.528502] Using IPI No-Shortcut mode
[    0.528608] registered taskstats version 1
[    0.628031] XENBUS: Device with no driver: device/vbd/51712
[    0.628040] XENBUS: Device with no driver: device/console/0
[    0.628061] /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    0.628086] Initalizing network drop monitor service
[    0.628172] Freeing unused kernel memory: 412k freed
[    0.628969] Write protecting the kernel text: 2704k
[    0.629233] Write protecting the kernel read-only data: 1016k
Loading, please wait...
[    0.660285] udev[46]: starting version 164
[    0.725951] sky2: driver version 1.28
[    0.741101] sky2 0000:00:00.0: BAR 0: set to [mem 0xfeb00000-0xfeb03fff 64bit] (PCI address [0xfeb00000-0xfeb03fff])
[    0.741187] sky2 0000:00:00.0: BAR 2: set to [io  0xe800-0xe8ff] (PCI address [0xe800-0xe8ff])
[    0.741258] sky2 0000:00:00.0: enabling device (0000 -> 0003)
[    0.741382] sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
[    0.741554] sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
[    0.742751] sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43
[    0.751244] blkfront: xvda: barriers enabled
[    0.753096]  xvda: xvda1 xvda2
[    0.871721] ------------[ cut here ]------------
[    0.871737] WARNING: at /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/block/blk-core.c:337 blk_start_queue+0x1d/0x31()
[    0.871750] Modules linked in: sky2 xen_blkfront
[    0.871763] Pid: 0, comm: swapper Not tainted 2.6.37-trunk-686-bigmem #1
[    0.871769] Call Trace:
[    0.871778]  [<c1036005>] ? warn_slowpath_common+0x6a/0x7b
[    0.871785]  [<c1139fd9>] ? blk_start_queue+0x1d/0x31
[    0.871792]  [<c1036023>] ? warn_slowpath_null+0xd/0x10
[    0.871800]  [<c1139fd9>] ? blk_start_queue+0x1d/0x31
[    0.871808]  [<e102a723>] ? kick_pending_request_queues+0x19/0x27 [xen_blkfront]
[    0.871817]  [<e102a925>] ? blkif_interrupt+0x1c8/0x1e3 [xen_blkfront]
[    0.871827]  [<c10753dc>] ? handle_IRQ_event+0x45/0xef
[    0.871835]  [<c1076c3d>] ? handle_fasteoi_irq+0x80/0xb1
[    0.871843]  [<c11a85d1>] ? __xen_evtchn_do_upcall+0xd0/0x155
[    0.871851]  [<c11a8cce>] ? xen_evtchn_do_upcall+0x18/0x26
[    0.871858]  [<c1009117>] ? xen_do_upcall+0x7/0xc
[    0.871865]  [<c10023a7>] ? hypercall_page+0x3a7/0x1001
[    0.871873]  [<c1006246>] ? xen_safe_halt+0xf/0x1b
[    0.871879]  [<c100e276>] ? default_idle+0x3f/0x5d
[    0.871886]  [<c1007dc4>] ? cpu_idle+0x98/0xdc
[    0.871893]  [<c14058a0>] ? start_kernel+0x320/0x325
[    0.871900]  [<c1407479>] ? xen_start_kernel+0x5a1/0x5a8
[    0.871907]  [<c1409046>] ? trap_init+0x1fe/0x367
[    0.871913] ---[ end trace 4924792c31d7209b ]---
[    0.871919] ------------[ cut here ]------------
[    0.871925] WARNING: at /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/block/blk-core.c:245 blk_remove_plug+0x1d/0x78()
[    0.871937] Modules linked in: sky2 xen_blkfront
[    0.871948] Pid: 0, comm: swapper Tainted: G        W   2.6.37-trunk-686-bigmem #1
[    0.871955] Call Trace:
[    0.871960]  [<c1036005>] ? warn_slowpath_common+0x6a/0x7b
[    0.871967]  [<c1139e6e>] ? blk_remove_plug+0x1d/0x78
[    0.871974]  [<c1036023>] ? warn_slowpath_null+0xd/0x10
[    0.871981]  [<c1139e6e>] ? blk_remove_plug+0x1d/0x78
[    0.871990]  [<c1139ed1>] ? __blk_run_queue+0x8/0x5b
[    0.871997]  [<e102a723>] ? kick_pending_request_queues+0x19/0x27 [xen_blkfront]
[    0.872007]  [<e102a925>] ? blkif_interrupt+0x1c8/0x1e3 [xen_blkfront]
[    0.872015]  [<c10753dc>] ? handle_IRQ_event+0x45/0xef
[    0.872022]  [<c1076c3d>] ? handle_fasteoi_irq+0x80/0xb1
[    0.872029]  [<c11a85d1>] ? __xen_evtchn_do_upcall+0xd0/0x155
[    0.872037]  [<c11a8cce>] ? xen_evtchn_do_upcall+0x18/0x26
[    0.872044]  [<c1009117>] ? xen_do_upcall+0x7/0xc
[    0.872050]  [<c10023a7>] ? hypercall_page+0x3a7/0x1001
[    0.872057]  [<c1006246>] ? xen_safe_halt+0xf/0x1b
[    0.872063]  [<c100e276>] ? default_idle+0x3f/0x5d
[    0.872071]  [<c1007dc4>] ? cpu_idle+0x98/0xdc
[    0.872078]  [<c14058a0>] ? start_kernel+0x320/0x325
[    0.872087]  [<c1407479>] ? xen_start_kernel+0x5a1/0x5a8
[    0.872098]  [<c1409046>] ? trap_init+0x1fe/0x367
[    0.872104] ---[ end trace 4924792c31d7209c ]---
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... [    0.986899] device-mapper: uevent: version 1.0.3
[    0.987478] device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@redhat.com
done.
Begin: Running /scripts/local-premount ... done.
[    1.121342] EXT3-fs: barriers not enabled
[    1.121674] kjournald starting.  Commit interval 5 seconds
[    1.121691] EXT3-fs (dm-0): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd[    1.897070] udev[161]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[    2.183288] input: PC Speaker as /devices/platform/pcspkr/input/input0
done.
Activating swap...done.
[    5.581677] EXT3-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
[    5.624960] EXT3-fs (dm-0): using internal journal
Loading kernel modules...done.
Cleaning up ifupdown....
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
Configuring network interfaces...[    6.098215] sky2 0000:00:00.0: eth0: enabling interface
[    6.098298] ADDRCONF(NETDEV_UP): eth0: link is not ready
done.
Cleaning up temporary files....
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 6.0 xps-106 hvc0

xps-106 login: [    8.916909] sky2 0000:00:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow control both
[    8.916977] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

[-- Attachment #4: 512M_domU_proc_outputs.txt --]
[-- Type: text/plain, Size: 1733 bytes --]

# cat /proc/interrupts
            CPU0
  55:         12  xen-pirq-pcifront-msi  sky2@pci:0000:00:00.0
1271:        972   xen-dyn-event     blkif
1272:        139   xen-dyn-event     hvc_console
1273:          1   xen-dyn-event     pcifront
1274:        260   xen-dyn-event     xenbus
1275:          0  xen-percpu-ipi       callfuncsingle0
1276:          0  xen-percpu-virq      debug0
1277:          0  xen-percpu-ipi       callfunc0
1278:          0  xen-percpu-ipi       resched0
1279:       1895  xen-percpu-virq      timer0
 NMI:          0   Non-maskable interrupts
 LOC:          0   Local timer interrupts
 SPU:          0   Spurious interrupts
 PMI:          0   Performance monitoring interrupts
 IWI:          0   IRQ work interrupts
 RES:          0   Rescheduling interrupts
 CAL:          0   Function call interrupts
 TLB:          0   TLB shootdowns
 TRM:          0   Thermal event interrupts
 THR:          0   Threshold APIC interrupts
 MCE:          0   Machine check exceptions
 MCP:          0   Machine check polls
 ERR:          0
 MIS:          0

# cat /proc/iomem
00000000-0000ffff : reserved
00010000-0009ffff : System RAM
000a0000-000fffff : reserved
  000f0000-000fffff : System ROM
00100000-207fffff : System RAM
  01000000-012a3d3f : Kernel code
  012a3d40-014046df : Kernel data
  01470000-015215ab : Kernel bss
feb00000-feb03fff : 0000:00:00.0
  feb00000-feb03fff : sky2

# cat /proc/ioports
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
0060-0060 : keyboard
0064-0064 : keyboard
0070-0071 : rtc_cmos
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
e800-e8ff : 0000:00:00.0
  e800-e8ff : sky2

[-- Attachment #5: 256M_domU_console_logs_2.txt --]
[-- Type: text/plain, Size: 15291 bytes --]

# xm create /cluster/xen/xps-106.cfg -c
Using config file "/cluster/xen/xps-106.cfg".
[218927.183880] pciback 0000:04:00.0: device has been assigned to another domain! Over-writting the ownership, but beware.
Started domain xps-106 (id=24)
(XEN) traps.c:2308:d24 Domain attempted WRMSR 00000000c0010004 from 0000c0b2:cbfa8461 to 00000000:0000abcd.
[    0.000000] Reserving virtual address space above 0xf5800000
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.37-trunk-686-bigmem (Debian 2.6.37-1~experimental.1) (ben@decadent.org.uk) (gcc version 4.4.5 (Debian 4.4.5-10) ) #1 SMP Thu Jan 6 15:21:44 UTC 2011
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] released 0 pages of unused memory
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 0000000010800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] last_pfn = 0x10800 max_arch_pfn = 0x1000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000010800000
[    0.000000] RAMDISK: 01793000 - 02ecc000
[    0.000000] 0MB HIGHMEM available.
[    0.000000] 264MB LOWMEM available.
[    0.000000]   mapped low ram: 0 - 10800000
[    0.000000]   low ram: 0 - 10800000
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   Normal   0x00001000 -> 0x00010800
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x00010800
[    0.000000] Using APIC driver default
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] SMP: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] Local APIC disabled by BIOS -- you can enable it with "lapic"
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] Allocating PCI resources starting at 10800000 (gap: 10800000:ef800000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.0.1 (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:32 nr_cpumask_bits:32 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Embedded 13 pages/cpu @cfddf000 s30336 r0 d22912 u53248
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 66944
[    0.000000] Kernel command line: root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force console=hvc0 xencons=tty
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Initializing CPU#0
[    0.000000] Placing 64MB software IO TLB between cbd7ee40 - cfd7ee40
[    0.000000] software IO TLB at phys 0xbd7ee40 - 0xfd7ee40
[    0.000000] Initializing HighMem for node 0 (00000000:00000000)
[    0.000000] Memory: 163268k/270336k available (2703k kernel code, 106620k reserved, 1410k data, 412k init, 0k highmem)
[    0.000000] virtual kernel memory layout:
[    0.000000]     fixmap  : 0xf5536000 - 0xf57ff000   (2852 kB)
[    0.000000]     pkmap   : 0xf5200000 - 0xf5400000   (2048 kB)
[    0.000000]     vmalloc : 0xd1000000 - 0xf51fe000   ( 577 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xd0800000   ( 264 MB)
[    0.000000]       .init : 0xc1405000 - 0xc146c000   ( 412 kB)
[    0.000000]       .data : 0xc12a3d40 - 0xc14046e0   (1410 kB)
[    0.000000]       .text : 0xc1000000 - 0xc12a3d40   (2703 kB)
[    0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000]  RCU-based detection of stalled CPUs is disabled.
[    0.000000] NR_IRQS:1280
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] Detected 2506.222 MHz processor.
[    0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 5012.44 BogoMIPS (lpj=10024888)
[    0.004000] pid_max: default: 32768 minimum: 301
[    0.004000] Security Framework initialized
[    0.004000] SELinux:  Disabled at boot.
[    0.004000] Mount-cache hash table entries: 512
[    0.004000] Initializing cgroup subsys ns
[    0.004000] ns_cgroup deprecated: consider using the 'clone_children' flag without the ns_cgroup.
[    0.004000] Initializing cgroup subsys cpuacct
[    0.004000] Initializing cgroup subsys devices
[    0.004000] Initializing cgroup subsys freezer
[    0.004000] Initializing cgroup subsys net_cls
[    0.004000] Initializing cgroup subsys blkio
[    0.004000] CPU: Physical Processor ID: 0
[    0.004000] CPU: Processor Core ID: 0
[    0.004000] Performance Events:
[    0.004000] no APIC, boot with the "lapic" boot parameter to force-enable it.
[    0.004000] no hardware sampling interrupt available.
[    0.004000] Broken PMU hardware detected, software events only.
[    0.004000] SMP alternatives: switching to UP code
[    0.016000] Freeing SMP alternatives: 16k freed
[    0.016233] NMI watchdog disabled for cpu0: unable to create perf event: -2
[    0.016345] Brought up 1 CPUs
[    0.016428] devtmpfs: initialized
[    0.016745] Grant table initialized
[    0.016753] regulator: core version 0.5
[    0.016807] regulator: dummy:
[    0.016854] NET: Registered protocol family 16
[    0.017058] PCI: setting up Xen PCI frontend stub
[    0.017058] bio: create slab <bio-0> at 0
[    0.017058] ACPI: Interpreter disabled.
[    0.017058] xen_balloon: Initialising balloon driver.
[    0.017058] last_pfn = 0x10800 max_arch_pfn = 0x1000000
[    0.017058] vgaarb: loaded
[    0.017058] PCI: System does not support PCI
[    0.017058] PCI: System does not support PCI
[    0.017058] Switching to clocksource xen
[    0.017417] pnp: PnP ACPI: disabled
[    0.017425] PnPBIOS: Disabled
[    0.019715] NET: Registered protocol family 2
[    0.019763] IP route cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.019948] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[    0.020022] TCP bind hash table entries: 16384 (order: 5, 131072 bytes)
[    0.020083] TCP: Hash tables configured (established 16384 bind 16384)
[    0.020091] TCP reno registered
[    0.020097] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.020107] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.020173] NET: Registered protocol family 1
[    0.020230] Unpacking initramfs...
[    0.046821] Freeing initrd memory: 23780k freed
[    0.054855] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.055094] audit: initializing netlink socket (disabled)
[    0.055112] type=2000 audit(1296645206.868:1): initialized
[    0.063463] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.065902] VFS: Disk quotas dquot_6.5.2
[    0.065967] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.066064] msgmni has been set to 365
[    0.066254] alg: No test for stdrng (krng)
[    0.066324] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.066333] io scheduler noop registered
[    0.066338] io scheduler deadline registered
[    0.066354] io scheduler cfq registered (default)
[    0.066589] isapnp: Scanning for PnP cards...
[    0.515649] isapnp: No Plug & Play device found
[    0.521110] Linux agpgart interface v0.103
[    0.521204] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.521479] PNP: No PS/2 controller found. Probing ports directly.
[    0.522297] i8042.c: No controller found.
[    0.522371] mice: PS/2 mouse device common for all mice
[    0.522460] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.522509] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.522537] cpuidle: using governor ladder
[    0.522543] cpuidle: using governor menu
[    0.522819] pcifront pci-0: Installing PCI frontend
[    0.522830] TCP cubic registered
[    0.522998] NET: Registered protocol family 10
[    0.523038] pcifront pci-0: Creating PCI Frontend Bus 0000:00
[    0.539171] lo: Disabled Privacy Extensions
[    0.539379] Mobile IPv6
[    0.539385] NET: Registered protocol family 17
[    0.539394] Registering the dns_resolver key type
[    0.539411] Using IPI No-Shortcut mode
[    0.539534] registered taskstats version 1
[    0.541437] pcifront pci-0: claiming resource 0000:00:00.0/0
[    0.541445] pcifront pci-0: claiming resource 0000:00:00.0/2
[    0.636034] XENBUS: Device with no driver: device/vbd/51712
[    0.636043] XENBUS: Device with no driver: device/console/0
[    0.636064] /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    0.636089] Initalizing network drop monitor service
[    0.636174] Freeing unused kernel memory: 412k freed
[    0.636935] Write protecting the kernel text: 2704k
[    0.637179] Write protecting the kernel read-only data: 1016k
Loading, please wait...
[    0.668289] udev[46]: starting version 164
[    0.719079] sky2: driver version 1.28
[    0.719174] sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
[    0.719346] sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
[    0.733989] sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43
[    0.743630] blkfront: xvda: barriers enabled
[    0.745505]  xvda: xvda1 xvda2
[    0.865767] ------------[ cut here ]------------
[    0.865782] WARNING: at /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/block/blk-core.c:337 blk_start_queue+0x1d/0x31()
[    0.865796] Modules linked in: sky2 xen_blkfront
[    0.865808] Pid: 0, comm: swapper Not tainted 2.6.37-trunk-686-bigmem #1
[    0.865815] Call Trace:
[    0.865824]  [<c1036005>] ? warn_slowpath_common+0x6a/0x7b
[    0.865832]  [<c1139fd9>] ? blk_start_queue+0x1d/0x31
[    0.865839]  [<c1036023>] ? warn_slowpath_null+0xd/0x10
[    0.865847]  [<c1139fd9>] ? blk_start_queue+0x1d/0x31
[    0.865855]  [<d102a723>] ? kick_pending_request_queues+0x19/0x27 [xen_blkfront]
[    0.865865]  [<d102a925>] ? blkif_interrupt+0x1c8/0x1e3 [xen_blkfront]
[    0.865875]  [<c10753dc>] ? handle_IRQ_event+0x45/0xef
[    0.865883]  [<c1076c3d>] ? handle_fasteoi_irq+0x80/0xb1
[    0.865892]  [<c11a85d1>] ? __xen_evtchn_do_upcall+0xd0/0x155
[    0.865900]  [<c11a8cce>] ? xen_evtchn_do_upcall+0x18/0x26
[    0.865910]  [<c1009117>] ? xen_do_upcall+0x7/0xc
[    0.865918]  [<c10023a7>] ? hypercall_page+0x3a7/0x1001
[    0.865926]  [<c1006246>] ? xen_safe_halt+0xf/0x1b
[    0.865933]  [<c100e276>] ? default_idle+0x3f/0x5d
[    0.865940]  [<c1007dc4>] ? cpu_idle+0x98/0xdc
[    0.865948]  [<c14058a0>] ? start_kernel+0x320/0x325
[    0.865955]  [<c1407479>] ? xen_start_kernel+0x5a1/0x5a8
[    0.865962]  [<c1409046>] ? trap_init+0x1fe/0x367
[    0.865969] ---[ end trace b6935cbfedb011b5 ]---
[    0.865974] ------------[ cut here ]------------
[    0.865981] WARNING: at /build/buildd-linux-2.6_2.6.37-1~experimental.1-i386-rePuD0/linux-2.6-2.6.37/debian/build/source_i386_none/block/blk-core.c:245 blk_remove_plug+0x1d/0x78()
[    0.865993] Modules linked in: sky2 xen_blkfront
[    0.866005] Pid: 0, comm: swapper Tainted: G        W   2.6.37-trunk-686-bigmem #1
[    0.866012] Call Trace:
[    0.866018]  [<c1036005>] ? warn_slowpath_common+0x6a/0x7b
[    0.866025]  [<c1139e6e>] ? blk_remove_plug+0x1d/0x78
[    0.866032]  [<c1036023>] ? warn_slowpath_null+0xd/0x10
[    0.866040]  [<c1139e6e>] ? blk_remove_plug+0x1d/0x78
[    0.866047]  [<c1139ed1>] ? __blk_run_queue+0x8/0x5b
[    0.866054]  [<d102a723>] ? kick_pending_request_queues+0x19/0x27 [xen_blkfront]
[    0.866064]  [<d102a925>] ? blkif_interrupt+0x1c8/0x1e3 [xen_blkfront]
[    0.866076]  [<c10753dc>] ? handle_IRQ_event+0x45/0xef
[    0.866083]  [<c1076c3d>] ? handle_fasteoi_irq+0x80/0xb1
[    0.866091]  [<c11a85d1>] ? __xen_evtchn_do_upcall+0xd0/0x155
[    0.866098]  [<c11a8cce>] ? xen_evtchn_do_upcall+0x18/0x26
[    0.866108]  [<c1009117>] ? xen_do_upcall+0x7/0xc
[    0.866115]  [<c10023a7>] ? hypercall_page+0x3a7/0x1001
[    0.866122]  [<c1006246>] ? xen_safe_halt+0xf/0x1b
[    0.866129]  [<c100e276>] ? default_idle+0x3f/0x5d
[    0.866135]  [<c1007dc4>] ? cpu_idle+0x98/0xdc
[    0.866142]  [<c14058a0>] ? start_kernel+0x320/0x325
[    0.866150]  [<c1407479>] ? xen_start_kernel+0x5a1/0x5a8
[    0.866157]  [<c1409046>] ? trap_init+0x1fe/0x367
[    0.866162] ---[ end trace b6935cbfedb011b6 ]---
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... [    0.977508] device-mapper: uevent: version 1.0.3
[    0.978078] device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@redhat.com
done.
Begin: Running /scripts/local-premount ... done.
[    1.125678] EXT3-fs: barriers not enabled
[    1.131378] kjournald starting.  Commit interval 5 seconds
[    1.131395] EXT3-fs (dm-0): mounted filesystem with ordered data mode
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
INIT: version 2.88 booting
Using makefile-style concurrent boot in runlevel S.
Starting the hotplug events dispatcher: udevd[    1.957027] udev[160]: starting version 164
.
Synthesizing the initial hotplug events...done.
Waiting for /dev to be fully populated...[    2.239895] input: PC Speaker as /devices/platform/pcspkr/input/input0
done.
Activating swap...done.
[    5.367412] EXT3-fs (dm-0): warning: maximal mount count reached, running e2fsck is recommended
[    5.408349] EXT3-fs (dm-0): using internal journal
Cleaning up ifupdown....
Loading kernel modules...done.
Setting up networking....
Activating lvm and md swap...done.
Checking file systems...fsck from util-linux-ng 2.17.2
done.
Mounting local filesystems...done.
Activating swapfile swap...done.
Cleaning up temporary files....
Setting kernel variables ...done.
Configuring network interfaces...[    5.895422] sky2 0000:00:00.0: eth0: enabling interface
[    5.895488] ADDRCONF(NETDEV_UP): eth0: link is not ready
done.
Cleaning up temporary files....
INIT: Entering runlevel: 2
Using makefile-style concurrent boot in runlevel 2.
Starting enhanced syslogd: rsyslogd.
Starting periodic command scheduler: cron.

Debian GNU/Linux 6.0 xps-106 hvc0

xps-106 login: [    8.332107] sky2 0000:00:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow control both
[    8.332161] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

[-- Attachment #6: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

> > Ping tests work whatever can be packet size.
> > 
> > If I understood well what you explain to me, it's now clear that the
> > problem is somewhat Xen related, isn't it ?
> 
> It certainly seems that way. I'm not 100% sure that swiotlb=force will
> have actually made the driver use the swiotlb, it may just have forced

It should have.

> the swiotlb to be allocated. Konrad?

Both. It would allocate it and force all DMA operations to go through
the bounce buffer.

Let me take a look at the driver itself. Ah, so I had a similar card: 3c59x
which hit the same type of failure way back in 2.6.31 time. It had the
pci_dma_sync_single_for_cpu(.. PCI_DMA_FROMDEVICE) turned around so it would never
copy properly. But in my tree (2.6.38) the sky driver looks to be doing just
right.. It has PCI_DMA_FROMDEVICE..

Well this looks odd:

2330         skb = netdev_alloc_skb_ip_align(sky2->netdev, length);
2331         if (likely(skb)) {
2332                 pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
2333                                             length, PCI_DMA_FROMDEVICE); 
=======>             skb_copy_from_linear_data(re->skb, skb->data, length); <====
2335                 skb->ip_summed = re->skb->ip_summed;
2336                 skb->csum = re->skb->csum;
2337                 pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,
2338                                                length, PCI_DMA_FROMDEVICE); 
2339                 re->skb->ip_summed = CHECKSUM_NONE;
2340                 skb_put(skb, length);
2341         }

It copies from skb->data (just allocated) to re->skb (seems that the re->skb->data
has been earlier DMA-mapped).

But there is no data in skb->data as we just allocated it? Shouldn't this
be the other way around? Like this:


diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..37c0631 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2,
 	if (likely(skb)) {
 		pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
 					    length, PCI_DMA_FROMDEVICE);
-		skb_copy_from_linear_data(re->skb, skb->data, length);
+		skb_copy_from_linear_data(skb, re->skb->data, length);
 		skb->ip_summed = re->skb->ip_summed;
 		skb->csum = re->skb->csum;
 		pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,

> 
> > I'll be able to continue any tests you want tomorrow.
> > Do you still need me to compile domU kernel with "CONFIG_DMA_API_DEBUG"
> > enabled ?
> 
> Yes please.
> 
> > If so, what tools will I need to get debug informations ?
> 
> Uh, Konrad?

Here, try this patch. Run it under baremetal and Xen and we can compare the
outputs .. but I think the issue here is not off mis-using the DMA API but
rather a driver bug.


/*
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License v2.0 as published by
 * the Free Software Foundation
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 */

#include <linux/module.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/init.h>
#include <linux/stat.h>
#include <linux/err.h>
#include <linux/ctype.h>
#include <linux/slab.h>
#include <linux/limits.h>
#include <linux/device.h>
#include <linux/pci.h>
#include <linux/blkdev.h>
#include <linux/device.h>

#include <linux/init.h>
#include <linux/mm.h>
#include <linux/fcntl.h>
#include <linux/slab.h>
#include <linux/kmod.h>
#include <linux/major.h>
#include <linux/smp_lock.h>
#include <linux/highmem.h>
#include <linux/blkdev.h>
#include <linux/module.h>
#include <linux/blkpg.h>
#include <linux/buffer_head.h>
#include <linux/mpage.h>
#include <linux/mount.h>
#include <linux/uio.h>
#include <linux/namei.h>
#include <asm/uaccess.h>

#include <linux/pagemap.h>
#include <linux/pagevec.h>

#include <linux/dma-debug.h>

#define DUMP_DMA_FUN  "0.1"

MODULE_AUTHOR("Konrad Rzeszutek Wilk <konrad@virtualiron>");
MODULE_DESCRIPTION("dump dma");
MODULE_LICENSE("GPL");
MODULE_VERSION(DUMP_DMA_FUN);

static int __init dump_dma_init(void)
{
	debug_dma_dump_mappings(NULL);
	return 0;
}

static void __exit dump_dma_exit(void)
{
}

module_init(dump_dma_init);
module_exit(dump_dma_exit);
> 
> Ian.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Ian Campbell]

On Wed, 2011-02-02 at 15:38 +0000, Konrad Rzeszutek Wilk wrote:
> > > Ping tests work whatever can be packet size.
> > > 
> > > If I understood well what you explain to me, it's now clear that the
> > > problem is somewhat Xen related, isn't it ?
> > 
> > It certainly seems that way. I'm not 100% sure that swiotlb=force will
> > have actually made the driver use the swiotlb, it may just have forced
> 
> It should have.
> 
> > the swiotlb to be allocated. Konrad?
> 
> Both. It would allocate it and force all DMA operations to go through
> the bounce buffer.
> 
> Let me take a look at the driver itself. Ah, so I had a similar card: 3c59x
> which hit the same type of failure way back in 2.6.31 time. It had the
> pci_dma_sync_single_for_cpu(.. PCI_DMA_FROMDEVICE) turned around so it would never
> copy properly. But in my tree (2.6.38) the sky driver looks to be doing just
> right.. It has PCI_DMA_FROMDEVICE..
> 
> Well this looks odd:
> 
> 2330         skb = netdev_alloc_skb_ip_align(sky2->netdev, length);
> 2331         if (likely(skb)) {
> 2332                 pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
> 2333                                             length, PCI_DMA_FROMDEVICE); 
> =======>             skb_copy_from_linear_data(re->skb, skb->data, length); <====
> 2335                 skb->ip_summed = re->skb->ip_summed;
> 2336                 skb->csum = re->skb->csum;
> 2337                 pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,
> 2338                                                length, PCI_DMA_FROMDEVICE); 
> 2339                 re->skb->ip_summed = CHECKSUM_NONE;
> 2340                 skb_put(skb, length);
> 2341         }
> 
> It copies from skb->data (just allocated) to re->skb (seems that the re->skb->data
> has been earlier DMA-mapped).
> 
> But there is no data in skb->data as we just allocated it? Shouldn't this
> be the other way around? Like this:

skb_copy_from_linear_data's arguments are the other way round to what
you may be expecting:
static inline void skb_copy_from_linear_data(const struct sk_buff *skb,
					     void *to,
					     const unsigned int len)

So the above line is correctly copying from re->skb->data to skb->data
not the other way round.

I stared at that exact line for ages this morning ;-)

Ian.

Re: PCI passthrough issue

[Jean Baptiste Favre]

> >> Ping tests work whatever can be packet size.
> >>
> >> If I understood well what you explain to me, it's now clear that the
> >> problem is somewhat Xen related, isn't it ?
> >
> > It certainly seems that way. I'm not 100% sure that swiotlb=force will
> > have actually made the driver use the swiotlb, it may just have forced
>
> It should have.
>
> > the swiotlb to be allocated. Konrad?
>
> Both. It would allocate it and force all DMA operations to go through
> the bounce buffer.
>
> Let me take a look at the driver itself. Ah, so I had a similar card:
> 3c59x
> which hit the same type of failure way back in 2.6.31 time. It had the
> pci_dma_sync_single_for_cpu(.. PCI_DMA_FROMDEVICE) turned around so it
> would never
> copy properly. But in my tree (2.6.38) the sky driver looks to be
> doing just
> right.. It has PCI_DMA_FROMDEVICE..
>
> Well this looks odd:
>
> 2330         skb = netdev_alloc_skb_ip_align(sky2->netdev, length);
> 2331         if (likely(skb)) {
> 2332                 pci_dma_sync_single_for_cpu(sky2->hw->pdev,
> re->data_addr,
> 2333                                             length,
> PCI_DMA_FROMDEVICE);
> =======>             skb_copy_from_linear_data(re->skb, skb->data,
> length); <====
> 2335                 skb->ip_summed = re->skb->ip_summed;
> 2336                 skb->csum = re->skb->csum;
> 2337                 pci_dma_sync_single_for_device(sky2->hw->pdev,
> re->data_addr,
> 2338                                                length,
> PCI_DMA_FROMDEVICE);
> 2339                 re->skb->ip_summed = CHECKSUM_NONE;
> 2340                 skb_put(skb, length);
> 2341         }
>
> It copies from skb->data (just allocated) to re->skb (seems that the
> re->skb->data
> has been earlier DMA-mapped).
>
> But there is no data in skb->data as we just allocated it? Shouldn't this
> be the other way around? Like this:
>
>
> diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
> index 7d85a38..37c0631 100644
> --- a/drivers/net/sky2.c
> +++ b/drivers/net/sky2.c
> @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct
> sky2_port *sky2,
>      if (likely(skb)) {
>          pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
>                          length, PCI_DMA_FROMDEVICE);
> -        skb_copy_from_linear_data(re->skb, skb->data, length);
> +        skb_copy_from_linear_data(skb, re->skb->data, length);
>          skb->ip_summed = re->skb->ip_summed;
>          skb->csum = re->skb->csum;
>          pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,
>
But, how could a "simple" driver bug automagically appear with Xen and
not as Baremetal ?
And why only with 32bits kernels ?

BTW, I'll try your patch as soon as I can.

> >
> >> I'll be able to continue any tests you want tomorrow.
> >> Do you still need me to compile domU kernel with "CONFIG_DMA_API_DEBUG"
> >> enabled ?
> >
> > Yes please.
> >
> >> If so, what tools will I need to get debug informations ?
> >
> > Uh, Konrad?
>
> Here, try this patch. Run it under baremetal and Xen and we can
> compare the
> outputs .. but I think the issue here is not off mis-using the DMA API but
> rather a driver bug.
OK, so if you first patch does not solve problem, I'll compile this one
as kernel module and load it, both on 32bits Baremetal and in 32bits DomU

Regards,
JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Wed, Feb 02, 2011 at 11:24:51AM +0100, Jean Baptiste Favre wrote:
> Hello Ian,
> 
> My domU config file:
> 
> # cat /cluster/xen/xps-106.cfg
> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
> builder      = 'linux'
> memory       = '398'
> vcpus        = '1'
> cpus         = '2'
> localtime    = 0
> serial       = 'pty'
> boot         = 'cdn'
> disk         = [ 'drbd:xps-106,xvda,w' ]
> on_poweroff  = 'destroy'
> on_reboot    = 'restart'
> on_crash     = 'restart'
> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
> console=hvc0 xencons=tty"
> pci = [ '04:00.0' ]
> name         = 'xps-106'
> hostname     = 'xps-106.clichy.jbfavre.org'
> 
> 
> Le 02/02/2011 10:27, Ian Campbell a écrit :
> > On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:
> > > Le 01/02/2011 17:23, Ian Campbell a écrit :
> >
> > >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> > >> you post a full guest console log at some point? Comparing the logs for
> > >> the 256MB, 398MB and 512MB guest RAM case might be useful.
> > > No sure I've ever posted that logs. But I can redo my tests :)
> >
> > yes, please do that.
> Please find attached both console startup logs with 256M and 512M:
> 256M_domU_console_logs.txt
> 512M_domU_console_logs.txt
> 
> For 512M, I saw some kernel CallTrace I can not explain. There are not
> present with 256M.
> 
> For 398M memory, I can't even start domU :
> # xm create /cluster/xen/xps-106.cfg -c
> Using config file "/cluster/xen/xps-106.cfg".
> [215739.007871] pciback 0000:04:00.0: device has been assigned to
> another domain! Over-writting the ownership, but beware.
> Started domain xps-106 (id=23)
> (XEN) mm.c:798:d23 Non-privileged (23) attempt to map I/O space 00000000
> (XEN) mm.c:4644:d23 ptwr_emulate: could not get_page_from_l1e()
> 
> As I told you, I'm still using Debian 2.6.37 kernel because I've some
> problem to compile 2.6.32.27 from Jeremy's git repository.
> I hope I can get it compiled today so I'll be able to test with that
> kernel as well.

So I've tried this on my Abit IP-35 box which has a 

04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13)
        Subsystem: ABIT Computer Corp. Device 1085
        Flags: bus master, fast devsel, latency 0, IRQ 29
        Memory at fdefc000 (64-bit, non-prefetchable) [size=16K]
        I/O ports at be00 [size=256]
        Expansion ROM at <ignored> [disabled]
        Capabilities: [48] Power Management version 3
        Capabilities: [50] Vital Product Data
        Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: sky2
        Kernel modules: sky2

And when I launch this guest with a 32-bit DomU:
kernel="/mnt/lab/latest/vmlinuz"
ramdisk="/mnt/lab/latest/initramfs.cpio.gz"
extra="console=hvc0 debug iommu=soft"
memory=320
vcpus=1
cpu='2'
on_crash="preserve"
#vif = [ 'bridge=switch' ]
pci = ["04:00.0"]
vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1']

And played around with the 'extra' to add 'swiotlb=force'.

The moment I had 'swiotlb=force' I could not get any DHCP
address from the NIC. If I did not have 'swiotlb=force' it would
work just fine (can ping any size, etc, this is with 320MB)


For fun, I upped the memory (320->720) and kept 'swiotlb=force' in effect.
Same effect: can't do DHCP.

I look to have a different issue than you, which is that whenever I use
swiotlb=force, things go haywire.

Fyi, this is what DomU tells me:

12:22:41 # 9 :~/ 
> dmesg |grep Memor
[    0.000000] Memory: 145640k/335872k available (3731k kernel code, 189784k reserved, 1565k data, 436k init, 0k highmem)

12:22:56 # 10 :~/ 
> uname -a
Linux (none) 2.6.38-rc2-00028-gf2a2d8b #2 SMP Wed Feb 2 12:10:25 EST 2011 i686 i686 i386 GNU/Linux
rnet driver.
[    0.921406] udevd (1126): /proc/1126/oom_adj is deprecated, please use /proc/1126/oom_score_adj instead.
[    0.984886] sky2: driver version 1.28
[    0.995595] sky2 0000:04:00.0: BAR 0: set to [mem 0xfdefc000-0xfdefffff 64bit] (PCI address [0xfdefc000-0xfdefffff])
[    0.995662] sky2 0000:04:00.0: BAR 2: set to [io  0xbe00-0xbeff] (PCI address [0xbe00-0xbeff])
[    0.995697] sky2 0000:04:00.0: enabling device (0000 -> 0003)
[    0.996440] sky2 0000:04:00.0: Xen PCI mapped GSI18 to IRQ27

(This is the #master branch from git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git)

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
Sorry for not answering earlier.

I made a test with OpenWRT, applying patches bellow.
DomU boots, can connect, but nothing changed.
I still have incoming packets dropped without any console log.

Looking deeper, I can see ARP request coming in on my GW, replies coming
out.
On the domU, arp command output is:
# arp
IP address       HW type     Flags       HW address            Mask
Device
10.0.0.1         0x1         0x0         00:00:00:00:00:00     *        eth0

So, now I've an Layer 2 issue as well :(

I'm still trying to make my Debian kernel working so that I can test on
Debian too.

Regards,
JB


>From Konrad:
diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..37c0631 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct
sky2_port *sky2,
 	if (likely(skb)) {
 		pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
 					    length, PCI_DMA_FROMDEVICE);
-		skb_copy_from_linear_data(re->skb, skb->data, length);
+		skb_copy_from_linear_data(skb, re->skb->data, length);
 		skb->ip_summed = re->skb->ip_summed;
 		skb->csum = re->skb->csum;
 		pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,


And from Ian:
diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..645d9e9 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2339,6 +2339,7 @@ static struct sk_buff *receive_copy(struct
sky2_port *sky2,
 		re->skb->ip_summed = CHECKSUM_NONE;
 		skb_put(skb, length);
 	}
+	WARN(skb == NULL, "sky2: receive_copy failed netdev_alloc_skb_ip_align
length %u\n", length);
 	return skb;
 }

@@ -2386,10 +2387,15 @@ static struct sk_buff *receive_new(struct
sky2_port *sky2,

 	nre.skb = sky2_rx_alloc(sky2);
 	if (unlikely(!nre.skb))
+	{
+		WARN(1, "sky2: receive_new failed sky2_rx_alloc\n");
 		goto nobuf;
-
+	}
 	if (sky2_rx_map_skb(sky2->hw->pdev, &nre, hdr_space))
+	{
+		WARN(1, "sky2: receive_new failed sky2_rx_map_skb\n");
 		goto nomap;
+	}

 	skb = re->skb;
 	sky2_rx_unmap_skb(sky2->hw->pdev, re);
diff --git a/net/core/dev.c b/net/core/dev.c
index 54277df..9c99612 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1516,6 +1516,7 @@ int dev_forward_skb(struct net_device *dev, struct
sk_buff *skb)

 	if (unlikely(!(dev->flags & IFF_UP) ||
 		     (skb->len > (dev->mtu + dev->hard_header_len + VLAN_HLEN)))) {
+		printk(KERN_CRIT "dev_forward_skb dropping skb\n");
 		atomic_long_inc(&dev->rx_dropped);
 		kfree_skb(skb);
 		return NET_RX_DROP;
@@ -2700,6 +2701,7 @@ enqueue:

 	local_irq_restore(flags);

+	printk(KERN_CRIT "enqueue_to_backlog dropping skb\n");
 	atomic_long_inc(&skb->dev->rx_dropped);
 	kfree_skb(skb);
 	return NET_RX_DROP;
@@ -3125,6 +3127,7 @@ ncls:
 	if (pt_prev) {
 		ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
 	} else {
+		printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x\n", type);
 		atomic_long_inc(&skb->dev->rx_dropped);
 		kfree_skb(skb);
 		/* Jamal, now you will not able to escape explaining



Le 02/02/2011 18:42, Konrad Rzeszutek Wilk a écrit :
> On Wed, Feb 02, 2011 at 11:24:51AM +0100, Jean Baptiste Favre wrote:
>> Hello Ian,
>>
>> My domU config file:
>>
>> # cat /cluster/xen/xps-106.cfg
>> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
>> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
>> builder      = 'linux'
>> memory       = '398'
>> vcpus        = '1'
>> cpus         = '2'
>> localtime    = 0
>> serial       = 'pty'
>> boot         = 'cdn'
>> disk         = [ 'drbd:xps-106,xvda,w' ]
>> on_poweroff  = 'destroy'
>> on_reboot    = 'restart'
>> on_crash     = 'restart'
>> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
>> console=hvc0 xencons=tty"
>> pci = [ '04:00.0' ]
>> name         = 'xps-106'
>> hostname     = 'xps-106.clichy.jbfavre.org'
>>
>>
>> Le 02/02/2011 10:27, Ian Campbell a écrit :
>>> On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:
>>>> Le 01/02/2011 17:23, Ian Campbell a écrit :
>>>
>>>>> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
>>>>> you post a full guest console log at some point? Comparing the logs for
>>>>> the 256MB, 398MB and 512MB guest RAM case might be useful.
>>>> No sure I've ever posted that logs. But I can redo my tests :)
>>>
>>> yes, please do that.
>> Please find attached both console startup logs with 256M and 512M:
>> 256M_domU_console_logs.txt
>> 512M_domU_console_logs.txt
>>
>> For 512M, I saw some kernel CallTrace I can not explain. There are not
>> present with 256M.
>>
>> For 398M memory, I can't even start domU :
>> # xm create /cluster/xen/xps-106.cfg -c
>> Using config file "/cluster/xen/xps-106.cfg".
>> [215739.007871] pciback 0000:04:00.0: device has been assigned to
>> another domain! Over-writting the ownership, but beware.
>> Started domain xps-106 (id=23)
>> (XEN) mm.c:798:d23 Non-privileged (23) attempt to map I/O space 00000000
>> (XEN) mm.c:4644:d23 ptwr_emulate: could not get_page_from_l1e()
>>
>> As I told you, I'm still using Debian 2.6.37 kernel because I've some
>> problem to compile 2.6.32.27 from Jeremy's git repository.
>> I hope I can get it compiled today so I'll be able to test with that
>> kernel as well.
> 
> So I've tried this on my Abit IP-35 box which has a 
> 
> 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13)
>         Subsystem: ABIT Computer Corp. Device 1085
>         Flags: bus master, fast devsel, latency 0, IRQ 29
>         Memory at fdefc000 (64-bit, non-prefetchable) [size=16K]
>         I/O ports at be00 [size=256]
>         Expansion ROM at <ignored> [disabled]
>         Capabilities: [48] Power Management version 3
>         Capabilities: [50] Vital Product Data
>         Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+
>         Capabilities: [e0] Express Legacy Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting
>         Kernel driver in use: sky2
>         Kernel modules: sky2
> 
> And when I launch this guest with a 32-bit DomU:
> kernel="/mnt/lab/latest/vmlinuz"
> ramdisk="/mnt/lab/latest/initramfs.cpio.gz"
> extra="console=hvc0 debug iommu=soft"
> memory=320
> vcpus=1
> cpu='2'
> on_crash="preserve"
> #vif = [ 'bridge=switch' ]
> pci = ["04:00.0"]
> vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1']
> 
> And played around with the 'extra' to add 'swiotlb=force'.
> 
> The moment I had 'swiotlb=force' I could not get any DHCP
> address from the NIC. If I did not have 'swiotlb=force' it would
> work just fine (can ping any size, etc, this is with 320MB)
> 
> 
> For fun, I upped the memory (320->720) and kept 'swiotlb=force' in effect.
> Same effect: can't do DHCP.
> 
> I look to have a different issue than you, which is that whenever I use
> swiotlb=force, things go haywire.
> 
> Fyi, this is what DomU tells me:
> 
> 12:22:41 # 9 :~/ 
>> dmesg |grep Memor
> [    0.000000] Memory: 145640k/335872k available (3731k kernel code, 189784k reserved, 1565k data, 436k init, 0k highmem)
> 
> 12:22:56 # 10 :~/ 
>> uname -a
> Linux (none) 2.6.38-rc2-00028-gf2a2d8b #2 SMP Wed Feb 2 12:10:25 EST 2011 i686 i686 i386 GNU/Linux
> rnet driver.
> [    0.921406] udevd (1126): /proc/1126/oom_adj is deprecated, please use /proc/1126/oom_score_adj instead.
> [    0.984886] sky2: driver version 1.28
> [    0.995595] sky2 0000:04:00.0: BAR 0: set to [mem 0xfdefc000-0xfdefffff 64bit] (PCI address [0xfdefc000-0xfdefffff])
> [    0.995662] sky2 0000:04:00.0: BAR 2: set to [io  0xbe00-0xbeff] (PCI address [0xbe00-0xbeff])
> [    0.995697] sky2 0000:04:00.0: enabling device (0000 -> 0003)
> [    0.996440] sky2 0000:04:00.0: Xen PCI mapped GSI18 to IRQ27
> 
> (This is the #master branch from git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git)
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Ian Campbell]

On Fri, 2011-02-04 at 08:43 +0000, Jean Baptiste Favre wrote:

> >From Konrad:
> diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
> index 7d85a38..37c0631 100644
> --- a/drivers/net/sky2.c
> +++ b/drivers/net/sky2.c
> @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct
> sky2_port *sky2,
>  	if (likely(skb)) {
>  		pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
>  					    length, PCI_DMA_FROMDEVICE);
> -		skb_copy_from_linear_data(re->skb, skb->data, length);
> +		skb_copy_from_linear_data(skb, re->skb->data, length);
>  		skb->ip_summed = re->skb->ip_summed;
>  		skb->csum = re->skb->csum;
>  		pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,

Please don't apply this bit, the argument order to skb_copy.... can be a
bit surprising if you assume it it is like memcpy. The original code is
correct. This is probably where your L2 issue came from.

Please can you try again with the just the WARN patch.

Ian.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,

Le 04/02/2011 09:53, Ian Campbell a écrit :
> On Fri, 2011-02-04 at 08:43 +0000, Jean Baptiste Favre wrote:
> 
>> >From Konrad:
>> diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
>> index 7d85a38..37c0631 100644
>> --- a/drivers/net/sky2.c
>> +++ b/drivers/net/sky2.c
>> @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct
>> sky2_port *sky2,
>>  	if (likely(skb)) {
>>  		pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
>>  					    length, PCI_DMA_FROMDEVICE);
>> -		skb_copy_from_linear_data(re->skb, skb->data, length);
>> +		skb_copy_from_linear_data(skb, re->skb->data, length);
>>  		skb->ip_summed = re->skb->ip_summed;
>>  		skb->csum = re->skb->csum;
>>  		pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,
> 
> Please don't apply this bit, the argument order to skb_copy.... can be a
> bit surprising if you assume it it is like memcpy. The original code is
> correct. This is probably where your L2 issue came from.
> 
> Please can you try again with the just the WARN patch.
Sure I can :)
Did not understood Konrad's patch was incorrect, sorry.

Regards,
JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello Ian,
Applyed your patches.
Now, I've:
# ping -s86 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 86 data bytes
__netif_receive_skb dropping skb proto 0x20


So problem seems to occur in net/core/dev.c file, according to the patch
bellow

@@ -3125,6 +3127,7 @@ ncls:
if (pt_prev) {
 		ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
 	} else {
		printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x\n", type);
 		atomic_long_inc(&skb->dev->rx_dropped);
 		kfree_skb(skb);
 		/* Jamal, now you will not able to escape explaining

Regards,
JB

Le 04/02/2011 09:54, Jean Baptiste Favre a écrit :
> Hello,
> 
> Le 04/02/2011 09:53, Ian Campbell a écrit :
>> On Fri, 2011-02-04 at 08:43 +0000, Jean Baptiste Favre wrote:
>>
>>> >From Konrad:
>>> diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
>>> index 7d85a38..37c0631 100644
>>> --- a/drivers/net/sky2.c
>>> +++ b/drivers/net/sky2.c
>>> @@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct
>>> sky2_port *sky2,
>>>  	if (likely(skb)) {
>>>  		pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
>>>  					    length, PCI_DMA_FROMDEVICE);
>>> -		skb_copy_from_linear_data(re->skb, skb->data, length);
>>> +		skb_copy_from_linear_data(skb, re->skb->data, length);
>>>  		skb->ip_summed = re->skb->ip_summed;
>>>  		skb->csum = re->skb->csum;
>>>  		pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,
>>
>> Please don't apply this bit, the argument order to skb_copy.... can be a
>> bit surprising if you assume it it is like memcpy. The original code is
>> correct. This is probably where your L2 issue came from.
>>
>> Please can you try again with the just the WARN patch.
> Sure I can :)
> Did not understood Konrad's patch was incorrect, sorry.
> 
> Regards,
> JB
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Ian Campbell]

On Fri, 2011-02-04 at 10:12 +0000, Jean Baptiste Favre wrote:
> Hello Ian,
> Applyed your patches.

Thanks.

> Now, I've:
> # ping -s86 10.0.0.1
> PING 10.0.0.1 (10.0.0.1): 86 data bytes
> __netif_receive_skb dropping skb proto 0x20
> 
> 
> So problem seems to occur in net/core/dev.c file, according to the patch
> bellow

Interesting. the number printed in the warning is type == skb->protocol
== 0x20 which is not a valid protocol that I can find anywhere (nor is
0x2000 in case I'm mixing my endianesses up). Neither is 0x20 it a valid
Ethernet frame length (min 64) so it's not that sort of confusion
AFAICT.

skb->protocol is initialised in sky2_status_intr with the return value
of "eth_type_trans(skb, dev)" which as far as I can tell cannot return
0x20.

The domU network configuration is using the sky2 device directly, no
bridging, VLAN, tunnels or anything else like that?

Please can you post the output of "ethtool -k <sky2-device>".

If either tx or rx checksumming are enabled please can you try disabling
them ("ethtool -K <sky2-device> [tx|rx] off"). This setting controls the
code path in sky2_skb_rx which can go either to netif_receive_skb or
napi_gro_receive.

The tcpdump captures from both ends would still be interesting to see.

The following patch enhances the previous one with a few checks for
protocol 0x20 getting set.

Ian.


diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..727922f 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2339,6 +2339,7 @@ static struct sk_buff *receive_copy(struct sky2_port *sky2,
 		re->skb->ip_summed = CHECKSUM_NONE;
 		skb_put(skb, length);
 	}
+	WARN(skb == NULL, "sky2: receive_copy failed netdev_alloc_skb_ip_align length %u\n", length);
 	return skb;
 }
 
@@ -2386,10 +2387,15 @@ static struct sk_buff *receive_new(struct sky2_port *sky2,
 
 	nre.skb = sky2_rx_alloc(sky2);
 	if (unlikely(!nre.skb))
+	{
+		WARN(1, "sky2: receive_new failed sky2_rx_alloc\n");
 		goto nobuf;
-
+	}
 	if (sky2_rx_map_skb(sky2->hw->pdev, &nre, hdr_space))
+	{
+		WARN(1, "sky2: receive_new failed sky2_rx_map_skb\n");
 		goto nomap;
+	}
 
 	skb = re->skb;
 	sky2_rx_unmap_skb(sky2->hw->pdev, re);
@@ -2600,6 +2606,7 @@ static int sky2_status_intr(struct sky2_hw *hw, int to_do, u16 idx)
 			}
 
 			skb->protocol = eth_type_trans(skb, dev);
+			WARN_ONCE(skb->protocol == 0x20, "sky2_status_intr: skb->protocol is 0x20. ip_summed is %d\n", skb->ip_summed);
 
 			sky2_skb_rx(sky2, status, skb);
 
diff --git a/net/core/dev.c b/net/core/dev.c
index 54277df..5d786b6 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1516,6 +1516,7 @@ int dev_forward_skb(struct net_device *dev, struct sk_buff *skb)
 
 	if (unlikely(!(dev->flags & IFF_UP) ||
 		     (skb->len > (dev->mtu + dev->hard_header_len + VLAN_HLEN)))) {
+		printk(KERN_CRIT "dev_forward_skb dropping skb\n");
 		atomic_long_inc(&dev->rx_dropped);
 		kfree_skb(skb);
 		return NET_RX_DROP;
@@ -1524,6 +1525,7 @@ int dev_forward_skb(struct net_device *dev, struct sk_buff *skb)
 	skb->tstamp.tv64 = 0;
 	skb->pkt_type = PACKET_HOST;
 	skb->protocol = eth_type_trans(skb, dev);
+	WARN_ONCE(skb->protocol == 0x20, "dev_forward_skb protocol 0x20 ip_summed %d\n", skb->ip_summed);
 	return netif_rx(skb);
 }
 EXPORT_SYMBOL_GPL(dev_forward_skb);
@@ -2700,6 +2702,7 @@ enqueue:
 
 	local_irq_restore(flags);
 
+	printk(KERN_CRIT "enqueue_to_backlog dropping skb\n");
 	atomic_long_inc(&skb->dev->rx_dropped);
 	kfree_skb(skb);
 	return NET_RX_DROP;
@@ -3010,6 +3013,8 @@ static int __netif_receive_skb(struct sk_buff *skb)
 	int ret = NET_RX_DROP;
 	__be16 type;
 
+	WARN_ONCE(type == 0x20, "__netif_receive_skb protocol 0x20 ip summed %d at line %d\n", skb->ip_summed, __LINE__);
+
 	if (!netdev_tstamp_prequeue)
 		net_timestamp_check(skb);
 
@@ -3111,6 +3116,7 @@ ncls:
 	}
 
 	type = skb->protocol;
+	WARN_ONCE(type == 0x20, "__netif_receive_skb protocol 0x20 ip summed %d at line %d\n", skb->ip_summed, __LINE__);
 	list_for_each_entry_rcu(ptype,
 			&ptype_base[ntohs(type) & PTYPE_HASH_MASK], list) {
 		if (ptype->type == type && (ptype->dev == null_or_orig ||
@@ -3125,6 +3131,7 @@ ncls:
 	if (pt_prev) {
 		ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
 	} else {
+		printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x ip summed %d\n", type, skb->ip_summed);
 		atomic_long_inc(&skb->dev->rx_dropped);
 		kfree_skb(skb);
 		/* Jamal, now you will not able to escape explaining
@@ -3447,6 +3454,7 @@ gro_result_t napi_frags_finish(struct napi_struct *napi, struct sk_buff *skb,
 	case GRO_NORMAL:
 	case GRO_HELD:
 		skb->protocol = eth_type_trans(skb, skb->dev);
+		WARN_ONCE(skb->protocol == 0x20, "napi_frags_finish protocol 0x20 ip_summed %d\n", skb->ip_summed);
 
 		if (ret == GRO_HELD)
 			skb_gro_pull(skb, -ETH_HLEN);
@@ -3498,6 +3506,7 @@ struct sk_buff *napi_frags_skb(struct napi_struct *napi)
 	 * special handling.  We'll fix it up properly at the end.
 	 */
 	skb->protocol = eth->h_proto;
+	WARN_ONCE(skb->protocol == 0x20, "napi_frags_skb protocol 0x20 ip_summed %d\n", skb->ip_summed);
 
 out:
 	return skb;

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,

Le 04/02/2011 12:04, Ian Campbell a écrit :
> On Fri, 2011-02-04 at 10:12 +0000, Jean Baptiste Favre wrote:
>> Hello Ian,
>> Applyed your patches.
> 
> Thanks.
> 
>> Now, I've:
>> # ping -s86 10.0.0.1
>> PING 10.0.0.1 (10.0.0.1): 86 data bytes
>> __netif_receive_skb dropping skb proto 0x20
>>
>>
>> So problem seems to occur in net/core/dev.c file, according to the patch
>> bellow
> 
> Interesting. the number printed in the warning is type == skb->protocol
> == 0x20 which is not a valid protocol that I can find anywhere (nor is
> 0x2000 in case I'm mixing my endianesses up). Neither is 0x20 it a valid
> Ethernet frame length (min 64) so it's not that sort of confusion
> AFAICT.
> 
> skb->protocol is initialised in sky2_status_intr with the return value
> of "eth_type_trans(skb, dev)" which as far as I can tell cannot return
> 0x20.
> 
> The domU network configuration is using the sky2 device directly, no
> bridging, VLAN, tunnels or anything else like that?
At boot it uses bridge.
For the test, I delete bridge and set IP address directly on eth0

root@OpenWrt:/# ifconfig br-lan down
br-lan: port 1(eth0) entering forwarding state

root@OpenWrt:/# brctl delbr br-lan
device eth0 left promiscuous mode
br-lan: port 1(eth0) entering disabled state

root@OpenWrt:/# ifconfig eth0 10.0.0.8 netmask 255.255.255.0
root@OpenWrt:/# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: seq=0 ttl=64 time=10.455 ms
64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.870 ms
^C
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.870/5.662/10.455 ms
root@OpenWrt:/# ping -s86 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 86 data bytes
^C
--- 10.0.0.1 ping statistics ---
12 packets transmitted, 0 packets received, 100% packet loss


> 
> Please can you post the output of "ethtool -k <sky2-device>".
root@OpenWrt:/# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: on


> If either tx or rx checksumming are enabled please can you try disabling
> them ("ethtool -K <sky2-device> [tx|rx] off"). This setting controls the
> code path in sky2_skb_rx which can go either to netif_receive_skb or
> napi_gro_receive.
root@OpenWrt:/# ethtool -K eth0 tx off
root@OpenWrt:/# ethtool -K eth0 rx off
root@OpenWrt:/# ping -s86 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 86 data bytes
^C
--- 10.0.0.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
root@OpenWrt:/# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
ntuple-filters: off
receive-hashing: on


> The tcpdump captures from both ends would still be interesting to see.
> 
> The following patch enhances the previous one with a few checks for
> protocol 0x20 getting set.
> 
> Ian.
What is a bit strange here is that I don't any more the KERN_CRIT printk
message.
Could be a false positive ?

I'm currently compiling new kernel with your last patch. will keep you
updated

Regards,
JB

Re: PCI passthrough issue

[Ian Campbell]

On Fri, 2011-02-04 at 11:25 +0000, Jean Baptiste Favre wrote:
> Hello,
> 
> Le 04/02/2011 12:04, Ian Campbell a écrit :
> > On Fri, 2011-02-04 at 10:12 +0000, Jean Baptiste Favre wrote:
> >> Hello Ian,
> >> Applyed your patches.
> > 
> > Thanks.
> > 
> >> Now, I've:
> >> # ping -s86 10.0.0.1
> >> PING 10.0.0.1 (10.0.0.1): 86 data bytes
> >> __netif_receive_skb dropping skb proto 0x20
> >>
> >>
> >> So problem seems to occur in net/core/dev.c file, according to the patch
> >> bellow
> > 
> > Interesting. the number printed in the warning is type == skb->protocol
> > == 0x20 which is not a valid protocol that I can find anywhere (nor is
> > 0x2000 in case I'm mixing my endianesses up). Neither is 0x20 it a valid
> > Ethernet frame length (min 64) so it's not that sort of confusion
> > AFAICT.
> > 
> > skb->protocol is initialised in sky2_status_intr with the return value
> > of "eth_type_trans(skb, dev)" which as far as I can tell cannot return
> > 0x20.
> > 
> > The domU network configuration is using the sky2 device directly, no
> > bridging, VLAN, tunnels or anything else like that?
> At boot it uses bridge.
> For the test, I delete bridge and set IP address directly on eth0

OK, good.

[...]
> generic-segmentation-offload: on
[...]
> receive-hashing: on

Can you also try turning these two off (independently and together).

> What is a bit strange here is that I don't any more the KERN_CRIT printk
> message.
> Could be a false positive ?

Worth bearing in mind, lets see what the next test run produces.

> I'm currently compiling new kernel with your last patch. will keep you
> updated

Thanks.

Please gather the tcpdump's too.

Ian.

Re: PCI passthrough issue

[Jean Baptiste Favre]

[-- Attachment #1: Type: text/plain, Size: 2447 bytes --]

Le 04/02/2011 12:28, Ian Campbell a écrit :
> On Fri, 2011-02-04 at 11:25 +0000, Jean Baptiste Favre wrote:
>> Hello,
>>
>> Le 04/02/2011 12:04, Ian Campbell a écrit :
>>> On Fri, 2011-02-04 at 10:12 +0000, Jean Baptiste Favre wrote:
>>>> Hello Ian,
>>>> Applyed your patches.
>>>
>>> Thanks.
>>>
>>>> Now, I've:
>>>> # ping -s86 10.0.0.1
>>>> PING 10.0.0.1 (10.0.0.1): 86 data bytes
>>>> __netif_receive_skb dropping skb proto 0x20
>>>>
>>>>
>>>> So problem seems to occur in net/core/dev.c file, according to the patch
>>>> bellow
>>>
>>> Interesting. the number printed in the warning is type == skb->protocol
>>> == 0x20 which is not a valid protocol that I can find anywhere (nor is
>>> 0x2000 in case I'm mixing my endianesses up). Neither is 0x20 it a valid
>>> Ethernet frame length (min 64) so it's not that sort of confusion
>>> AFAICT.
>>>
>>> skb->protocol is initialised in sky2_status_intr with the return value
>>> of "eth_type_trans(skb, dev)" which as far as I can tell cannot return
>>> 0x20.
>>>
>>> The domU network configuration is using the sky2 device directly, no
>>> bridging, VLAN, tunnels or anything else like that?
>> At boot it uses bridge.
>> For the test, I delete bridge and set IP address directly on eth0
> 
> OK, good.
> 
> [...]
>> generic-segmentation-offload: on
> [...]
>> receive-hashing: on
> 
> Can you also try turning these two off (independently and together).
No change with ethtool -K gso off
Can not change receive-hashing:
# ethtool -K eth1 rxhash off
Cannot set device flag settings: Invalid argument



>> What is a bit strange here is that I don't any more the KERN_CRIT printk
>> message.
>> Could be a false positive ?
> 
> Worth bearing in mind, lets see what the next test run produces.
Seems that I got this messge only with copybreak=0.
With default value (128), no such message

More, with copybreak=0, all packets are dropped (even a ping with
default packet size is dropped. Same with ping -s1)

>> I'm currently compiling new kernel with your last patch. will keep you
>> updated
No new messages, despite your patch :(

> Thanks.
> 
> Please gather the tcpdump's too.
Both tcpdump from GW and domU are Attached.

Commands were:

domU# tcpdump -n -w domU.cap -s0 -i eth0 ether host 00:1f:c6:eb:71:43 or
ether broadcast

gw# tcpdump -n -w gw.cap -s0 -i br0 ether host 00:1f:c6:eb:71:43 or
ether broadcast


[-- Attachment #2: gw.cap --]
[-- Type: application/octet-stream, Size: 8906 bytes --]

[-- Attachment #3: domU.cap --]
[-- Type: application/octet-stream, Size: 7484 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Ian Campbell]

On Fri, 2011-02-04 at 13:15 +0000, Jean Baptiste Favre wrote:

> 
> >> What is a bit strange here is that I don't any more the KERN_CRIT printk
> >> message.
> >> Could be a false positive ?
> > 
> > Worth bearing in mind, lets see what the next test run produces.
> Seems that I got this messge only with copybreak=0.
> With default value (128), no such message
> 
> More, with copybreak=0, all packets are dropped (even a ping with
> default packet size is dropped. Same with ping -s1)

Hang on, I thought you previously said copybreak=0 made everything work
ok. If that isn't definitely the case then we may be following a red
herring.

Are you saying that copybreak=0 + this patch breaks? That would be very
surprising since the patch doesn't cause any flow control differences.

Perhaps there is some difference between your self-built kernels and the
Debian kernels you started with? Perhaps you should try the self built
kernel with no patches, just to confirm it behaves the same as the
Debian kernels?

> > Thanks.
> > 
> > Please gather the tcpdump's too.
> Both tcpdump from GW and domU are Attached.

Were these collected with or without patches? With or without ethtool -K
options? With or without copybreak?

Please try and be explicit about everything you post, there are lots of
variables in the air.

> Commands were:
> 
> domU# tcpdump -n -w domU.cap -s0 -i eth0 ether host 00:1f:c6:eb:71:43 or
> ether broadcast
> 
> gw# tcpdump -n -w gw.cap -s0 -i br0 ether host 00:1f:c6:eb:71:43 or
> ether broadcast

So no 0x20 anywhere in what the gw sent as the reply and still no clue
where the value came from...

Ian.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 04/02/2011 14:50, Ian Campbell a écrit :
> On Fri, 2011-02-04 at 13:15 +0000, Jean Baptiste Favre wrote:
> 
>>
>>>> What is a bit strange here is that I don't any more the KERN_CRIT printk
>>>> message.
>>>> Could be a false positive ?
>>>
>>> Worth bearing in mind, lets see what the next test run produces.
>> Seems that I got this messge only with copybreak=0.
>> With default value (128), no such message
>>
>> More, with copybreak=0, all packets are dropped (even a ping with
>> default packet size is dropped. Same with ping -s1)
> 
> Hang on, I thought you previously said copybreak=0 made everything work
> ok. If that isn't definitely the case then we may be following a red
> herring.
That's something I'm investigating.
Under Debian, copybreak=0 solve the problem
Under OpenWRT, copybreak=0 + patch breaks. Will try without patch.

> Are you saying that copybreak=0 + this patch breaks? That would be very
> surprising since the patch doesn't cause any flow control differences.
> 
> Perhaps there is some difference between your self-built kernels and the
> Debian kernels you started with? Perhaps you should try the self built
> kernel with no patches, just to confirm it behaves the same as the
> Debian kernels?
Under Debian, I use 2.6.37 from experimental
Under OpenWRT, use legacy 2.6.37, build env applies patches for OpenWRT
and compile.

OpenWRT provides complete build env, as I still have problem compiling
Debian 32bits kernel from 64bits env. That's why I switched back to
openWRT for testing.


>>> Thanks.
>>>
>>> Please gather the tcpdump's too.
>> Both tcpdump from GW and domU are Attached.
> 
> Were these collected with or without patches? With or without ethtool -K
> options? With or without copybreak?
> 
> Please try and be explicit about everything you post, there are lots of
> variables in the air.
OK, sorry. Will redo all tests

Regards,
JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
Sorry for long silent period.

I'm taking night classes to get an engineering degree and am in the
midst of exams this whole week.

I'll be back available next week to perform required tests and will
provide you an update as soon as possible.

Regards,
JB

Le 04/02/2011 15:01, Jean Baptiste Favre a écrit :
> Le 04/02/2011 14:50, Ian Campbell a écrit :
>> On Fri, 2011-02-04 at 13:15 +0000, Jean Baptiste Favre wrote:
>>
>>>
>>>>> What is a bit strange here is that I don't any more the KERN_CRIT printk
>>>>> message.
>>>>> Could be a false positive ?
>>>>
>>>> Worth bearing in mind, lets see what the next test run produces.
>>> Seems that I got this messge only with copybreak=0.
>>> With default value (128), no such message
>>>
>>> More, with copybreak=0, all packets are dropped (even a ping with
>>> default packet size is dropped. Same with ping -s1)
>>
>> Hang on, I thought you previously said copybreak=0 made everything work
>> ok. If that isn't definitely the case then we may be following a red
>> herring.
> That's something I'm investigating.
> Under Debian, copybreak=0 solve the problem
> Under OpenWRT, copybreak=0 + patch breaks. Will try without patch.
> 
>> Are you saying that copybreak=0 + this patch breaks? That would be very
>> surprising since the patch doesn't cause any flow control differences.
>>
>> Perhaps there is some difference between your self-built kernels and the
>> Debian kernels you started with? Perhaps you should try the self built
>> kernel with no patches, just to confirm it behaves the same as the
>> Debian kernels?
> Under Debian, I use 2.6.37 from experimental
> Under OpenWRT, use legacy 2.6.37, build env applies patches for OpenWRT
> and compile.
> 
> OpenWRT provides complete build env, as I still have problem compiling
> Debian 32bits kernel from 64bits env. That's why I switched back to
> openWRT for testing.
> 
> 
>>>> Thanks.
>>>>
>>>> Please gather the tcpdump's too.
>>> Both tcpdump from GW and domU are Attached.
>>
>> Were these collected with or without patches? With or without ethtool -K
>> options? With or without copybreak?
>>
>> Please try and be explicit about everything you post, there are lots of
>> variables in the air.
> OK, sorry. Will redo all tests
> 
> Regards,
> JB
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
Back online after my exams :)

Had some time to perform tests with my Debian Squeeze 32bits domU and
2.6.37 kernel from experimental.
DomU config is:
*****************************************************************
kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
#kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem-sky2'
#ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem-sky2'
builder      = 'linux'
memory=268
vcpus        = '1'
cpus         = '2'
localtime    = 0
serial       = 'pty'
boot         = 'cdn'
disk         = [ 'drbd:xps-106,xvda,w' ]
on_poweroff  = 'destroy'
on_reboot    = 'restart'
on_crash     = 'restart'
name         = 'xps-106'
hostname     = 'xps-106.clichy.jbfavre.org'

extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
console=hvc0 xencons=tty"
pci = [ '04:00.0' ]
*****************************************************************

Tests were done first with "legacy" kernel from Debian and with patched
kernel (patches from Ian to get verbose debug).
Each test consist in executing following command:
ping -c5 -s150 10.0.0.1
ping -c5 -s85 10.0.0.1

10.0.0.1 is my gateway

As you can see bellow, all verbose debug display "__netif_receive_skb
dropping skb" with various "proto" value and always "ip summed 2".
I can provide full detailled output if you want.

Here are the results:
Memory	2.6.37	2.6.37-patched
128	Works	Works
130	NoBoot	NoBoot
132	Works	Works
134	NoBoot	NoBoot
136	Works	Works
138	NoBoot	NoBoot
140	Fails	__netif_receive_skb dropping skb
142	NoBoot	NoBoot
144	Fails	__netif_receive_skb dropping skb
146	NoBoot	NoBoot
148	Fails	__netif_receive_skb dropping skb
150	NoBoot	NoBoot
152	Fails	__netif_receive_skb dropping skb
154	NoBoot	NoBoot
156	Fails	__netif_receive_skb dropping skb
158	NoBoot	NoBoot
160	Fails	__netif_receive_skb dropping skb
192	Fails	__netif_receive_skb dropping skb
224	Fails	__netif_receive_skb dropping skb
256	Fails	__netif_receive_skb dropping skb
264	Fails	__netif_receive_skb dropping skb
266	NoBoot	NoBoot
268	Works	Works
272	Works	Works
288	Works	Works
320	Works	Works
352	Works	Works
384	Works	Works
416	Works	Works
448	Works	Works
480	Works	Works
512	Works	Works

Now, I'll try to do the same tests with OpenWRT.
I hope I'll have similar results :)

Regards,
JB



Le 09/02/2011 10:59, Jean Baptiste Favre a écrit :
> Hello,
> Sorry for long silent period.
> 
> I'm taking night classes to get an engineering degree and am in the
> midst of exams this whole week.
> 
> I'll be back available next week to perform required tests and will
> provide you an update as soon as possible.
> 
> Regards,
> JB
> 
> Le 04/02/2011 15:01, Jean Baptiste Favre a écrit :
>> Le 04/02/2011 14:50, Ian Campbell a écrit :
>>> On Fri, 2011-02-04 at 13:15 +0000, Jean Baptiste Favre wrote:
>>>
>>>>
>>>>>> What is a bit strange here is that I don't any more the KERN_CRIT printk
>>>>>> message.
>>>>>> Could be a false positive ?
>>>>>
>>>>> Worth bearing in mind, lets see what the next test run produces.
>>>> Seems that I got this messge only with copybreak=0.
>>>> With default value (128), no such message
>>>>
>>>> More, with copybreak=0, all packets are dropped (even a ping with
>>>> default packet size is dropped. Same with ping -s1)
>>>
>>> Hang on, I thought you previously said copybreak=0 made everything work
>>> ok. If that isn't definitely the case then we may be following a red
>>> herring.
>> That's something I'm investigating.
>> Under Debian, copybreak=0 solve the problem
>> Under OpenWRT, copybreak=0 + patch breaks. Will try without patch.
>>
>>> Are you saying that copybreak=0 + this patch breaks? That would be very
>>> surprising since the patch doesn't cause any flow control differences.
>>>
>>> Perhaps there is some difference between your self-built kernels and the
>>> Debian kernels you started with? Perhaps you should try the self built
>>> kernel with no patches, just to confirm it behaves the same as the
>>> Debian kernels?
>> Under Debian, I use 2.6.37 from experimental
>> Under OpenWRT, use legacy 2.6.37, build env applies patches for OpenWRT
>> and compile.
>>
>> OpenWRT provides complete build env, as I still have problem compiling
>> Debian 32bits kernel from 64bits env. That's why I switched back to
>> openWRT for testing.
>>
>>
>>>>> Thanks.
>>>>>
>>>>> Please gather the tcpdump's too.
>>>> Both tcpdump from GW and domU are Attached.
>>>
>>> Were these collected with or without patches? With or without ethtool -K
>>> options? With or without copybreak?
>>>
>>> Please try and be explicit about everything you post, there are lots of
>>> variables in the air.
>> OK, sorry. Will redo all tests
>>
>> Regards,
>> JB
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Fri, Feb 18, 2011 at 10:14:13PM +0100, Jean Baptiste Favre wrote:
> Hello,
> Back online after my exams :)
> 
> Had some time to perform tests with my Debian Squeeze 32bits domU and
> 2.6.37 kernel from experimental.

I am bit lost now.. Can you refresh my memory whether the 'copy_break'
parameter worked or not?

Experimental is the stock kernel or was that a new proposed kernel?
Where are the sources for the experimental kernel?

> DomU config is:
> *****************************************************************
> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
> #kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem-sky2'
> #ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem-sky2'
> builder      = 'linux'
> memory=268
> vcpus        = '1'
> cpus         = '2'
> localtime    = 0
> serial       = 'pty'
> boot         = 'cdn'
> disk         = [ 'drbd:xps-106,xvda,w' ]
> on_poweroff  = 'destroy'
> on_reboot    = 'restart'
> on_crash     = 'restart'
> name         = 'xps-106'
> hostname     = 'xps-106.clichy.jbfavre.org'

I had a box with a sky2 adapter that looked to have a similar issue but
found the culprit to be the switch. So at this point I am having no luck
reproducing this. Would it be possible for you to stick the kernel + dist
image somewhere so I can try it out on my box?

> 
> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
> console=hvc0 xencons=tty"

Try without 'swiotlb=force' on any kernel that is PVOPS. Only the older
ones (lenny) required that. And you don't need 'xencons=tty' either with
PVOPS kernels.

Re: PCI passthrough issue

[Jean Baptiste Favre]

Le 25/02/2011 15:40, Konrad Rzeszutek Wilk a écrit :
> On Fri, Feb 18, 2011 at 10:14:13PM +0100, Jean Baptiste Favre wrote:
>> Hello,
>> Back online after my exams :)
>>
>> Had some time to perform tests with my Debian Squeeze 32bits domU and
>> 2.6.37 kernel from experimental.
> 
> I am bit lost now.. Can you refresh my memory whether the 'copy_break'
> parameter worked or not?
Copybreak param still solve problem:
# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
__netif_receive_skb dropping skb proto 0x6aac ip summed 2
__netif_receive_skb dropping skb proto 0x620c ip summed 2
__netif_receive_skb dropping skb proto 0x6aac ip summed 2
^C
--- 10.0.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

root@xps-106:~# rmmod sky2
[   23.137515] sky2 0000:00:00.0: eth0: disabling interface
root@xps-106:~# modprobe sky2 copybreak=0
 sky2: driver version 1.28
 sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
 xen_map_pirq_gsi: returning irq 18 for gsi 18
 sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
 xen_map_pirq_gsi: returning irq 55 for gsi 55
 sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43
 sky2 0000:00:00.0: eth0: enabling interface
 ADDRCONF(NETDEV_UP): eth0: link is not ready
 sky2 0000:00:00.0: eth0: Link is up at 1000 Mbps, full duplex, flow
control both
 ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
root@xps-106:~# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_req=1 ttl=64 time=4.09 ms
64 bytes from 10.0.0.1: icmp_req=2 ttl=64 time=0.814 ms
^C
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.814/2.456/4.098/1.642 ms

> 
> Experimental is the stock kernel or was that a new proposed kernel?
> Where are the sources for the experimental kernel?
Kernel 2.6.37 has been installed from Debian experimental repository.
Did not have time to compile my own from vanilla sources.
Maybe will have some time this week, but not sure.


>> DomU config is:
>> *****************************************************************
>> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
>> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
>> #kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem-sky2'
>> #ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem-sky2'
>> builder      = 'linux'
>> memory=268
>> vcpus        = '1'
>> cpus         = '2'
>> localtime    = 0
>> serial       = 'pty'
>> boot         = 'cdn'
>> disk         = [ 'drbd:xps-106,xvda,w' ]
>> on_poweroff  = 'destroy'
>> on_reboot    = 'restart'
>> on_crash     = 'restart'
>> name         = 'xps-106'
>> hostname     = 'xps-106.clichy.jbfavre.org'
> 
> I had a box with a sky2 adapter that looked to have a similar issue but
> found the culprit to be the switch. So at this point I am having no luck
> reproducing this. Would it be possible for you to stick the kernel + dist
> image somewhere so I can try it out on my box?
As requested by you (or Ian, I don't remeber exactly), I've tried
booting the server with 32bits 2.6.37 kernel and 32bits Debian Squeeze.
Even with mem=256 boot param, everything worked fine, so I'm not sure
switch can be the culprit.

>> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
>> console=hvc0 xencons=tty"
> 
> Try without 'swiotlb=force' on any kernel that is PVOPS. Only the older
> ones (lenny) required that. And you don't need 'xencons=tty' either with
> PVOPS kernels.
OK, will try that.

Regards,
JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,


Le 25/02/2011 15:40, Konrad Rzeszutek Wilk a écrit :
> On Fri, Feb 18, 2011 at 10:14:13PM +0100, Jean Baptiste Favre wrote:
>> Hello,
>> Back online after my exams :)
>>
>> Had some time to perform tests with my Debian Squeeze 32bits domU and
>> 2.6.37 kernel from experimental.
> 
> I am bit lost now.. Can you refresh my memory whether the 'copy_break'
> parameter worked or not?
> 
> Experimental is the stock kernel or was that a new proposed kernel?
> Where are the sources for the experimental kernel?
> 
>> DomU config is:
>> *****************************************************************
>> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
>> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
>> #kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem-sky2'
>> #ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem-sky2'
>> builder      = 'linux'
>> memory=268
>> vcpus        = '1'
>> cpus         = '2'
>> localtime    = 0
>> serial       = 'pty'
>> boot         = 'cdn'
>> disk         = [ 'drbd:xps-106,xvda,w' ]
>> on_poweroff  = 'destroy'
>> on_reboot    = 'restart'
>> on_crash     = 'restart'
>> name         = 'xps-106'
>> hostname     = 'xps-106.clichy.jbfavre.org'
> 
> I had a box with a sky2 adapter that looked to have a similar issue but
> found the culprit to be the switch. So at this point I am having no luck
> reproducing this. Would it be possible for you to stick the kernel + dist
> image somewhere so I can try it out on my box?
> 
>>
>> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
>> console=hvc0 xencons=tty"
> 
> Try without 'swiotlb=force' on any kernel that is PVOPS. Only the older
> ones (lenny) required that. And you don't need 'xencons=tty' either with
> PVOPS kernels.
Thought I already tried without 'swiotlb=force', but seems not...

Just tried it on my Debian domU, and everything works now, whatever
copybreak value can be.

But trying the same with OpenWRT gives following results:
- With 'swiotlb=force': fails for size equal or greater than 128 bytes
(or ping -s86)
- Without 'swiotlb=force': always fails

I can provide OpenWRT disk img and kernel. For Debian domU, I can
provide a dd from LVM and kernel. Let me know which ones you want.

Regards,
JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

> >> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
> >> console=hvc0 xencons=tty"
> > 
> > Try without 'swiotlb=force' on any kernel that is PVOPS. Only the older
> > ones (lenny) required that. And you don't need 'xencons=tty' either with
> > PVOPS kernels.
> Thought I already tried without 'swiotlb=force', but seems not...
> 
> Just tried it on my Debian domU, and everything works now, whatever
> copybreak value can be.
> 
> But trying the same with OpenWRT gives following results:
> - With 'swiotlb=force': fails for size equal or greater than 128 bytes
> (or ping -s86)
> - Without 'swiotlb=force': always fails
> 
> I can provide OpenWRT disk img and kernel. For Debian domU, I can
> provide a dd from LVM and kernel. Let me know which ones you want.

Lets do both. Do you know where the sources for OpenWRT are located?

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,

Le 28/02/2011 16:01, Konrad Rzeszutek Wilk a écrit :
>>>> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
>>>> console=hvc0 xencons=tty"
>>>
>>> Try without 'swiotlb=force' on any kernel that is PVOPS. Only the older
>>> ones (lenny) required that. And you don't need 'xencons=tty' either with
>>> PVOPS kernels.
>> Thought I already tried without 'swiotlb=force', but seems not...
>>
>> Just tried it on my Debian domU, and everything works now, whatever
>> copybreak value can be.
>>
>> But trying the same with OpenWRT gives following results:
>> - With 'swiotlb=force': fails for size equal or greater than 128 bytes
>> (or ping -s86)
>> - Without 'swiotlb=force': always fails
>>
>> I can provide OpenWRT disk img and kernel. For Debian domU, I can
>> provide a dd from LVM and kernel. Let me know which ones you want.
> 
> Lets do both. Do you know where the sources for OpenWRT are located?

Openwrt .img and PV kernel are available at
http://downloads.jbfavre.org/openwrt.tar.gz

I'm making dd from my debian DomU and will upload it as debian.tar.gz at
the same place. Will update you when completed.

Openwrt source tree is available here:
svn://svn.openwrt.org/openwrt/trunk

You have all information to build it here:
http://wiki.openwrt.org/doc/howto/build

Or you have all steps I followed here:
http://publications.jbfavre.org/virtualisation/xen_openwrt_domu_pci_passthrough.en

Regards,
JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

> >> Lets do both. Do you know where the sources for OpenWRT are located?
> > 
> > Openwrt .img and PV kernel are available at
> > http://downloads.jbfavre.org/openwrt.tar.gz

Using that, and this xm file

kernel="/mnt/tmp/openwrt/openwrt-x86-xen_domu-vmlinuz"
root='/dev/xvda2 rw'
memory=256
vcpus=1
localtime=0
disk=['phy:/dev/sdc,xvda,w']
extra="console=hvc0 debug loglevel=10 iommu=soft"
name="openwrt"
on_crash="preserve"
vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1']
pci = ['04:00.0']

where
04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13)

I can't get the sky2 adapter to work at all.

Just to make sure it wasn't your build ...
> > 
> > I'm making dd from my debian DomU and will upload it as debian.tar.gz at
> > the same place. Will update you when completed.
> > 
> > Openwrt source tree is available here:
> > svn://svn.openwrt.org/openwrt/trunk
> > 
> > You have all information to build it here:
> > http://wiki.openwrt.org/doc/howto/build
> > 
> > Or you have all steps I followed here:
> > http://publications.jbfavre.org/virtualisation/xen_openwrt_domu_pci_passthrough.en

... I tried to follow those directions and found that it would not work.
I can't get the xen-pcifront.ko file at all on any of the *combined.img images.


I made this patch thinking it was due to the name of the module being different:

Index: target/linux/x86/Makefile
===================================================================
--- target/linux/x86/Makefile	(revision 25855)
+++ target/linux/x86/Makefile	(working copy)
@@ -12,7 +12,7 @@
 FEATURES:=squashfs jffs2 ext4 vdi vmdk pcmcia targz
 SUBTARGETS=generic olpc xen_domu ep80579 net5501 kvm_guest geos
 
-LINUX_VERSION:=2.6.32.29
+LINUX_VERSION:=2.6.37
 
 include $(INCLUDE_DIR)/target.mk
 
Index: package/kernel/modules/virtual.mk
===================================================================
--- package/kernel/modules/virtual.mk	(revision 25855)
+++ package/kernel/modules/virtual.mk	(working copy)
@@ -168,7 +168,7 @@
   TITLE:=Xen PCI device frontend
   DEPENDS:=@TARGET_x86_xen_domu @LINUX_2_6_37||LINUX_2_6_38
   KCONFIG:=CONFIG_XEN_PCIDEV_FRONTEND
-  FILES:=$(LINUX_DIR)/drivers/xen/platform-pci.ko
+  FILES:=$(LINUX_DIR)/drivers/pci/xen-pcifront.ko
   AUTOLOAD:=$(call AutoLoad,10,xen-pcifront)
 endef

but it still would not include the xen-pcifront.ko file on the *combined-ext4.img.gz
file. Any ideas what I am doing wrong?

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,

Le 03/03/2011 23:12, Konrad Rzeszutek Wilk a écrit :
>>>> Lets do both. Do you know where the sources for OpenWRT are located?
>>>
>>> Openwrt .img and PV kernel are available at
>>> http://downloads.jbfavre.org/openwrt.tar.gz
> 
> Using that, and this xm file
> 
> kernel="/mnt/tmp/openwrt/openwrt-x86-xen_domu-vmlinuz"
> root='/dev/xvda2 rw'
> memory=256
> vcpus=1
> localtime=0
> disk=['phy:/dev/sdc,xvda,w']
> extra="console=hvc0 debug loglevel=10 iommu=soft"
> name="openwrt"
> on_crash="preserve"
> vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1']
> pci = ['04:00.0']
> 
> where
> 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13)
> 
> I can't get the sky2 adapter to work at all.
> 
> Just to make sure it wasn't your build ...
>>>
>>> I'm making dd from my debian DomU and will upload it as debian.tar.gz at
>>> the same place. Will update you when completed.
>>>
>>> Openwrt source tree is available here:
>>> svn://svn.openwrt.org/openwrt/trunk
>>>
>>> You have all information to build it here:
>>> http://wiki.openwrt.org/doc/howto/build
>>>
>>> Or you have all steps I followed here:
>>> http://publications.jbfavre.org/virtualisation/xen_openwrt_domu_pci_passthrough.en
> 
> ... I tried to follow those directions and found that it would not work.
> I can't get the xen-pcifront.ko file at all on any of the *combined.img images.
> 
> 
> I made this patch thinking it was due to the name of the module being different:
> 
> Index: target/linux/x86/Makefile
> ===================================================================
> --- target/linux/x86/Makefile	(revision 25855)
> +++ target/linux/x86/Makefile	(working copy)
> @@ -12,7 +12,7 @@
>  FEATURES:=squashfs jffs2 ext4 vdi vmdk pcmcia targz
>  SUBTARGETS=generic olpc xen_domu ep80579 net5501 kvm_guest geos
>  
> -LINUX_VERSION:=2.6.32.29
> +LINUX_VERSION:=2.6.37
>  
>  include $(INCLUDE_DIR)/target.mk
>  
> Index: package/kernel/modules/virtual.mk
> ===================================================================
> --- package/kernel/modules/virtual.mk	(revision 25855)
> +++ package/kernel/modules/virtual.mk	(working copy)
> @@ -168,7 +168,7 @@
>    TITLE:=Xen PCI device frontend
>    DEPENDS:=@TARGET_x86_xen_domu @LINUX_2_6_37||LINUX_2_6_38
>    KCONFIG:=CONFIG_XEN_PCIDEV_FRONTEND
> -  FILES:=$(LINUX_DIR)/drivers/xen/platform-pci.ko
> +  FILES:=$(LINUX_DIR)/drivers/pci/xen-pcifront.ko
>    AUTOLOAD:=$(call AutoLoad,10,xen-pcifront)
>  endef
> 
> but it still would not include the xen-pcifront.ko file on the *combined-ext4.img.gz
> file. Any ideas what I am doing wrong?

I don't remember such problem. Could this module be

BTW, I'm checking that rebuilding OpenWRT.

When you execute make menuconfig, choose Target=x86 and SubTarget="Xen
paravirt Guest". Then go to Kernel Modules -> Virtualisation Support
Xen PCI frontend is disabled by default. Did you activated it ?

OpenWRT global config cat be found in .config:
# grep xen -i .config
CONFIG_TARGET_x86_xen_domu=y
CONFIG_TARGET_x86_xen_domu_Default=y
CONFIG_DEFAULT_kmod-xen-evtchn=y
CONFIG_DEFAULT_kmod-xen-fs=y
CONFIG_DEFAULT_kmod-xen-kbddev=y
CONFIG_DEFAULT_kmod-xen-netdev=y
CONFIG_X86_GRUB_BOOTOPTS="xencons=hvc"
CONFIG_PACKAGE_kmod-xen-evtchn=y
# CONFIG_PACKAGE_kmod-xen-fbdev is not set
CONFIG_PACKAGE_kmod-xen-fs=y
CONFIG_PACKAGE_kmod-xen-kbddev=y
CONFIG_PACKAGE_kmod-xen-netdev=y
CONFIG_PACKAGE_kmod-xen-pcidev=y


Other way may be to fill target/linux/x86/xen_domu/config-default with
Xen CONFIG_* values. This will be used as kernel config

Regards,
JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

> I don't remember such problem. Could this module be
> 
> BTW, I'm checking that rebuilding OpenWRT.

OK.

To eliminate this being the kernel, I took the

trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37
(which has a bunch of patches on top of stock 2.6.37)

compiled it outside the OpenWRT and tried to passthrough the same
adapter. It worked. But let me try to follow your directions
below tomorrow and see what progress I can make.

> 
> When you execute make menuconfig, choose Target=x86 and SubTarget="Xen
> paravirt Guest". Then go to Kernel Modules -> Virtualisation Support
> Xen PCI frontend is disabled by default. Did you activated it ?
> 
> OpenWRT global config cat be found in .config:
> # grep xen -i .config
> CONFIG_TARGET_x86_xen_domu=y
> CONFIG_TARGET_x86_xen_domu_Default=y
> CONFIG_DEFAULT_kmod-xen-evtchn=y
> CONFIG_DEFAULT_kmod-xen-fs=y
> CONFIG_DEFAULT_kmod-xen-kbddev=y
> CONFIG_DEFAULT_kmod-xen-netdev=y
> CONFIG_X86_GRUB_BOOTOPTS="xencons=hvc"
> CONFIG_PACKAGE_kmod-xen-evtchn=y
> # CONFIG_PACKAGE_kmod-xen-fbdev is not set
> CONFIG_PACKAGE_kmod-xen-fs=y
> CONFIG_PACKAGE_kmod-xen-kbddev=y
> CONFIG_PACKAGE_kmod-xen-netdev=y
> CONFIG_PACKAGE_kmod-xen-pcidev=y
> 
> 
> Other way may be to fill target/linux/x86/xen_domu/config-default with
> Xen CONFIG_* values. This will be used as kernel config
> 
> Regards,
> JB

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello Konrad,

Le 03/03/2011 23:58, Konrad Rzeszutek Wilk a écrit :
>> I don't remember such problem. Could this module be
>>
>> BTW, I'm checking that rebuilding OpenWRT.
> 
> OK.
> 
> To eliminate this being the kernel, I took the
> 
> trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37
> (which has a bunch of patches on top of stock 2.6.37)
> 
> compiled it outside the OpenWRT and tried to passthrough the same
> adapter. It worked. But let me try to follow your directions
> below tomorrow and see what progress I can make.
Hello,
One tought or two between wake-up and leaving for work :-)

trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37
 is kernel builddir. So, if it works from here, there's no reason it
finaly doesn't.

One tought: in "make menuconfig", you will setup the Openwrt's
package/features list. If you have one package marked "M", that means
it'll be build as separate package (*.opkg). If marked with "y", it'll
be integrated within FS image.

For you kernel module xen-pcifront, you have to make sure that:
- it's marked in trunk/.config
- it's marked in kernel config file

You can check after compliation:
ls -l build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/
total 12
drwxr-xr-x 2 jbfavre jbfavre 4096 Mar  4 01:09 CONTROL
drwxr-xr-x 3 jbfavre jbfavre 4096 Mar  4 01:09 etc
drwxr-xr-x 3 jbfavre jbfavre 4096 Mar  4 01:09 lib
jbfavre@xps-103:~/openwrt/trunk$ ls -l
build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko

-rw-r--r-- 1 jbfavre jbfavre 4936 Mar  4 01:09
build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko

So, not sure you patch is mandatory, is it ?

Regards,
JB

Re: PCI passthrough issue

[Konrad Rzeszutek Wilk]

On Fri, Mar 04, 2011 at 08:25:49AM +0100, Jean Baptiste Favre wrote:
> Hello Konrad,
> 
> Le 03/03/2011 23:58, Konrad Rzeszutek Wilk a écrit :
> >> I don't remember such problem. Could this module be
> >>
> >> BTW, I'm checking that rebuilding OpenWRT.
> > 
> > OK.
> > 
> > To eliminate this being the kernel, I took the
> > 
> > trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37
> > (which has a bunch of patches on top of stock 2.6.37)
> > 
> > compiled it outside the OpenWRT and tried to passthrough the same
> > adapter. It worked. But let me try to follow your directions
> > below tomorrow and see what progress I can make.
> Hello,
> One tought or two between wake-up and leaving for work :-)

I haven't gotten back on this - sorry about this.

The other thing that might be going wrong is that CONFIG_PCI_QUIRKS is not
turned on in the OpenWRT build. Perhaps there are some work-arounds for the
driver that aren't enabled with OpenWRT build? You could try to
build a debian kernel using the OpenWRT .config and seeing if
it works right?

> 
> trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37
>  is kernel builddir. So, if it works from here, there's no reason it
> finaly doesn't.
> 
> One tought: in "make menuconfig", you will setup the Openwrt's
> package/features list. If you have one package marked "M", that means
> it'll be build as separate package (*.opkg). If marked with "y", it'll
> be integrated within FS image.
> 
> For you kernel module xen-pcifront, you have to make sure that:
> - it's marked in trunk/.config
> - it's marked in kernel config file
> 
> You can check after compliation:
> ls -l build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/
> total 12
> drwxr-xr-x 2 jbfavre jbfavre 4096 Mar  4 01:09 CONTROL
> drwxr-xr-x 3 jbfavre jbfavre 4096 Mar  4 01:09 etc
> drwxr-xr-x 3 jbfavre jbfavre 4096 Mar  4 01:09 lib
> jbfavre@xps-103:~/openwrt/trunk$ ls -l
> build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko
> 
> -rw-r--r-- 1 jbfavre jbfavre 4936 Mar  4 01:09
> build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko
> 
> So, not sure you patch is mandatory, is it ?

Probably not.

Will get back to this after the XenHackothon.
> 
> Regards,
> JB
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: PCI passthrough issue

[Jean Baptiste Favre]

Hello,
Sorry for long silence: had some hollidays :)

On 16/03/2011 04:14, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 04, 2011 at 08:25:49AM +0100, Jean Baptiste Favre wrote:
>> Hello Konrad,
>>
>> Le 03/03/2011 23:58, Konrad Rzeszutek Wilk a écrit :
>>>> I don't remember such problem. Could this module be
>>>>
>>>> BTW, I'm checking that rebuilding OpenWRT.
>>>
>>> OK.
>>>
>>> To eliminate this being the kernel, I took the
>>>
>>> trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37
>>> (which has a bunch of patches on top of stock 2.6.37)
>>>
>>> compiled it outside the OpenWRT and tried to passthrough the same
>>> adapter. It worked. But let me try to follow your directions
>>> below tomorrow and see what progress I can make.
>> Hello,
>> One tought or two between wake-up and leaving for work :-)
> 
> I haven't gotten back on this - sorry about this.
> 
> The other thing that might be going wrong is that CONFIG_PCI_QUIRKS is not
> turned on in the OpenWRT build. Perhaps there are some work-arounds for the
> driver that aren't enabled with OpenWRT build? You could try to
> build a debian kernel using the OpenWRT .config and seeing if
> it works right?
In my setup, CONFIG_PCI_QUIRKS is enabled. As default compilation did
not worked, I took all XEN and PCI related config options from Debian
kernel and manually added them in target/linux/x86/xen_domu/config-default

As a consequence, its activated in final kernel but with no change :(


>> trunk/build_dir/toolchain-i386_gcc-linaro_uClibc-0.9.32/linux-2.6.37
>>  is kernel builddir. So, if it works from here, there's no reason it
>> finaly doesn't.
>>
>> One tought: in "make menuconfig", you will setup the Openwrt's
>> package/features list. If you have one package marked "M", that means
>> it'll be build as separate package (*.opkg). If marked with "y", it'll
>> be integrated within FS image.
>>
>> For you kernel module xen-pcifront, you have to make sure that:
>> - it's marked in trunk/.config
>> - it's marked in kernel config file
>>
>> You can check after compliation:
>> ls -l build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/
>> total 12
>> drwxr-xr-x 2 jbfavre jbfavre 4096 Mar  4 01:09 CONTROL
>> drwxr-xr-x 3 jbfavre jbfavre 4096 Mar  4 01:09 etc
>> drwxr-xr-x 3 jbfavre jbfavre 4096 Mar  4 01:09 lib
>> jbfavre@xps-103:~/openwrt/trunk$ ls -l
>> build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko
>>
>> -rw-r--r-- 1 jbfavre jbfavre 4936 Mar  4 01:09
>> build_dir/linux-x86_xen_domu/packages/ipkg-x86/kmod-xen-pcidev/lib/modules/2.6.37/platform-pci.ko
>>
>> So, not sure you patch is mandatory, is it ?
> 
> Probably not.
> 
> Will get back to this after the XenHackothon.
Hope this goes well :)

Regards,
JB