Re: Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

Re: Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 3582 bytes --]

 F11 PV DomU running for about 1 hr at Xen 3.4.1 Dom0 (2.6.30-rc6-tip) , been built
on top of F11. During this time frame:-
1.  4GB ISO file scp'ed from Dom0 to DomU at 30 MB/sec
2.  Same file scp'ed from DomU to Dom0 at  17-24 MB/sec
"yum update" was running at DomU in parallel with both "scp" about 20-25 min and succeeded. File /var/log/messages captured and attached. It doesn't look to me to have network problems. However, speed of data transfer from DomU to Dom0 is noticable
lower then vice versa.

Hardware: Q9550,8GB RAM, SATA Drive 500 GB ( attached to ICH10R).
2 GB and 2 vcpus been allocated  for F11 DomU

--- On Thu, 5/28/09, M A Young <m.a.young@durham.ac.uk> wrote:

From: M A Young <m.a.young@durham.ac.uk>
Subject: [Xen-devel] Network drop on domU (netfront: rx->offset: 0, size: 4294967295)
To: xen-devel@lists.xensource.com
Date: Thursday, May 28, 2009, 6:02 PM

I have started to see this error as well now on a Fedora Rawhide + xen-tip/next dom0 kernel on Fedora 11 and a Fedora 11 domU. The base system has just been reinstalled and used to be running Fedora 8 dom0, the domU machine hasn't changed. This is on a fairly old machine with 512M memory shared between the two instances.

/var/log/messages is packed with errors like
net eth0: rx->offset: 0, size: 4294967295
with the more occasional
__ratelimit: 25 callbacks suppressed
about every 10 lines, while updating some packages using rpm where the packages are on an NFS mount, which is maybe running at 1/10th or less the speed than I would expect. The network setup on dom0 is

bridge name    bridge id        STP enabled    interfaces
eth0        8000.0011110a766f    no        peth0
                            vif4.0

with only a handful of dropped packets on vif4.0
vif4.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:524716 errors:0 dropped:0 overruns:0 frame:0
          TX packets:658136 errors:0 dropped:39 overruns:0 carrier:0
          collisions:0 txqueuelen:32
          RX bytes:36703881 (35.0 MiB)  TX bytes:862875521 (822.9 MiB)

On domU however ifconfig reports
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:567462 errors:73814 dropped:0 overruns:0 frame:0
          TX packets:511318 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:719940314 (686.5 MiB)  TX bytes:42982009 (40.9 MiB)
          Interrupt:9

I tried enabling debugging in drivers/xen/netback/netback.c by setting
#define NETBE_DEBUG_INTERRUPT
but when I try to build the kernel I get the error
drivers/xen/blkback/xenbus.c: In function 'blkif_xenbus_init':
drivers/xen/blkback/xenbus.c:541: warning: ignoring return value of 'xenbus_register_backend', declared with attribute warn_unused_result
drivers/xen/netback/netback.c: In function 'netback_init':
drivers/xen/netback/netback.c:1503: error: 'SA_SHIRQ' undeclared (first use in this function)
drivers/xen/netback/netback.c:1503: error: (Each undeclared identifier is reported only once
drivers/xen/netback/netback.c:1503: error: for each function it appears in.)
drivers/xen/netback/netback.c:1505: warning: passing argument 3 of 'bind_virq_to_irqhandler' from incompatible pointer type

    Michael Young

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 4801 bytes --]

[-- Attachment #2: var-log-messages-PV.gz --]
[-- Type: application/x-gzip, Size: 17810 bytes --]

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 6018 bytes --]

Xen Host failure during VNC session. Requires reboot to bring box back to LAN

Kernel failure message 1:

------------[ cut here ]------------
WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0xcf/0x12c()
Hardware name: P5Q-E
NETDEV WATCHDOG: peth0 (sky2): transmit timed out
Modules linked in: xt_physdev nls_utf8 fuse sco bnep l2cap bluetooth sunrpc ipv6 dm_multipath uinput snd_hda_codec_analog snd_hda_intel snd_hda_codec snd_hwdep snd_pcm firewire_ohci snd_timer firewire_core i2c_i801 snd sky2 asus_atk0110 iTCO_wdt i2c_core hwmon pcspkr soundcore iTCO_vendor_support crc_itu_t skge snd_page_alloc [last unloaded: microcode]
Pid: 0, comm: swapper Not tainted 2.6.30-rc6-tip #1
Call Trace:
<IRQ>  [<ffffffff8104f251>] warn_slowpath_common+0x7c/0x94
[<ffffffff8104f2c0>] warn_slowpath_fmt+0x41/0x43
[<ffffffff8134e9e1>] ? netif_tx_lock+0x44/0x6c
[<ffffffff8134eb25>] dev_watchdog+0xcf/0x12c
[<ffffffff81059823>] run_timer_softirq+0x1a4/0x225
[<ffffffff8100eeef>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff81055225>] __do_softirq+0xc3/0x1ab
[<ffffffff8123bb7b>] ? unmask_evtchn+0x24/0xb3
[<ffffffff81012dbc>] call_softirq+0x1c/0x30
[<ffffffff810142f4>] do_softirq+0x50/0xb1
[<ffffffff81054c4f>] irq_exit+0x53/0x90
[<ffffffff8123c338>] xen_evtchn_do_upcall+0x178/0x194
[<ffffffff8100eeef>] ? xen_restore_fl_direct_end+0x0/0x1
[<ffffffff81012e0e>] xen_do_hypervisor_callback+0x1e/0x30
<EOI>  [<ffffffff810093aa>] ? _stext+0x3aa/0x100b
[<ffffffff810093aa>] ? _stext+0x3aa/0x100b
[<ffffffff8100e873>] ? xen_safe_halt+0x10/0x1a
[<ffffffff8100bfc9>] ? xen_idle+0x49/0x60
[<ffffffff81010e3e>] ? cpu_idle+0x67/0xb2
[<ffffffff813cc207>] ? rest_init+0x6b/0x6d
[<ffffffff81674ccd>] ? start_kernel+0x3bc/0x3c7
[<ffffffff816742c1>] ? x86_64_start_reservations+0xac/0xb0
[<ffffffff81677d16>] ? xen_start_kernel+0x5cc/0x5d3
---[ end trace 5120fb3b2e6f575c ]---


Boris.

--- On Wed, 6/10/09, Boris Derzhavets <bderzhavets@yahoo.com> wrote:

From: Boris Derzhavets <bderzhavets@yahoo.com>
Subject: Re: [Xen-devel] Network drop on domU (netfront: rx->offset: 0, size: 4294967295)
To: xen-devel@lists.xensource.com, "M A Young" <m.a.young@durham.ac.uk>
Date: Wednesday, June 10, 2009, 1:14 PM

 F11 PV DomU running for about 1 hr at Xen 3.4.1 Dom0 (2.6.30-rc6-tip) , been built
on top of F11. During this time frame:-
1.  4GB ISO file scp'ed from Dom0 to DomU at 30 MB/sec
2.  Same file scp'ed from DomU to Dom0 at  17-24 MB/sec
"yum update" was running at DomU in parallel with both "scp" about 20-25 min and succeeded. File /var/log/messages captured and attached. It doesn't look to me to have network problems. However, speed of data transfer from DomU to Dom0 is noticable
lower then vice versa.

Hardware: Q9550,8GB RAM, SATA Drive 500 GB ( attached to ICH10R).
2 GB and 2 vcpus been allocated  for F11 DomU

--- On Thu, 5/28/09, M A Young <m.a.young@durham.ac.uk> wrote:

From: M A
 Young <m.a.young@durham.ac.uk>
Subject: [Xen-devel] Network drop on domU (netfront: rx->offset: 0, size: 4294967295)
To: xen-devel@lists.xensource.com
Date: Thursday, May 28, 2009, 6:02 PM

I have started to see this error as well now on a Fedora Rawhide + xen-tip/next dom0 kernel on Fedora 11 and a Fedora 11 domU. The base system has just been reinstalled and used to be running Fedora 8 dom0, the domU machine hasn't changed. This is on a fairly old machine with 512M memory shared between the two instances.

/var/log/messages is packed with errors like
net eth0: rx->offset: 0, size: 4294967295
with the more occasional
__ratelimit: 25 callbacks suppressed
about every 10 lines, while updating some packages using rpm where the packages are on an NFS mount, which is maybe running at 1/10th or less the speed than I would expect. The network setup on dom0 is

bridge
 name    bridge id        STP enabled    interfaces
eth0        8000.0011110a766f    no        peth0
                            vif4.0

with only a handful of dropped packets on vif4.0
vif4.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:524716 errors:0 dropped:0 overruns:0 frame:0
          TX packets:658136 errors:0 dropped:39 overruns:0 carrier:0
          collisions:0
 txqueuelen:32
          RX bytes:36703881 (35.0 MiB)  TX bytes:862875521 (822.9 MiB)

On domU however ifconfig reports
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:567462 errors:73814 dropped:0 overruns:0 frame:0
          TX packets:511318 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:719940314 (686.5 MiB)  TX bytes:42982009 (40.9 MiB)
          Interrupt:9

I tried enabling debugging in drivers/xen/netback/netback.c by setting
#define NETBE_DEBUG_INTERRUPT
but when I try to build the kernel I get the error
drivers/xen/blkback/xenbus.c: In function 'blkif_xenbus_init':
drivers/xen/blkback/xenbus.c:541: warning: ignoring return
 value of 'xenbus_register_backend', declared with attribute warn_unused_result
drivers/xen/netback/netback.c: In function 'netback_init':
drivers/xen/netback/netback.c:1503: error: 'SA_SHIRQ' undeclared (first use in this function)
drivers/xen/netback/netback.c:1503: error: (Each undeclared identifier is reported only once
drivers/xen/netback/netback.c:1503: error: for each function it appears in.)
drivers/xen/netback/netback.c:1505: warning: passing argument 3 of 'bind_virq_to_irqhandler' from incompatible pointer type

    Michael Young

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      
-----Inline Attachment Follows-----

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 8128 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

[James Harper]

> 
> Not sure, it will help :-
> 
> http://hightechsorcery.com/2008/03/virtualization-tip-always-disable-
> checksumming-virtual-ethernet-devices
> 
> Personally , i used to manage this way.
> 

When it works, checksum offload and [GTL]SO can make a big difference in
performance when the packets are staying inside the same physical
machine, and can still make a big difference even when they are leaving
via a physical adapter.

Two servers I look after have no problems at all with checksum or large
send offloads.

One server gets upset though and forgets to make the checksums correct
when it routes the packets over a GRE tunnel. It's strange though - it
will be fine for weeks and then suddenly starts forgetting to fix up the
checksums for 30 minutes or so, then comes good again.

If network performance isn't a big part of what you are doing (eg your
VM is serving web pages at mbit rather than gbit speeds), then turning
off the offloads is probably reasonable. If you are doing file serving
or something then it's probably worth finding a configuration that
works.

It can be horribly frustrating though - I recently spent hours on a
non-virtualised windows server that was hanging connections all over the
place. It turned out to be a checksum offload problem and turning off
the feature on the card fixed it. I think it was an interaction with the
firewall or something.

James

Re: Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1438 bytes --]

Not sure, it will help :-

http://hightechsorcery.com/2008/03/virtualization-tip-always-disable-checksumming-virtual-ethernet-devices

Personally , i used to manage this way.

Boris.

--- On Fri, 5/29/09, M A Young <m.a.young@durham.ac.uk> wrote:

From: M A Young <m.a.young@durham.ac.uk>
Subject: Re: [Xen-devel] Network drop on domU (netfront: rx->offset: 0, size: 4294967295)
To: "Dennis Krul" <dweazle@gmail.com>
Cc: xen-devel@lists.xensource.com
Date: Friday, May 29, 2009, 11:13 AM

On Fri, 29 May 2009, M A Young wrote:

> On Fri, 29 May 2009, Dennis Krul wrote:
>
>> We've encountered the same problem. (We're using your packages btw, thanks
>> for publishing them ;)
>> 
>> For what it's worth, you can work around the speed issue by disabling TCP
>> segmentation offloading (ethtool -K ethX tso off).
>> (You should disable it in your dom0 on all interfaces and in the domU as
>> well.)
>> 
>> Unfortunately there is no apparent way to disable tso when kickstarting a
>> domU, but other than that the speeds are back to normal.
>
> Thanks for that, it fixes my problem as well.

Actually, I spoke too soon, the problems are still there (I think I was 
fooled by a quiet period straight after applying the changes).

     Michael Young

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 2080 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

[M A Young]

On Fri, 29 May 2009, M A Young wrote:

> On Fri, 29 May 2009, Dennis Krul wrote:
>
>> We've encountered the same problem. (We're using your packages btw, thanks
>> for publishing them ;)
>> 
>> For what it's worth, you can work around the speed issue by disabling TCP
>> segmentation offloading (ethtool -K ethX tso off).
>> (You should disable it in your dom0 on all interfaces and in the domU as
>> well.)
>> 
>> Unfortunately there is no apparent way to disable tso when kickstarting a
>> domU, but other than that the speeds are back to normal.
>
> Thanks for that, it fixes my problem as well.

Actually, I spoke too soon, the problems are still there (I think I was 
fooled by a quiet period straight after applying the changes).

 	Michael Young

Re: Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

[M A Young]

On Fri, 29 May 2009, Dennis Krul wrote:

> We've encountered the same problem. (We're using your packages btw, thanks
> for publishing them ;)
> 
> For what it's worth, you can work around the speed issue by disabling TCP
> segmentation offloading (ethtool -K ethX tso off).
> (You should disable it in your dom0 on all interfaces and in the domU as
> well.)
> 
> Unfortunately there is no apparent way to disable tso when kickstarting a
> domU, but other than that the speeds are back to normal.

Thanks for that, it fixes my problem as well.

 	Michael Young

Re: Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

[Dennis Krul]

[-- Attachment #1.1: Type: text/plain, Size: 3341 bytes --]

Michael,

We've encountered the same problem. (We're using your packages btw, thanks
for publishing them ;)

For what it's worth, you can work around the speed issue by disabling TCP
segmentation offloading (ethtool -K ethX tso off).
(You should disable it in your dom0 on all interfaces and in the domU as
well.)

Unfortunately there is no apparent way to disable tso when kickstarting a
domU, but other than that the speeds are back to normal.

Regards,
-- Dennis Krul <dweazle@gmail.com>


On Fri, May 29, 2009 at 12:02 AM, M A Young <m.a.young@durham.ac.uk> wrote:

> I have started to see this error as well now on a Fedora Rawhide +
> xen-tip/next dom0 kernel on Fedora 11 and a Fedora 11 domU. The base system
> has just been reinstalled and used to be running Fedora 8 dom0, the domU
> machine hasn't changed. This is on a fairly old machine with 512M memory
> shared between the two instances.
>
> /var/log/messages is packed with errors like
> net eth0: rx->offset: 0, size: 4294967295
> with the more occasional
> __ratelimit: 25 callbacks suppressed
> about every 10 lines, while updating some packages using rpm where the
> packages are on an NFS mount, which is maybe running at 1/10th or less the
> speed than I would expect. The network setup on dom0 is
>
> bridge name     bridge id               STP enabled     interfaces
> eth0            8000.0011110a766f       no              peth0
>                                                        vif4.0
>
> with only a handful of dropped packets on vif4.0
> vif4.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>          inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
>          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
>          RX packets:524716 errors:0 dropped:0 overruns:0 frame:0
>          TX packets:658136 errors:0 dropped:39 overruns:0 carrier:0
>          collisions:0 txqueuelen:32
>          RX bytes:36703881 (35.0 MiB)  TX bytes:862875521 (822.9 MiB)
>
> On domU however ifconfig reports
>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>          RX packets:567462 errors:73814 dropped:0 overruns:0 frame:0
>          TX packets:511318 errors:0 dropped:0 overruns:0 carrier:0
>          collisions:0 txqueuelen:1000
>          RX bytes:719940314 (686.5 MiB)  TX bytes:42982009 (40.9 MiB)
>          Interrupt:9
>
> I tried enabling debugging in drivers/xen/netback/netback.c by setting
> #define NETBE_DEBUG_INTERRUPT
> but when I try to build the kernel I get the error
> drivers/xen/blkback/xenbus.c: In function 'blkif_xenbus_init':
> drivers/xen/blkback/xenbus.c:541: warning: ignoring return value of
> 'xenbus_register_backend', declared with attribute warn_unused_result
> drivers/xen/netback/netback.c: In function 'netback_init':
> drivers/xen/netback/netback.c:1503: error: 'SA_SHIRQ' undeclared (first use
> in this function)
> drivers/xen/netback/netback.c:1503: error: (Each undeclared identifier is
> reported only once
> drivers/xen/netback/netback.c:1503: error: for each function it appears
> in.)
> drivers/xen/netback/netback.c:1505: warning: passing argument 3 of
> 'bind_virq_to_irqhandler' from incompatible pointer type
>
>        Michael Young
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 4016 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Network drop on domU (netfront: rx->offset: 0, size: 4294967295)

[M A Young]

I have started to see this error as well now on a Fedora Rawhide + 
xen-tip/next dom0 kernel on Fedora 11 and a Fedora 11 domU. The base 
system has just been reinstalled and used to be running Fedora 8 dom0, the 
domU machine hasn't changed. This is on a fairly old machine with 512M 
memory shared between the two instances.

/var/log/messages is packed with errors like
net eth0: rx->offset: 0, size: 4294967295
with the more occasional
__ratelimit: 25 callbacks suppressed
about every 10 lines, while updating some packages using rpm where the 
packages are on an NFS mount, which is maybe running at 1/10th or less 
the speed than I would expect. The network setup on dom0 is

bridge name	bridge id		STP enabled	interfaces
eth0		8000.0011110a766f	no		peth0
 							vif4.0

with only a handful of dropped packets on vif4.0
vif4.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
           RX packets:524716 errors:0 dropped:0 overruns:0 frame:0
           TX packets:658136 errors:0 dropped:39 overruns:0 carrier:0
           collisions:0 txqueuelen:32
           RX bytes:36703881 (35.0 MiB)  TX bytes:862875521 (822.9 MiB)

On domU however ifconfig reports
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:567462 errors:73814 dropped:0 overruns:0 frame:0
           TX packets:511318 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:719940314 (686.5 MiB)  TX bytes:42982009 (40.9 MiB)
           Interrupt:9

I tried enabling debugging in drivers/xen/netback/netback.c by setting
#define NETBE_DEBUG_INTERRUPT
but when I try to build the kernel I get the error
drivers/xen/blkback/xenbus.c: In function 'blkif_xenbus_init':
drivers/xen/blkback/xenbus.c:541: warning: ignoring return value of 
'xenbus_register_backend', declared with attribute warn_unused_result
drivers/xen/netback/netback.c: In function 'netback_init':
drivers/xen/netback/netback.c:1503: error: 'SA_SHIRQ' undeclared (first 
use in this function)
drivers/xen/netback/netback.c:1503: error: (Each undeclared identifier is 
reported only once
drivers/xen/netback/netback.c:1503: error: for each function it appears 
in.)
drivers/xen/netback/netback.c:1505: warning: passing argument 3 of 
'bind_virq_to_irqhandler' from incompatible pointer type

 	Michael Young