* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
[not found] <8511913.uMAmUdIO30@eistomin.edss.local>
@ 2013-05-17 8:59 ` Wei Liu
2013-05-17 9:08 ` Jan Beulich
` (2 more replies)
0 siblings, 3 replies; 85+ messages in thread
From: Wei Liu @ 2013-05-17 8:59 UTC (permalink / raw)
To: Eugene Istomin; +Cc: wei.liu2, xen-devel
Moving discussion to Xen-devel
On Thu, May 16, 2013 at 10:29:56PM +0300, Eugene Istomin wrote:
> Hello,
>
> I tried to use 3.9.2 kernel with xen 4.2.2/4.3rc1 and in both variants leads
> to this error in network-intensive load (such as iperf, 100 nginx parallel
> requests to 1M files and so on):
>
It would be more helpful if you can provide info on your configurations
(Dom0 and DomU), your workload, how to reproduce the bug.
I run iperf and NFS to test Xen network, but never see any crash like
this myself.
> BUG: unable to handle kernel paging request at ffff8800795a3000
> [ 60.246945] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
> [ 60.246975] PGD a8a067 PUD a9a067 PMD 7fc27067 PTE
> 80100000795a3065
> [ 60.247004] Oops: 0003 [#1] SMP
> [ 60.247020] Modules linked in: af_packet hwmon domctl crc32_pclmul
> crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
> aes_x86_64 joydev xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua
> scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
> ata_piix
> [ 60.247144] CPU 0
> [ 60.247154] Pid: 0, comm: swapper/0 Not tainted 3.9.2-1.g04040b9-xen
> #1
> [ 60.247179] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
> netif_poll+0x49c/0xe80 [xennet]
> ...
Could you provide fuul stack trace? AFAICT there is no netif_poll in Xen
netfront/back.
Presumably this is Dom0 log? (from the domctl module)
>
> We have couple of production hypervisors on 3.4 kernels with high-
> throughput internal network (VM-to-VM in one Dom0), iperf on them is
> working well:
> [ 3] 0.0- 2.0 sec 3357 MBytes 14080 Mbits/sec
> [ 3] 2.0- 4.0 sec 2880 MBytes 12077 Mbits/sec
> [ 3] 4.0- 6.0 sec 2909 MBytes 12202 Mbits/sec
> [ 3] 6.0- 8.0 sec 2552 MBytes 10702 Mbits/sec
> [ 3] 8.0-10.0 sec 3616 MBytes 15166 Mbits/sec
> [ 3] 10.0-12.0 sec 3415 MBytes 14324 Mbits/sec
>
>
> Seems like a kernel bug, is this related to one of this fixes in linux-next or i
> need to create new bugreport?
>
> 1) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=1aaf6d3d3d1e95f4be07e32dd84aa1c93855fbbd
> 2) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=9ecd1a75d977e2e8c48139c7d3efed183f898d94
> 3) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=2810e5b9a7731ca5fce22bfbe12c96e16ac44b6f
> 4) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=03393fd5cc2b6cdeec32b704ecba64dbb0feae3c
> 5) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=59ccb4ebbc35e36a3c143f2d1355deb75c2e628f
>
I don't think these patches can fix your problem at first glance.
Wei.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 8:59 ` [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Wei Liu
@ 2013-05-17 9:08 ` Jan Beulich
2013-05-17 9:37 ` Eugene Istomin
2013-07-04 13:43 ` Dion Kant
2 siblings, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-05-17 9:08 UTC (permalink / raw)
To: wei.liu2; +Cc: Eugene Istomin, xen-devel
>>> On 17.05.13 at 10:59, Wei Liu <wei.liu2@citrix.com> wrote:
>> BUG: unable to handle kernel paging request at ffff8800795a3000
>> [ 60.246945] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
>> [ 60.246975] PGD a8a067 PUD a9a067 PMD 7fc27067 PTE
>> 80100000795a3065
>> [ 60.247004] Oops: 0003 [#1] SMP
>> [ 60.247020] Modules linked in: af_packet hwmon domctl crc32_pclmul
>> crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
>> aes_x86_64 joydev xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua
>> scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
>> ata_piix
>> [ 60.247144] CPU 0
>> [ 60.247154] Pid: 0, comm: swapper/0 Not tainted 3.9.2-1.g04040b9-xen
>> #1
>> [ 60.247179] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
>> netif_poll+0x49c/0xe80 [xennet]
>> ...
>
> Could you provide fuul stack trace? AFAICT there is no netif_poll in Xen
> netfront/back.
He failed to mention that this is an openSUSE kernel, i.e. not
pv-ops but forward ported.
> Presumably this is Dom0 log? (from the domctl module)
Indeed.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 8:59 ` [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Wei Liu
2013-05-17 9:08 ` Jan Beulich
@ 2013-05-17 9:37 ` Eugene Istomin
2013-05-17 9:40 ` Eugene Istomin
2013-05-17 10:45 ` Jan Beulich
2013-07-04 13:43 ` Dion Kant
2 siblings, 2 replies; 85+ messages in thread
From: Eugene Istomin @ 2013-05-17 9:37 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 5491 bytes --]
Ye, this is OpenSUSE 12.3 kernel (both Dom0/DomU is kernel
xen-3.9.2-8.1.g04040b9.x86_64) from
http://download.opensuse.org/repositories/Kernel:/HEAD/standard
First VM:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.202 port 5001 connected with 10.251.2.201 port
38196
##
#(after iperf -c 10.251.2.202 -i 2 -f m from the second VM)
##
[ 38.447860] BUG: unable to handle kernel paging request at
ffff88007928b000
[ 38.447898] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 38.447927] PGD a83067 PUD a93067 PMD 7fc28067 PTE
801000007928b065
[ 38.447955] Oops: 0003 [#1] SMP
[ 38.447970] Modules linked in: af_packet hwmon domctl crc32_pclmul
crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul joydev autofs4 scsi_dh_emc scsi_dh_alua
scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
ata_piix
[ 38.448091] CPU 0
[ 38.448100] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 38.448125] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 38.448158] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 38.448176] RAX: ffff88007da68cd0 RBX: ffff88007928aec0 RCX:
ffff88007928b000
This trace is viewed only using xl console, DomUs had no records in logs.
You are right, may be this is Dom0 trace.
Here is xl-test1.log from Dom0:
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1944130
wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl.c:998:domain_death_xswatch_callback:
[evg=0x19435e0:2] from domid=2 nentries=1 rc=1
libxl: debug: libxl.c:1009:domain_death_xswatch_callback:
[evg=0x19435e0:2] got=domaininfos[0] got->domain=2
libxl: debug: libxl.c:1036:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=30004
libxl: debug: libxl.c:1048:domain_death_xswatch_callback: shutdown
reporting
libxl: debug: libxl.c:1002:domain_death_xswatch_callback: [evg=0] all
reported
libxl: debug: libxl.c:1066:domain_death_xswatch_callback: domain death
search done
Domain 2 has shut down, reason code 3 0x3
Action for shutdown reason code 3 is restart
Domain 2 needs to be cleaned up: destroying the domain
libxl: debug: libxl.c:1250:libxl_domain_destroy: ao 0x19438d0: create:
how=(nil) callback=(nil) poller=0x19435a0
libxl: debug: libxl_dm.c:1266:libxl__destroy_device_model: Device Model
signaled
--
*/Best regards,/*
/Eugene Istomin/
> Moving discussion to Xen-devel
>
> On Thu, May 16, 2013 at 10:29:56PM +0300, Eugene Istomin wrote:
> > Hello,
> >
> > I tried to use 3.9.2 kernel with xen 4.2.2/4.3rc1 and in both variants
> > leads to this error in network-intensive load (such as iperf, 100 nginx
> > parallel
> > requests to 1M files and so on):
> It would be more helpful if you can provide info on your configurations
> (Dom0 and DomU), your workload, how to reproduce the bug.
>
> I run iperf and NFS to test Xen network, but never see any crash like
> this myself.
>
> > BUG: unable to handle kernel paging request at ffff8800795a3000
> > [ 60.246945] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
> > [ 60.246975] PGD a8a067 PUD a9a067 PMD 7fc27067 PTE
> > 80100000795a3065
> > [ 60.247004] Oops: 0003 [#1] SMP
> > [ 60.247020] Modules linked in: af_packet hwmon domctl
crc32_pclmul
> > crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
> > aes_x86_64 joydev xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua
> > scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
> > ata_piix
> > [ 60.247144] CPU 0
> > [ 60.247154] Pid: 0, comm: swapper/0 Not tainted 3.9.2-1.g04040b9-
xen
> > #1
> > [ 60.247179] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
> > netif_poll+0x49c/0xe80 [xennet]
> > ...
>
> Could you provide fuul stack trace? AFAICT there is no netif_poll in Xen
> netfront/back.
>
> Presumably this is Dom0 log? (from the domctl module)
>
> > We have couple of production hypervisors on 3.4 kernels with high-
> > throughput internal network (VM-to-VM in one Dom0), iperf on them is
> > working well:
> > [ 3] 0.0- 2.0 sec 3357 MBytes 14080 Mbits/sec
> > [ 3] 2.0- 4.0 sec 2880 MBytes 12077 Mbits/sec
> > [ 3] 4.0- 6.0 sec 2909 MBytes 12202 Mbits/sec
> > [ 3] 6.0- 8.0 sec 2552 MBytes 10702 Mbits/sec
> > [ 3] 8.0-10.0 sec 3616 MBytes 15166 Mbits/sec
> > [ 3] 10.0-12.0 sec 3415 MBytes 14324 Mbits/sec
> >
> >
> > Seems like a kernel bug, is this related to one of this fixes in
> > linux-next or i need to create new bugreport?
> >
> > 1)
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?
> > id=1aaf6d3d3d1e95f4be07e32dd84aa1c93855fbbd 2)
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?
> > id=9ecd1a75d977e2e8c48139c7d3efed183f898d94 3)
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?
> > id=2810e5b9a7731ca5fce22bfbe12c96e16ac44b6f 4)
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?
> > id=03393fd5cc2b6cdeec32b704ecba64dbb0feae3c 5)
> > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?
> > id=59ccb4ebbc35e36a3c143f2d1355deb75c2e628f
> I don't think these patches can fix your problem at first glance.
>
>
[-- Attachment #1.2: Type: text/html, Size: 25466 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 9:37 ` Eugene Istomin
@ 2013-05-17 9:40 ` Eugene Istomin
2013-05-17 10:45 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Eugene Istomin @ 2013-05-17 9:40 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 5112 bytes --]
>>Ye, this is OpenSUSE 12.3 kernel (both Dom0/DomU is
>>kernel-xen-3.9.2-8.1.g04040b9.x86_64) from
>>>>http://download.opensuse.org/repositories/Kernel:/HEAD/standard
Sorry, misspelled,
I tested kernel-xen-3.9.2-8.1.g04040b9.x86_64 and kernel-
xen-3.9.2-4.756ee56.x86_64 kernels - results are the same.
--
*/Best regards,/*
/Eugene Istomin/
Ye, this is OpenSUSE 12.3 kernel (both Dom0/DomU is kernel
xen-3.9.2-8.1.g04040b9.x86_64) from
http://download.opensuse.org/repositories/Kernel:/HEAD/standard
First VM:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.202 port 5001 connected with 10.251.2.201 port
38196
##
#(after iperf -c 10.251.2.202 -i 2 -f m from the second VM)
##
[ 38.447860] BUG: unable to handle kernel paging request at
ffff88007928b000
[ 38.447898] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 38.447927] PGD a83067 PUD a93067 PMD 7fc28067 PTE
801000007928b065
[ 38.447955] Oops: 0003 [#1] SMP
[ 38.447970] Modules linked in: af_packet hwmon domctl crc32_pclmul
crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul joydev autofs4 scsi_dh_emc scsi_dh_alua
scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
ata_piix
[ 38.448091] CPU 0
[ 38.448100] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 38.448125] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 38.448158] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 38.448176] RAX: ffff88007da68cd0 RBX: ffff88007928aec0 RCX:
ffff88007928b000
This trace is viewed only using xl console, DomUs had no records in logs.
You are right, may be this is Dom0 trace.
Here is xl-test1.log from Dom0:
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1944130
wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl.c:998:domain_death_xswatch_callback:
[evg=0x19435e0:2] from domid=2 nentries=1 rc=1
libxl: debug: libxl.c:1009:domain_death_xswatch_callback:
[evg=0x19435e0:2] got=domaininfos[0] got->domain=2
libxl: debug: libxl.c:1036:domain_death_xswatch_callback: exists
shutdown_reported=0 dominf.flags=30004
libxl: debug: libxl.c:1048:domain_death_xswatch_callback: shutdown
reporting
libxl: debug: libxl.c:1002:domain_death_xswatch_callback: [evg=0] all
reported
libxl: debug: libxl.c:1066:domain_death_xswatch_callback: domain death
search done
Domain 2 has shut down, reason code 3 0x3
Action for shutdown reason code 3 is restart
Domain 2 needs to be cleaned up: destroying the domain
libxl: debug: libxl.c:1250:libxl_domain_destroy: ao 0x19438d0: create:
how=(nil) callback=(nil) poller=0x19435a0
libxl: debug: libxl_dm.c:1266:libxl__destroy_device_model: Device Model
signaled
--
*/Best regards,/*
/Eugene Istomin/
> Moving discussion to Xen-devel
>
> On Thu, May 16, 2013 at 10:29:56PM +0300, Eugene Istomin wrote:
> > Hello,
> >
> > I tried to use 3.9.2 kernel with xen 4.2.2/4.3rc1 and in both variants
> > leads to this error in network-intensive load (such as iperf, 100 nginx
> > parallel
> > requests to 1M files and so on):
> It would be more helpful if you can provide info on your configurations
> (Dom0 and DomU), your workload, how to reproduce the bug.
>
> I run iperf and NFS to test Xen network, but never see any crash like
> this myself.
>
> > BUG: unable to handle kernel paging request at ffff8800795a3000
> > [ 60.246945] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
> > [ 60.246975] PGD a8a067 PUD a9a067 PMD 7fc27067 PTE
> > 80100000795a3065
> > [ 60.247004] Oops: 0003 [#1] SMP
> > [ 60.247020] Modules linked in: af_packet hwmon domctl
crc32_pclmul
> > crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
> > aes_x86_64 joydev xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua
> > scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
> > ata_piix
> > [ 60.247144] CPU 0
> > [ 60.247154] Pid: 0, comm: swapper/0 Not tainted 3.9.2-1.g04040b9-
xen
> > #1
> > [ 60.247179] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
> > netif_poll+0x49c/0xe80 [xennet]
> > ...
>
> Could you provide fuul stack trace? AFAICT there is no netif_poll in Xen
> netfront/back.
>
> Presumably this is Dom0 log? (from the domctl module)
>
> > We have couple of production hypervisors on 3.4 kernels with high-
> > throughput internal network (VM-to-VM in one Dom0), iperf on them is
> > working well:
> > [ 3] 0.0- 2.0 sec 3357 MBytes 14080 Mbits/sec
> > [ 3] 2.0- 4.0 sec 2880 MBytes 12077 Mbits/sec
> > [ 3] 4.0- 6.0 sec 2909 MBytes 12202 Mbits/sec
> > [ 3] 6.0- 8.0 sec 2552 MBytes 10702 Mbits/sec
> > [ 3] 8.0-10.0 sec 3616 MBytes 15166 Mbits/sec
> > [ 3] 10.0-12.0 sec 3415 MBytes 14324 Mbits/sec
> >
> >
> > Seems like a kernel bug, is this related to one of this fixes in
> > linux-next or i need to create new bugreport?
[-- Attachment #1.2: Type: text/html, Size: 28904 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 9:37 ` Eugene Istomin
2013-05-17 9:40 ` Eugene Istomin
@ 2013-05-17 10:45 ` Jan Beulich
2013-05-17 12:03 ` Eugene Istomin
1 sibling, 1 reply; 85+ messages in thread
From: Jan Beulich @ 2013-05-17 10:45 UTC (permalink / raw)
To: Eugene Istomin; +Cc: Wei Liu, xen-devel
>>> On 17.05.13 at 11:37, Eugene Istomin <e.istomin@edss.ee> wrote:
> [ 38.447860] BUG: unable to handle kernel paging request at
> ffff88007928b000
> [ 38.447898] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
> [ 38.447927] PGD a83067 PUD a93067 PMD 7fc28067 PTE
> 801000007928b065
> [ 38.447955] Oops: 0003 [#1] SMP
> [ 38.447970] Modules linked in: af_packet hwmon domctl crc32_pclmul
> crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
> aes_x86_64 xts gf128mul joydev autofs4 scsi_dh_emc scsi_dh_alua
> scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
> ata_piix
> [ 38.448091] CPU 0
> [ 38.448100] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
> #1
> [ 38.448125] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
> netif_poll+0x49c/0xe80 [xennet]
> [ 38.448158] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
> [ 38.448176] RAX: ffff88007da68cd0 RBX: ffff88007928aec0 RCX:
> ffff88007928b000
>
>
>
>
> This trace is viewed only using xl console, DomUs had no records in logs.
> You are right, may be this is Dom0 trace.
Output of "xl console" can hardly be Dom0 kernel output. The fact
that various modules (most prominently domctl) are present here that
would normally only be expected in Dom0 may be confusing us. Init
scripts might be loading those for no reason...
But - you still didn't give us a full guest kernel log, so we're still left in
the dark as to what may have gone wrong earlier, and what the call
stack is that got us here. Nor did you tell us what kernel version you
used last without seeing that issue.
Plus, should in depth analysis of the stack turn out necessary, using
a random build will make this more complicated than necessary (as
those builds vanish as newer ones appear).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 10:45 ` Jan Beulich
@ 2013-05-17 12:03 ` Eugene Istomin
2013-05-17 12:25 ` Jan Beulich
0 siblings, 1 reply; 85+ messages in thread
From: Eugene Istomin @ 2013-05-17 12:03 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 5377 bytes --]
All of kernels i mention before are official opensuse kernels (3.4 from
http://download.opensuse.org/repositories/Kernel:/openSUSE-12.2/standar
d
, 3.9 from
http://download.opensuse.org/repositories/Kernel:/stable/standard )
Also i try kernels from
http://download.opensuse.org/repositories/Kernel:/HEAD/standard for
double-checking the fact that is problem is kernel-related/opensuse-xen
build related.
We use 3.4 kernels in production, 3.4.37-1-xen and 3.4.45-2 are working
without this problems, a couple of days ago i started to test 3.9.
*VM configuration:*
name="test2"
memory = 2048
vcpus = 4
cpu_weight = 256
maxvcpus = 8
maxmem = 4096
superpages = 1
bootloader = "pygrub"
bootargs = "--entry=xvda1:/boot/vmlinuz-xen,/boot/initrd-xen"
vfb = [ 'type=vnc,vnclisten=0.0.0.0,vncdisplay=2,vncpasswd=vnc' ]
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
disk=[
"tap:raw:/storage/images/manual/test2.raw,xvda,w",
"tap:raw:/storage/images/manual/test2_storage.raw,xvdb,w",
]
vif =[
"script=/etc/xen/scripts/vif-
ovs,mac=AA:00:AA:BB:02:01,bridge=vlannet.2512.9000",
]
We use simple vif-to-OVS bash mapper and it work fine.
BTW, only VM runs iperf server (-s) is affected, VM runs iperf client is
working normally.
Logs (test1 - iperf server, test2 - iperf client):
Dom0:
1) tailf /var/log/xen/xl-test1.log (turned just before bug) - xl-test1.log
2) tailf /var/log/xen/xl-test2.log (turned just before bug) - xl-test2.log
3) tailf /var/log/messages (turned just before bug) - messages.log
4) xl create log - xl_creation.log
DomU:
1) dmesg - domu_dmesg
Do you need any kernel dumps?
--
*/Best regards,/*
/Eugene Istomin/
> >>> On 17.05.13 at 11:37, Eugene Istomin <e.istomin@edss.ee>
wrote:
> > [ 38.447860] BUG: unable to handle kernel paging request at
> > ffff88007928b000
> > [ 38.447898] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
> > [ 38.447927] PGD a83067 PUD a93067 PMD 7fc28067 PTE
> > 801000007928b065
> > [ 38.447955] Oops: 0003 [#1] SMP
> > [ 38.447970] Modules linked in: af_packet hwmon domctl
crc32_pclmul
> > crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
> > aes_x86_64 xts gf128mul joydev autofs4 scsi_dh_emc scsi_dh_alua
> > scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
> > ata_piix
> > [ 38.448091] CPU 0
> > [ 38.448100] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-
xen
> > #1
> > [ 38.448125] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
> > netif_poll+0x49c/0xe80 [xennet]
> > [ 38.448158] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
> > [ 38.448176] RAX: ffff88007da68cd0 RBX: ffff88007928aec0 RCX:
> > ffff88007928b000
> >
> >
> >
> >
> > This trace is viewed only using xl console, DomUs had no records in
logs.
> > You are right, may be this is Dom0 trace.
>
> Output of "xl console" can hardly be Dom0 kernel output. The fact
> that various modules (most prominently domctl) are present here that
> would normally only be expected in Dom0 may be confusing us. Init
> scripts might be loading those for no reason...
>
> But - you still didn't give us a full guest kernel log, so we're still left
> in the dark as to what may have gone wrong earlier, and what the call
stack
> is that got us here. Nor did you tell us what kernel version you used last
> without seeing that issue.
>
> Plus, should in depth analysis of the stack turn out necessary, using
> a random build will make this more complicated than necessary (as
> those builds vanish as newer ones appear).
>
> Jan
[-- Attachment #1.2: Type: text/html, Size: 22837 bytes --]
[-- Attachment #2: domu_dmesg.log --]
[-- Type: text/x-log, Size: 15474 bytes --]
[ 0.000000] BRK [0x00a83000, 0x00a83fff] PUD
[ 0.000000] BRK [0x00a84000, 0x00a84fff] PMD
[ 0.000000] BRK [0x00a85000, 0x00a92fff] PTE
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.9.2-4.756ee56-xen (geeko@buildhost) (gcc version 4.7.3 (SUSE Linux) ) #1 SMP Thu May 16 10:08:22 UTC 2013 (756ee56)
[ 0.000000] Command line: root=/dev/disk/by-label/system resume=/dev/disk/by-label/swap splash=silent quiet showopts ipv6.disable=1 elevator=noop vga=0x314
[ 0.000000] e820: Xen-provided physical RAM map:
[ 0.000000] Xen: [mem 0x0000000000000000-0x00000000ffffffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] e820: last_pfn = 0x100000 max_arch_pfn = 0x80000000
[ 0.000000] init_memory_mapping: [mem 0x7fe00000-0x7fffffff]
[ 0.000000] [mem 0x7fe00000-0x7fffffff] page 4k
[ 0.000000] BRK [0x00a93000, 0x00a93fff] PGTABLE
[ 0.000000] BRK [0x00a94000, 0x00a94fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0x7c000000-0x7fdfffff]
[ 0.000000] [mem 0x7c000000-0x7fdfffff] page 4k
[ 0.000000] BRK [0x00a95000, 0x00a95fff] PGTABLE
[ 0.000000] BRK [0x00a96000, 0x00a96fff] PGTABLE
[ 0.000000] BRK [0x00a97000, 0x00a97fff] PGTABLE
[ 0.000000] init_memory_mapping: [mem 0x00000000-0x7bffffff]
[ 0.000000] [mem 0x00000000-0x7bffffff] page 4k
[ 0.000000] init_memory_mapping: [mem 0x80000000-0xffffffff]
[ 0.000000] [mem 0x80000000-0xffffffff] page 4k
[ 0.000000] RAMDISK: [mem 0x00baa000-0x0168efff]
[ 0.000000] ACPI in unprivileged domain disabled
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x00000000-0x00ffffff]
[ 0.000000] DMA32 [mem 0x01000000-0xffffffff]
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x00000000-0xffffffff]
[ 0.000000] On node 0 totalpages: 1048576
[ 0.000000] free_area_init_node: node 0, pgdat ffffffff8091d980, node_mem_map ffff88007c010000
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 4096 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 14280 pages used for memmap
[ 0.000000] DMA32 zone: 1044480 pages, LIFO batch:31
[ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:8 nr_node_ids:1
[ 0.000000] PERCPU: Embedded 19 pages/cpu @ffff88007b400000 s47680 r8192 d21952 u262144
[ 0.000000] pcpu-alloc: s47680 r8192 d21952 u262144 alloc=1*2097152
[ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7
[ 0.000000] Swapping MFNs for PFN 93f and 7b407 (MFN 105f841 and 10e4e79)
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1034240
[ 0.000000] Kernel command line: root=/dev/disk/by-label/system resume=/dev/disk/by-label/swap splash=silent quiet showopts ipv6.disable=1 elevator=noop vga=0x314
[ 0.000000] bootsplash: silent mode.
[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[ 0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[ 0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[ 0.000000] __ex_table already sorted, skipping sort
[ 0.000000] allocated 16777216 bytes of page_cgroup
[ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[ 0.000000] Software IO TLB disabled
[ 0.000000] Memory: 1978108k/4194304k available (5336k kernel code, 0k absent, 2216196k reserved, 4091k data, 440k init)
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] RCU dyntick-idle grace-period acceleration is enabled.
[ 0.000000] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=8.
[ 0.000000] nr_pirqs: 256
[ 0.000000] NR_IRQS:67328 nr_irqs:2624 16
[ 0.000000] Xen reported: 2666.846 MHz processor.
[ 0.000000] Console: colour dummy device 80x25
[ 0.000000] console [tty0] enabled
[ 0.000000] console [xvc-1] enabled
[ 0.080001] Calibrating delay using timer specific routine.. 5403.97 BogoMIPS (lpj=10807950)
[ 0.080003] pid_max: default: 32768 minimum: 301
[ 0.080026] Security Framework initialized
[ 0.080036] AppArmor: AppArmor initialized
[ 0.080044] Mount-cache hash table entries: 256
[ 0.080165] Initializing cgroup subsys cpuacct
[ 0.080167] Initializing cgroup subsys memory
[ 0.080175] Initializing cgroup subsys devices
[ 0.080176] Initializing cgroup subsys freezer
[ 0.080178] Initializing cgroup subsys net_cls
[ 0.080179] Initializing cgroup subsys blkio
[ 0.080180] Initializing cgroup subsys perf_event
[ 0.080218] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[ 0.080218] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[ 0.080222] Last level iTLB entries: 4KB 512, 2MB 7, 4MB 7
[ 0.080222] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[ 0.080222] tlb_flushall_shift: 6
[ 0.110684] Brought up 1 CPUs
[ 0.110786] devtmpfs: initialized
[ 0.130801] RTC time: 165:165:165, date: 165/165/65
[ 0.130838] NET: Registered protocol family 16
[ 0.132005] SMP alternatives: switching to SMP code
[ 0.164163] Brought up 4 CPUs
[ 0.164376] PCI: Fatal: No config space access function found
[ 0.164381] PCI: setting up Xen PCI frontend stub
[ 0.164487] patch_vdso64: .altinstructions not found
[ 0.164489] patch_vdsox32: .altinstructions not found
[ 0.164816] bio: create slab <bio-0> at 0
[ 0.164816] ACPI: Interpreter disabled.
[ 0.164816] vgaarb: loaded
[ 0.164816] suspend: event channel 15
[ 0.164816] xen_mem: Initialising balloon driver.
[ 0.172927] Unable to read sysrq code in control/sysrq
[ 0.172927] SCSI subsystem initialized
[ 0.172927] libata version 3.00 loaded.
[ 0.172927] PCI: System does not support PCI
[ 0.172927] PCI: System does not support PCI
[ 0.172927] NetLabel: Initializing
[ 0.172927] NetLabel: domain hash size = 128
[ 0.172927] NetLabel: protocols = UNLABELED CIPSOv4
[ 0.172927] NetLabel: unlabeled traffic allowed by default
[ 0.172927] Switching to clocksource xen
[ 0.173707] AppArmor: AppArmor Filesystem Enabled
[ 0.173719] pnp: PnP ACPI: disabled
[ 0.174653] NET: Registered protocol family 2
[ 0.174798] TCP established hash table entries: 32768 (order: 7, 524288 bytes)
[ 0.174868] TCP bind hash table entries: 32768 (order: 7, 524288 bytes)
[ 0.174934] TCP: Hash tables configured (established 32768 bind 32768)
[ 0.174955] TCP: reno registered
[ 0.174958] UDP hash table entries: 2048 (order: 5, 131072 bytes)
[ 0.174978] UDP-Lite hash table entries: 2048 (order: 5, 131072 bytes)
[ 0.175053] NET: Registered protocol family 1
[ 0.175059] PCI: CLS 32 bytes
[ 0.175086] Unpacking initramfs...
[ 0.182596] Freeing initrd memory: 11156k freed
[ 0.186579] audit: initializing netlink socket (disabled)
[ 0.186590] type=2000 audit(1368791454.184:1): initialized
[ 0.216723] VFS: Disk quotas dquot_6.5.2
[ 0.216748] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 0.216931] msgmni has been set to 4117
[ 0.217165] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[ 0.217197] io scheduler noop registered (default)
[ 0.217199] io scheduler deadline registered
[ 0.217216] io scheduler cfq registered
[ 0.217366] Non-volatile memory driver v1.3
[ 0.217367] Linux agpgart interface v0.103
[ 0.217440] Xen virtual console successfully installed as xvc0
[ 0.219206] Console: switching to colour frame buffer device 100x37
[ 0.222889] console [tty0] enabled
[ 0.223852] input: Xen Virtual Keyboard as /devices/virtual/input/input0
[ 0.223875] input: Xen Virtual Pointer as /devices/virtual/input/input1
[ 0.226130] i8042: PNP: No PS/2 controller found. Probing ports directly.
[ 1.239115] i8042: No controller found
[ 1.239292] mousedev: PS/2 mouse device common for all mice
[ 1.239410] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.239515] TCP: cubic registered
[ 1.239516] IPv6: Loaded, but administratively disabled, reboot required to enable
[ 1.239526] Key type dns_resolver registered
[ 1.239629] registered taskstats version 1
[ 1.239699] Magic number: 1:252:3141
[ 1.239715] XENBUS: Device with no driver: device/vbd/51712
[ 1.239716] XENBUS: Device with no driver: device/vbd/51728
[ 1.239717] XENBUS: Device with no driver: device/vif/0
[ 1.239934] Freeing unused kernel memory: 440k freed
[ 1.240070] Write protecting the kernel read-only data: 8912k
[ 1.310915] Initialising virtual ethernet driver.
[ 1.414401] blkfront: device/vbd/51712: ring-pages=1 nr_ents=32
[ 1.421937] blkfront: device/vbd/51728: ring-pages=1 nr_ents=32
[ 1.423590] xen-vbd: registered block device major 202
[ 1.423628] blkfront: xvda: barrier or flush: disabled
[ 1.425229] xvda: xvda1 xvda2
[ 1.426265] Setting capacity to 20971520
[ 1.426270] xvda: detected capacity change from 0 to 10737418240
[ 1.427111] blkfront: xvdb: barrier or flush: disabled
[ 1.433071] xvdb: unknown partition table
[ 1.528825] hp_sw: device handler registered
[ 1.531766] rdac: device handler registered
[ 1.534773] alua: device handler registered
[ 1.537735] emc: device handler registered
[ 1.547178] systemd-udevd[111]: starting version 195
[ 1.843895] EXT4-fs (xvda2): mounted filesystem with ordered data mode. Opts: acl,user_xattr
[ 1.991518] EXT4-fs (xvda2): re-mounted. Opts: acl,user_xattr
[ 2.257074] systemd[1]: systemd 195 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ; suse)
[ 2.362136] systemd[1]: Inserted module 'autofs4'
[ 2.370140] systemd[1]: Set hostname to <template.edss.local>.
[ 2.754679] systemd[1]: Expecting device dev-xvc0.device...
[ 2.754708] systemd[1]: Starting Forward Password Requests to Wall Directory Watch.
[ 2.754827] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[ 2.754851] systemd[1]: Starting System Time Synchronized.
[ 2.754871] systemd[1]: Reached target System Time Synchronized.
[ 2.754886] systemd[1]: Starting Remote File Systems.
[ 2.754904] systemd[1]: Reached target Remote File Systems.
[ 2.754919] systemd[1]: Starting Syslog Socket.
[ 2.754981] systemd[1]: Listening on Syslog Socket.
[ 2.755007] systemd[1]: Starting Collect Read-Ahead Data...
[ 2.756219] systemd[1]: Starting Replay Read-Ahead Data...
[ 2.757217] systemd[1]: Starting Delayed Shutdown Socket.
[ 2.757278] systemd[1]: Listening on Delayed Shutdown Socket.
[ 2.757295] systemd[1]: Starting /dev/initctl Compatibility Named Pipe.
[ 2.757349] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[ 2.757383] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch.
[ 2.757472] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[ 2.757586] systemd[1]: Starting udev Kernel Socket.
[ 2.757627] systemd[1]: Listening on udev Kernel Socket.
[ 2.757724] systemd[1]: Starting udev Control Socket.
[ 2.757778] systemd[1]: Listening on udev Control Socket.
[ 2.757836] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
[ 2.757990] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[ 2.758010] systemd[1]: Starting Encrypted Volumes.
[ 2.758033] systemd[1]: Reached target Encrypted Volumes.
[ 2.758060] systemd[1]: Expecting device dev-disk-by\x2dlabel-swap.device...
[ 2.758096] systemd[1]: Starting Journal Socket.
[ 2.758201] systemd[1]: Listening on Journal Socket.
[ 2.758248] systemd[1]: Mounting POSIX Message Queue File System...
[ 2.759183] systemd[1]: Mounted Huge Pages File System.
[ 2.759220] systemd[1]: Starting Create dynamic rule for /dev/root link...
[ 2.760345] systemd[1]: Starting Journal Service...
[ 2.761603] systemd[1]: Started Journal Service.
[ 2.761805] systemd-readahead[208]: Bumped block_nr parameter of 202:0 to 20480. This is a temporary hack and should be removed one day.
[ 2.830611] EXT4-fs (xvda2): re-mounted. Opts: acl,user_xattr
[ 2.884995] systemd-udevd[232]: starting version 195
[ 2.913594] systemd-journald[211]: Received SIGUSR1
[ 2.919885] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 2.919887] EDD information not available.
[ 3.005770] type=1400 audit(1368791457.000:2): apparmor="STATUS" operation="profile_load" name="/{usr/,}bin/ping" pid=273 comm="apparmor_parser"
[ 3.021044] type=1400 audit(1368791457.016:3): apparmor="STATUS" operation="profile_load" name="/sbin/klogd" pid=299 comm="apparmor_parser"
[ 3.031577] type=1400 audit(1368791457.024:4): apparmor="STATUS" operation="profile_load" name="/sbin/syslog-ng" pid=329 comm="apparmor_parser"
[ 3.043335] type=1400 audit(1368791457.036:5): apparmor="STATUS" operation="profile_load" name="/sbin/syslogd" pid=339 comm="apparmor_parser"
[ 3.056589] type=1400 audit(1368791457.052:6): apparmor="STATUS" operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2" pid=356 comm="apparmor_parser"
[ 3.056854] type=1400 audit(1368791457.052:7): apparmor="STATUS" operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//DEFAULT_URI" pid=356 comm="apparmor_parser"
[ 3.057092] type=1400 audit(1368791457.052:8): apparmor="STATUS" operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//HANDLING_UNTRUSTED_INPUT" pid=356 comm="apparmor_parser"
[ 3.057308] type=1400 audit(1368791457.052:9): apparmor="STATUS" operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//phpsysinfo" pid=356 comm="apparmor_parser"
[ 3.065740] type=1400 audit(1368791457.060:10): apparmor="STATUS" operation="profile_load" name="/usr/lib/dovecot/deliver" pid=363 comm="apparmor_parser"
[ 5.963850] Adding 1051644k swap on /dev/xvda1. Priority:-1 extents:1 across:1051644k SS
[ 6.007235] audit_printk_skb: 57 callbacks suppressed
[ 6.007238] type=1400 audit(1368791460.000:30): apparmor="DENIED" operation="mknod" parent=1 profile="/usr/sbin/nscd" name="/var/run/nscd/netgroup" pid=468 comm="nscd" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[ 6.275196] type=1400 audit(1368791460.268:31): apparmor="DENIED" operation="capable" parent=1 profile="/usr/sbin/nscd" pid=468 comm="nscd" pid=468 comm="nscd" capability=36 capname="block_suspend"
[ 6.612452] NET: Registered protocol family 17
[ 9.077256] type=1400 audit(1368791463.072:32): apparmor="DENIED" operation="capable" parent=1 profile="/usr/sbin/nscd" pid=468 comm="nscd" pid=468 comm="nscd" capability=36 capname="block_suspend"
[ 9.113208] type=1400 audit(1368791463.108:33): apparmor="DENIED" operation="capable" parent=1 profile="/usr/sbin/nscd" pid=468 comm="nscd" pid=468 comm="nscd" capability=36 capname="block_suspend"
[ 36.008106] type=1400 audit(1368791490.004:34): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/nscd" name="/proc/sys/vm/overcommit_memory" pid=472 comm="nscd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 69.172553] type=1400 audit(1368791523.168:35): apparmor="DENIED" operation="capable" parent=1 profile="/usr/sbin/nscd" pid=468 comm="nscd" pid=468 comm="nscd" capability=36 capname="block_suspend"
[-- Attachment #3: xl_creation.log --]
[-- Type: text/x-log, Size: 17483 bytes --]
Parsing config from /storage/xl/test1.xl
libxl: debug: libxl_create.c:1192:do_domain_create: ao 0x1f824c0: create: how=(nil) callback=(nil) poller=0x1f839f0
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:175:disk_try_backend: Disk vdev=xvda, backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:265:libxl__device_disk_set_backend: Disk vdev=xvda, using backend tap
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvdb spec.backend=unknown
libxl: debug: libxl_device.c:175:disk_try_backend: Disk vdev=xvdb, backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:265:libxl__device_disk_set_backend: Disk vdev=xvdb, using backend tap
libxl: debug: libxl_create.c:694:initiate_domain_create: running bootloader
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=tap
libxl: debug: libxl.c:2318:libxl__device_disk_local_initiate_attach: locally attaching tap disk /storage/images/manual/test1.raw directly (ie without using blktap)
libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: pygrub
libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_create.c:1205:do_domain_create: ao 0x1f824c0: inprogress: poller=0x1f839f0, flags=i
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f82840 wpath=/local/domain/2 token=3/0: register slotnum=3
libxl: debug: libxl_event.c:1736:libxl__ao_progress_report: ao 0x1f824c0: progress report: ignored
libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.2.out
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.2.d
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /storage/images/manual/test1.raw
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f82840 wpath=/local/domain/2 token=3/0: event epath=/local/domain/2
libxl: debug: libxl_bootloader.c:634:bootloader_finished: bootloader completed
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained kernel /var/run/xen/bootloader.2.d/boot_kernel.JoFGbt
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained ramdisk /var/run/xen/bootloader.2.d/boot_ramdisk.Vg4_jl
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained args root=/dev/disk/by-label/system resume=/dev/disk/by-label/swap splash=silent quiet showopts ipv6.disable=1 elevator=noop vga=0x314
libxl: debug: libxl_bootloader.c:647:bootloader_finished: bootloader execution successful
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f82840 wpath=/local/domain/2 token=3/0: deregister slotnum=3
libxl: debug: libxl_numa.c:435:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=12, nr_vcpus=16, free_memkb=46390
libxl: detail: libxl_dom.c:193:numa_place_domain: NUMA placement candidate with 1 nodes, 12 cpus and 46390 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/disk/by-label/system resume=/dev/disk/by-label/swap splash=silent quiet showopts ipv6.disable=1 elevator=noop vga=0x314", features="(null)"
libxl: debug: libxl_dom.c:383:libxl__build_pv: pv kernel mapped 1 path /var/run/xen/bootloader.2.d/boot_kernel.JoFGbt
domainbuilder: detail: xc_dom_kernel_mem: called
domainbuilder: detail: xc_dom_ramdisk_mem: called
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.2, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_malloc : 11958 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x43c02b -> 0xbada10
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x2000 memsz=0x8b4000
xc: detail: elf_parse_binary: phdr: paddr=0x8b6000 memsz=0x810f0
xc: detail: elf_parse_binary: phdr: paddr=0x938000 memsz=0xba40
xc: detail: elf_parse_binary: phdr: paddr=0x944000 memsz=0x266000
xc: detail: elf_parse_binary: memory: 0x2000 -> 0xbaa000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff80002000
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80003000
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: MOD_START_PFN = 0x1
xc: detail: elf_xen_parse_note: INIT_P2M = 0xffffea0000000000
xc: detail: elf_xen_parse_note: FEATURES = "writable_page_tables|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
xc: detail: elf_xen_parse_note: SUPPORTED_FEATURES = 0x80f
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail: virt_base = 0xffffffff80000000
xc: detail: elf_paddr_offset = 0x0
xc: detail: virt_offset = 0xffffffff80000000
xc: detail: virt_kstart = 0xffffffff80002000
xc: detail: virt_kend = 0xffffffff80baa000
xc: detail: virt_entry = 0xffffffff80002000
xc: detail: p2m_base = 0xffffea0000000000
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff80002000 -> 0xffffffff80baa000
domainbuilder: detail: xc_dom_mem_init: mem 2048 MB, pages 0x80000 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x80000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc : 4096 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment: kernel : 0xffffffff80002000 -> 0xffffffff80baa000 (pfn 0x2 + 0xba8 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x2+0xba8 at 0x7f63ab057000
xc: detail: elf_load_binary: phdr 0 at 0x0x7f63ab057000 -> 0x0x7f63ab90b000
xc: detail: elf_load_binary: phdr 1 at 0x0x7f63ab90b000 -> 0x0x7f63ab98c0f0
xc: detail: elf_load_binary: phdr 2 at 0x0x7f63ab98d000 -> 0x0x7f63ab998a40
xc: detail: elf_load_binary: phdr 3 at 0x0x7f63ab999000 -> 0x0x7f63aba02000
domainbuilder: detail: xc_dom_alloc_segment: ramdisk : 0xffffffff80baa000 -> 0xffffffff8168f000 (pfn 0xbaa + 0xae5 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0xbaa+0xae5 at 0x7f63aa572000
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x4bffdd -> 0xae4010
domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0xffffffff8168f000 -> 0xffffffff81a8f000 (pfn 0x168f + 0x400 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x168f+0x400 at 0x7f63aa172000
domainbuilder: detail: xc_dom_alloc_page : start info : 0xffffffff81a8f000 (pfn 0x1a8f)
domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xffffffff81a90000 (pfn 0x1a90)
domainbuilder: detail: xc_dom_alloc_page : console : 0xffffffff81a91000 (pfn 0x1a91)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 -> 0xffffffff81bfffff, 14 table(s)
domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xffffffff81a92000 -> 0xffffffff81aa3000 (pfn 0x1a92 + 0x11 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x1a92+0x11 at 0x7f63b3b92000
domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xffffffff81aa3000 (pfn 0x1aa3)
domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xffffffff81aa4000
domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0xffffffff81c00000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x80000
domainbuilder: detail: clear_page: pfn 0x1a91, mfn 0x823127
domainbuilder: detail: clear_page: pfn 0x1a90, mfn 0x823128
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x1a8f+0x1 at 0x7f63b3bab000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff80003000 pfn=0x3
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail: allocated
domainbuilder: detail: malloc : 16217 kB
domainbuilder: detail: anon mmap : 0 bytes
domainbuilder: detail: mapped
domainbuilder: detail: file mmap : 0 bytes
domainbuilder: detail: domU mmap : 27260 kB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xbf029
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x1a92 mfn 0x823126
domainbuilder: detail: launch_vm: called, ctxt=0x7fff4896ef90
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f84878 wpath=/local/domain/0/backend/vbd/2/51712/state token=3/1: register slotnum=3
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvdb spec.backend=tap
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvdb spec.backend=tap
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f84d48 wpath=/local/domain/0/backend/vbd/2/51728/state token=2/2: register slotnum=2
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f84878 wpath=/local/domain/0/backend/vbd/2/51712/state token=3/1: event epath=/local/domain/0/backend/vbd/2/51712/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f84878 wpath=/local/domain/0/backend/vbd/2/51712/state token=3/1: deregister slotnum=3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f84878: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/2/51712/state token=3/1: empty slot
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f84d48 wpath=/local/domain/0/backend/vbd/2/51728/state token=2/2: event epath=/local/domain/0/backend/vbd/2/51728/state
libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/51728/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f84d48 wpath=/local/domain/0/backend/vbd/2/51728/state token=2/2: event epath=/local/domain/0/backend/vbd/2/51728/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/51728/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f84d48 wpath=/local/domain/0/backend/vbd/2/51728/state token=2/2: deregister slotnum=2
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f84d48: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_dm.c:1160:libxl__spawn_local_dm: Spawning device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: /usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -xen-domid
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: 2
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -chardev
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-2,server,nowait
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -mon
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -xen-attach
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -name
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: test1
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: 0.0.0.0:1,password,to=99
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -M
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: xenpv
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -m
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: 4097
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f82a78 wpath=/local/domain/0/device-model/2/state token=2/3: register slotnum=2
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f82a78 wpath=/local/domain/0/device-model/2/state token=2/3: event epath=/local/domain/0/device-model/2/state
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f82a78 wpath=/local/domain/0/device-model/2/state token=2/3: event epath=/local/domain/0/device-model/2/state
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f82a78 wpath=/local/domain/0/device-model/2/state token=2/3: deregister slotnum=2
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f82a78: deregister unregistered
libxl: debug: libxl_qmp.c:696:libxl__qmp_initialize: connected to /var/run/xen/qmp-libxl-2
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: qmp
libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{"execute":"qmp_capabilities","id":1}'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{"execute":"query-chardev","id":2}'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{"execute":"change","id":3,"arguments":{"device":"vnc","target":"password","arg":"vnc"}}'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{"execute":"query-vnc","id":4}'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f8a0c8 wpath=/local/domain/0/backend/vif/2/0/state token=2/4: register slotnum=2
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f8a0c8 wpath=/local/domain/0/backend/vif/2/0/state token=2/4: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f8a0c8 wpath=/local/domain/0/backend/vif/2/0/state token=2/4: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f8a0c8 wpath=/local/domain/0/backend/vif/2/0/state token=2/4: deregister slotnum=2
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f8a0c8: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-ovs online
libxl: debug: libxl_event.c:1736:libxl__ao_progress_report: ao 0x1f824c0: progress report: ignored
libxl: debug: libxl_event.c:1568:libxl__ao_complete: ao 0x1f824c0: complete, rc=0
libxl: debug: libxl_event.c:1540:libxl__ao__destroy: ao 0x1f824c0: destroy
Daemon running with PID 4439
xc: debug: hypercall buffer: total allocations:369 total releases:369
xc: debug: hypercall buffer: current allocations:0 maximum allocations:4
xc: debug: hypercall buffer: cache current size:4
xc: debug: hypercall buffer: cache hits:358 misses:4 toobig:7
[-- Attachment #4: xl-test1.log --]
[-- Type: text/x-log, Size: 1219 bytes --]
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xabeff0 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl.c:998:domain_death_xswatch_callback: [evg=0xabe4a0:1] from domid=1 nentries=1 rc=1
libxl: debug: libxl.c:1009:domain_death_xswatch_callback: [evg=0xabe4a0:1] got=domaininfos[0] got->domain=1
libxl: debug: libxl.c:1036:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0010
libxl: debug: libxl.c:1002:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl.c:1066:domain_death_xswatch_callback: domain death search done
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0xabeff0 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl.c:998:domain_death_xswatch_callback: [evg=0xabe4a0:1] from domid=1 nentries=1 rc=1
libxl: debug: libxl.c:1009:domain_death_xswatch_callback: [evg=0xabe4a0:1] got=domaininfos[0] got->domain=1
libxl: debug: libxl.c:1036:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0010
libxl: debug: libxl.c:1002:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl.c:1066:domain_death_xswatch_callback: domain death search done
[-- Attachment #5: xl-test2.log --]
[-- Type: text/x-log, Size: 22012 bytes --]
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f8a0d0 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl.c:998:domain_death_xswatch_callback: [evg=0x1f89580:2] from domid=2 nentries=1 rc=1
libxl: debug: libxl.c:1009:domain_death_xswatch_callback: [evg=0x1f89580:2] got=domaininfos[0] got->domain=2
libxl: debug: libxl.c:1036:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=30004
libxl: debug: libxl.c:1048:domain_death_xswatch_callback: shutdown reporting
libxl: debug: libxl.c:1002:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl.c:1066:domain_death_xswatch_callback: domain death search done
Domain 2 has shut down, reason code 3 0x3
Action for shutdown reason code 3 is restart
Domain 2 needs to be cleaned up: destroying the domain
libxl: debug: libxl.c:1250:libxl_domain_destroy: ao 0x1f83dc0: create: how=(nil) callback=(nil) poller=0x1f89540
libxl: debug: libxl_dm.c:1266:libxl__destroy_device_model: Device Model signaled
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f899d8 wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: register slotnum=2
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f89b78 wpath=/local/domain/0/backend/vbd/2/51728/state token=1/2: register slotnum=1
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f85138 wpath=/local/domain/0/backend/vif/2/0/state token=0/3: register slotnum=0
libxl: debug: libxl.c:1259:libxl_domain_destroy: ao 0x1f83dc0: inprogress: poller=0x1f89540, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f899d8 wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: event epath=/local/domain/0/backend/vbd/2/51712/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/51712/state wanted state 6 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f899d8 wpath=/local/domain/0/backend/vbd/2/51712/state token=2/1: deregister slotnum=2
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f899d8: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove
libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/2/51712/state token=2/1: empty slot
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f89b78 wpath=/local/domain/0/backend/vbd/2/51728/state token=1/2: event epath=/local/domain/0/backend/vbd/2/51728/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/51728/state wanted state 6 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f89b78 wpath=/local/domain/0/backend/vbd/2/51728/state token=1/2: deregister slotnum=1
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f89b78: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove
libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/2/51728/state token=1/2: empty slot
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f85138 wpath=/local/domain/0/backend/vif/2/0/state token=0/3: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:646:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 6 still waiting state 5
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f85138 wpath=/local/domain/0/backend/vif/2/0/state token=0/3: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 6 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f85138 wpath=/local/domain/0/backend/vif/2/0/state token=0/3: deregister slotnum=0
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f85138: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-ovs offline
libxl: debug: libxl_event.c:1568:libxl__ao_complete: ao 0x1f83dc0: complete, rc=0
libxl: debug: libxl_event.c:1540:libxl__ao__destroy: ao 0x1f83dc0: destroy
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f8a0d0 wpath=@releaseDomain token=3/0: deregister slotnum=3
Done. Rebooting now
libxl: debug: libxl_create.c:1192:do_domain_create: ao 0x1f86fa0: create: how=(nil) callback=(nil) poller=0x1f89540
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:175:disk_try_backend: Disk vdev=xvda, backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:265:libxl__device_disk_set_backend: Disk vdev=xvda, using backend tap
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvdb spec.backend=unknown
libxl: debug: libxl_device.c:175:disk_try_backend: Disk vdev=xvdb, backend phy unsuitable as phys path not a block device
libxl: debug: libxl_device.c:265:libxl__device_disk_set_backend: Disk vdev=xvdb, using backend tap
libxl: debug: libxl_create.c:694:initiate_domain_create: running bootloader
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=(null) spec.backend=tap
libxl: debug: libxl.c:2318:libxl__device_disk_local_initiate_attach: locally attaching tap disk /storage/images/manual/test1.raw directly (ie without using blktap)
libxl: debug: libxl_bootloader.c:409:bootloader_disk_attached_cb: Config bootloader value: pygrub
libxl: debug: libxl_bootloader.c:425:bootloader_disk_attached_cb: Checking for bootloader in libexec path: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_create.c:1205:do_domain_create: ao 0x1f86fa0: inprogress: poller=0x1f89540, flags=i
libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=@releaseDomain token=3/0: empty slot
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f85a90 wpath=/local/domain/3 token=3/4: register slotnum=3
libxl: debug: libxl_event.c:1736:libxl__ao_progress_report: ao 0x1f86fa0: progress report: ignored
libxl: debug: libxl_bootloader.c:535:bootloader_gotptys: executing bootloader: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /usr/lib64/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output=/var/run/xen/bootloader.3.out
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-format=simple0
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: --output-directory=/var/run/xen/bootloader.3.d
libxl: debug: libxl_bootloader.c:539:bootloader_gotptys: bootloader arg: /storage/images/manual/test1.raw
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f85a90 wpath=/local/domain/3 token=3/4: event epath=/local/domain/3
libxl: debug: libxl_bootloader.c:634:bootloader_finished: bootloader completed
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained kernel /var/run/xen/bootloader.3.d/boot_kernel.8zO6CU
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained ramdisk /var/run/xen/bootloader.3.d/boot_ramdisk.x8kIrK
libxl: debug: libxl_bootloader.c:136:bootloader_result_command: bootloader output contained args root=/dev/disk/by-label/system resume=/dev/disk/by-label/swap splash=silent quiet showopts ipv6.disable=1 elevator=noop vga=0x314
libxl: debug: libxl_bootloader.c:647:bootloader_finished: bootloader execution successful
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f85a90 wpath=/local/domain/3 token=3/4: deregister slotnum=3
libxl: debug: libxl_numa.c:435:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=12, nr_vcpus=16, free_memkb=46390
libxl: detail: libxl_dom.c:193:numa_place_domain: NUMA placement candidate with 1 nodes, 12 cpus and 46390 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="root=/dev/disk/by-label/system resume=/dev/disk/by-label/swap splash=silent quiet showopts ipv6.disable=1 elevator=noop vga=0x314", features="(null)"
libxl: debug: libxl_dom.c:383:libxl__build_pv: pv kernel mapped 1 path /var/run/xen/bootloader.3.d/boot_kernel.8zO6CU
domainbuilder: detail: xc_dom_kernel_mem: called
domainbuilder: detail: xc_dom_ramdisk_mem: called
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.2, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ...
domainbuilder: detail: xc_dom_malloc : 11958 kB
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x43c02b -> 0xbada10
domainbuilder: detail: loader probe OK
xc: detail: elf_parse_binary: phdr: paddr=0x2000 memsz=0x8b4000
xc: detail: elf_parse_binary: phdr: paddr=0x8b6000 memsz=0x810f0
xc: detail: elf_parse_binary: phdr: paddr=0x938000 memsz=0xba40
xc: detail: elf_parse_binary: phdr: paddr=0x944000 memsz=0x266000
xc: detail: elf_parse_binary: memory: 0x2000 -> 0xbaa000
xc: detail: elf_xen_parse_note: GUEST_OS = "linux"
xc: detail: elf_xen_parse_note: GUEST_VERSION = "2.6"
xc: detail: elf_xen_parse_note: XEN_VERSION = "xen-3.0"
xc: detail: elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
xc: detail: elf_xen_parse_note: PADDR_OFFSET = 0x0
xc: detail: elf_xen_parse_note: ENTRY = 0xffffffff80002000
xc: detail: elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80003000
xc: detail: elf_xen_parse_note: unknown xen elf note (0xd)
xc: detail: elf_xen_parse_note: MOD_START_PFN = 0x1
xc: detail: elf_xen_parse_note: INIT_P2M = 0xffffea0000000000
xc: detail: elf_xen_parse_note: FEATURES = "writable_page_tables|writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel"
xc: detail: elf_xen_parse_note: SUPPORTED_FEATURES = 0x80f
xc: detail: elf_xen_parse_note: LOADER = "generic"
xc: detail: elf_xen_parse_note: SUSPEND_CANCEL = 0x1
xc: detail: elf_xen_addr_calc_check: addresses:
xc: detail: virt_base = 0xffffffff80000000
xc: detail: elf_paddr_offset = 0x0
xc: detail: virt_offset = 0xffffffff80000000
xc: detail: virt_kstart = 0xffffffff80002000
xc: detail: virt_kend = 0xffffffff80baa000
xc: detail: virt_entry = 0xffffffff80002000
xc: detail: p2m_base = 0xffffea0000000000
domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0xffffffff80002000 -> 0xffffffff80baa000
domainbuilder: detail: xc_dom_mem_init: mem 2048 MB, pages 0x80000 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x80000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64
domainbuilder: detail: xc_dom_malloc : 4096 kB
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_alloc_segment: kernel : 0xffffffff80002000 -> 0xffffffff80baa000 (pfn 0x2 + 0xba8 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x2+0xba8 at 0x7f63b03f6000
xc: detail: elf_load_binary: phdr 0 at 0x0x7f63b03f6000 -> 0x0x7f63b0caa000
xc: detail: elf_load_binary: phdr 1 at 0x0x7f63b0caa000 -> 0x0x7f63b0d2b0f0
xc: detail: elf_load_binary: phdr 2 at 0x0x7f63b0d2c000 -> 0x0x7f63b0d37a40
xc: detail: elf_load_binary: phdr 3 at 0x0x7f63b0d38000 -> 0x0x7f63b0da1000
domainbuilder: detail: xc_dom_alloc_segment: ramdisk : 0xffffffff80baa000 -> 0xffffffff8168f000 (pfn 0xbaa + 0xae5 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0xbaa+0xae5 at 0x7f63ab51b000
domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x4bffdd -> 0xae4010
domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0xffffffff8168f000 -> 0xffffffff81a8f000 (pfn 0x168f + 0x400 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x168f+0x400 at 0x7f63ab11b000
domainbuilder: detail: xc_dom_alloc_page : start info : 0xffffffff81a8f000 (pfn 0x1a8f)
domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xffffffff81a90000 (pfn 0x1a90)
domainbuilder: detail: xc_dom_alloc_page : console : 0xffffffff81a91000 (pfn 0x1a91)
domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0xffff000000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0xffffff8000000000 -> 0xffffffffffffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0xffffffff80000000 -> 0xffffffffbfffffff, 1 table(s)
domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0xffffffff80000000 -> 0xffffffff81bfffff, 14 table(s)
domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xffffffff81a92000 -> 0xffffffff81aa3000 (pfn 0x1a92 + 0x11 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x1a92+0x11 at 0x7f63b3b92000
domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xffffffff81aa3000 (pfn 0x1aa3)
domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xffffffff81aa4000
domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0xffffffff81c00000
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: arch_setup_bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x80000
domainbuilder: detail: clear_page: pfn 0x1a91, mfn 0x822526
domainbuilder: detail: clear_page: pfn 0x1a90, mfn 0x822527
domainbuilder: detail: xc_dom_pfn_to_ptr: domU mapping: pfn 0x1a8f+0x1 at 0x7f63b3bab000
domainbuilder: detail: start_info_x86_64: called
domainbuilder: detail: setup_hypercall_page: vaddr=0xffffffff80003000 pfn=0x3
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail: allocated
domainbuilder: detail: malloc : 16217 kB
domainbuilder: detail: anon mmap : 0 bytes
domainbuilder: detail: mapped
domainbuilder: detail: file mmap : 0 bytes
domainbuilder: detail: domU mmap : 27260 kB
domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xbf029
domainbuilder: detail: shared_info_x86_64: called
domainbuilder: detail: vcpu_x86_64: called
domainbuilder: detail: vcpu_x86_64: cr3: pfn 0x1a92 mfn 0x822525
domainbuilder: detail: launch_vm: called, ctxt=0x7fff4896ef90
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=tap
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f89288 wpath=/local/domain/0/backend/vbd/3/51712/state token=3/5: register slotnum=3
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvdb spec.backend=tap
libxl: debug: libxl_device.c:229:libxl__device_disk_set_backend: Disk vdev=xvdb spec.backend=tap
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f876c8 wpath=/local/domain/0/backend/vbd/3/51728/state token=0/6: register slotnum=0
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f89288 wpath=/local/domain/0/backend/vbd/3/51712/state token=3/5: event epath=/local/domain/0/backend/vbd/3/51712/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vbd/3/51712/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f89288 wpath=/local/domain/0/backend/vbd/3/51712/state token=3/5: deregister slotnum=3
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f89288: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/3/51712/state token=3/5: empty slot
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f876c8 wpath=/local/domain/0/backend/vbd/3/51728/state token=0/6: event epath=/local/domain/0/backend/vbd/3/51728/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vbd/3/51728/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f876c8 wpath=/local/domain/0/backend/vbd/3/51728/state token=0/6: deregister slotnum=0
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f876c8: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/3/51728/state token=0/6: empty slot
libxl: debug: libxl_dm.c:1160:libxl__spawn_local_dm: Spawning device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: /usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -xen-domid
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: 3
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -chardev
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-3,server,nowait
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -mon
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -xen-attach
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -name
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: test1
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -vnc
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: 0.0.0.0:1,password,to=99
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -M
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: xenpv
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: -m
libxl: debug: libxl_dm.c:1162:libxl__spawn_local_dm: 4097
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f85cc8 wpath=/local/domain/0/device-model/3/state token=0/7: register slotnum=0
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f85cc8 wpath=/local/domain/0/device-model/3/state token=0/7: event epath=/local/domain/0/device-model/3/state
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f85cc8 wpath=/local/domain/0/device-model/3/state token=0/7: event epath=/local/domain/0/device-model/3/state
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f85cc8 wpath=/local/domain/0/device-model/3/state token=0/7: deregister slotnum=0
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f85cc8: deregister unregistered
libxl: debug: libxl_qmp.c:696:libxl__qmp_initialize: connected to /var/run/xen/qmp-libxl-3
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: qmp
libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{"execute":"qmp_capabilities","id":1}'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{"execute":"query-chardev","id":2}'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{"execute":"change","id":3,"arguments":{"device":"vnc","target":"password","arg":"vnc"}}'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:556:qmp_send_prepare: next qmp command: '{"execute":"query-vnc","id":4}'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f81c68 wpath=/local/domain/0/backend/vif/3/0/state token=0/8: register slotnum=0
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f81c68 wpath=/local/domain/0/backend/vif/3/0/state token=0/8: event epath=/local/domain/0/backend/vif/3/0/state
libxl: debug: libxl_event.c:642:devstate_watch_callback: backend /local/domain/0/backend/vif/3/0/state wanted state 2 ok
libxl: debug: libxl_event.c:595:libxl__ev_xswatch_deregister: watch w=0x1f81c68 wpath=/local/domain/0/backend/vif/3/0/state token=0/8: deregister slotnum=0
libxl: debug: libxl_event.c:607:libxl__ev_xswatch_deregister: watch w=0x1f81c68: deregister unregistered
libxl: debug: libxl_device.c:917:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-ovs online
libxl: debug: libxl_event.c:1736:libxl__ao_progress_report: ao 0x1f86fa0: progress report: ignored
libxl: debug: libxl_event.c:1568:libxl__ao_complete: ao 0x1f86fa0: complete, rc=0
libxl: debug: libxl_event.c:1540:libxl__ao__destroy: ao 0x1f86fa0: destroy
Waiting for domain test1 (domid 3) to die [pid 4440]
libxl: debug: libxl_event.c:558:libxl__ev_xswatch_register: watch w=0x1f8a0d0 wpath=@releaseDomain token=0/9: register slotnum=0
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x1f8a0d0 wpath=@releaseDomain token=0/9: event epath=@releaseDomain
libxl: debug: libxl.c:998:domain_death_xswatch_callback: [evg=0x1f80a80:3] from domid=3 nentries=1 rc=1
libxl: debug: libxl.c:1009:domain_death_xswatch_callback: [evg=0x1f80a80:3] got=domaininfos[0] got->domain=3
libxl: debug: libxl.c:1036:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0020
libxl: debug: libxl.c:1002:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl.c:1066:domain_death_xswatch_callback: domain death search done
[-- Attachment #6: messages.log --]
[-- Type: text/x-log, Size: 15840 bytes --]
2013-05-17T14:54:44.306147+03:00 hv04 kernel: [ 909.307861] device vif2.0 left promiscuous mode
2013-05-17T14:54:44.325228+03:00 hv04 local: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/2/51712
2013-05-17T14:54:44.326123+03:00 hv04 local: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/2/51728
2013-05-17T14:54:44.334134+03:00 hv04 tapdisk2[4133]: received 'pid' message (uuid = 0)
2013-05-17T14:54:44.334667+03:00 hv04 tapdisk2[4133]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:44.335095+03:00 hv04 tapdisk2[4136]: received 'pid' message (uuid = 0)
2013-05-17T14:54:44.335536+03:00 hv04 tapdisk2[4136]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:44.335968+03:00 hv04 tapdisk2[4292]: received 'pid' message (uuid = 0)
2013-05-17T14:54:44.336414+03:00 hv04 tapdisk2[4292]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:44.336932+03:00 hv04 tapdisk2[4297]: received 'pid' message (uuid = 0)
2013-05-17T14:54:44.337392+03:00 hv04 tapdisk2[4297]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:44.337822+03:00 hv04 tapdisk2[4133]: received 'list' message (uuid = 65535)
2013-05-17T14:54:44.338350+03:00 hv04 tapdisk2[4133]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.338872+03:00 hv04 tapdisk2[4133]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.339541+03:00 hv04 tapdisk2[4136]: received 'list' message (uuid = 65535)
2013-05-17T14:54:44.340338+03:00 hv04 tapdisk2[4136]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.343005+03:00 hv04 tapdisk2[4136]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.343017+03:00 hv04 tapdisk2[4292]: received 'list' message (uuid = 65535)
2013-05-17T14:54:44.343022+03:00 hv04 tapdisk2[4292]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.343987+03:00 hv04 kernel: [ 909.342243] blktap_ring_vm_close: 6 callbacks suppressed
2013-05-17T14:54:44.343995+03:00 hv04 kernel: [ 909.342248] blktap_ring_vm_close: unmapping ring 2
2013-05-17T14:54:44.343997+03:00 hv04 kernel: [ 909.342252] blktap_ring_release: freeing device 2
2013-05-17T14:54:44.343998+03:00 hv04 kernel: [ 909.342991] blktap_device_destroy: destroy device 2 users 0
2013-05-17T14:54:44.344010+03:00 hv04 tapdisk2[4292]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.344018+03:00 hv04 tapdisk2[4297]: received 'list' message (uuid = 65535)
2013-05-17T14:54:44.344023+03:00 hv04 tapdisk2[4297]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.344441+03:00 hv04 tapdisk2[4297]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.344818+03:00 hv04 tapdisk2[4292]: received 'close' message (uuid = 2)
2013-05-17T14:54:44.344956+03:00 hv04 tapdisk2[4292]: closed image /storage/images/manual/test1.raw (0 users, state: 0x00000000, type: 0)
2013-05-17T14:54:44.345150+03:00 hv04 tapdisk2[4292]: sending 'close response' message (uuid = 2)
2013-05-17T14:54:44.345289+03:00 hv04 tapdisk2[4292]: received 'detach' message (uuid = 2)
2013-05-17T14:54:44.345422+03:00 hv04 tapdisk2[4292]: sending 'detach response' message (uuid = 2)
2013-05-17T14:54:44.345553+03:00 hv04 local: /etc/xen/scripts/vif-ovs: offline type_if=vif XENBUS_PATH=backend/vif/2/0
2013-05-17T14:54:44.345694+03:00 hv04 tapdisk2[4133]: received 'pid' message (uuid = 0)
2013-05-17T14:54:44.346120+03:00 hv04 tapdisk2[4133]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:44.346500+03:00 hv04 tapdisk2[4136]: received 'pid' message (uuid = 0)
2013-05-17T14:54:44.346878+03:00 hv04 tapdisk2[4136]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:44.347352+03:00 hv04 tapdisk2[4297]: received 'pid' message (uuid = 0)
2013-05-17T14:54:44.347771+03:00 hv04 tapdisk2[4297]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:44.348148+03:00 hv04 tapdisk2[4133]: received 'list' message (uuid = 65535)
2013-05-17T14:54:44.348530+03:00 hv04 tapdisk2[4133]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.348916+03:00 hv04 tapdisk2[4133]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.349301+03:00 hv04 tapdisk2[4136]: received 'list' message (uuid = 65535)
2013-05-17T14:54:44.349682+03:00 hv04 tapdisk2[4136]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.350069+03:00 hv04 tapdisk2[4136]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.350122+03:00 hv04 kernel: [ 909.350488] blktap_ring_vm_close: unmapping ring 3
2013-05-17T14:54:44.350131+03:00 hv04 kernel: [ 909.350492] blktap_ring_release: freeing device 3
2013-05-17T14:54:44.350590+03:00 hv04 tapdisk2[4297]: received 'list' message (uuid = 65535)
2013-05-17T14:54:44.350804+03:00 hv04 tapdisk2[4297]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.350996+03:00 hv04 tapdisk2[4297]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:44.351175+03:00 hv04 tapdisk2[4297]: received 'close' message (uuid = 3)
2013-05-17T14:54:44.351367+03:00 hv04 tapdisk2[4297]: closed image /storage/images/manual/test1_storage.raw (0 users, state: 0x00000000, type: 0)
2013-05-17T14:54:44.351549+03:00 hv04 tapdisk2[4297]: sending 'close response' message (uuid = 3)
2013-05-17T14:54:44.351729+03:00 hv04 tapdisk2[4297]: received 'detach' message (uuid = 3)
2013-05-17T14:54:44.351994+03:00 hv04 tapdisk2[4297]: sending 'detach response' message (uuid = 3)
2013-05-17T14:54:44.354133+03:00 hv04 kernel: [ 909.352103] blktap_device_destroy: destroy device 3 users 0
2013-05-17T14:54:44.368669+03:00 hv04 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl del-port vlannet vif2.0
2013-05-17T14:54:44.404213+03:00 hv04 local: /etc/xen/scripts/vif-ovs: ip link set vif2.0 down failed
2013-05-17T14:54:44.405650+03:00 hv04 local: /etc/xen/scripts/vif-ovs: Successful vif-ovs offline for switch vlannet dev vif2.0, vlan 2512, mtu 9000
2013-05-17T14:54:53.136729+03:00 hv04 tapdisk2[4133]: received 'pid' message (uuid = 0)
2013-05-17T14:54:53.137411+03:00 hv04 tapdisk2[4133]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:53.137945+03:00 hv04 tapdisk2[4136]: received 'pid' message (uuid = 0)
2013-05-17T14:54:53.138135+03:00 hv04 kernel: [ 918.138974] blktap_control_allocate_tap: allocated tap ffff8800ea8a8000
2013-05-17T14:54:53.138590+03:00 hv04 tapdisk2[4136]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:53.138993+03:00 hv04 tapdisk2[4133]: received 'list' message (uuid = 65535)
2013-05-17T14:54:53.139410+03:00 hv04 tapdisk2[4133]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.139824+03:00 hv04 tapdisk2[4133]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.140303+03:00 hv04 tapdisk2[4136]: received 'list' message (uuid = 65535)
2013-05-17T14:54:53.140728+03:00 hv04 tapdisk2[4136]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.141144+03:00 hv04 tapdisk2[4136]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.141543+03:00 hv04 tapdisk2[4695]: I/O queue driver: lio
2013-05-17T14:54:53.142019+03:00 hv04 tapdisk2[4695]: received 'attach' message (uuid = 2)
2013-05-17T14:54:53.142123+03:00 hv04 kernel: [ 918.143614] blktap_ring_open: opening device blktap2
2013-05-17T14:54:53.142130+03:00 hv04 kernel: [ 918.143617] blktap_ring_open: opened device 2
2013-05-17T14:54:53.142131+03:00 hv04 kernel: [ 918.143636] blktap_ring_mmap: blktap: mapping pid is 4696
2013-05-17T14:54:53.142133+03:00 hv04 kernel: [ 918.143790] blktap_validate_params: aio:/storage/images/manual/test1.raw: capacity: 20971520, sector-size: 512
2013-05-17T14:54:53.142134+03:00 hv04 kernel: [ 918.143793] blktap_validate_params: aio:/storage/images/manual/test1.raw: capacity: 20971520, sector-size: 512
2013-05-17T14:54:53.142135+03:00 hv04 kernel: [ 918.143794] blktap_device_create: minor 2 sectors 20971520 sector-size 512
2013-05-17T14:54:53.142136+03:00 hv04 kernel: [ 918.143973] blktap_device_create: creation of 253:2: 0
2013-05-17T14:54:53.142439+03:00 hv04 tapdisk2[4695]: sending 'attach response' message (uuid = 2)
2013-05-17T14:54:53.143007+03:00 hv04 tapdisk2[4695]: received 'open' message (uuid = 2)
2013-05-17T14:54:53.143514+03:00 hv04 tapdisk2[4695]: block-aio open('/storage/images/manual/test1.raw')
2013-05-17T14:54:53.144159+03:00 hv04 tapdisk2[4695]: open(/storage/images/manual/test1.raw) with O_DIRECT
2013-05-17T14:54:53.144654+03:00 hv04 tapdisk2[4695]: Image size:
2013-05-17T14:54:53.145141+03:00 hv04 tapdisk2[4695]: opened image /storage/images/manual/test1.raw (1 users, state: 0x00000001, type: 0)
2013-05-17T14:54:53.145556+03:00 hv04 tapdisk2[4695]: VBD CHAIN:
2013-05-17T14:54:53.146045+03:00 hv04 tapdisk2[4695]: /storage/images/manual/test1.raw: 0
2013-05-17T14:54:53.146456+03:00 hv04 tapdisk2[4695]: sending 'open response' message (uuid = 2)
2013-05-17T14:54:53.148803+03:00 hv04 tapdisk2[4133]: received 'pid' message (uuid = 0)
2013-05-17T14:54:53.149375+03:00 hv04 tapdisk2[4133]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:53.149855+03:00 hv04 tapdisk2[4136]: received 'pid' message (uuid = 0)
2013-05-17T14:54:53.150365+03:00 hv04 tapdisk2[4136]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:53.150846+03:00 hv04 tapdisk2[4695]: received 'pid' message (uuid = 0)
2013-05-17T14:54:53.151329+03:00 hv04 tapdisk2[4695]: sending 'pid response' message (uuid = 0)
2013-05-17T14:54:53.151800+03:00 hv04 tapdisk2[4133]: received 'list' message (uuid = 65535)
2013-05-17T14:54:53.152392+03:00 hv04 tapdisk2[4133]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.152893+03:00 hv04 tapdisk2[4133]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.153406+03:00 hv04 tapdisk2[4136]: received 'list' message (uuid = 65535)
2013-05-17T14:54:53.153897+03:00 hv04 tapdisk2[4136]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.154130+03:00 hv04 kernel: [ 918.153577] blktap_control_allocate_tap: allocated tap ffff8800f4eba800
2013-05-17T14:54:53.154412+03:00 hv04 tapdisk2[4136]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.155063+03:00 hv04 tapdisk2[4695]: received 'list' message (uuid = 65535)
2013-05-17T14:54:53.155582+03:00 hv04 tapdisk2[4695]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.156209+03:00 hv04 tapdisk2[4695]: sending 'list response' message (uuid = 65535)
2013-05-17T14:54:53.156694+03:00 hv04 tapdisk2[4700]: I/O queue driver: lio
2013-05-17T14:54:53.157207+03:00 hv04 tapdisk2[4700]: received 'attach' message (uuid = 3)
2013-05-17T14:54:53.157696+03:00 hv04 tapdisk2[4700]: sending 'attach response' message (uuid = 3)
2013-05-17T14:54:53.158125+03:00 hv04 kernel: [ 918.158869] blktap_ring_open: opening device blktap3
2013-05-17T14:54:53.158297+03:00 hv04 tapdisk2[4700]: received 'open' message (uuid = 3)
2013-05-17T14:54:53.158940+03:00 hv04 tapdisk2[4700]: block-aio open('/storage/images/manual/test1_storage.raw')
2013-05-17T14:54:53.159465+03:00 hv04 tapdisk2[4700]: open(/storage/images/manual/test1_storage.raw) with O_DIRECT
2013-05-17T14:54:53.159953+03:00 hv04 tapdisk2[4700]: Image size:
2013-05-17T14:54:53.160443+03:00 hv04 tapdisk2[4700]: opened image /storage/images/manual/test1_storage.raw (1 users, state: 0x00000001, type: 0)
2013-05-17T14:54:53.160932+03:00 hv04 tapdisk2[4700]: VBD CHAIN:
2013-05-17T14:54:53.161420+03:00 hv04 tapdisk2[4700]: /storage/images/manual/test1_storage.raw: 0
2013-05-17T14:54:53.161988+03:00 hv04 tapdisk2[4700]: sending 'open response' message (uuid = 3)
2013-05-17T14:54:53.190990+03:00 hv04 local: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/51712
2013-05-17T14:54:53.193395+03:00 hv04 local: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/3/51728
2013-05-17T14:54:53.271920+03:00 hv04 local: /etc/xen/scripts/vif-ovs: online type_if=vif XENBUS_PATH=backend/vif/3/0
2013-05-17T14:54:53.302602+03:00 hv04 ovs-vsctl: 00001|vsctl|INFO|Called as ovs-vsctl add-port vlannet vif3.0 tag=2512
2013-05-17T14:54:53.306135+03:00 hv04 kernel: [ 918.307947] device vif3.0 entered promiscuous mode
2013-05-17T14:54:53.355759+03:00 hv04 local: /etc/xen/scripts/vif-ovs: Successful vif-ovs online for switch vlannet dev vif3.0, vlan 2512, mtu 9000
2013-05-17T14:54:53.357293+03:00 hv04 local: /etc/xen/scripts/vif-ovs: Writing backend/vif/3/0/hotplug-status connected to xenstore.
2013-05-17T14:54:54.878304+03:00 hv04 kernel: [ 919.878284] blkback: event-channel 19
2013-05-17T14:54:54.878328+03:00 hv04 kernel: [ 919.878385] blkback: protocol 1 (x86_64-abi)
2013-05-17T14:54:54.878331+03:00 hv04 kernel: [ 919.878562] blkback: ring-ref 770
2013-05-17T14:54:54.886195+03:00 hv04 kernel: [ 919.887871] blkback: event-channel 20
2013-05-17T14:54:54.886209+03:00 hv04 kernel: [ 919.887966] blkback: protocol 1 (x86_64-abi)
2013-05-17T14:54:54.890191+03:00 hv04 kernel: [ 919.888212] blkback: ring-ref 771
[-- Attachment #7: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 12:03 ` Eugene Istomin
@ 2013-05-17 12:25 ` Jan Beulich
2013-05-17 12:30 ` Eugene Istomin
0 siblings, 1 reply; 85+ messages in thread
From: Jan Beulich @ 2013-05-17 12:25 UTC (permalink / raw)
To: Eugene Istomin; +Cc: Wei Liu, xen-devel
>>> On 17.05.13 at 14:03, Eugene Istomin <e.istomin@edss.ee> wrote:
> We use 3.4 kernels in production, 3.4.37-1-xen and 3.4.45-2 are working
> without this problems, a couple of days ago i started to test 3.9.
That's quite big a jump.
> Logs (test1 - iperf server, test2 - iperf client):
>
> Dom0:
> 1) tailf /var/log/xen/xl-test1.log (turned just before bug) - xl-test1.log
> 2) tailf /var/log/xen/xl-test2.log (turned just before bug) - xl-test2.log
> 3) tailf /var/log/messages (turned just before bug) - messages.log
> 4) xl create log - xl_creation.log
>
> DomU:
> 1) dmesg - domu_dmesg
Did you look at these before sending - there's no sign of an oops
anywhere afaict.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 12:25 ` Jan Beulich
@ 2013-05-17 12:30 ` Eugene Istomin
2013-05-17 12:36 ` Jan Beulich
0 siblings, 1 reply; 85+ messages in thread
From: Eugene Istomin @ 2013-05-17 12:30 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 2780 bytes --]
> That's quite big a jump.
Kernels from 3.5 to 3.8 had OCFS2 bugs, in 3.9 all OCFS2 patches are
merged.
> Did you look at these before sending - there's no sign of an oops
> anywhere afaict.
Yes, there are no oops in logs. oop is listed only in xl console, and this is
looks strage to me.
Welcome to openSUSE 12.3 "Dartmouth" - Kernel 3.9.2-4.756ee56-xen
(xvc0).
template login: local
Password:
Last login: Fri May 17 14:34:27 on xvc0
Have a lot of fun...
local@template:~> sudo su
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
43162
[ 221.826637] BUG: unable to handle kernel paging request at
ffff880078b85000
[ 221.826674] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 221.826703] PGD a83067 PUD a93067 PMD 7fc2c067 PTE
8010000078b85065
[ 221.826732] Oops: 0003 [#1] SMP
[ 221.826748] Modules linked in: af_packet hwmon domctl crc32_pclmul
crc32c_intel ghash_clmulni_intel joydev aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
[ 221.826875] CPU 0
[ 221.826885] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 221.826911] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 221.826945] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 221.826964] RAX: ffff88007da9b438 RBX: ffff880078b84ec0 RCX:
ffff880078b85000
[ 221.826989] RDX: ffff880078bdf7c0 RSI: ffff880078b84ec0 RDI:
ffff880078536580
[ 221.827014] RBP: ffff880078479a80 R08: 0000000000000248 R09:
0000000000000000
[ 221.827039] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000011
[ 221.827064] R13: 00000000000030f0 R14: ffff88007b403dd8 R15:
ffff880078be8800
--
*/Best regards,/*
/Eugene Istomin/
> >>> On 17.05.13 at 14:03, Eugene Istomin <e.istomin@edss.ee>
wrote:
> > We use 3.4 kernels in production, 3.4.37-1-xen and 3.4.45-2 are
working
> > without this problems, a couple of days ago i started to test 3.9.
>
> That's quite big a jump.
>
> > Logs (test1 - iperf server, test2 - iperf client):
> >
> > Dom0:
> > 1) tailf /var/log/xen/xl-test1.log (turned just before bug) - xl-test1.log
> > 2) tailf /var/log/xen/xl-test2.log (turned just before bug) - xl-test2.log
> > 3) tailf /var/log/messages (turned just before bug) - messages.log
> > 4) xl create log - xl_creation.log
> >
> > DomU:
> > 1) dmesg - domu_dmesg
>
> Did you look at these before sending - there's no sign of an oops
> anywhere afaict.
>
> Jan
[-- Attachment #1.2: Type: text/html, Size: 13246 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 12:30 ` Eugene Istomin
@ 2013-05-17 12:36 ` Jan Beulich
2013-05-17 12:52 ` Eugene Istomin
0 siblings, 1 reply; 85+ messages in thread
From: Jan Beulich @ 2013-05-17 12:36 UTC (permalink / raw)
To: Eugene Istomin; +Cc: Wei Liu, xen-devel
>>> On 17.05.13 at 14:30, Eugene Istomin <e.istomin@edss.ee> wrote:
>> That's quite big a jump.
> Kernels from 3.5 to 3.8 had OCFS2 bugs, in 3.9 all OCFS2 patches are
> merged.
>
>> Did you look at these before sending - there's no sign of an oops
>> anywhere afaict.
> Yes, there are no oops in logs. oop is listed only in xl console, and this
> is
> looks strage to me.
That likely depends on some configuration setting within the guest.
> [ 221.826637] BUG: unable to handle kernel paging request at
> ffff880078b85000
> [ 221.826674] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
> [ 221.826703] PGD a83067 PUD a93067 PMD 7fc2c067 PTE
> 8010000078b85065
> [ 221.826732] Oops: 0003 [#1] SMP
> [ 221.826748] Modules linked in: af_packet hwmon domctl crc32_pclmul
> crc32c_intel ghash_clmulni_intel joydev aesni_intel ablk_helper cryptd lrw
> aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
> scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
> [ 221.826875] CPU 0
> [ 221.826885] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
> #1
> [ 221.826911] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
> netif_poll+0x49c/0xe80 [xennet]
> [ 221.826945] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
> [ 221.826964] RAX: ffff88007da9b438 RBX: ffff880078b84ec0 RCX:
> ffff880078b85000
> [ 221.826989] RDX: ffff880078bdf7c0 RSI: ffff880078b84ec0 RDI:
> ffff880078536580
> [ 221.827014] RBP: ffff880078479a80 R08: 0000000000000248 R09:
> 0000000000000000
> [ 221.827039] R10: 0000000000000000 R11: 0000000000000001 R12:
> 0000000000000011
> [ 221.827064] R13: 00000000000030f0 R14: ffff88007b403dd8 R15:
> ffff880078be8800
Sorry, but as said before, this is incomplete (stack dump, no call stack).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 12:36 ` Jan Beulich
@ 2013-05-17 12:52 ` Eugene Istomin
2013-05-17 13:00 ` Eugene Istomin
0 siblings, 1 reply; 85+ messages in thread
From: Eugene Istomin @ 2013-05-17 12:52 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 5360 bytes --]
Jan,
after 10 seconds of working (speed is 33 Mb/s!!) here is new oops:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61806
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 5.4 sec 21.6 MBytes 33.5 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61807
[ 5] 0.0-10.0 sec 39.5 MBytes 33.0 Mbits/sec
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61808
[ 4] 0.0-10.0 sec 149 MBytes 124 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61809
[ 118.230386] BUG: unable to handle kernel paging request at
ffff8800791e4000
[ 118.230426] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 118.230501] PGD a83067 PUD a93067 PMD 7fc29067 PTE
80100000791e4065
[ 118.230534] Oops: 0003 [#1] SMP
[ 118.230552] Modules linked in: af_packet hwmon domctl crc32_pclmul
joydev crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
[ 118.230690] CPU 0
[ 118.230700] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 118.230729] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 118.230766] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 118.230787] RAX: ffff88007da89500 RBX: ffff8800791e3ec0 RCX:
ffff8800791e4000
[ 118.230814] RDX: ffff880078446150 RSI: ffff8800791e3ec0 RDI:
ffff880078434880
[ 118.230841] RBP: ffff88007852db80 R08: 0000000000000ba8 R09:
0000000000000000
[ 118.230867] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000011
[ 118.230895] R13: 0000000000025322 R14: ffff88007b403dd8 R15:
ffff880078470800
[ 118.230925] FS: 00007ff5457b37c0(0000) GS:ffff88007b400000(0000)
knlGS:0000000000000000
[ 118.230956] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 118.230978] CR2: ffff8800791e4000 CR3: 0000000078c97000 CR4:
0000000000002660
[ 118.231629] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 118.232272] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 118.232906] Process swapper/0 (pid: 0, threadinfo ffffffff808b6000, task
ffffffff808c1960)
[ 118.233554] Stack:
[ 118.234194] ffff88007b403d38 ffffffff80060845 ffff88007b403d68
ffff88007b403db8
[ 118.234373] ffff88007b403d98 0000001200025311 ffff880078471d50
0000000000000000
[ 118.234373] ffff880078470858 ffff88007b40e878 0000000100000012
0000004000025323
[ 118.234373] Call Trace:
[ 118.234373] [<ffffffff80441070>] net_rx_action+0x170/0x2c0
[ 118.234373] [<ffffffff80036dbe>] __do_softirq+0xee/0x230
[ 118.234373] [<ffffffff80037085>] irq_exit+0x95/0xa0
[ 118.234373] [<ffffffff803933ed>] evtchn_do_upcall+0x2ad/0x2f0
[ 118.234373] [<ffffffff80534d5e>] do_hypervisor_callback+0x1e/0x30
[ 118.234373] [<ffffffff800033aa>] HYPERVISOR_sched_op_new+0xa/0x20
[ 118.234373] [<ffffffff8000e671>] xen_idle+0x41/0x110
[ 118.234373] [<ffffffff8000e7ef>] cpu_idle+0xaf/0x110
[ 118.234373] [<ffffffff80944b1f>] start_kernel+0x424/0x42f
I restarting Dom0 server now and will try to get first case call trace.
--
*/Best regards,/*
/Eugene Istomin/
> >>> On 17.05.13 at 14:30, Eugene Istomin <e.istomin@edss.ee>
wrote:
> >> That's quite big a jump.
> >
> > Kernels from 3.5 to 3.8 had OCFS2 bugs, in 3.9 all OCFS2 patches are
> > merged.
> >
> >> Did you look at these before sending - there's no sign of an oops
> >> anywhere afaict.
> >
> > Yes, there are no oops in logs. oop is listed only in xl console, and this
> > is
> > looks strage to me.
>
> That likely depends on some configuration setting within the guest.
>
> > [ 221.826637] BUG: unable to handle kernel paging request at
> > ffff880078b85000
> > [ 221.826674] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80
[xennet]
> > [ 221.826703] PGD a83067 PUD a93067 PMD 7fc2c067 PTE
> > 8010000078b85065
> > [ 221.826732] Oops: 0003 [#1] SMP
> > [ 221.826748] Modules linked in: af_packet hwmon domctl
crc32_pclmul
> > crc32c_intel ghash_clmulni_intel joydev aesni_intel ablk_helper cryptd
lrw
> > aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua
scsi_dh_rdac
> > scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
> > [ 221.826875] CPU 0
> > [ 221.826885] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-
xen
> > #1
> > [ 221.826911] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
> > netif_poll+0x49c/0xe80 [xennet]
> > [ 221.826945] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
> > [ 221.826964] RAX: ffff88007da9b438 RBX: ffff880078b84ec0 RCX:
> > ffff880078b85000
> > [ 221.826989] RDX: ffff880078bdf7c0 RSI: ffff880078b84ec0 RDI:
> > ffff880078536580
> > [ 221.827014] RBP: ffff880078479a80 R08: 0000000000000248 R09:
> > 0000000000000000
> > [ 221.827039] R10: 0000000000000000 R11: 0000000000000001
R12:
> > 0000000000000011
> > [ 221.827064] R13: 00000000000030f0 R14: ffff88007b403dd8 R15:
> > ffff880078be8800
>
> Sorry, but as said before, this is incomplete (stack dump, no call stack).
>
> Jan
[-- Attachment #1.2: Type: text/html, Size: 24095 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 12:52 ` Eugene Istomin
@ 2013-05-17 13:00 ` Eugene Istomin
2013-05-18 10:06 ` Eugene Istomin
2013-05-21 9:55 ` Jan Beulich
0 siblings, 2 replies; 85+ messages in thread
From: Eugene Istomin @ 2013-05-17 13:00 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 6008 bytes --]
Bump, here it is:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
23902
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.1 sec 38.5 MBytes 32.0 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
23903
[ 124.555698] BUG: unable to handle kernel paging request at
ffff880078453000
[ 124.555760] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 124.555813] PGD a83067 PUD a93067 PMD 7fc2f067 PTE
8010000078453065
[ 124.555866] Oops: 0003 [#1] SMP
[ 124.555894] Modules linked in: af_packet hwmon domctl crc32_pclmul
crc32c_intel ghash_clmulni_intel joydev aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
[ 124.556126] CPU 0
[ 124.556143] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 124.556190] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 124.556252] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 124.556286] RAX: ffff88007da5f650 RBX: ffff880078452ec0 RCX:
ffff880078453000
[ 124.556332] RDX: ffff880078876810 RSI: ffff880078452ec0 RDI:
ffff880078451580
[ 124.556377] RBP: ffff88007802ac80 R08: 00000000000009e0 R09:
0000000000000000
[ 124.556423] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000011
[ 124.556468] R13: 0000000000005afa R14: ffff88007b403dd8 R15:
ffff880078440800
[ 124.556518] FS: 00007faae5316700(0000) GS:ffff88007b400000(0000)
knlGS:0000000000000000
[ 124.556569] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 124.557637] CR2: ffff880078453000 CR3: 0000000079f61000 CR4:
0000000000002660
[ 124.558714] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 124.559681] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 124.559681] Process swapper/0 (pid: 0, threadinfo ffffffff808b6000, task
ffffffff808c1960)
[ 124.559681] Stack:
[ 124.559681] ffff8800786afb80 000000007953e510
000000000000ad00 ffff88007b403db8
[ 124.559681] ffff88007b403d98 0000001200005ae9 ffff880078441d50
0000000000000000
[ 124.559681] ffff880078440858 ffff88007b40e878 0000000100000012
0000004000005afb
[ 124.559681] Call Trace:
[ 124.559681] [<ffffffff80441070>] net_rx_action+0x170/0x2c0
[ 124.559681] [<ffffffff80036dbe>] __do_softirq+0xee/0x230
[ 124.559681] [<ffffffff80037085>] irq_exit+0x95/0xa0
[ 124.559681] [<ffffffff803933ed>] evtchn_do_upcall+0x2ad/0x2f0
[ 124.559681] [<ffffffff80534d5e>] do_hypervisor_callback+0x1e/0x30
[ 124.559681] [<ffffffff800033aa>] HYPERVISOR_sched_op_new+0xa/0x20
[ 124.559681] [<ffffffff8000e671>] xen_idle+0x41/0x110
[ 124.559681] [<ffffffff8000e7ef>] cpu_idle+0xaf/0x110
[ 124.559681] [<ffffffff80944b1f>] start_kernel+0x424/0x42f
[ 124.559681] Code: 44 21 ea 48 8d 54 d0 40 8b 87 d8 00 00 00 44 0f bf
42 06 44 0f b7 4a 02 48 8b 44 01 30 49 63 cc 48 83 c1 03 48 c1 e1 04 48
01 f1 <48> 89 01 44 89 49 08 44 89 41 0c 48 8b 08 80 e5 80 0f 85 54 09
[ 124.559681] RIP [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 124.559681] RSP <ffff88007b403d18>
[ 124.559681] CR2: ffff880078453000
[ 124.559681] ---[ end trace d5028239de5a7a42 ]---
[ 124.559681] Kernel panic - not syncing: Fatal exception in interrupt
--
*/Best regards,/*
/Eugene Istomin/
Jan,
after 10 seconds of working (speed is 33 Mb/s!!) here is new oops:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61806
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 5.4 sec 21.6 MBytes 33.5 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61807
[ 5] 0.0-10.0 sec 39.5 MBytes 33.0 Mbits/sec
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61808
[ 4] 0.0-10.0 sec 149 MBytes 124 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61809
[ 118.230386] BUG: unable to handle kernel paging request at
ffff8800791e4000
[ 118.230426] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 118.230501] PGD a83067 PUD a93067 PMD 7fc29067 PTE
80100000791e4065
[ 118.230534] Oops: 0003 [#1] SMP
[ 118.230552] Modules linked in: af_packet hwmon domctl crc32_pclmul
joydev crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
[ 118.230690] CPU 0
[ 118.230700] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 118.230729] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 118.230766] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 118.230787] RAX: ffff88007da89500 RBX: ffff8800791e3ec0 RCX:
ffff8800791e4000
[ 118.230814] RDX: ffff880078446150 RSI: ffff8800791e3ec0 RDI:
ffff880078434880
[ 118.230841] RBP: ffff88007852db80 R08: 0000000000000ba8 R09:
0000000000000000
[ 118.230867] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000011
[ 118.230895] R13: 0000000000025322 R14: ffff88007b403dd8 R15:
ffff880078470800
[ 118.230925] FS: 00007ff5457b37c0(0000) GS:ffff88007b400000(0000)
knlGS:0000000000000000
[ 118.230956] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 118.230978] CR2: ffff8800791e4000 CR3: 0000000078c97000 CR4:
0000000000002660
[ 118.231629] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 118.232272] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[-- Attachment #1.2: Type: text/html, Size: 39897 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 13:00 ` Eugene Istomin
@ 2013-05-18 10:06 ` Eugene Istomin
2013-05-20 6:52 ` Eugene Istomin
2013-05-21 9:55 ` Jan Beulich
1 sibling, 1 reply; 85+ messages in thread
From: Eugene Istomin @ 2013-05-18 10:06 UTC (permalink / raw)
To: xen-devel; +Cc: Wei Liu, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 5682 bytes --]
Hello,
Do you need any other logs?
--
*/Best regards,/*
/Eugene Istomin/
Bump, here it is:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
23902
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.1 sec 38.5 MBytes 32.0 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
23903
[ 124.555698] BUG: unable to handle kernel paging request at
ffff880078453000
[ 124.555760] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 124.555813] PGD a83067 PUD a93067 PMD 7fc2f067 PTE
8010000078453065
[ 124.555866] Oops: 0003 [#1] SMP
[ 124.555894] Modules linked in: af_packet hwmon domctl crc32_pclmul
crc32c_intel ghash_clmulni_intel joydev aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
[ 124.556126] CPU 0
[ 124.556143] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 124.556190] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 124.556252] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 124.556286] RAX: ffff88007da5f650 RBX: ffff880078452ec0 RCX:
ffff880078453000
[ 124.556332] RDX: ffff880078876810 RSI: ffff880078452ec0 RDI:
ffff880078451580
[ 124.556377] RBP: ffff88007802ac80 R08: 00000000000009e0 R09:
0000000000000000
[ 124.556423] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000011
[ 124.556468] R13: 0000000000005afa R14: ffff88007b403dd8 R15:
ffff880078440800
[ 124.556518] FS: 00007faae5316700(0000) GS:ffff88007b400000(0000)
knlGS:0000000000000000
[ 124.556569] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 124.557637] CR2: ffff880078453000 CR3: 0000000079f61000 CR4:
0000000000002660
[ 124.558714] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 124.559681] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 124.559681] Process swapper/0 (pid: 0, threadinfo ffffffff808b6000, task
ffffffff808c1960)
[ 124.559681] Stack:
[ 124.559681] ffff8800786afb80 000000007953e510
000000000000ad00 ffff88007b403db8
[ 124.559681] ffff88007b403d98 0000001200005ae9 ffff880078441d50
0000000000000000
[ 124.559681] ffff880078440858 ffff88007b40e878 0000000100000012
0000004000005afb
[ 124.559681] Call Trace:
[ 124.559681] [<ffffffff80441070>] net_rx_action+0x170/0x2c0
[ 124.559681] [<ffffffff80036dbe>] __do_softirq+0xee/0x230
[ 124.559681] [<ffffffff80037085>] irq_exit+0x95/0xa0
[ 124.559681] [<ffffffff803933ed>] evtchn_do_upcall+0x2ad/0x2f0
[ 124.559681] [<ffffffff80534d5e>] do_hypervisor_callback+0x1e/0x30
[ 124.559681] [<ffffffff800033aa>] HYPERVISOR_sched_op_new+0xa/0x20
[ 124.559681] [<ffffffff8000e671>] xen_idle+0x41/0x110
[ 124.559681] [<ffffffff8000e7ef>] cpu_idle+0xaf/0x110
[ 124.559681] [<ffffffff80944b1f>] start_kernel+0x424/0x42f
[ 124.559681] Code: 44 21 ea 48 8d 54 d0 40 8b 87 d8 00 00 00 44 0f bf
42 06 44 0f b7 4a 02 48 8b 44 01 30 49 63 cc 48 83 c1 03 48 c1 e1 04 48
01 f1 <48> 89 01 44 89 49 08 44 89 41 0c 48 8b 08 80 e5 80 0f 85 54 09
[ 124.559681] RIP [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 124.559681] RSP <ffff88007b403d18>
[ 124.559681] CR2: ffff880078453000
[ 124.559681] ---[ end trace d5028239de5a7a42 ]---
[ 124.559681] Kernel panic - not syncing: Fatal exception in interrupt
--
*/Best regards,/*
/Eugene Istomin/
Jan,
after 10 seconds of working (speed is 33 Mb/s!!) here is new oops:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61806
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 5.4 sec 21.6 MBytes 33.5 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61807
[ 5] 0.0-10.0 sec 39.5 MBytes 33.0 Mbits/sec
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61808
[ 4] 0.0-10.0 sec 149 MBytes 124 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61809
[ 118.230386] BUG: unable to handle kernel paging request at
ffff8800791e4000
[ 118.230426] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 118.230501] PGD a83067 PUD a93067 PMD 7fc29067 PTE
80100000791e4065
[ 118.230534] Oops: 0003 [#1] SMP
[ 118.230552] Modules linked in: af_packet hwmon domctl crc32_pclmul
joydev crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
[ 118.230690] CPU 0
[ 118.230700] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 118.230729] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 118.230766] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 118.230787] RAX: ffff88007da89500 RBX: ffff8800791e3ec0 RCX:
ffff8800791e4000
[ 118.230814] RDX: ffff880078446150 RSI: ffff8800791e3ec0 RDI:
ffff880078434880
[ 118.230841] RBP: ffff88007852db80 R08: 0000000000000ba8 R09:
0000000000000000
[ 118.230867] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000011
[ 118.230895] R13: 0000000000025322 R14: ffff88007b403dd8 R15:
ffff880078470800
[-- Attachment #1.2: Type: text/html, Size: 42146 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-18 10:06 ` Eugene Istomin
@ 2013-05-20 6:52 ` Eugene Istomin
2013-05-20 8:58 ` Ian Campbell
2013-05-20 9:06 ` Wei Liu
0 siblings, 2 replies; 85+ messages in thread
From: Eugene Istomin @ 2013-05-20 6:52 UTC (permalink / raw)
To: xen-devel; +Cc: Wei Liu, Jan Beulich
[-- Attachment #1.1: Type: text/plain, Size: 5402 bytes --]
Ping
--
*/Best regards,/*
/Eugene Istomin/
Hello,
Do you need any other logs?
--
*/Best regards,/*
/Eugene Istomin/
Bump, here it is:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
23902
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.1 sec 38.5 MBytes 32.0 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
23903
[ 124.555698] BUG: unable to handle kernel paging request at
ffff880078453000
[ 124.555760] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 124.555813] PGD a83067 PUD a93067 PMD 7fc2f067 PTE
8010000078453065
[ 124.555866] Oops: 0003 [#1] SMP
[ 124.555894] Modules linked in: af_packet hwmon domctl crc32_pclmul
crc32c_intel ghash_clmulni_intel joydev aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
[ 124.556126] CPU 0
[ 124.556143] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 124.556190] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 124.556252] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 124.556286] RAX: ffff88007da5f650 RBX: ffff880078452ec0 RCX:
ffff880078453000
[ 124.556332] RDX: ffff880078876810 RSI: ffff880078452ec0 RDI:
ffff880078451580
[ 124.556377] RBP: ffff88007802ac80 R08: 00000000000009e0 R09:
0000000000000000
[ 124.556423] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000011
[ 124.556468] R13: 0000000000005afa R14: ffff88007b403dd8 R15:
ffff880078440800
[ 124.556518] FS: 00007faae5316700(0000) GS:ffff88007b400000(0000)
knlGS:0000000000000000
[ 124.556569] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 124.557637] CR2: ffff880078453000 CR3: 0000000079f61000 CR4:
0000000000002660
[ 124.558714] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 124.559681] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 124.559681] Process swapper/0 (pid: 0, threadinfo ffffffff808b6000, task
ffffffff808c1960)
[ 124.559681] Stack:
[ 124.559681] ffff8800786afb80 000000007953e510
000000000000ad00 ffff88007b403db8
[ 124.559681] ffff88007b403d98 0000001200005ae9 ffff880078441d50
0000000000000000
[ 124.559681] ffff880078440858 ffff88007b40e878 0000000100000012
0000004000005afb
[ 124.559681] Call Trace:
[ 124.559681] [<ffffffff80441070>] net_rx_action+0x170/0x2c0
[ 124.559681] [<ffffffff80036dbe>] __do_softirq+0xee/0x230
[ 124.559681] [<ffffffff80037085>] irq_exit+0x95/0xa0
[ 124.559681] [<ffffffff803933ed>] evtchn_do_upcall+0x2ad/0x2f0
[ 124.559681] [<ffffffff80534d5e>] do_hypervisor_callback+0x1e/0x30
[ 124.559681] [<ffffffff800033aa>] HYPERVISOR_sched_op_new+0xa/0x20
[ 124.559681] [<ffffffff8000e671>] xen_idle+0x41/0x110
[ 124.559681] [<ffffffff8000e7ef>] cpu_idle+0xaf/0x110
[ 124.559681] [<ffffffff80944b1f>] start_kernel+0x424/0x42f
[ 124.559681] Code: 44 21 ea 48 8d 54 d0 40 8b 87 d8 00 00 00 44 0f bf
42 06 44 0f b7 4a 02 48 8b 44 01 30 49 63 cc 48 83 c1 03 48 c1 e1 04 48
01 f1 <48> 89 01 44 89 49 08 44 89 41 0c 48 8b 08 80 e5 80 0f 85 54 09
[ 124.559681] RIP [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 124.559681] RSP <ffff88007b403d18>
[ 124.559681] CR2: ffff880078453000
[ 124.559681] ---[ end trace d5028239de5a7a42 ]---
[ 124.559681] Kernel panic - not syncing: Fatal exception in interrupt
--
*/Best regards,/*
/Eugene Istomin/
Jan,
after 10 seconds of working (speed is 33 Mb/s!!) here is new oops:
template:/home/local # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61806
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 5.4 sec 21.6 MBytes 33.5 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61807
[ 5] 0.0-10.0 sec 39.5 MBytes 33.0 Mbits/sec
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61808
[ 4] 0.0-10.0 sec 149 MBytes 124 Mbits/sec
[ 5] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
61809
[ 118.230386] BUG: unable to handle kernel paging request at
ffff8800791e4000
[ 118.230426] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 118.230501] PGD a83067 PUD a93067 PMD 7fc29067 PTE
80100000791e4065
[ 118.230534] Oops: 0003 [#1] SMP
[ 118.230552] Modules linked in: af_packet hwmon domctl crc32_pclmul
joydev crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
aes_x86_64 xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua scsi_dh_rdac
scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic ata_piix
[ 118.230690] CPU 0
[ 118.230700] Pid: 0, comm: swapper/0 Not tainted 3.9.2-4.756ee56-xen
#1
[ 118.230729] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 118.230766] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 118.230787] RAX: ffff88007da89500 RBX: ffff8800791e3ec0 RCX:
ffff8800791e4000
[-- Attachment #1.2: Type: text/html, Size: 44240 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-20 6:52 ` Eugene Istomin
@ 2013-05-20 8:58 ` Ian Campbell
2013-05-20 9:06 ` Wei Liu
1 sibling, 0 replies; 85+ messages in thread
From: Ian Campbell @ 2013-05-20 8:58 UTC (permalink / raw)
To: Eugene Istomin; +Cc: Wei Liu, Jan Beulich, xen-devel
On Mon, 2013-05-20 at 09:52 +0300, Eugene Istomin wrote:
> Ping
It's been less than two days (over a weekend), not to mention your
initial ping on a Saturday morning less than a day after posting on a
Friday afternoon -- be a little patient please.
Ian.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-20 6:52 ` Eugene Istomin
2013-05-20 8:58 ` Ian Campbell
@ 2013-05-20 9:06 ` Wei Liu
2013-05-20 9:17 ` Eugene Istomin
1 sibling, 1 reply; 85+ messages in thread
From: Wei Liu @ 2013-05-20 9:06 UTC (permalink / raw)
To: Eugene Istomin; +Cc: Wei Liu, Jan Beulich, xen-devel
On Mon, May 20, 2013 at 09:52:21AM +0300, Eugene Istomin wrote:
> Ping
I might be missing something, but you only seemed to provide links to
the RPM package you use.
The kernel you're using, as Jan pointed out earlier, is a nightly build
of SuSE kernel. It is impossible to tell what's going on without source
code. And generally speaking, I don't think using that kernel in your
production environment a good idea.
Wei.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-20 9:06 ` Wei Liu
@ 2013-05-20 9:17 ` Eugene Istomin
2013-05-20 9:24 ` Wei Liu
0 siblings, 1 reply; 85+ messages in thread
From: Eugene Istomin @ 2013-05-20 9:17 UTC (permalink / raw)
To: Wei Liu; +Cc: Jan Beulich, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 745 bytes --]
Wei,
i use all opensuse available 3.9.2 kernels in tests:
1) stable ones
(http://download.opensuse.org/repositories/Kernel:/stable/standard/x86_64
),
2) head opensuse release
(http://download.opensuse.org/repositories/Kernel:/HEAD/standard/x86_64)
--
*/Best regards,/*
/Eugene Istomin/
> On Mon, May 20, 2013 at 09:52:21AM +0300, Eugene Istomin wrote:
> > Ping
>
> I might be missing something, but you only seemed to provide links to
> the RPM package you use.
>
> The kernel you're using, as Jan pointed out earlier, is a nightly build
> of SuSE kernel. It is impossible to tell what's going on without source
> code. And generally speaking, I don't think using that kernel in your
> production environment a good idea.
>
>
> Wei.
[-- Attachment #1.2: Type: text/html, Size: 5386 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-20 9:17 ` Eugene Istomin
@ 2013-05-20 9:24 ` Wei Liu
0 siblings, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-05-20 9:24 UTC (permalink / raw)
To: Eugene Istomin; +Cc: Wei Liu, Jan Beulich, xen-devel
On Mon, May 20, 2013 at 12:17:44PM +0300, Eugene Istomin wrote:
> Wei,
>
> i use all opensuse available 3.9.2 kernels in tests:
> 1) stable ones
> (http://download.opensuse.org/repositories/Kernel:/stable/standard/x86_64
> ),
> 2) head opensuse release
> (http://download.opensuse.org/repositories/Kernel:/HEAD/standard/x86_64)
>
Those are the binary packages, not the source code.
Wei.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 13:00 ` Eugene Istomin
2013-05-18 10:06 ` Eugene Istomin
@ 2013-05-21 9:55 ` Jan Beulich
2013-05-21 11:09 ` Eugene Istomin
2013-05-21 15:28 ` Wei Liu
1 sibling, 2 replies; 85+ messages in thread
From: Jan Beulich @ 2013-05-21 9:55 UTC (permalink / raw)
To: Ian Campbell, Wei Liu, Eugene Istomin; +Cc: xen-devel
>>> On 17.05.13 at 15:00, Eugene Istomin <e.istomin@edss.ee> wrote:
> Bump, here it is:
Okay, but I think we're still lacking information on what your
Dom0 kernel is.
Ian, Wei - looking at the forward ported as well as the upstream
frontends, I'm wondering if there isn't a fundamental flaw in
..._get_responses(): It allows up to MAX_SKB_FRAGS + 1 slots/
frags, and ..._fill_frags() then fills as many fragments as got
queued onto the respective list. Only after both of them are done,
__pskb_pull_tail() gets invoked reducing the fragment count by
one if the condition is met that made ..._get_responses() bump
the limit by one.
Am I overlooking something? I'm asking because working through
disassembly and register values of the dump Eugene had sent I
clearly see that ..._fill_frags() is processing the 18th fragment,
while in that kernel version MAX_SKB_FRAGS is only 17 (but
possibly, hence the first question above, the Dom0 kernel still is
one with MAX_SKB_FRAGS being 18, or the packet turns out to
be such that it fills 17 fragments and the header is smaller than
RX_COPY_THRESHOLD).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-21 9:55 ` Jan Beulich
@ 2013-05-21 11:09 ` Eugene Istomin
2013-05-21 11:49 ` Jan Beulich
2013-05-21 15:28 ` Wei Liu
1 sibling, 1 reply; 85+ messages in thread
From: Eugene Istomin @ 2013-05-21 11:09 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, Ian Campbell, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 5665 bytes --]
Jan,
i updated to 3.9.3 from stable branch
*Dom0:*
#hv04: xl info
host : hv04.edss.local
release : 3.9.3-1.g0b5d8f5-xen
version : #1 SMP Mon May 20 09:43:24 UTC 2013 (0b5d8f5)
machine : x86_64
nr_cpus : 24
max_cpu_id : 23
nr_nodes : 2
cores_per_socket : 6
threads_per_core : 2
cpu_mhz : 2666
hw_caps :
bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:000
00000
virt_caps : hvm hvm_directio
total_memory : 98294
free_memory : 92985
sharing_freed_memory : 0
sharing_used_memory : 0
free_cpus : 0
xen_major : 4
xen_minor : 2
xen_extra : .2_01-240.2
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : 26064
xen_commandline : vga=mode-0x314 dom0_max_vcpus=8
dom0_mem=4G,max:4G msi=on iommu=on console=ttyS0,115200 xen-
netback.netback_max_groups=6
cc_compiler : gcc (SUSE Linux) 4.7.2 20130108 [gcc-4_7-branch
revision 195012
cc_compile_by : abuild
cc_compile_domain :
cc_compile_date : Fri May 10 15:05:06 UTC 2013
xend_config_format : 4
*Both DomU:*
test:/home/local # uname -a
Linux test.edss.local 3.9.3-1.g0b5d8f5-xen #1 SMP Mon May 20 09:43:24
UTC 2013 (0b5d8f5) x86_64 x86_64 x86_64 GNU/Linux
*From xl console*
[ 4] local 10.251.2.201 port 5001 connected with 10.251.2.202 port
26946
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 2.0 sec 8.38 MBytes 35.2 Mbits/sec
[ 4] 2.0- 4.0 sec 90.3 MBytes 379 Mbits/sec
[ 4] 4.0- 6.0 sec 7.30 MBytes 30.6 Mbits/sec
[ 105.391855] BUG: unable to handle kernel paging request at
ffff880078c6f000
[ 105.391918] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 105.391970] PGD a85067 PUD a95067 PMD 7fc2b067 PTE
8010000078c6f065
[ 105.392029] Oops: 0003 [#1] SMP
[ 105.392058] Modules linked in: iptable_filter ip_tables x_tables af_packet
hwmon domctl crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel
ablk_helper joydev cryptd lrw aes_x86_64 xts gf128mul autofs4
scsi_dh_emc scsi_dh_alua scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk
cdrom xennet ata_generic ata_piix
[ 105.392319] CPU 0
[ 105.392336] Pid: 0, comm: swapper/0 Not tainted 3.9.3-1.g0b5d8f5-xen
#1
[ 105.392383] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
netif_poll+0x49c/0xe80 [xennet]
[ 105.392450] RSP: e02b:ffff88007b403d18 EFLAGS: 00010286
[ 105.392485] RAX: ffff88007da85638 RBX: ffff880078c6eec0 RCX:
ffff880078c6f000
[ 105.392531] RDX: ffff880078b54170 RSI: ffff880078c6eec0 RDI:
ffff880078479280
[ 105.392576] RBP: ffff88007869c6c0 R08: 0000000000000dc0 R09:
0000000000000000
[ 105.392622] R10: 0000000000000000 R11: 0000000000000001 R12:
0000000000000011
[ 105.392667] R13: 000000000000b826 R14: ffff88007b403dd8 R15:
ffff880078428800
[ 105.393781] FS: 00007f11b3900700(0000) GS:ffff88007b400000(0000)
knlGS:0000000000000000
[ 105.394907] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 105.395840] CR2: ffff880078c6f000 CR3: 00000000797f5000 CR4:
0000000000002660
[ 105.395840] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 105.395840] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 105.395840] Process swapper/0 (pid: 0, threadinfo ffffffff808b8000, task
ffffffff808c3960)
[ 105.395840] Stack:
[ 105.395840] ffffffff00000269 0000002200000022 ffff8800784288d0
ffff88007b403db8
[ 105.395840] ffff88007b403d98 000000150000b815 ffff880078429d50
0000000000000000
[ 105.395840] ffff880078428858 ffff88007b40e878 0000000000000012
000000400000b82a
[ 105.395840] Call Trace:
[ 105.395840] [<ffffffff804419c0>] net_rx_action+0x170/0x2c0
[ 105.395840] [<ffffffff80036dae>] __do_softirq+0xee/0x230
[ 105.395840] [<ffffffff80037075>] irq_exit+0x95/0xa0
[ 105.395840] [<ffffffff803937cd>] evtchn_do_upcall+0x2ad/0x2f0
[ 105.395840] [<ffffffff80535a0e>] do_hypervisor_callback+0x1e/0x30
[ 105.395840] [<ffffffff800033aa>] HYPERVISOR_sched_op_new+0xa/0x20
[ 105.395840] [<ffffffff8000e671>] xen_idle+0x41/0x110
[ 105.395840] [<ffffffff8000e7ef>] cpu_idle+0xaf/0x110
[ 105.395840] [<ffffffff80946b1f>] start_kernel+0x424/0x42f
[ 105.395840] Code: 44 21 ea 48 8d 54 d0 40 8b 87 d8 00 00 00 44 0f bf
42 06 44 0f b7 4a 02 48 8b 44 01 30 49 63 cc 48 83 c1 03 48 c1 e1 04 48
01 f1 <48> 89 01 44 89 49 08 44 89 41 0c 48 8b 08 80 e5 80 0f 85 54 09
[ 105.395840] RIP [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
[ 105.395840] RSP <ffff88007b403d18>
[ 105.395840] CR2: ffff880078c6f000
[ 105.395840] ---[ end trace 67dcc8bdc485cab5 ]---
[ 105.395840] Kernel panic - not syncing: Fatal exception in interrupt
--
*/Best regards,/*
/Eugene Istomin/
> >>> On 17.05.13 at 15:00, Eugene Istomin <e.istomin@edss.ee>
wrote:
> > Bump, here it is:
> Okay, but I think we're still lacking information on what your
> Dom0 kernel is.
>
> Ian, Wei - looking at the forward ported as well as the upstream
> frontends, I'm wondering if there isn't a fundamental flaw in
> ..._get_responses(): It allows up to MAX_SKB_FRAGS + 1 slots/
> frags, and ..._fill_frags() then fills as many fragments as got
> queued onto the respective list. Only after both of them are done,
> __pskb_pull_tail() gets invoked reducing the fragment count by
> one if the condition is met that made ..._get_responses() bump
[-- Attachment #1.2: Type: text/html, Size: 28734 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-21 11:09 ` Eugene Istomin
@ 2013-05-21 11:49 ` Jan Beulich
2013-05-21 16:19 ` Eugene Istomin
0 siblings, 1 reply; 85+ messages in thread
From: Jan Beulich @ 2013-05-21 11:49 UTC (permalink / raw)
To: Eugene Istomin; +Cc: Wei Liu, Ian Campbell, xen-devel
>>> On 21.05.13 at 13:09, Eugene Istomin <e.istomin@edss.ee> wrote:
> i updated to 3.9.3 from stable branch
I think I had pointed this out already - I'm not sure the stable branch
is really being kept up to date wrt Xen patches. I'm only updating
the master branch, so if you really don't want to stick to released
kernels, for Xen it may be better if you track the master branch (and
simply don't update as long as that one's on *-rc* for the next
upstream version).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-21 9:55 ` Jan Beulich
2013-05-21 11:09 ` Eugene Istomin
@ 2013-05-21 15:28 ` Wei Liu
2013-05-31 15:47 ` Wei Liu
1 sibling, 1 reply; 85+ messages in thread
From: Wei Liu @ 2013-05-21 15:28 UTC (permalink / raw)
To: Jan Beulich; +Cc: Eugene Istomin, Wei Liu, Ian Campbell, xen-devel
On Tue, May 21, 2013 at 10:55:00AM +0100, Jan Beulich wrote:
> >>> On 17.05.13 at 15:00, Eugene Istomin <e.istomin@edss.ee> wrote:
> > Bump, here it is:
>
> Okay, but I think we're still lacking information on what your
> Dom0 kernel is.
>
> Ian, Wei - looking at the forward ported as well as the upstream
> frontends, I'm wondering if there isn't a fundamental flaw in
> ..._get_responses(): It allows up to MAX_SKB_FRAGS + 1 slots/
> frags, and ..._fill_frags() then fills as many fragments as got
> queued onto the respective list. Only after both of them are done,
> __pskb_pull_tail() gets invoked reducing the fragment count by
> one if the condition is met that made ..._get_responses() bump
> the limit by one.
>
> Am I overlooking something? I'm asking because working through
> disassembly and register values of the dump Eugene had sent I
> clearly see that ..._fill_frags() is processing the 18th fragment,
> while in that kernel version MAX_SKB_FRAGS is only 17 (but
> possibly, hence the first question above, the Dom0 kernel still is
> one with MAX_SKB_FRAGS being 18, or the packet turns out to
I have the same suspection. However I just did a quick hack to bump
MAX_SKB_FRAGS to 20 in backend, DomU worked just fine.
> be such that it fills 17 fragments and the header is smaller than
> RX_COPY_THRESHOLD).
It would be better to know more info about Dom0. Will need some time to
figure out what's going on there.
Wei.
> Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-21 11:49 ` Jan Beulich
@ 2013-05-21 16:19 ` Eugene Istomin
2013-05-31 3:37 ` Eugene Istomin
0 siblings, 1 reply; 85+ messages in thread
From: Eugene Istomin @ 2013-05-21 16:19 UTC (permalink / raw)
To: Jan Beulich, Wei Liu; +Cc: Ian Campbell, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 908 bytes --]
Jan,
thanks for the explanation,
right now xen code and patches in stable branch
(http://kernel.opensuse.org/cgit/kernel/log/?h=stable&qt=grep&q=xen) is
in-sync with master
(http://kernel.opensuse.org/cgit/kernel/log/?qt=grep&q=xen)
Wei,
>>It would be better to know more info about Dom0.
i added Dom0 "xl info" in previous mail, do you need any other info?
--
*/Best regards,/*
/Eugene Istomin/
> >>> On 21.05.13 at 13:09, Eugene Istomin <e.istomin@edss.ee>
wrote:
> > i updated to 3.9.3 from stable branch
>
> I think I had pointed this out already - I'm not sure the stable branch
> is really being kept up to date wrt Xen patches. I'm only updating
> the master branch, so if you really don't want to stick to released
> kernels, for Xen it may be better if you track the master branch (and
> simply don't update as long as that one's on *-rc* for the next
> upstream version).
>
> Jan
[-- Attachment #1.2: Type: text/html, Size: 6910 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-21 16:19 ` Eugene Istomin
@ 2013-05-31 3:37 ` Eugene Istomin
2013-05-31 6:19 ` Jan Beulich
0 siblings, 1 reply; 85+ messages in thread
From: Eugene Istomin @ 2013-05-31 3:37 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, Ian Campbell, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 1029 bytes --]
Hello Jan,
do you need any other info? Can i help with this bug resolving?
--
*/Best regards,/*
/Eugene Istomin/
Jan,
thanks for the explanation,
right now xen code and patches in stable branch
(http://kernel.opensuse.org/cgit/kernel/log/?h=stable&qt=grep&q=xen) is
in-sync with master
(http://kernel.opensuse.org/cgit/kernel/log/?qt=grep&q=xen)
Wei,
>>It would be better to know more info about Dom0.
i added Dom0 "xl info" in previous mail, do you need any other info?
--
*/Best regards,/*
/Eugene Istomin/
> >>> On 21.05.13 at 13:09, Eugene Istomin <e.istomin@edss.ee>
wrote:
> > i updated to 3.9.3 from stable branch
>
> I think I had pointed this out already - I'm not sure the stable branch
> is really being kept up to date wrt Xen patches. I'm only updating
> the master branch, so if you really don't want to stick to released
> kernels, for Xen it may be better if you track the master branch (and
> simply don't update as long as that one's on *-rc* for the next
> upstream version).
>
> Jan
[-- Attachment #1.2: Type: text/html, Size: 9155 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-31 3:37 ` Eugene Istomin
@ 2013-05-31 6:19 ` Jan Beulich
2013-05-31 6:22 ` Eugene Istomin
2013-05-31 8:20 ` Wei Liu
0 siblings, 2 replies; 85+ messages in thread
From: Jan Beulich @ 2013-05-31 6:19 UTC (permalink / raw)
To: Eugene Istomin; +Cc: Wei Liu, Ian Campbell, xen-devel
>>> On 31.05.13 at 05:37, Eugene Istomin <e.istomin@edss.ee> wrote:
> do you need any other info? Can i help with this bug resolving?
No, it's a time allocation thing. For one I'm waiting for Wei to find
time to look at whether the issue I spotted is a real one. And I'm
myself short on time preventing me to get to merging the grant
copy changes into our netback. Unless you can do either of these,
you'll have to be patient I'm afraid.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-31 6:19 ` Jan Beulich
@ 2013-05-31 6:22 ` Eugene Istomin
2013-05-31 8:20 ` Wei Liu
1 sibling, 0 replies; 85+ messages in thread
From: Eugene Istomin @ 2013-05-31 6:22 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, Ian Campbell, xen-devel
[-- Attachment #1.1: Type: text/plain, Size: 523 bytes --]
Ok,
thank you.
--
*/Best regards,/*
/Eugene Istomin/
> >>> On 31.05.13 at 05:37, Eugene Istomin <e.istomin@edss.ee>
wrote:
> > do you need any other info? Can i help with this bug resolving?
>
> No, it's a time allocation thing. For one I'm waiting for Wei to find
> time to look at whether the issue I spotted is a real one. And I'm
> myself short on time preventing me to get to merging the grant
> copy changes into our netback. Unless you can do either of these,
> you'll have to be patient I'm afraid.
>
> Jan
[-- Attachment #1.2: Type: text/html, Size: 4065 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-31 6:19 ` Jan Beulich
2013-05-31 6:22 ` Eugene Istomin
@ 2013-05-31 8:20 ` Wei Liu
1 sibling, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-05-31 8:20 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, Eugene Istomin, Ian Campbell, xen-devel
On Fri, May 31, 2013 at 07:19:51AM +0100, Jan Beulich wrote:
> >>> On 31.05.13 at 05:37, Eugene Istomin <e.istomin@edss.ee> wrote:
> > do you need any other info? Can i help with this bug resolving?
>
> No, it's a time allocation thing. For one I'm waiting for Wei to find
> time to look at whether the issue I spotted is a real one. And I'm
> myself short on time preventing me to get to merging the grant
> copy changes into our netback. Unless you can do either of these,
> you'll have to be patient I'm afraid.
>
Sorry for the delay. I might be able to allocate some time today to have
a look at the issue you spotted.
Wei.
> Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-21 15:28 ` Wei Liu
@ 2013-05-31 15:47 ` Wei Liu
2013-05-31 16:00 ` Jan Beulich
0 siblings, 1 reply; 85+ messages in thread
From: Wei Liu @ 2013-05-31 15:47 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, Eugene Istomin, Ian Campbell, xen-devel
On Tue, May 21, 2013 at 04:28:47PM +0100, Wei Liu wrote:
> On Tue, May 21, 2013 at 10:55:00AM +0100, Jan Beulich wrote:
> > >>> On 17.05.13 at 15:00, Eugene Istomin <e.istomin@edss.ee> wrote:
> > > Bump, here it is:
> >
> > Okay, but I think we're still lacking information on what your
> > Dom0 kernel is.
> >
> > Ian, Wei - looking at the forward ported as well as the upstream
> > frontends, I'm wondering if there isn't a fundamental flaw in
> > ..._get_responses(): It allows up to MAX_SKB_FRAGS + 1 slots/
> > frags, and ..._fill_frags() then fills as many fragments as got
> > queued onto the respective list. Only after both of them are done,
> > __pskb_pull_tail() gets invoked reducing the fragment count by
> > one if the condition is met that made ..._get_responses() bump
> > the limit by one.
> >
> > Am I overlooking something? I'm asking because working through
> > disassembly and register values of the dump Eugene had sent I
> > clearly see that ..._fill_frags() is processing the 18th fragment,
>
> > while in that kernel version MAX_SKB_FRAGS is only 17 (but
> > possibly, hence the first question above, the Dom0 kernel still is
> > one with MAX_SKB_FRAGS being 18, or the packet turns out to
>
> I have the same suspection. However I just did a quick hack to bump
> MAX_SKB_FRAGS to 20 in backend, DomU worked just fine.
>
> > be such that it fills 17 fragments and the header is smaller than
> > RX_COPY_THRESHOLD).
>
Eugene,
I have more or less the same suspension as Jan did, but as I could not
effectively reproduce this problem upstream, could you please apply
following patch to opensuse kernel and see what happens.
Thanks
Wei.
-----
diff --git a/drivers/xen/netfront/netfront.c b/drivers/xen/netfront/netfront.c
index 6e5d233..ab7e35b 100644
--- a/drivers/xen/netfront/netfront.c
+++ b/drivers/xen/netfront/netfront.c
@@ -1319,6 +1319,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
kfree_skb(nskb);
nr_frags++;
+ BUG_ON(nr_frags >= MAX_SKB_FRAGS);
}
shinfo->nr_frags = nr_frags;
^ permalink raw reply related [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-31 15:47 ` Wei Liu
@ 2013-05-31 16:00 ` Jan Beulich
2013-05-31 16:05 ` Wei Liu
0 siblings, 1 reply; 85+ messages in thread
From: Jan Beulich @ 2013-05-31 16:00 UTC (permalink / raw)
To: Wei Liu; +Cc: Eugene Istomin, IanCampbell, xen-devel
>>> On 31.05.13 at 17:47, Wei Liu <wei.liu2@citrix.com> wrote:
> --- a/drivers/xen/netfront/netfront.c
> +++ b/drivers/xen/netfront/netfront.c
> @@ -1319,6 +1319,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
> kfree_skb(nskb);
>
> nr_frags++;
> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
Isn't this going to see false positives? I would have thought such a
check would need to go at the top of the loop body.
Jan
> }
>
> shinfo->nr_frags = nr_frags;
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-31 16:00 ` Jan Beulich
@ 2013-05-31 16:05 ` Wei Liu
0 siblings, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-05-31 16:05 UTC (permalink / raw)
To: Jan Beulich; +Cc: Eugene Istomin, Wei Liu, IanCampbell, xen-devel
On Fri, May 31, 2013 at 05:00:32PM +0100, Jan Beulich wrote:
> >>> On 31.05.13 at 17:47, Wei Liu <wei.liu2@citrix.com> wrote:
> > --- a/drivers/xen/netfront/netfront.c
> > +++ b/drivers/xen/netfront/netfront.c
> > @@ -1319,6 +1319,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
> > kfree_skb(nskb);
> >
> > nr_frags++;
> > + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
>
> Isn't this going to see false positives? I would have thought such a
> check would need to go at the top of the loop body.
>
> Jan
>
Ha, yes.
Please use this one.
Wei.
-----
diff --git a/drivers/xen/netfront/netfront.c b/drivers/xen/netfront/netfront.c
index 6e5d233..9583011 100644
--- a/drivers/xen/netfront/netfront.c
+++ b/drivers/xen/netfront/netfront.c
@@ -1306,6 +1306,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
struct sk_buff *nskb;
while ((nskb = __skb_dequeue(list))) {
+ BUG_ON(nr_frags >= MAX_SKB_FRAGS);
struct netif_rx_response *rx =
RING_GET_RESPONSE(&np->rx, ++cons);
^ permalink raw reply related [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-05-17 8:59 ` [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Wei Liu
2013-05-17 9:08 ` Jan Beulich
2013-05-17 9:37 ` Eugene Istomin
@ 2013-07-04 13:43 ` Dion Kant
2013-07-04 14:18 ` Jan Beulich
2013-07-04 15:01 ` Wei Liu
2 siblings, 2 replies; 85+ messages in thread
From: Dion Kant @ 2013-07-04 13:43 UTC (permalink / raw)
To: xen-users, xen-devel
Hello Wei and all other interested people,
I saw this thread from around May. It got silent on this thread after
your post on May 31.
Is there any progress on this problem?
I am running into this issue as well with the openSUSE 12.3
distribution. This is with their 3.7.10-1.16-xen kernel and Xen version
4.2.1_12-1.12.10. On the net I see some discussion of people hitting
this issue but not that much. E.g., one of the symptoms is that a guest
crashes when running zypper install or zypper update when the Internet
connection is fast enough.
OpenSUSE 3.4.X kernels are running ok as guest on top of the openSUSE
12.3 Xen distribution, but apparently since 3.7.10 and higher there is
this issue.
I spent already quite some time in getting grip on the issue. I added a
bug to bugzilla.novell.com but no response. See
https://bugzilla.novell.com/show_bug.cgi?id=826374 for details.
Apparently for hitting this bug (i.e. make it all the way to the crash),
it is required to use some hardware which performs not too slow. With
this I mean it is easy to find hardware which is unable to reproduce the
issue.
In one of my recent experiments I changed the SLAB allocater to SLUB
which provides more detailed kernel logging. Here is the log output
after the first detected issue regarding xennet:
2013-07-03T23:51:16.560229+02:00 domUA kernel: [ 97.562370] netfront:
Too many frags
2013-07-03T23:51:17.228143+02:00 domUA kernel: [ 98.230466] netfront:
Too many frags
2013-07-03T23:51:17.596074+02:00 domUA kernel: [ 98.597300] netfront:
Too many frags
2013-07-03T23:51:18.740215+02:00 domUA kernel: [ 99.743080]
net_ratelimit: 2 callbacks suppressed
2013-07-03T23:51:18.740242+02:00 domUA kernel: [ 99.743084] netfront:
Too many frags
2013-07-03T23:51:19.104100+02:00 domUA kernel: [ 100.104281] netfront:
Too many frags
2013-07-03T23:51:19.760134+02:00 domUA kernel: [ 100.760594] netfront:
Too many frags
2013-07-03T23:51:21.820154+02:00 domUA kernel: [ 102.821202] netfront:
Too many frags
2013-07-03T23:51:22.192188+02:00 domUA kernel: [ 103.192655] netfront:
Too many frags
2013-07-03T23:51:26.060144+02:00 domUA kernel: [ 107.062447] netfront:
Too many frags
2013-07-03T23:51:26.412116+02:00 domUA kernel: [ 107.415165] netfront:
Too many frags
2013-07-03T23:51:27.092147+02:00 domUA kernel: [ 108.094615] netfront:
Too many frags
2013-07-03T23:51:27.492112+02:00 domUA kernel: [ 108.494255] netfront:
Too many frags
2013-07-03T23:51:27.520194+02:00 domUA kernel: [ 108.522445]
=============================================================================
2013-07-03T23:51:27.520206+02:00 domUA kernel: [ 108.522448] BUG
kmalloc-1024 (Tainted: G W ): Redzone overwritten
2013-07-03T23:51:27.520209+02:00 domUA kernel: [ 108.522450]
-----------------------------------------------------------------------------
2013-07-03T23:51:27.520212+02:00 domUA kernel: [ 108.522450]
2013-07-03T23:51:27.520215+02:00 domUA kernel: [ 108.522452] Disabling
lock debugging due to kernel taint
2013-07-03T23:51:27.520217+02:00 domUA kernel: [ 108.522454] INFO:
0xffff8800f66068f8-0xffff8800f66068ff. First byte 0x0 instead of 0xcc
2013-07-03T23:51:27.520220+02:00 domUA kernel: [ 108.522461] INFO:
Allocated in __alloc_skb+0x88/0x260 age=11 cpu=0 pid=1325
2013-07-03T23:51:27.520223+02:00 domUA kernel: [ 108.522466]
set_track+0x6c/0x190
2013-07-03T23:51:27.520225+02:00 domUA kernel: [ 108.522470]
alloc_debug_processing+0x83/0x109
2013-07-03T23:51:27.520228+02:00 domUA kernel: [ 108.522472]
__slab_alloc.constprop.48+0x523/0x593
2013-07-03T23:51:27.520231+02:00 domUA kernel: [ 108.522474]
__kmalloc_track_caller+0xb4/0x200
2013-07-03T23:51:27.520233+02:00 domUA kernel: [ 108.522477]
__kmalloc_reserve+0x3c/0xa0
2013-07-03T23:51:27.520236+02:00 domUA kernel: [ 108.522478]
__alloc_skb+0x88/0x260
2013-07-03T23:51:27.520239+02:00 domUA kernel: [ 108.522483]
network_alloc_rx_buffers+0x76/0x5f0 [xennet]
2013-07-03T23:51:27.520241+02:00 domUA kernel: [ 108.522486]
netif_poll+0xcf4/0xf30 [xennet]
2013-07-03T23:51:27.520243+02:00 domUA kernel: [ 108.522489]
net_rx_action+0xf0/0x2e0
2013-07-03T23:51:27.520246+02:00 domUA kernel: [ 108.522493]
__do_softirq+0x127/0x280
2013-07-03T23:51:27.520248+02:00 domUA kernel: [ 108.522496]
call_softirq+0x1c/0x30
2013-07-03T23:51:27.520251+02:00 domUA kernel: [ 108.522499]
do_softirq+0x56/0xd0
2013-07-03T23:51:27.520253+02:00 domUA kernel: [ 108.522501]
irq_exit+0x52/0xd0
2013-07-03T23:51:27.520256+02:00 domUA kernel: [ 108.522503]
evtchn_do_upcall+0x281/0x2e7
2013-07-03T23:51:27.520258+02:00 domUA kernel: [ 108.522505]
do_hypervisor_callback+0x1e/0x30
2013-07-03T23:51:27.520261+02:00 domUA kernel: [ 108.522507]
0x7f45f0a2f1e0
2013-07-03T23:51:27.520263+02:00 domUA kernel: [ 108.522509] INFO:
Freed in skb_free_head+0x5c/0x70 age=14 cpu=0 pid=1325
2013-07-03T23:51:27.520266+02:00 domUA kernel: [ 108.522512]
set_track+0x6c/0x190
2013-07-03T23:51:27.520269+02:00 domUA kernel: [ 108.522513]
free_debug_processing+0x151/0x201
2013-07-03T23:51:27.520271+02:00 domUA kernel: [ 108.522515]
__slab_free+0x47/0x499
2013-07-03T23:51:27.520274+02:00 domUA kernel: [ 108.522517]
kfree+0x1df/0x230
2013-07-03T23:51:27.520276+02:00 domUA kernel: [ 108.522519]
skb_free_head+0x5c/0x70
2013-07-03T23:51:27.520279+02:00 domUA kernel: [ 108.522521]
skb_release_data+0xea/0xf0
2013-07-03T23:51:27.520281+02:00 domUA kernel: [ 108.522522]
__kfree_skb+0x1e/0xb0
2013-07-03T23:51:27.520284+02:00 domUA kernel: [ 108.522524]
kfree_skb+0x80/0xc0
2013-07-03T23:51:27.520286+02:00 domUA kernel: [ 108.522527]
netif_poll+0x824/0xf30 [xennet]
2013-07-03T23:51:27.520289+02:00 domUA kernel: [ 108.522529]
net_rx_action+0xf0/0x2e0
2013-07-03T23:51:27.520291+02:00 domUA kernel: [ 108.522530]
__do_softirq+0x127/0x280
2013-07-03T23:51:27.520294+02:00 domUA kernel: [ 108.522532]
call_softirq+0x1c/0x30
2013-07-03T23:51:27.520296+02:00 domUA kernel: [ 108.522534]
do_softirq+0x56/0xd0
2013-07-03T23:51:27.520299+02:00 domUA kernel: [ 108.522536]
irq_exit+0x52/0xd0
2013-07-03T23:51:27.520302+02:00 domUA kernel: [ 108.522538]
evtchn_do_upcall+0x281/0x2e7
2013-07-03T23:51:27.520304+02:00 domUA kernel: [ 108.522539]
do_hypervisor_callback+0x1e/0x30
2013-07-03T23:51:27.520307+02:00 domUA kernel: [ 108.522541] INFO: Slab
0xffff8800ffd78100 objects=12 used=7 fp=0xffff8800f66074d0
flags=0x400000000000408
2013-07-03T23:51:27.520310+02:00 domUA kernel: [ 108.522543] INFO:
Object 0xffff8800f66064f8 @offset=9464 fp=0x0000018800000000
2013-07-03T23:51:27.520312+02:00 domUA kernel: [ 108.522543]
2013-07-03T23:51:27.520315+02:00 domUA kernel: [ 108.522546] Bytes b4
ffff8800f66064e8: 4a 40 ff ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a
J@......ZZZZZZZZ
2013-07-03T23:51:27.520318+02:00 domUA kernel: [ 108.522548] Object
ffff8800f66064f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
2013-07-03T23:51:27.520320+02:00 domUA kernel: [ 108.522549] Object
ffff8800f6606508: 00 16 3e 29 7e 3c 00 25 90 69 ea 4e 08 00 45 08
..>)~<.%.i.N..E.
2013-07-03T23:51:27.520323+02:00 domUA kernel: [ 108.522551] Object
ffff8800f6606518: fe bc 46 d7 40 00 40 06 d3 69 0a 57 06 91 0a 57
..F.@.@..i.W...W
2013-07-03T23:51:27.520326+02:00 domUA kernel: [ 108.522553] Object
ffff8800f6606528: 06 b4 9b 86 00 16 57 4d 5e bd 89 4c 40 ad 80 10
......WM^..L@...
2013-07-03T23:51:27.520329+02:00 domUA kernel: [ 108.522554] Object
ffff8800f6606538: 00 a6 20 a2 00 00 01 01 08 0a 01 eb 40 a7 ff ff ..
.........@...
2013-07-03T23:51:27.520332+02:00 domUA kernel: [ 108.522556] Object
ffff8800f6606548: 44 fa 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
D.kkkkkkkkkkkkkk
2013-07-03T23:51:27.520335+02:00 domUA kernel: [ 108.522557] Object
ffff8800f6606558: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
2013-07-03T23:51:27.520337+02:00 domUA kernel: [ 108.522559] Object
ffff8800f6606568: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
Skipping some of the object dumping.......
2013-07-03T23:51:27.520583+02:00 domUA kernel: [ 108.522644] Object
ffff8800f66068d8: 00 d7 e4 ff 00 88 ff ff 00 00 00 00 00 10 00 00
................
2013-07-03T23:51:27.520586+02:00 domUA kernel: [ 108.522646] Object
ffff8800f66068e8: 00 92 dd ff 00 88 ff ff 00 00 00 00 88 01 00 00
................
2013-07-03T23:51:27.520588+02:00 domUA kernel: [ 108.522647] Redzone
ffff8800f66068f8: 00 92 dd ff 00 88 ff ff ........
2013-07-03T23:51:27.520591+02:00 domUA kernel: [ 108.522649] Padding
ffff8800f6606a38: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
2013-07-03T23:51:27.520594+02:00 domUA kernel: [ 108.522651] Pid: 1325,
comm: sshd Tainted: G B W 3.7.10-1.16-dbg-xen #3
2013-07-03T23:51:27.520597+02:00 domUA kernel: [ 108.522652] Call Trace:
2013-07-03T23:51:27.520599+02:00 domUA kernel: [ 108.522658]
[<ffffffff8000b097>] try_stack_unwind+0x87/0x1c0
2013-07-03T23:51:27.520602+02:00 domUA kernel: [ 108.522662]
[<ffffffff80008fa5>] dump_trace+0xd5/0x250
2013-07-03T23:51:27.520605+02:00 domUA kernel: [ 108.522665]
[<ffffffff8000b22c>] show_trace_log_lvl+0x5c/0x80
2013-07-03T23:51:27.520608+02:00 domUA kernel: [ 108.522668]
[<ffffffff8000b265>] show_trace+0x15/0x20
2013-07-03T23:51:27.520610+02:00 domUA kernel: [ 108.522672]
[<ffffffff80553a69>] dump_stack+0x77/0x80
2013-07-03T23:51:27.520612+02:00 domUA kernel: [ 108.522676]
[<ffffffff801491b1>] print_trailer+0x131/0x140
2013-07-03T23:51:27.520615+02:00 domUA kernel: [ 108.522680]
[<ffffffff80149709>] check_bytes_and_report+0xc9/0x120
2013-07-03T23:51:27.520617+02:00 domUA kernel: [ 108.522683]
[<ffffffff8014a7f6>] check_object+0x56/0x240
2013-07-03T23:51:27.520620+02:00 domUA kernel: [ 108.522687]
[<ffffffff805575b6>] free_debug_processing+0xc4/0x201
2013-07-03T23:51:27.520622+02:00 domUA kernel: [ 108.522690]
[<ffffffff8055773a>] __slab_free+0x47/0x499
2013-07-03T23:51:27.520625+02:00 domUA kernel: [ 108.522694]
[<ffffffff8014beff>] kfree+0x1df/0x230
2013-07-03T23:51:27.520627+02:00 domUA kernel: [ 108.522697]
[<ffffffff8044a8cc>] skb_free_head+0x5c/0x70
2013-07-03T23:51:27.520630+02:00 domUA kernel: [ 108.522701]
[<ffffffff8044a9ca>] skb_release_data+0xea/0xf0
2013-07-03T23:51:27.520632+02:00 domUA kernel: [ 108.522704]
[<ffffffff8044a9ee>] __kfree_skb+0x1e/0xb0
2013-07-03T23:51:27.520635+02:00 domUA kernel: [ 108.522709]
[<ffffffff8049fa2a>] tcp_recvmsg+0x99a/0xd50
2013-07-03T23:51:27.520637+02:00 domUA kernel: [ 108.522714]
[<ffffffff804c796d>] inet_recvmsg+0xed/0x110
2013-07-03T23:51:27.520640+02:00 domUA kernel: [ 108.522718]
[<ffffffff80440be8>] sock_aio_read+0x158/0x190
2013-07-03T23:51:27.520642+02:00 domUA kernel: [ 108.522722]
[<ffffffff8015cb68>] do_sync_read+0x98/0xf0
2013-07-03T23:51:27.520645+02:00 domUA kernel: [ 108.522726]
[<ffffffff8015d32d>] vfs_read+0xbd/0x180
2013-07-03T23:51:27.520647+02:00 domUA kernel: [ 108.522729]
[<ffffffff8015d442>] sys_read+0x52/0xa0
2013-07-03T23:51:27.520650+02:00 domUA kernel: [ 108.522733]
[<ffffffff8056ab3b>] system_call_fastpath+0x1a/0x1f
2013-07-03T23:51:27.520652+02:00 domUA kernel: [ 108.522736]
[<00007f45ef74c960>] 0x7f45ef74c95f
2013-07-03T23:51:27.520655+02:00 domUA kernel: [ 108.522738] FIX
kmalloc-1024: Restoring 0xffff8800f66068f8-0xffff8800f66068ff=0xcc
2013-07-03T23:51:27.520657+02:00 domUA kernel: [ 108.522738]
2013-07-03T23:51:27.679444+02:00 domUA kernel: [ 108.671750]
=============================================================================
2013-07-03T23:51:27.679454+02:00 domUA kernel: [ 108.671753] BUG
kmalloc-1024 (Tainted: G B W ): Redzone overwritten
2013-07-03T23:51:27.679456+02:00 domUA kernel: [ 108.671754]
-----------------------------------------------------------------------------
2013-07-03T23:51:27.679458+02:00 domUA kernel: [ 108.671754]
2013-07-03T23:51:27.679460+02:00 domUA kernel: [ 108.671757] INFO:
0xffff8800f66068f8-0xffff8800f66068ff. First byte 0xcc instead of 0xbb
2013-07-03T23:51:27.679462+02:00 domUA kernel: [ 108.671762] INFO:
Allocated in __alloc_skb+0x88/0x260 age=48 cpu=0 pid=1325
2013-07-03T23:51:27.679464+02:00 domUA kernel: [ 108.671765]
set_track+0x6c/0x190
2013-07-03T23:51:27.679466+02:00 domUA kernel: [ 108.671767]
alloc_debug_processing+0x83/0x109
2013-07-03T23:51:27.679468+02:00 domUA kernel: [ 108.671769]
__slab_alloc.constprop.48+0x523/0x593
2013-07-03T23:51:27.679469+02:00 domUA kernel: [ 108.671771]
__kmalloc_track_caller+0xb4/0x200
2013-07-03T23:51:27.679471+02:00 domUA kernel: [ 108.671773]
__kmalloc_reserve+0x3c/0xa0
2013-07-03T23:51:27.679473+02:00 domUA kernel: [ 108.671775]
__alloc_skb+0x88/0x260
2013-07-03T23:51:27.679475+02:00 domUA kernel: [ 108.671778]
network_alloc_rx_buffers+0x76/0x5f0 [xennet]
2013-07-03T23:51:27.679476+02:00 domUA kernel: [ 108.671781]
netif_poll+0xcf4/0xf30 [xennet]
2013-07-03T23:51:27.679478+02:00 domUA kernel: [ 108.671783]
net_rx_action+0xf0/0x2e0
I noticed that after turning on all this debugging stuff, a real panic
does not appear any more.
This happens while copying a file with scp from dom0 to the guest (scp
bigfile domu:/dev/null).
In my lab, I am currently experimenting with a SuperMicro based system
with Xen showing the following characteristics:
__ __ _ _ ____ _ _ ____ _ _ ____ _ ___
\ \/ /___ _ __ | || | |___ \ / | / |___ \ / | / |___ \ / |/ _ \
\ // _ \ '_ \ | || |_ __) | | | | | __) |__| | | | __) | | | | | |
/ \ __/ | | | |__ _| / __/ _| | | |/ __/|__| |_| |/ __/ _| | |_| |
/_/\_\___|_| |_| |_|(_)_____(_)_|___|_|_____| |_(_)_|_____(_)_|\___/
|_____|
(XEN) Xen version 4.2.1_12-1.12.10 (abuild@) (gcc (SUSE Linux) 4.7.2
20130108 [gcc-4_7-branch revision 195012]) Wed May 29 20:31:49 UTC 2013
(XEN) Latest ChangeSet: 25952
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: dom0_mem=2048M,max:2048M loglvl=all guest_loglvl=all
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 4 MBR signatures
(XEN) Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 0000000000096400 (usable)
(XEN) 0000000000096400 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000bf780000 (usable)
(XEN) 00000000bf78e000 - 00000000bf790000 type 9
(XEN) 00000000bf790000 - 00000000bf79e000 (ACPI data)
(XEN) 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
(XEN) 00000000bf7d0000 - 00000000bf7e0000 (reserved)
(XEN) 00000000bf7ec000 - 00000000c0000000 (reserved)
(XEN) 00000000e0000000 - 00000000f0000000 (reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 00000000ffc00000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000340000000 (usable)
Skipping ACPI en SRAT
(XEN) System RAM: 12279MB (12573784kB)
(XEN) NUMA: Allocated memnodemap from 33e38a000 - 33e38e000
(XEN) NUMA: Using 8 for the hash shift.
(XEN) Domain heap initialised DMA width 30 bits
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Enabling APIC mode: Phys. Using 2 I/O APICs
(XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
(XEN) Failed to get Error Log Address Range.
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 24 CPUs (8 hotplug CPUs)
(XEN) IRQ limits: 48 GSI, 3040 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2400.115 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:1238: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0
extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 128 KiB.
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 16 CPUs
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, lsb, paddr 0x2000 -> 0xa65000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000336000000->0000000337000000 (516915 pages
to be allocated)
(XEN) Init. ramdisk: 000000033f333000->0000000340000000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff80002000->ffffffff80a65000
(XEN) Init. ramdisk: 0000000000000000->0000000000000000
(XEN) Phys-Mach map: ffffea0000000000->ffffea0000400000
(XEN) Start info: ffffffff80a65000->ffffffff80a654b4
(XEN) Page tables: ffffffff80a66000->ffffffff80a6f000
(XEN) Boot stack: ffffffff80a6f000->ffffffff80a70000
(XEN) TOTAL: ffffffff80000000->ffffffff80c00000
(XEN) ENTRY ADDRESS: ffffffff80002000
(XEN) Dom0 has maximum 16 VCPUs
(XEN) Scrubbing Free RAM:
.....................................................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) ACPI: RSDP 000FACE0, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT BF790100, 008C (r1 SMCI 20110827 MSFT 97)
(XEN) ACPI: FACP BF790290, 00F4 (r4 082711 FACP1638 20110827 MSFT 97)
(XEN) ACPI: DSDT BF7906A0, 6563 (r2 10600 10600000 0 INTL 20051117)
(XEN) ACPI: FACS BF79E000, 0040
(XEN) ACPI: APIC BF790390, 011E (r2 082711 APIC1638 20110827 MSFT 97)
(XEN) ACPI: MCFG BF7904B0, 003C (r1 082711 OEMMCFG 20110827 MSFT 97)
(XEN) ACPI: SLIT BF7904F0, 0030 (r1 082711 OEMSLIT 20110827 MSFT 97)
(XEN) ACPI: OEMB BF79E040, 0085 (r1 082711 OEMB1638 20110827 MSFT 97)
(XEN) ACPI: SRAT BF79A6A0, 01D0 (r2 082711 OEMSRAT 1 INTL 1)
(XEN) ACPI: HPET BF79A870, 0038 (r1 082711 OEMHPET 20110827 MSFT 97)
(XEN) ACPI: DMAR BF79E0D0, 0130 (r1 AMI OEMDMAR 1 MSFT 97)
(XEN) ACPI: SSDT BF7A1B30, 0363 (r1 DpgPmm CpuPm 12 INTL 20051117)
(XEN) ACPI: EINJ BF79A8B0, 0130 (r1 AMIER AMI_EINJ 20110827 MSFT 97)
(XEN) ACPI: BERT BF79AA40, 0030 (r1 AMIER AMI_BERT 20110827 MSFT 97)
(XEN) ACPI: ERST BF79AA70, 01B0 (r1 AMIER AMI_ERST 20110827 MSFT 97)
(XEN) ACPI: HEST BF79AC20, 00A8 (r1 AMIER ABC_HEST 20110827 MSFT 97)
(XEN) System RAM: 12279MB (12573784kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 20 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 19 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 21 -> Node 0
I am happy to assist in more kernel probing. It is even possible for me
to setup access for someone to this machine.
Best regards,
Dion Kant
On 05/17/2013 10:59 AM, Wei Liu wrote:
> Moving discussion to Xen-devel
>
> On Thu, May 16, 2013 at 10:29:56PM +0300, Eugene Istomin wrote:
>> Hello,
>>
>> I tried to use 3.9.2 kernel with xen 4.2.2/4.3rc1 and in both variants leads
>> to this error in network-intensive load (such as iperf, 100 nginx parallel
>> requests to 1M files and so on):
>>
> It would be more helpful if you can provide info on your configurations
> (Dom0 and DomU), your workload, how to reproduce the bug.
>
> I run iperf and NFS to test Xen network, but never see any crash like
> this myself.
>
>> BUG: unable to handle kernel paging request at ffff8800795a3000
>> [ 60.246945] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
>> [ 60.246975] PGD a8a067 PUD a9a067 PMD 7fc27067 PTE
>> 80100000795a3065
>> [ 60.247004] Oops: 0003 [#1] SMP
>> [ 60.247020] Modules linked in: af_packet hwmon domctl crc32_pclmul
>> crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
>> aes_x86_64 joydev xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua
>> scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
>> ata_piix
>> [ 60.247144] CPU 0
>> [ 60.247154] Pid: 0, comm: swapper/0 Not tainted 3.9.2-1.g04040b9-xen
>> #1
>> [ 60.247179] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
>> netif_poll+0x49c/0xe80 [xennet]
>> ...
> Could you provide fuul stack trace? AFAICT there is no netif_poll in Xen
> netfront/back.
>
> Presumably this is Dom0 log? (from the domctl module)
>
>> We have couple of production hypervisors on 3.4 kernels with high-
>> throughput internal network (VM-to-VM in one Dom0), iperf on them is
>> working well:
>> [ 3] 0.0- 2.0 sec 3357 MBytes 14080 Mbits/sec
>> [ 3] 2.0- 4.0 sec 2880 MBytes 12077 Mbits/sec
>> [ 3] 4.0- 6.0 sec 2909 MBytes 12202 Mbits/sec
>> [ 3] 6.0- 8.0 sec 2552 MBytes 10702 Mbits/sec
>> [ 3] 8.0-10.0 sec 3616 MBytes 15166 Mbits/sec
>> [ 3] 10.0-12.0 sec 3415 MBytes 14324 Mbits/sec
>>
>>
>> Seems like a kernel bug, is this related to one of this fixes in linux-next or i
>> need to create new bugreport?
>>
>> 1) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=1aaf6d3d3d1e95f4be07e32dd84aa1c93855fbbd
>> 2) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=9ecd1a75d977e2e8c48139c7d3efed183f898d94
>> 3) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=2810e5b9a7731ca5fce22bfbe12c96e16ac44b6f
>> 4) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=03393fd5cc2b6cdeec32b704ecba64dbb0feae3c
>> 5) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=59ccb4ebbc35e36a3c143f2d1355deb75c2e628f
>>
> I don't think these patches can fix your problem at first glance.
>
>
> Wei.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-04 13:43 ` Dion Kant
@ 2013-07-04 14:18 ` Jan Beulich
2013-07-04 15:01 ` Wei Liu
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-04 14:18 UTC (permalink / raw)
To: Dion Kant; +Cc: xen-devel
>>> On 04.07.13 at 15:43, Dion Kant <g.w.kant@hunenet.nl> wrote:
> Hello Wei and all other interested people,
>
> I saw this thread from around May. It got silent on this thread after
> your post on May 31.
>
> Is there any progress on this problem?
We didn't receive feedback on the effect of the debugging patch
Wei had posted on that day
(http://lists.xenproject.org/archives/html/xen-devel/2013-05/msg03138.html).
Without knowing whether the possible issue pointed out earlier in
the thread is an actual one, or whether we need to look for
something else, we can't really make any progress here.
> I spent already quite some time in getting grip on the issue. I added a
> bug to bugzilla.novell.com but no response. See
> https://bugzilla.novell.com/show_bug.cgi?id=826374 for details.
> Apparently for hitting this bug (i.e. make it all the way to the crash),
> it is required to use some hardware which performs not too slow. With
> this I mean it is easy to find hardware which is unable to reproduce the
> issue.
You having picked the wrong component (Kernel instead of Xen)
may be part of the problem.
> 2013-07-03T23:51:16.560229+02:00 domUA kernel: [ 97.562370] netfront:
> Too many frags
The fix for this is already in the queue (and half of it already
committed at least to our master branch). But this isn't necessarily
connected to the crash/corruption you observe (but I do think that
crash and corruption are just two different manifestations of the
same problem).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-04 13:43 ` Dion Kant
2013-07-04 14:18 ` Jan Beulich
@ 2013-07-04 15:01 ` Wei Liu
2013-07-05 9:32 ` [PATCH] xen-netfront: pull on receive skb may need to happen earlier Jan Beulich
` (3 more replies)
1 sibling, 4 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-04 15:01 UTC (permalink / raw)
To: Dion Kant; +Cc: wei.liu2, jbeulich, xen-devel
Dropping Xen-user, CCing Jan. This should be discussed on Xen-devel.
On Thu, Jul 04, 2013 at 03:43:59PM +0200, Dion Kant wrote:
> Hello Wei and all other interested people,
>
> I saw this thread from around May. It got silent on this thread after
> your post on May 31.
>
> Is there any progress on this problem?
>
Sorry, no. I haven't been able to allocate time slots, and as this is
OpenSUSE kernel with forward ported patches it takes a little bit more
time to follow the code.
Last time I requested the reporter to run a test but he has not come
back until now.
> I am running into this issue as well with the openSUSE 12.3
> distribution. This is with their 3.7.10-1.16-xen kernel and Xen version
> 4.2.1_12-1.12.10. On the net I see some discussion of people hitting
> this issue but not that much. E.g., one of the symptoms is that a guest
> crashes when running zypper install or zypper update when the Internet
> connection is fast enough.
>
Do you have references to other reports?
> OpenSUSE 3.4.X kernels are running ok as guest on top of the openSUSE
> 12.3 Xen distribution, but apparently since 3.7.10 and higher there is
> this issue.
>
> I spent already quite some time in getting grip on the issue. I added a
> bug to bugzilla.novell.com but no response. See
> https://bugzilla.novell.com/show_bug.cgi?id=826374 for details.
> Apparently for hitting this bug (i.e. make it all the way to the crash),
> it is required to use some hardware which performs not too slow. With
> this I mean it is easy to find hardware which is unable to reproduce the
able?
> issue.
>
I'm not quite sure about what you mean. Do you mean this bug can only
be triggered when your receive path has real hardware NIC invloved?
And reading your test case below it doesn't seem so. Dom0 to DomU
transmission crashes the guest per your example.
> In one of my recent experiments I changed the SLAB allocater to SLUB
> which provides more detailed kernel logging. Here is the log output
> after the first detected issue regarding xennet:
>
But the log below is not about SLUB. I cannot understand why SLAB v.s
SLUB makes a difference.
> 2013-07-03T23:51:16.560229+02:00 domUA kernel: [ 97.562370] netfront:
> Too many frags
> 2013-07-03T23:51:17.228143+02:00 domUA kernel: [ 98.230466] netfront:
> Too many frags
> 2013-07-03T23:51:17.596074+02:00 domUA kernel: [ 98.597300] netfront:
> Too many frags
> 2013-07-03T23:51:18.740215+02:00 domUA kernel: [ 99.743080]
> net_ratelimit: 2 callbacks suppressed
> 2013-07-03T23:51:18.740242+02:00 domUA kernel: [ 99.743084] netfront:
> Too many frags
> 2013-07-03T23:51:19.104100+02:00 domUA kernel: [ 100.104281] netfront:
> Too many frags
> 2013-07-03T23:51:19.760134+02:00 domUA kernel: [ 100.760594] netfront:
> Too many frags
> 2013-07-03T23:51:21.820154+02:00 domUA kernel: [ 102.821202] netfront:
> Too many frags
> 2013-07-03T23:51:22.192188+02:00 domUA kernel: [ 103.192655] netfront:
> Too many frags
> 2013-07-03T23:51:26.060144+02:00 domUA kernel: [ 107.062447] netfront:
> Too many frags
> 2013-07-03T23:51:26.412116+02:00 domUA kernel: [ 107.415165] netfront:
> Too many frags
> 2013-07-03T23:51:27.092147+02:00 domUA kernel: [ 108.094615] netfront:
> Too many frags
> 2013-07-03T23:51:27.492112+02:00 domUA kernel: [ 108.494255] netfront:
> Too many frags
> 2013-07-03T23:51:27.520194+02:00 domUA kernel: [ 108.522445]
"Too many frags" means your frontend is generating malformed packets.
This is not normal. And apparently you didn't use the latest kernel in
tree because the log message should be "Too many slots" in the latest
OpenSuSE kernel.
> =============================================================================
> 2013-07-03T23:51:27.520206+02:00 domUA kernel: [ 108.522448] BUG
> kmalloc-1024 (Tainted: G W ): Redzone overwritten
> 2013-07-03T23:51:27.520209+02:00 domUA kernel: [ 108.522450]
> -----------------------------------------------------------------------------
> 2013-07-03T23:51:27.520212+02:00 domUA kernel: [ 108.522450]
> 2013-07-03T23:51:27.520215+02:00 domUA kernel: [ 108.522452] Disabling
> lock debugging due to kernel taint
> 2013-07-03T23:51:27.520217+02:00 domUA kernel: [ 108.522454] INFO:
> 0xffff8800f66068f8-0xffff8800f66068ff. First byte 0x0 instead of 0xcc
> 2013-07-03T23:51:27.520220+02:00 domUA kernel: [ 108.522461] INFO:
> Allocated in __alloc_skb+0x88/0x260 age=11 cpu=0 pid=1325
> 2013-07-03T23:51:27.520223+02:00 domUA kernel: [ 108.522466]
> set_track+0x6c/0x190
> 2013-07-03T23:51:27.520225+02:00 domUA kernel: [ 108.522470]
> alloc_debug_processing+0x83/0x109
> 2013-07-03T23:51:27.520228+02:00 domUA kernel: [ 108.522472]
> __slab_alloc.constprop.48+0x523/0x593
> 2013-07-03T23:51:27.520231+02:00 domUA kernel: [ 108.522474]
> __kmalloc_track_caller+0xb4/0x200
> 2013-07-03T23:51:27.520233+02:00 domUA kernel: [ 108.522477]
> __kmalloc_reserve+0x3c/0xa0
> 2013-07-03T23:51:27.520236+02:00 domUA kernel: [ 108.522478]
> __alloc_skb+0x88/0x260
> 2013-07-03T23:51:27.520239+02:00 domUA kernel: [ 108.522483]
> network_alloc_rx_buffers+0x76/0x5f0 [xennet]
> 2013-07-03T23:51:27.520241+02:00 domUA kernel: [ 108.522486]
> netif_poll+0xcf4/0xf30 [xennet]
> 2013-07-03T23:51:27.520243+02:00 domUA kernel: [ 108.522489]
> net_rx_action+0xf0/0x2e0
> 2013-07-03T23:51:27.520246+02:00 domUA kernel: [ 108.522493]
> __do_softirq+0x127/0x280
> 2013-07-03T23:51:27.520248+02:00 domUA kernel: [ 108.522496]
> call_softirq+0x1c/0x30
> 2013-07-03T23:51:27.520251+02:00 domUA kernel: [ 108.522499]
> do_softirq+0x56/0xd0
> 2013-07-03T23:51:27.520253+02:00 domUA kernel: [ 108.522501]
> irq_exit+0x52/0xd0
> 2013-07-03T23:51:27.520256+02:00 domUA kernel: [ 108.522503]
> evtchn_do_upcall+0x281/0x2e7
> 2013-07-03T23:51:27.520258+02:00 domUA kernel: [ 108.522505]
> do_hypervisor_callback+0x1e/0x30
> 2013-07-03T23:51:27.520261+02:00 domUA kernel: [ 108.522507]
> 0x7f45f0a2f1e0
> 2013-07-03T23:51:27.520263+02:00 domUA kernel: [ 108.522509] INFO:
> Freed in skb_free_head+0x5c/0x70 age=14 cpu=0 pid=1325
> 2013-07-03T23:51:27.520266+02:00 domUA kernel: [ 108.522512]
> set_track+0x6c/0x190
> 2013-07-03T23:51:27.520269+02:00 domUA kernel: [ 108.522513]
> free_debug_processing+0x151/0x201
> 2013-07-03T23:51:27.520271+02:00 domUA kernel: [ 108.522515]
> __slab_free+0x47/0x499
> 2013-07-03T23:51:27.520274+02:00 domUA kernel: [ 108.522517]
> kfree+0x1df/0x230
> 2013-07-03T23:51:27.520276+02:00 domUA kernel: [ 108.522519]
> skb_free_head+0x5c/0x70
> 2013-07-03T23:51:27.520279+02:00 domUA kernel: [ 108.522521]
> skb_release_data+0xea/0xf0
> 2013-07-03T23:51:27.520281+02:00 domUA kernel: [ 108.522522]
> __kfree_skb+0x1e/0xb0
> 2013-07-03T23:51:27.520284+02:00 domUA kernel: [ 108.522524]
> kfree_skb+0x80/0xc0
> 2013-07-03T23:51:27.520286+02:00 domUA kernel: [ 108.522527]
> netif_poll+0x824/0xf30 [xennet]
> 2013-07-03T23:51:27.520289+02:00 domUA kernel: [ 108.522529]
> net_rx_action+0xf0/0x2e0
> 2013-07-03T23:51:27.520291+02:00 domUA kernel: [ 108.522530]
> __do_softirq+0x127/0x280
> 2013-07-03T23:51:27.520294+02:00 domUA kernel: [ 108.522532]
> call_softirq+0x1c/0x30
> 2013-07-03T23:51:27.520296+02:00 domUA kernel: [ 108.522534]
> do_softirq+0x56/0xd0
> 2013-07-03T23:51:27.520299+02:00 domUA kernel: [ 108.522536]
> irq_exit+0x52/0xd0
> 2013-07-03T23:51:27.520302+02:00 domUA kernel: [ 108.522538]
> evtchn_do_upcall+0x281/0x2e7
> 2013-07-03T23:51:27.520304+02:00 domUA kernel: [ 108.522539]
> do_hypervisor_callback+0x1e/0x30
> 2013-07-03T23:51:27.520307+02:00 domUA kernel: [ 108.522541] INFO: Slab
> 0xffff8800ffd78100 objects=12 used=7 fp=0xffff8800f66074d0
> flags=0x400000000000408
> 2013-07-03T23:51:27.520310+02:00 domUA kernel: [ 108.522543] INFO:
> Object 0xffff8800f66064f8 @offset=9464 fp=0x0000018800000000
> 2013-07-03T23:51:27.520312+02:00 domUA kernel: [ 108.522543]
> 2013-07-03T23:51:27.520315+02:00 domUA kernel: [ 108.522546] Bytes b4
> ffff8800f66064e8: 4a 40 ff ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a
> J@......ZZZZZZZZ
> 2013-07-03T23:51:27.520318+02:00 domUA kernel: [ 108.522548] Object
> ffff8800f66064f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> kkkkkkkkkkkkkkkk
> 2013-07-03T23:51:27.520320+02:00 domUA kernel: [ 108.522549] Object
> ffff8800f6606508: 00 16 3e 29 7e 3c 00 25 90 69 ea 4e 08 00 45 08
> ..>)~<.%.i.N..E.
> 2013-07-03T23:51:27.520323+02:00 domUA kernel: [ 108.522551] Object
> ffff8800f6606518: fe bc 46 d7 40 00 40 06 d3 69 0a 57 06 91 0a 57
> ..F.@.@..i.W...W
> 2013-07-03T23:51:27.520326+02:00 domUA kernel: [ 108.522553] Object
> ffff8800f6606528: 06 b4 9b 86 00 16 57 4d 5e bd 89 4c 40 ad 80 10
> ......WM^..L@...
> 2013-07-03T23:51:27.520329+02:00 domUA kernel: [ 108.522554] Object
> ffff8800f6606538: 00 a6 20 a2 00 00 01 01 08 0a 01 eb 40 a7 ff ff ..
> .........@...
> 2013-07-03T23:51:27.520332+02:00 domUA kernel: [ 108.522556] Object
> ffff8800f6606548: 44 fa 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> D.kkkkkkkkkkkkkk
> 2013-07-03T23:51:27.520335+02:00 domUA kernel: [ 108.522557] Object
> ffff8800f6606558: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> kkkkkkkkkkkkkkkk
> 2013-07-03T23:51:27.520337+02:00 domUA kernel: [ 108.522559] Object
> ffff8800f6606568: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
> kkkkkkkkkkkkkkkk
>
> Skipping some of the object dumping.......
>
> 2013-07-03T23:51:27.520583+02:00 domUA kernel: [ 108.522644] Object
> ffff8800f66068d8: 00 d7 e4 ff 00 88 ff ff 00 00 00 00 00 10 00 00
> ................
> 2013-07-03T23:51:27.520586+02:00 domUA kernel: [ 108.522646] Object
> ffff8800f66068e8: 00 92 dd ff 00 88 ff ff 00 00 00 00 88 01 00 00
> ................
> 2013-07-03T23:51:27.520588+02:00 domUA kernel: [ 108.522647] Redzone
> ffff8800f66068f8: 00 92 dd ff 00 88 ff ff ........
> 2013-07-03T23:51:27.520591+02:00 domUA kernel: [ 108.522649] Padding
> ffff8800f6606a38: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
> 2013-07-03T23:51:27.520594+02:00 domUA kernel: [ 108.522651] Pid: 1325,
> comm: sshd Tainted: G B W 3.7.10-1.16-dbg-xen #3
> 2013-07-03T23:51:27.520597+02:00 domUA kernel: [ 108.522652] Call Trace:
> 2013-07-03T23:51:27.520599+02:00 domUA kernel: [ 108.522658]
> [<ffffffff8000b097>] try_stack_unwind+0x87/0x1c0
> 2013-07-03T23:51:27.520602+02:00 domUA kernel: [ 108.522662]
> [<ffffffff80008fa5>] dump_trace+0xd5/0x250
> 2013-07-03T23:51:27.520605+02:00 domUA kernel: [ 108.522665]
> [<ffffffff8000b22c>] show_trace_log_lvl+0x5c/0x80
> 2013-07-03T23:51:27.520608+02:00 domUA kernel: [ 108.522668]
> [<ffffffff8000b265>] show_trace+0x15/0x20
> 2013-07-03T23:51:27.520610+02:00 domUA kernel: [ 108.522672]
> [<ffffffff80553a69>] dump_stack+0x77/0x80
> 2013-07-03T23:51:27.520612+02:00 domUA kernel: [ 108.522676]
> [<ffffffff801491b1>] print_trailer+0x131/0x140
> 2013-07-03T23:51:27.520615+02:00 domUA kernel: [ 108.522680]
> [<ffffffff80149709>] check_bytes_and_report+0xc9/0x120
> 2013-07-03T23:51:27.520617+02:00 domUA kernel: [ 108.522683]
> [<ffffffff8014a7f6>] check_object+0x56/0x240
> 2013-07-03T23:51:27.520620+02:00 domUA kernel: [ 108.522687]
> [<ffffffff805575b6>] free_debug_processing+0xc4/0x201
> 2013-07-03T23:51:27.520622+02:00 domUA kernel: [ 108.522690]
> [<ffffffff8055773a>] __slab_free+0x47/0x499
> 2013-07-03T23:51:27.520625+02:00 domUA kernel: [ 108.522694]
> [<ffffffff8014beff>] kfree+0x1df/0x230
> 2013-07-03T23:51:27.520627+02:00 domUA kernel: [ 108.522697]
> [<ffffffff8044a8cc>] skb_free_head+0x5c/0x70
> 2013-07-03T23:51:27.520630+02:00 domUA kernel: [ 108.522701]
> [<ffffffff8044a9ca>] skb_release_data+0xea/0xf0
> 2013-07-03T23:51:27.520632+02:00 domUA kernel: [ 108.522704]
> [<ffffffff8044a9ee>] __kfree_skb+0x1e/0xb0
> 2013-07-03T23:51:27.520635+02:00 domUA kernel: [ 108.522709]
> [<ffffffff8049fa2a>] tcp_recvmsg+0x99a/0xd50
> 2013-07-03T23:51:27.520637+02:00 domUA kernel: [ 108.522714]
> [<ffffffff804c796d>] inet_recvmsg+0xed/0x110
> 2013-07-03T23:51:27.520640+02:00 domUA kernel: [ 108.522718]
> [<ffffffff80440be8>] sock_aio_read+0x158/0x190
> 2013-07-03T23:51:27.520642+02:00 domUA kernel: [ 108.522722]
> [<ffffffff8015cb68>] do_sync_read+0x98/0xf0
> 2013-07-03T23:51:27.520645+02:00 domUA kernel: [ 108.522726]
> [<ffffffff8015d32d>] vfs_read+0xbd/0x180
> 2013-07-03T23:51:27.520647+02:00 domUA kernel: [ 108.522729]
> [<ffffffff8015d442>] sys_read+0x52/0xa0
> 2013-07-03T23:51:27.520650+02:00 domUA kernel: [ 108.522733]
> [<ffffffff8056ab3b>] system_call_fastpath+0x1a/0x1f
> 2013-07-03T23:51:27.520652+02:00 domUA kernel: [ 108.522736]
> [<00007f45ef74c960>] 0x7f45ef74c95f
> 2013-07-03T23:51:27.520655+02:00 domUA kernel: [ 108.522738] FIX
> kmalloc-1024: Restoring 0xffff8800f66068f8-0xffff8800f66068ff=0xcc
> 2013-07-03T23:51:27.520657+02:00 domUA kernel: [ 108.522738]
> 2013-07-03T23:51:27.679444+02:00 domUA kernel: [ 108.671750]
> =============================================================================
> 2013-07-03T23:51:27.679454+02:00 domUA kernel: [ 108.671753] BUG
> kmalloc-1024 (Tainted: G B W ): Redzone overwritten
> 2013-07-03T23:51:27.679456+02:00 domUA kernel: [ 108.671754]
> -----------------------------------------------------------------------------
> 2013-07-03T23:51:27.679458+02:00 domUA kernel: [ 108.671754]
> 2013-07-03T23:51:27.679460+02:00 domUA kernel: [ 108.671757] INFO:
> 0xffff8800f66068f8-0xffff8800f66068ff. First byte 0xcc instead of 0xbb
> 2013-07-03T23:51:27.679462+02:00 domUA kernel: [ 108.671762] INFO:
> Allocated in __alloc_skb+0x88/0x260 age=48 cpu=0 pid=1325
> 2013-07-03T23:51:27.679464+02:00 domUA kernel: [ 108.671765]
> set_track+0x6c/0x190
> 2013-07-03T23:51:27.679466+02:00 domUA kernel: [ 108.671767]
> alloc_debug_processing+0x83/0x109
> 2013-07-03T23:51:27.679468+02:00 domUA kernel: [ 108.671769]
> __slab_alloc.constprop.48+0x523/0x593
> 2013-07-03T23:51:27.679469+02:00 domUA kernel: [ 108.671771]
> __kmalloc_track_caller+0xb4/0x200
> 2013-07-03T23:51:27.679471+02:00 domUA kernel: [ 108.671773]
> __kmalloc_reserve+0x3c/0xa0
> 2013-07-03T23:51:27.679473+02:00 domUA kernel: [ 108.671775]
> __alloc_skb+0x88/0x260
> 2013-07-03T23:51:27.679475+02:00 domUA kernel: [ 108.671778]
> network_alloc_rx_buffers+0x76/0x5f0 [xennet]
> 2013-07-03T23:51:27.679476+02:00 domUA kernel: [ 108.671781]
> netif_poll+0xcf4/0xf30 [xennet]
> 2013-07-03T23:51:27.679478+02:00 domUA kernel: [ 108.671783]
> net_rx_action+0xf0/0x2e0
>
Seems like there's memory corruption in guest RX path.
> I noticed that after turning on all this debugging stuff, a real panic
> does not appear any more.
>
> This happens while copying a file with scp from dom0 to the guest (scp
> bigfile domu:/dev/null).
>
> In my lab, I am currently experimenting with a SuperMicro based system
> with Xen showing the following characteristics:
>
> __ __ _ _ ____ _ _ ____ _ _ ____ _ ___
> \ \/ /___ _ __ | || | |___ \ / | / |___ \ / | / |___ \ / |/ _ \
> \ // _ \ '_ \ | || |_ __) | | | | | __) |__| | | | __) | | | | | |
> / \ __/ | | | |__ _| / __/ _| | | |/ __/|__| |_| |/ __/ _| | |_| |
> /_/\_\___|_| |_| |_|(_)_____(_)_|___|_|_____| |_(_)_|_____(_)_|\___/
> |_____|
> (XEN) Xen version 4.2.1_12-1.12.10 (abuild@) (gcc (SUSE Linux) 4.7.2
> 20130108 [gcc-4_7-branch revision 195012]) Wed May 29 20:31:49 UTC 2013
> (XEN) Latest ChangeSet: 25952
> (XEN) Bootloader: GNU GRUB 0.97
> (XEN) Command line: dom0_mem=2048M,max:2048M loglvl=all guest_loglvl=all
> (XEN) Video information:
> (XEN) VGA is text mode 80x25, font 8x16
> (XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
> (XEN) Disc information:
> (XEN) Found 4 MBR signatures
> (XEN) Found 4 EDD information structures
> (XEN) Xen-e820 RAM map:
> (XEN) 0000000000000000 - 0000000000096400 (usable)
> (XEN) 0000000000096400 - 00000000000a0000 (reserved)
> (XEN) 00000000000e0000 - 0000000000100000 (reserved)
> (XEN) 0000000000100000 - 00000000bf780000 (usable)
> (XEN) 00000000bf78e000 - 00000000bf790000 type 9
> (XEN) 00000000bf790000 - 00000000bf79e000 (ACPI data)
> (XEN) 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
> (XEN) 00000000bf7d0000 - 00000000bf7e0000 (reserved)
> (XEN) 00000000bf7ec000 - 00000000c0000000 (reserved)
> (XEN) 00000000e0000000 - 00000000f0000000 (reserved)
> (XEN) 00000000fee00000 - 00000000fee01000 (reserved)
> (XEN) 00000000ffc00000 - 0000000100000000 (reserved)
> (XEN) 0000000100000000 - 0000000340000000 (usable)
>
> Skipping ACPI en SRAT
>
> (XEN) System RAM: 12279MB (12573784kB)
>
> (XEN) NUMA: Allocated memnodemap from 33e38a000 - 33e38e000
> (XEN) NUMA: Using 8 for the hash shift.
> (XEN) Domain heap initialised DMA width 30 bits
> (XEN) found SMP MP-table at 000ff780
> (XEN) DMI present.
>
> (XEN) Enabling APIC mode: Phys. Using 2 I/O APICs
> (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
> (XEN) Failed to get Error Log Address Range.
> (XEN) Using ACPI (MADT) for SMP configuration information
> (XEN) SMP: Allowing 24 CPUs (8 hotplug CPUs)
> (XEN) IRQ limits: 48 GSI, 3040 MSI/MSI-X
> (XEN) Using scheduler: SMP Credit Scheduler (credit)
> (XEN) Detected 2400.115 MHz processor.
> (XEN) Initing memory sharing.
> (XEN) mce_intel.c:1238: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0
> extended MCE MSR 0
> (XEN) Intel machine check reporting enabled
> (XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
> (XEN) PCI: MCFG area at e0000000 reserved in E820
> (XEN) PCI: Using MCFG for segment 0000 bus 00-ff
> (XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
> (XEN) Intel VT-d Snoop Control enabled.
> (XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
> (XEN) Intel VT-d Queued Invalidation enabled.
> (XEN) Intel VT-d Interrupt Remapping enabled.
> (XEN) Intel VT-d Shared EPT tables not enabled.
> (XEN) I/O virtualisation enabled
> (XEN) - Dom0 mode: Relaxed
> (XEN) Interrupt remapping enabled
> (XEN) Enabled directed EOI with ioapic_ack_old on!
> (XEN) ENABLING IO-APIC IRQs
> (XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
> (XEN) Platform timer is 14.318MHz HPET
> (XEN) Allocated console ring of 128 KiB.
> (XEN) VMX: Supported advanced features:
> (XEN) - APIC MMIO access virtualisation
> (XEN) - APIC TPR shadow
> (XEN) - Extended Page Tables (EPT)
> (XEN) - Virtual-Processor Identifiers (VPID)
> (XEN) - Virtual NMI
> (XEN) - MSR direct-access bitmap
> (XEN) - Unrestricted Guest
> (XEN) HVM: ASIDs enabled.
> (XEN) HVM: VMX enabled
> (XEN) HVM: Hardware Assisted Paging (HAP) detected
> (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
> (XEN) Brought up 16 CPUs
> (XEN) ACPI sleep modes: S3
> (XEN) mcheck_poll: Machine check polling timer started.
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Xen kernel: 64-bit, lsb, compat32
> (XEN) Dom0 kernel: 64-bit, lsb, paddr 0x2000 -> 0xa65000
> (XEN) PHYSICAL MEMORY ARRANGEMENT:
> (XEN) Dom0 alloc.: 0000000336000000->0000000337000000 (516915 pages
> to be allocated)
> (XEN) Init. ramdisk: 000000033f333000->0000000340000000
> (XEN) VIRTUAL MEMORY ARRANGEMENT:
> (XEN) Loaded kernel: ffffffff80002000->ffffffff80a65000
> (XEN) Init. ramdisk: 0000000000000000->0000000000000000
> (XEN) Phys-Mach map: ffffea0000000000->ffffea0000400000
> (XEN) Start info: ffffffff80a65000->ffffffff80a654b4
> (XEN) Page tables: ffffffff80a66000->ffffffff80a6f000
> (XEN) Boot stack: ffffffff80a6f000->ffffffff80a70000
> (XEN) TOTAL: ffffffff80000000->ffffffff80c00000
> (XEN) ENTRY ADDRESS: ffffffff80002000
> (XEN) Dom0 has maximum 16 VCPUs
> (XEN) Scrubbing Free RAM:
> .....................................................................................................done.
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> (XEN) Std. Loglevel: All
> (XEN) Guest Loglevel: All
> (XEN) Xen is relinquishing VGA console.
>
>
>
> (XEN) ACPI: RSDP 000FACE0, 0024 (r2 ACPIAM)
> (XEN) ACPI: XSDT BF790100, 008C (r1 SMCI 20110827 MSFT 97)
> (XEN) ACPI: FACP BF790290, 00F4 (r4 082711 FACP1638 20110827 MSFT 97)
> (XEN) ACPI: DSDT BF7906A0, 6563 (r2 10600 10600000 0 INTL 20051117)
> (XEN) ACPI: FACS BF79E000, 0040
> (XEN) ACPI: APIC BF790390, 011E (r2 082711 APIC1638 20110827 MSFT 97)
> (XEN) ACPI: MCFG BF7904B0, 003C (r1 082711 OEMMCFG 20110827 MSFT 97)
> (XEN) ACPI: SLIT BF7904F0, 0030 (r1 082711 OEMSLIT 20110827 MSFT 97)
> (XEN) ACPI: OEMB BF79E040, 0085 (r1 082711 OEMB1638 20110827 MSFT 97)
> (XEN) ACPI: SRAT BF79A6A0, 01D0 (r2 082711 OEMSRAT 1 INTL 1)
> (XEN) ACPI: HPET BF79A870, 0038 (r1 082711 OEMHPET 20110827 MSFT 97)
> (XEN) ACPI: DMAR BF79E0D0, 0130 (r1 AMI OEMDMAR 1 MSFT 97)
> (XEN) ACPI: SSDT BF7A1B30, 0363 (r1 DpgPmm CpuPm 12 INTL 20051117)
> (XEN) ACPI: EINJ BF79A8B0, 0130 (r1 AMIER AMI_EINJ 20110827 MSFT 97)
> (XEN) ACPI: BERT BF79AA40, 0030 (r1 AMIER AMI_BERT 20110827 MSFT 97)
> (XEN) ACPI: ERST BF79AA70, 01B0 (r1 AMIER AMI_ERST 20110827 MSFT 97)
> (XEN) ACPI: HEST BF79AC20, 00A8 (r1 AMIER ABC_HEST 20110827 MSFT 97)
> (XEN) System RAM: 12279MB (12573784kB)
> (XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 18 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 20 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 19 -> Node 0
> (XEN) SRAT: PXM 0 -> APIC 21 -> Node 0
>
> I am happy to assist in more kernel probing. It is even possible for me
> to setup access for someone to this machine.
>
Excellent. Last time Jan suspected that we potentially overrun the frag
list of a skb (which would corrupt memory) but it has not been verified.
I also skimmed your bug report on novell bugzilla which did suggest
memory corruption.
I wrote a patch to crash the kernel immediately when looping over the
frag list, probably we could start from there? (You might need to adjust
context, but it is only a one-liner which should be easy).
Wei.
======
diff --git a/drivers/xen/netfront/netfront.c b/drivers/xen/netfront/netfront.c
index 6e5d233..9583011 100644
--- a/drivers/xen/netfront/netfront.c
+++ b/drivers/xen/netfront/netfront.c
@@ -1306,6 +1306,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
struct sk_buff *nskb;
while ((nskb = __skb_dequeue(list))) {
+ BUG_ON(nr_frags >= MAX_SKB_FRAGS);
struct netif_rx_response *rx =
RING_GET_RESPONSE(&np->rx, ++cons);
^ permalink raw reply related [flat|nested] 85+ messages in thread
* [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-04 15:01 ` Wei Liu
@ 2013-07-05 9:32 ` Jan Beulich
2013-07-05 14:53 ` Wei Liu
2013-07-05 14:53 ` Wei Liu
2013-07-05 9:32 ` Jan Beulich
` (2 subsequent siblings)
3 siblings, 2 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-05 9:32 UTC (permalink / raw)
To: davem; +Cc: Ian Campbell, wei.liu2, Dion Kant, xen-devel, netdev, stable
Due to commit 3683243b ("xen-netfront: use __pskb_pull_tail to ensure
linear area is big enough on RX") xennet_fill_frags() may end up
filling MAX_SKB_FRAGS + 1 fragments in a receive skb, and only reduce
the fragment count subsequently via __pskb_pull_tail(). That's a
result of xennet_get_responses() allowing a maximum of one more slot to
be consumed (and intermediately transformed into a fragment) if the
head slot has a size less than or equal to RX_COPY_THRESHOLD.
Hence we need to adjust xennet_fill_frags() to pull earlier if we
reached the maximum fragment count - due to the described behavior of
xennet_get_responses() this guarantees that at least the first fragment
will get completely consumed, and hence the fragment count reduced.
In order to not needlessly call __pskb_pull_tail() twice, make the
original call conditional upon the pull target not having been reached
yet, and defer the newly added one as much as possible (an alternative
would have been to always call the function right before the call to
xennet_fill_frags(), but that would imply more frequent cases of
needing to call it twice).
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: stable@vger.kernel.org (3.6 onwards)
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
RING_GET_RESPONSE(&np->rx, ++cons);
skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
+ if (nr_frags == MAX_SKB_FRAGS) {
+ unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
+
+ BUG_ON(pull_to <= skb_headlen(skb));
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ nr_frags = shinfo->nr_frags;
+ }
+ BUG_ON(nr_frags >= MAX_SKB_FRAGS);
+
__skb_fill_page_desc(skb, nr_frags,
skb_frag_page(nfrag),
rx->offset, rx->status);
@@ -929,7 +938,8 @@ static int handle_incoming_queue(struct
while ((skb = __skb_dequeue(rxq)) != NULL) {
int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
- __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ if (pull_to > skb_headlen(skb))
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
/* Ethernet work: Delayed to here as it peeks the header. */
skb->protocol = eth_type_trans(skb, dev);
^ permalink raw reply [flat|nested] 85+ messages in thread
* [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-04 15:01 ` Wei Liu
2013-07-05 9:32 ` [PATCH] xen-netfront: pull on receive skb may need to happen earlier Jan Beulich
@ 2013-07-05 9:32 ` Jan Beulich
2013-07-05 9:36 ` kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Jan Beulich
2013-07-05 10:40 ` [Xen-users] " Dion Kant
3 siblings, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-05 9:32 UTC (permalink / raw)
To: davem; +Cc: wei.liu2, Ian Campbell, netdev, stable, xen-devel, Dion Kant
Due to commit 3683243b ("xen-netfront: use __pskb_pull_tail to ensure
linear area is big enough on RX") xennet_fill_frags() may end up
filling MAX_SKB_FRAGS + 1 fragments in a receive skb, and only reduce
the fragment count subsequently via __pskb_pull_tail(). That's a
result of xennet_get_responses() allowing a maximum of one more slot to
be consumed (and intermediately transformed into a fragment) if the
head slot has a size less than or equal to RX_COPY_THRESHOLD.
Hence we need to adjust xennet_fill_frags() to pull earlier if we
reached the maximum fragment count - due to the described behavior of
xennet_get_responses() this guarantees that at least the first fragment
will get completely consumed, and hence the fragment count reduced.
In order to not needlessly call __pskb_pull_tail() twice, make the
original call conditional upon the pull target not having been reached
yet, and defer the newly added one as much as possible (an alternative
would have been to always call the function right before the call to
xennet_fill_frags(), but that would imply more frequent cases of
needing to call it twice).
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: stable@vger.kernel.org (3.6 onwards)
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
RING_GET_RESPONSE(&np->rx, ++cons);
skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
+ if (nr_frags == MAX_SKB_FRAGS) {
+ unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
+
+ BUG_ON(pull_to <= skb_headlen(skb));
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ nr_frags = shinfo->nr_frags;
+ }
+ BUG_ON(nr_frags >= MAX_SKB_FRAGS);
+
__skb_fill_page_desc(skb, nr_frags,
skb_frag_page(nfrag),
rx->offset, rx->status);
@@ -929,7 +938,8 @@ static int handle_incoming_queue(struct
while ((skb = __skb_dequeue(rxq)) != NULL) {
int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
- __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ if (pull_to > skb_headlen(skb))
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
/* Ethernet work: Delayed to here as it peeks the header. */
skb->protocol = eth_type_trans(skb, dev);
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-04 15:01 ` Wei Liu
2013-07-05 9:32 ` [PATCH] xen-netfront: pull on receive skb may need to happen earlier Jan Beulich
2013-07-05 9:32 ` Jan Beulich
@ 2013-07-05 9:36 ` Jan Beulich
2013-07-05 10:40 ` [Xen-users] " Dion Kant
3 siblings, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-05 9:36 UTC (permalink / raw)
To: Eugene Istomin, Dion Kant; +Cc: wei.liu2, xen-devel
[-- Attachment #1: Type: text/plain, Size: 364 bytes --]
Attached the openSUSE 12.3 patch corresponding to the one I just
sent against the upstream kernel, in case you want to give this a
try (this is in fact against the master branch, bit I verified that it
applies reasonably cleanly to the 12.3 branch too); neither Wei nor
me having been able to reproduce this would make your testing
pretty valuable.
Jan
[-- Attachment #2: xen3-patch-3.6-netfront-pull.patch --]
[-- Type: text/plain, Size: 1111 bytes --]
--- head.orig/drivers/xen/netfront/netfront.c 2013-05-10 14:43:24.000000000 +0200
+++ head/drivers/xen/netfront/netfront.c 2013-07-05 10:56:41.000000000 +0200
@@ -1309,6 +1309,15 @@ static RING_IDX xennet_fill_frags(struct
struct netif_rx_response *rx =
RING_GET_RESPONSE(&np->rx, ++cons);
+ if (nr_frags == MAX_SKB_FRAGS) {
+ unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
+
+ BUG_ON(pull_to <= skb_headlen(skb));
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ nr_frags = shinfo->nr_frags;
+ }
+ BUG_ON(nr_frags >= MAX_SKB_FRAGS);
+
__skb_fill_page_desc(skb, nr_frags,
skb_frag_page(skb_shinfo(nskb)->frags),
rx->offset, rx->status);
@@ -1480,7 +1489,8 @@ err:
while ((skb = __skb_dequeue(&rxq)) != NULL) {
unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
- __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ if (pull_to > skb_headlen(skb))
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
/* Ethernet work: Delayed to here as it peeks the header. */
skb->protocol = eth_type_trans(skb, dev);
[-- Attachment #3: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-04 15:01 ` Wei Liu
` (2 preceding siblings ...)
2013-07-05 9:36 ` kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Jan Beulich
@ 2013-07-05 10:40 ` Dion Kant
2013-07-05 10:54 ` Wei Liu
2013-07-05 10:56 ` Jan Beulich
3 siblings, 2 replies; 85+ messages in thread
From: Dion Kant @ 2013-07-05 10:40 UTC (permalink / raw)
To: Wei Liu; +Cc: jbeulich, xen-devel
On 07/04/2013 05:01 PM, Wei Liu wrote:
>
>> I am running into this issue as well with the openSUSE 12.3
>> distribution. This is with their 3.7.10-1.16-xen kernel and Xen version
>> 4.2.1_12-1.12.10. On the net I see some discussion of people hitting
>> this issue but not that much. E.g., one of the symptoms is that a guest
>> crashes when running zypper install or zypper update when the Internet
>> connection is fast enough.
>>
> Do you have references to other reports?
I will gather them and post them later.
>
>> OpenSUSE 3.4.X kernels are running ok as guest on top of the openSUSE
>> 12.3 Xen distribution, but apparently since 3.7.10 and higher there is
>> this issue.
>>
>> I spent already quite some time in getting grip on the issue. I added a
>> bug to bugzilla.novell.com but no response. See
>> https://bugzilla.novell.com/show_bug.cgi?id=826374 for details.
>> Apparently for hitting this bug (i.e. make it all the way to the crash),
>> it is required to use some hardware which performs not too slow. With
>> this I mean it is easy to find hardware which is unable to reproduce the
> able?
>> issue.
>>
> I'm not quite sure about what you mean. Do you mean this bug can only
> be triggered when your receive path has real hardware NIC invloved?
>
> And reading your test case below it doesn't seem so. Dom0 to DomU
> transmission crashes the guest per your example.
Yes, a physical network card is not required. If you do send data to the
guest over a physical Ethernet card, it is required to operate it in 1
GbE mode. With a 100 MbE link I am unable to crash the guest.
If you do use vif interfaces only, the data rate will be high enough to
crash it.
However, I have also openSUSE 12.3 Xen configurations running which do
not have this issue. It is my feeling that smaller systems (in the sense
of less CPU cores and/or less memory bandwidth) do not reveal the issue.
>
>> In one of my recent experiments I changed the SLAB allocater to SLUB
>> which provides more detailed kernel logging. Here is the log output
>> after the first detected issue regarding xennet:
>>
> But the log below is not about SLUB. I cannot understand why SLAB v.s
> SLUB makes a difference.
I switched to SLUB from SLAB for its debugging functionality. The
openSUSE stock kernel used SLAB.
>
>> Too many frags
>> 2013-07-03T23:51:27.092147+02:00 domUA kernel: [ 108.094615] netfront:
>> Too many frags
>> 2013-07-03T23:51:27.492112+02:00 domUA kernel: [ 108.494255] netfront:
>> Too many frags
>> 2013-07-03T23:51:27.520194+02:00 domUA kernel: [ 108.522445]
> "Too many frags" means your frontend is generating malformed packets.
> This is not normal. And apparently you didn't use the latest kernel in
> tree because the log message should be "Too many slots" in the latest
> OpenSuSE kernel.
Yes, I have seen that, but I used the latest openSUSE kernel which
belongs to openSUSE 12.3.
>> network_alloc_rx_buffers+0x76/0x5f0 [xennet]
>> 2013-07-03T23:51:27.679476+02:00 domUA kernel: [ 108.671781]
>> netif_poll+0xcf4/0xf30 [xennet]
>> 2013-07-03T23:51:27.679478+02:00 domUA kernel: [ 108.671783]
>> net_rx_action+0xf0/0x2e0
>>
> Seems like there's memory corruption in guest RX path.
As Jan already mentioned, it could be related to the kernel panics I
obtain, however it may be a different issue as well.
>>
>>
>> I am happy to assist in more kernel probing. It is even possible for me
>> to setup access for someone to this machine.
>>
> Excellent. Last time Jan suspected that we potentially overrun the frag
> list of a skb (which would corrupt memory) but it has not been verified.
>
> I also skimmed your bug report on novell bugzilla which did suggest
> memory corruption.
>
> I wrote a patch to crash the kernel immediately when looping over the
> frag list, probably we could start from there? (You might need to adjust
> context, but it is only a one-liner which should be easy).
>
>
> Wei.
>
> ======
> diff --git a/drivers/xen/netfront/netfront.c b/drivers/xen/netfront/netfront.c
> index 6e5d233..9583011 100644
> --- a/drivers/xen/netfront/netfront.c
> +++ b/drivers/xen/netfront/netfront.c
> @@ -1306,6 +1306,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
> struct sk_buff *nskb;
>
> while ((nskb = __skb_dequeue(list))) {
> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> struct netif_rx_response *rx =
> RING_GET_RESPONSE(&np->rx, ++cons);
>
Integrated the patch. I obtained a crash dump and the log in it did not
show this BUG_ON. Here is the relevant section from the log
var/lib/xen/dump/domUA # crash /root/vmlinux-p1
2013-0705-1347.43-domUA.1.core
[ 7.670132] Adding 4192252k swap on /dev/xvda1. Priority:-1
extents:1 across:4192252k SS
[ 10.204340] NET: Registered protocol family 17
[ 481.534979] netfront: Too many frags
[ 487.543946] netfront: Too many frags
[ 491.049458] netfront: Too many frags
[ 491.491153] ------------[ cut here ]------------
[ 491.491628] kernel BUG at drivers/xen/netfront/netfront.c:1295!
[ 491.492056] invalid opcode: 0000 [#1] SMP
[ 491.492056] Modules linked in: af_packet autofs4 xennet xenblk cdrom
[ 491.492056] CPU 0
[ 491.492056] Pid: 1471, comm: sshd Not tainted 3.7.10-1.16-dbg-p1-xen #8
[ 491.492056] RIP: e030:[<ffffffffa0023aef>] [<ffffffffa0023aef>]
netif_poll+0xe4f/0xf90 [xennet]
[ 491.492056] RSP: e02b:ffff8801f5803c60 EFLAGS: 00010202
[ 491.492056] RAX: ffff8801f5803da0 RBX: ffff8801f1a082c0 RCX:
0000000180200010
[ 491.492056] RDX: ffff8801f5803da0 RSI: ffff8801fe83ec80 RDI:
ffff8801f03b2900
[ 491.492056] RBP: ffff8801f5803e20 R08: 0000000000000001 R09:
0000000000000000
[ 491.492056] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff8801f03b3400
[ 491.492056] R13: 0000000000000011 R14: 000000000004327e R15:
ffff8801f06009c0
[ 491.492056] FS: 00007fc519f3d7c0(0000) GS:ffff8801f5800000(0000)
knlGS:0000000000000000
[ 491.492056] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 491.492056] CR2: 00007fc51410c400 CR3: 00000001f1430000 CR4:
0000000000002660
[ 491.492056] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 491.492056] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 491.492056] Process sshd (pid: 1471, threadinfo ffff8801f1264000,
task ffff8801f137bf00)
[ 491.492056] Stack:
[ 491.492056] ffff8801f5803d60 ffffffff8008503e ffff8801f0600a40
ffff8801f0600000
[ 491.492056] 0004328000000040 0000001200000000 ffff8801f5810570
ffff8801f0600a78
[ 491.492056] 0000000000000000 ffff8801f0601fb0 0004326e00000012
ffff8801f5803d00
[ 491.492056] Call Trace:
[ 491.492056] [<ffffffff8041ee35>] net_rx_action+0xd5/0x250
[ 491.492056] [<ffffffff800376d8>] __do_softirq+0xe8/0x230
[ 491.492056] [<ffffffff8051151c>] call_softirq+0x1c/0x30
[ 491.492056] [<ffffffff80008a75>] do_softirq+0x75/0xd0
[ 491.492056] [<ffffffff800379f5>] irq_exit+0xb5/0xc0
[ 491.492056] [<ffffffff8036c225>] evtchn_do_upcall+0x295/0x2d0
[ 491.492056] [<ffffffff8051114e>] do_hypervisor_callback+0x1e/0x30
[ 491.492056] [<00007fc519f97700>] 0x7fc519f976ff
[ 491.492056] Code: ff 0f 1f 00 e8 a3 c1 40 e0 85 c0 90 75 69 44 89 ea
4c 89 f6 4c 89 ff e8 f0 cb ff ff c7 85 80 fe ff ff ea ff ff ff e9 7c f4
ff ff <0f> 0b ba 12 00 00 00 48 01 d0 48 39 c1 0f 82 bd fc ff ff e9 e9
[ 491.492056] RIP [<ffffffffa0023aef>] netif_poll+0xe4f/0xf90 [xennet]
[ 491.492056] RSP <ffff8801f5803c60>
[ 491.511975] ---[ end trace c9e37475f12e1aaf ]---
[ 491.512877] Kernel panic - not syncing: Fatal exception in interrupt
In the mean time Jan took the bug in bugzilla
(https://bugzilla.novell.com/show_bug.cgi?id=826374) and created a first
patch. I propose we continue the discussion there and post the
conclusion in this list to finish this thread here as well.
Dion
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-05 10:40 ` [Xen-users] " Dion Kant
@ 2013-07-05 10:54 ` Wei Liu
2013-07-05 19:11 ` Dion Kant
2013-07-05 10:56 ` Jan Beulich
1 sibling, 1 reply; 85+ messages in thread
From: Wei Liu @ 2013-07-05 10:54 UTC (permalink / raw)
To: Dion Kant; +Cc: Wei Liu, jbeulich, xen-devel
On Fri, Jul 05, 2013 at 12:40:02PM +0200, Dion Kant wrote:
> > ======
> > diff --git a/drivers/xen/netfront/netfront.c b/drivers/xen/netfront/netfront.c
> > index 6e5d233..9583011 100644
> > --- a/drivers/xen/netfront/netfront.c
> > +++ b/drivers/xen/netfront/netfront.c
> > @@ -1306,6 +1306,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
> > struct sk_buff *nskb;
> >
> > while ((nskb = __skb_dequeue(list))) {
> > + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> > struct netif_rx_response *rx =
> > RING_GET_RESPONSE(&np->rx, ++cons);
> >
>
> Integrated the patch. I obtained a crash dump and the log in it did not
> show this BUG_ON. Here is the relevant section from the log
>
> var/lib/xen/dump/domUA # crash /root/vmlinux-p1
> 2013-0705-1347.43-domUA.1.core
>
> [ 7.670132] Adding 4192252k swap on /dev/xvda1. Priority:-1
> extents:1 across:4192252k SS
> [ 10.204340] NET: Registered protocol family 17
> [ 481.534979] netfront: Too many frags
> [ 487.543946] netfront: Too many frags
> [ 491.049458] netfront: Too many frags
> [ 491.491153] ------------[ cut here ]------------
> [ 491.491628] kernel BUG at drivers/xen/netfront/netfront.c:1295!
So what's the code around line 1295? There must be a BUG_ON there. It's
normal you didn't see the same line number in my patch (1306) because we
were using different kernel.
> [ 491.492056] invalid opcode: 0000 [#1] SMP
> [ 491.492056] Modules linked in: af_packet autofs4 xennet xenblk cdrom
> [ 491.492056] CPU 0
> [ 491.492056] Pid: 1471, comm: sshd Not tainted 3.7.10-1.16-dbg-p1-xen #8
> [ 491.492056] RIP: e030:[<ffffffffa0023aef>] [<ffffffffa0023aef>]
> netif_poll+0xe4f/0xf90 [xennet]
> [ 491.492056] RSP: e02b:ffff8801f5803c60 EFLAGS: 00010202
> [ 491.492056] RAX: ffff8801f5803da0 RBX: ffff8801f1a082c0 RCX:
> 0000000180200010
> [ 491.492056] RDX: ffff8801f5803da0 RSI: ffff8801fe83ec80 RDI:
> ffff8801f03b2900
> [ 491.492056] RBP: ffff8801f5803e20 R08: 0000000000000001 R09:
> 0000000000000000
> [ 491.492056] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffff8801f03b3400
> [ 491.492056] R13: 0000000000000011 R14: 000000000004327e R15:
> ffff8801f06009c0
> [ 491.492056] FS: 00007fc519f3d7c0(0000) GS:ffff8801f5800000(0000)
> knlGS:0000000000000000
> [ 491.492056] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 491.492056] CR2: 00007fc51410c400 CR3: 00000001f1430000 CR4:
> 0000000000002660
> [ 491.492056] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 491.492056] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 491.492056] Process sshd (pid: 1471, threadinfo ffff8801f1264000,
> task ffff8801f137bf00)
> [ 491.492056] Stack:
> [ 491.492056] ffff8801f5803d60 ffffffff8008503e ffff8801f0600a40
> ffff8801f0600000
> [ 491.492056] 0004328000000040 0000001200000000 ffff8801f5810570
> ffff8801f0600a78
> [ 491.492056] 0000000000000000 ffff8801f0601fb0 0004326e00000012
> ffff8801f5803d00
> [ 491.492056] Call Trace:
> [ 491.492056] [<ffffffff8041ee35>] net_rx_action+0xd5/0x250
> [ 491.492056] [<ffffffff800376d8>] __do_softirq+0xe8/0x230
> [ 491.492056] [<ffffffff8051151c>] call_softirq+0x1c/0x30
> [ 491.492056] [<ffffffff80008a75>] do_softirq+0x75/0xd0
> [ 491.492056] [<ffffffff800379f5>] irq_exit+0xb5/0xc0
> [ 491.492056] [<ffffffff8036c225>] evtchn_do_upcall+0x295/0x2d0
> [ 491.492056] [<ffffffff8051114e>] do_hypervisor_callback+0x1e/0x30
> [ 491.492056] [<00007fc519f97700>] 0x7fc519f976ff
> [ 491.492056] Code: ff 0f 1f 00 e8 a3 c1 40 e0 85 c0 90 75 69 44 89 ea
> 4c 89 f6 4c 89 ff e8 f0 cb ff ff c7 85 80 fe ff ff ea ff ff ff e9 7c f4
> ff ff <0f> 0b ba 12 00 00 00 48 01 d0 48 39 c1 0f 82 bd fc ff ff e9 e9
> [ 491.492056] RIP [<ffffffffa0023aef>] netif_poll+0xe4f/0xf90 [xennet]
> [ 491.492056] RSP <ffff8801f5803c60>
> [ 491.511975] ---[ end trace c9e37475f12e1aaf ]---
> [ 491.512877] Kernel panic - not syncing: Fatal exception in interrupt
>
> In the mean time Jan took the bug in bugzilla
> (https://bugzilla.novell.com/show_bug.cgi?id=826374) and created a first
> patch. I propose we continue the discussion there and post the
> conclusion in this list to finish this thread here as well.
>
I'm not so sure what's the best way. In general it would be much a
burden for developers to look at every bugzilla, let alone register
account for each bugzilla.
Anyway, as we already started the discussion here, I think we should
just continue it here. We could do this the other way around, discuss
here and post conclusion / references there.
Wei.
>
> Dion
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-05 10:40 ` [Xen-users] " Dion Kant
2013-07-05 10:54 ` Wei Liu
@ 2013-07-05 10:56 ` Jan Beulich
2013-07-05 19:46 ` Dion Kant
1 sibling, 1 reply; 85+ messages in thread
From: Jan Beulich @ 2013-07-05 10:56 UTC (permalink / raw)
To: Dion Kant; +Cc: Wei Liu, xen-devel
>>> On 05.07.13 at 12:40, Dion Kant <g.w.kant@hunenet.nl> wrote:
> On 07/04/2013 05:01 PM, Wei Liu wrote:
>> --- a/drivers/xen/netfront/netfront.c
>> +++ b/drivers/xen/netfront/netfront.c
>> @@ -1306,6 +1306,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
>> struct sk_buff *nskb;
>>
>> while ((nskb = __skb_dequeue(list))) {
>> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
>> struct netif_rx_response *rx =
>> RING_GET_RESPONSE(&np->rx, ++cons);
>>
>
> Integrated the patch. I obtained a crash dump and the log in it did not
> show this BUG_ON. Here is the relevant section from the log
>
> var/lib/xen/dump/domUA # crash /root/vmlinux-p1
> 2013-0705-1347.43-domUA.1.core
>
> [ 7.670132] Adding 4192252k swap on /dev/xvda1. Priority:-1 extents:1 across:4192252k SS
> [ 10.204340] NET: Registered protocol family 17
> [ 481.534979] netfront: Too many frags
> [ 487.543946] netfront: Too many frags
> [ 491.049458] netfront: Too many frags
> [ 491.491153] ------------[ cut here ]------------
> [ 491.491628] kernel BUG at drivers/xen/netfront/netfront.c:1295!
So if not the BUG_ON() from the patch above, what else does that
line have in your sources?
Jan
> [ 491.492056] invalid opcode: 0000 [#1] SMP
> [ 491.492056] Modules linked in: af_packet autofs4 xennet xenblk cdrom
> [ 491.492056] CPU 0
> [ 491.492056] Pid: 1471, comm: sshd Not tainted 3.7.10-1.16-dbg-p1-xen #8
> [ 491.492056] RIP: e030:[<ffffffffa0023aef>] [<ffffffffa0023aef>]
> netif_poll+0xe4f/0xf90 [xennet]
> [ 491.492056] RSP: e02b:ffff8801f5803c60 EFLAGS: 00010202
> [ 491.492056] RAX: ffff8801f5803da0 RBX: ffff8801f1a082c0 RCX:
> 0000000180200010
> [ 491.492056] RDX: ffff8801f5803da0 RSI: ffff8801fe83ec80 RDI:
> ffff8801f03b2900
> [ 491.492056] RBP: ffff8801f5803e20 R08: 0000000000000001 R09:
> 0000000000000000
> [ 491.492056] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffff8801f03b3400
> [ 491.492056] R13: 0000000000000011 R14: 000000000004327e R15:
> ffff8801f06009c0
> [ 491.492056] FS: 00007fc519f3d7c0(0000) GS:ffff8801f5800000(0000)
> knlGS:0000000000000000
> [ 491.492056] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 491.492056] CR2: 00007fc51410c400 CR3: 00000001f1430000 CR4:
> 0000000000002660
> [ 491.492056] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 491.492056] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 491.492056] Process sshd (pid: 1471, threadinfo ffff8801f1264000,
> task ffff8801f137bf00)
> [ 491.492056] Stack:
> [ 491.492056] ffff8801f5803d60 ffffffff8008503e ffff8801f0600a40
> ffff8801f0600000
> [ 491.492056] 0004328000000040 0000001200000000 ffff8801f5810570
> ffff8801f0600a78
> [ 491.492056] 0000000000000000 ffff8801f0601fb0 0004326e00000012
> ffff8801f5803d00
> [ 491.492056] Call Trace:
> [ 491.492056] [<ffffffff8041ee35>] net_rx_action+0xd5/0x250
> [ 491.492056] [<ffffffff800376d8>] __do_softirq+0xe8/0x230
> [ 491.492056] [<ffffffff8051151c>] call_softirq+0x1c/0x30
> [ 491.492056] [<ffffffff80008a75>] do_softirq+0x75/0xd0
> [ 491.492056] [<ffffffff800379f5>] irq_exit+0xb5/0xc0
> [ 491.492056] [<ffffffff8036c225>] evtchn_do_upcall+0x295/0x2d0
> [ 491.492056] [<ffffffff8051114e>] do_hypervisor_callback+0x1e/0x30
> [ 491.492056] [<00007fc519f97700>] 0x7fc519f976ff
> [ 491.492056] Code: ff 0f 1f 00 e8 a3 c1 40 e0 85 c0 90 75 69 44 89 ea
> 4c 89 f6 4c 89 ff e8 f0 cb ff ff c7 85 80 fe ff ff ea ff ff ff e9 7c f4
> ff ff <0f> 0b ba 12 00 00 00 48 01 d0 48 39 c1 0f 82 bd fc ff ff e9 e9
> [ 491.492056] RIP [<ffffffffa0023aef>] netif_poll+0xe4f/0xf90 [xennet]
> [ 491.492056] RSP <ffff8801f5803c60>
> [ 491.511975] ---[ end trace c9e37475f12e1aaf ]---
> [ 491.512877] Kernel panic - not syncing: Fatal exception in interrupt
>
> In the mean time Jan took the bug in bugzilla
> (https://bugzilla.novell.com/show_bug.cgi?id=826374) and created a first
> patch. I propose we continue the discussion there and post the
> conclusion in this list to finish this thread here as well.
>
>
> Dion
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-05 9:32 ` [PATCH] xen-netfront: pull on receive skb may need to happen earlier Jan Beulich
2013-07-05 14:53 ` Wei Liu
@ 2013-07-05 14:53 ` Wei Liu
2013-07-07 1:10 ` David Miller
` (3 more replies)
1 sibling, 4 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-05 14:53 UTC (permalink / raw)
To: Jan Beulich
Cc: davem, Ian Campbell, wei.liu2, Dion Kant, xen-devel, netdev, stable
On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
> Due to commit 3683243b ("xen-netfront: use __pskb_pull_tail to ensure
> linear area is big enough on RX") xennet_fill_frags() may end up
> filling MAX_SKB_FRAGS + 1 fragments in a receive skb, and only reduce
> the fragment count subsequently via __pskb_pull_tail(). That's a
> result of xennet_get_responses() allowing a maximum of one more slot to
> be consumed (and intermediately transformed into a fragment) if the
> head slot has a size less than or equal to RX_COPY_THRESHOLD.
>
> Hence we need to adjust xennet_fill_frags() to pull earlier if we
> reached the maximum fragment count - due to the described behavior of
> xennet_get_responses() this guarantees that at least the first fragment
> will get completely consumed, and hence the fragment count reduced.
>
> In order to not needlessly call __pskb_pull_tail() twice, make the
> original call conditional upon the pull target not having been reached
> yet, and defer the newly added one as much as possible (an alternative
> would have been to always call the function right before the call to
> xennet_fill_frags(), but that would imply more frequent cases of
> needing to call it twice).
>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> Cc: Wei Liu <wei.liu2@citrix.com>
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: stable@vger.kernel.org (3.6 onwards)
>
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
> RING_GET_RESPONSE(&np->rx, ++cons);
> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>
> + if (nr_frags == MAX_SKB_FRAGS) {
> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> +
> + BUG_ON(pull_to <= skb_headlen(skb));
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
skb_headlen is in fact "skb->len - skb->data_len". Looking at the
caller code:
while loop {
skb_shinfo(skb)->frags[0].page_offset = rx->offset;
skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
skb->data_len = rx->status;
i = xennet_fill_frags(np, skb, &tmpq);
/*
* Truesize is the actual allocation size, even if the
* allocation is only partially used.
*/
skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
skb->len += skb->data_len;
}
handle_incoming_packet();
You seem to be altering the behavior of the original code, because in
your patch the skb->len is incremented before use, while in the original
code (which calls skb_headlen in handle_incoming_packet) the skb->len is
correctly set.
> + nr_frags = shinfo->nr_frags;
> + }
> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> +
> __skb_fill_page_desc(skb, nr_frags,
> skb_frag_page(nfrag),
> rx->offset, rx->status);
> @@ -929,7 +938,8 @@ static int handle_incoming_queue(struct
> while ((skb = __skb_dequeue(rxq)) != NULL) {
> int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>
> - __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + if (pull_to > skb_headlen(skb))
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>
> /* Ethernet work: Delayed to here as it peeks the header. */
> skb->protocol = eth_type_trans(skb, dev);
>
>
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-05 9:32 ` [PATCH] xen-netfront: pull on receive skb may need to happen earlier Jan Beulich
@ 2013-07-05 14:53 ` Wei Liu
2013-07-05 14:53 ` Wei Liu
1 sibling, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-05 14:53 UTC (permalink / raw)
To: Jan Beulich
Cc: wei.liu2, Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
> Due to commit 3683243b ("xen-netfront: use __pskb_pull_tail to ensure
> linear area is big enough on RX") xennet_fill_frags() may end up
> filling MAX_SKB_FRAGS + 1 fragments in a receive skb, and only reduce
> the fragment count subsequently via __pskb_pull_tail(). That's a
> result of xennet_get_responses() allowing a maximum of one more slot to
> be consumed (and intermediately transformed into a fragment) if the
> head slot has a size less than or equal to RX_COPY_THRESHOLD.
>
> Hence we need to adjust xennet_fill_frags() to pull earlier if we
> reached the maximum fragment count - due to the described behavior of
> xennet_get_responses() this guarantees that at least the first fragment
> will get completely consumed, and hence the fragment count reduced.
>
> In order to not needlessly call __pskb_pull_tail() twice, make the
> original call conditional upon the pull target not having been reached
> yet, and defer the newly added one as much as possible (an alternative
> would have been to always call the function right before the call to
> xennet_fill_frags(), but that would imply more frequent cases of
> needing to call it twice).
>
> Signed-off-by: Jan Beulich <jbeulich@suse.com>
> Cc: Wei Liu <wei.liu2@citrix.com>
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: stable@vger.kernel.org (3.6 onwards)
>
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
> RING_GET_RESPONSE(&np->rx, ++cons);
> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>
> + if (nr_frags == MAX_SKB_FRAGS) {
> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> +
> + BUG_ON(pull_to <= skb_headlen(skb));
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
skb_headlen is in fact "skb->len - skb->data_len". Looking at the
caller code:
while loop {
skb_shinfo(skb)->frags[0].page_offset = rx->offset;
skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
skb->data_len = rx->status;
i = xennet_fill_frags(np, skb, &tmpq);
/*
* Truesize is the actual allocation size, even if the
* allocation is only partially used.
*/
skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
skb->len += skb->data_len;
}
handle_incoming_packet();
You seem to be altering the behavior of the original code, because in
your patch the skb->len is incremented before use, while in the original
code (which calls skb_headlen in handle_incoming_packet) the skb->len is
correctly set.
> + nr_frags = shinfo->nr_frags;
> + }
> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> +
> __skb_fill_page_desc(skb, nr_frags,
> skb_frag_page(nfrag),
> rx->offset, rx->status);
> @@ -929,7 +938,8 @@ static int handle_incoming_queue(struct
> while ((skb = __skb_dequeue(rxq)) != NULL) {
> int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>
> - __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + if (pull_to > skb_headlen(skb))
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>
> /* Ethernet work: Delayed to here as it peeks the header. */
> skb->protocol = eth_type_trans(skb, dev);
>
>
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-05 10:54 ` Wei Liu
@ 2013-07-05 19:11 ` Dion Kant
0 siblings, 0 replies; 85+ messages in thread
From: Dion Kant @ 2013-07-05 19:11 UTC (permalink / raw)
To: Wei Liu; +Cc: jbeulich, xen-devel
On 07/05/2013 12:54 PM, Wei Liu wrote:
>>
>> [ 7.670132] Adding 4192252k swap on /dev/xvda1. Priority:-1
>> extents:1 across:4192252k SS
>> [ 10.204340] NET: Registered protocol family 17
>> [ 481.534979] netfront: Too many frags
>> [ 487.543946] netfront: Too many frags
>> [ 491.049458] netfront: Too many frags
>> [ 491.491153] ------------[ cut here ]------------
>> [ 491.491628] kernel BUG at drivers/xen/netfront/netfront.c:1295!
>
> So what's the code around line 1295? There must be a BUG_ON there. It's
> normal you didn't see the same line number in my patch (1306) because we
> were using different kernel.
>
Yes, stupid me. I was definitely in a wrong state when writing this.
Even more stupid, I did make the same stupid remark after obtaining
results with Jan's patch. 1000 times sorry... ( I was to much in a hurry
to get my response back to you likely because I was happy that you guys
could free some for this.)
So now we know that nr_frags >= MAX_SKB_FRAGS
1285 static RING_IDX xennet_fill_frags(struct netfront_info *np,
1286 struct sk_buff *skb,
1287 struct sk_buff_head *list)
1288 {
1289 struct skb_shared_info *shinfo = skb_shinfo(skb);
1290 int nr_frags = shinfo->nr_frags;
1291 RING_IDX cons = np->rx.rsp_cons;
1292 struct sk_buff *nskb;
1293
1294 while ((nskb = __skb_dequeue(list))) {
1295 BUG_ON(nr_frags >= MAX_SKB_FRAGS);
1296
1297 struct netif_rx_response *rx =
1298 RING_GET_RESPONSE(&np->rx, ++cons);
1299
>>
>> In the mean time Jan took the bug in bugzilla
>> (https://bugzilla.novell.com/show_bug.cgi?id=826374) and created a first
>> patch. I propose we continue the discussion there and post the
>> conclusion in this list to finish this thread here as well.
>>
>
> I'm not so sure what's the best way. In general it would be much a
> burden for developers to look at every bugzilla, let alone register
> account for each bugzilla.
>
> Anyway, as we already started the discussion here, I think we should
> just continue it here. We could do this the other way around, discuss
> here and post conclusion / references there.
This is oke with me. I was wondering what would be most convenient.
Dion
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-05 10:56 ` Jan Beulich
@ 2013-07-05 19:46 ` Dion Kant
2013-07-06 13:36 ` Wei Liu
2013-07-08 8:25 ` Jan Beulich
0 siblings, 2 replies; 85+ messages in thread
From: Dion Kant @ 2013-07-05 19:46 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, xen-devel
On 07/05/2013 12:56 PM, Jan Beulich wrote:
>>>> On 05.07.13 at 12:40, Dion Kant <g.w.kant@hunenet.nl> wrote:
>> On 07/04/2013 05:01 PM, Wei Liu wrote:
>>> --- a/drivers/xen/netfront/netfront.c
>>> +++ b/drivers/xen/netfront/netfront.c
>>> @@ -1306,6 +1306,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
>>> struct sk_buff *nskb;
>>>
>>> while ((nskb = __skb_dequeue(list))) {
>>> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
>>> struct netif_rx_response *rx =
>>> RING_GET_RESPONSE(&np->rx, ++cons);
>>>
>>
>> Integrated the patch. I obtained a crash dump and the log in it did not
>> show this BUG_ON. Here is the relevant section from the log
>>
>> var/lib/xen/dump/domUA # crash /root/vmlinux-p1
>> 2013-0705-1347.43-domUA.1.core
>>
>> [ 7.670132] Adding 4192252k swap on /dev/xvda1. Priority:-1 extents:1 across:4192252k SS
>> [ 10.204340] NET: Registered protocol family 17
>> [ 481.534979] netfront: Too many frags
>> [ 487.543946] netfront: Too many frags
>> [ 491.049458] netfront: Too many frags
>> [ 491.491153] ------------[ cut here ]------------
>> [ 491.491628] kernel BUG at drivers/xen/netfront/netfront.c:1295!
>
> So if not the BUG_ON() from the patch above, what else does that
> line have in your sources?
Nothing else, but thanks for pointing this out to me.
After obtaining results with your patch
1285 static RING_IDX xennet_fill_frags(struct netfront_info *np,
1286 struct sk_buff *skb,
1287 struct sk_buff_head *list)
1288 {
1289 struct skb_shared_info *shinfo = skb_shinfo(skb);
1290 int nr_frags = shinfo->nr_frags;
1291 RING_IDX cons = np->rx.rsp_cons;
1292 struct sk_buff *nskb;
1293
1294 while ((nskb = __skb_dequeue(list))) {
1295 struct netif_rx_response *rx =
1296 RING_GET_RESPONSE(&np->rx, ++cons);
1297
1298
1299 if (nr_frags == MAX_SKB_FRAGS) {
1300 unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
1301
1302 BUG_ON(pull_to <= skb_headlen(skb));
1303 __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
1304 nr_frags = shinfo->nr_frags;
1305 }
1306 BUG_ON(nr_frags >= MAX_SKB_FRAGS);
1307
1308 __skb_fill_page_desc(skb, nr_frags,
1309 skb_frag_page(skb_shinfo(nskb)->frags),
1310 rx->offset, rx->status);
Can I conclude that nr_frags == MAX_SKB_FRAGS, pull_to <=
skb_headlen(skb) and the panic happens before the next BUG_ON is reached ?
> [ 717.568040] netfront: Too many frags
> [ 723.768738] ------------[ cut here ]------------
> [ 723.769226] kernel BUG at drivers/xen/netfront/netfront.c:1302!
> [ 723.769724] invalid opcode: 0000 [#1] SMP
> [ 723.770203] Modules linked in: af_packet autofs4 xennet xenblk cdrom
> [ 723.770697] CPU 0
> [ 723.770710] Pid: 1309, comm: sshd Not tainted 3.7.10-1.16-dbg-jbp1-xen #9
> [ 723.771667] RIP: e030:[<ffffffffa0023b17>] [<ffffffffa0023b17>] netif_poll+0xe77/0xf70 [xennet]
> [ 723.772057] RSP: e02b:ffff8800fb403c60 EFLAGS: 00010213
> [ 723.772057] RAX: 00000000ffff0986 RBX: 0000000000049b7d RCX: ffff8800f78a5428
> [ 723.772057] RDX: 000000000000007d RSI: 0000000000000042 RDI: ffff8800f98026c0
> [ 723.772057] RBP: ffff8800fb403e20 R08: 0000000000000001 R09: 0000000000000000
> [ 723.772057] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800f79709c0
> [ 723.772057] R13: 0000000000000011 R14: ffff8800f866bc00 R15: ffff8800f85a5800
> [ 723.772057] FS: 00007fc8f97e87c0(0000) GS:ffff8800fb400000(0000) knlGS:0000000000000000
> [ 723.772057] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 723.772057] CR2: 00007fc8f3aafff0 CR3: 00000000f8a82000 CR4: 0000000000002660
> [ 723.772057] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 723.772057] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 723.772057] Process sshd (pid: 1309, threadinfo ffff8800f88f4000, task ffff8800f8279f80)
> [ 723.772057] Stack:
> [ 723.772057] ffff8800fb403d60 ffff8800f7970a40 ffff8800f7970000 0000004000000002
> [ 723.772057] 0000000000049b7e ffff8800fb410570 ffff8800f7970a78 0000000000000012
> [ 723.772057] ffff8800f7971fb0 0000001200000000 ffff8800f8b8d2c0 ffff8800fb403d50
> [ 723.772057] Call Trace:
> [ 723.772057] [<ffffffff8041ee35>] net_rx_action+0xd5/0x250
> [ 723.772057] [<ffffffff800376d8>] __do_softirq+0xe8/0x230
> [ 723.772057] [<ffffffff8051151c>] call_softirq+0x1c/0x30
> [ 723.772057] [<ffffffff80008a75>] do_softirq+0x75/0xd0
> [ 723.772057] [<ffffffff800379f5>] irq_exit+0xb5/0xc0
> [ 723.772057] [<ffffffff8036c225>] evtchn_do_upcall+0x295/0x2d0
> [ 723.772057] [<ffffffff8051114e>] do_hypervisor_callback+0x1e/0x30
> [ 723.772057] [<00007fc8f8c5529b>] 0x7fc8f8c5529a
> [ 723.772057] Code: 85 7c fe ff ff ea ff ff ff e9 69 f4 ff ff ba 12 00 00 00 48 01 d0 48 39 c1 0f
> 82 b2 fc ff ff e9 e0 fe ff ff ba 08 00 00 00 eb e8 <0f> 0b 0f b7 33 48 c7 c7
> 70 6c 02 a0 31 c0 e8 40 7a 4d e0 c7 85
> [ 723.772057] RIP [<ffffffffa0023b17>] netif_poll+0xe77/0xf70 [xennet]
> [ 723.772057] RSP <ffff8800fb403c60>
> [ 723.790939] ---[ end trace 846551a77e015655 ]---
> [ 723.791847] Kernel panic - not syncing: Fatal exception in interrupt
Dion.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-05 19:46 ` Dion Kant
@ 2013-07-06 13:36 ` Wei Liu
2013-07-08 8:25 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-06 13:36 UTC (permalink / raw)
To: Dion Kant; +Cc: Wei Liu, Jan Beulich, xen-devel
On Fri, Jul 05, 2013 at 09:46:25PM +0200, Dion Kant wrote:
> On 07/05/2013 12:56 PM, Jan Beulich wrote:
> >>>> On 05.07.13 at 12:40, Dion Kant <g.w.kant@hunenet.nl> wrote:
> >> On 07/04/2013 05:01 PM, Wei Liu wrote:
> >>> --- a/drivers/xen/netfront/netfront.c
> >>> +++ b/drivers/xen/netfront/netfront.c
> >>> @@ -1306,6 +1306,7 @@ static RING_IDX xennet_fill_frags(struct netfront_info *np,
> >>> struct sk_buff *nskb;
> >>>
> >>> while ((nskb = __skb_dequeue(list))) {
> >>> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> >>> struct netif_rx_response *rx =
> >>> RING_GET_RESPONSE(&np->rx, ++cons);
> >>>
> >>
> >> Integrated the patch. I obtained a crash dump and the log in it did not
> >> show this BUG_ON. Here is the relevant section from the log
> >>
> >> var/lib/xen/dump/domUA # crash /root/vmlinux-p1
> >> 2013-0705-1347.43-domUA.1.core
> >>
> >> [ 7.670132] Adding 4192252k swap on /dev/xvda1. Priority:-1 extents:1 across:4192252k SS
> >> [ 10.204340] NET: Registered protocol family 17
> >> [ 481.534979] netfront: Too many frags
> >> [ 487.543946] netfront: Too many frags
> >> [ 491.049458] netfront: Too many frags
> >> [ 491.491153] ------------[ cut here ]------------
> >> [ 491.491628] kernel BUG at drivers/xen/netfront/netfront.c:1295!
> >
> > So if not the BUG_ON() from the patch above, what else does that
> > line have in your sources?
>
> Nothing else, but thanks for pointing this out to me.
>
> After obtaining results with your patch
>
> 1285 static RING_IDX xennet_fill_frags(struct netfront_info *np,
> 1286 struct sk_buff *skb,
> 1287 struct sk_buff_head *list)
> 1288 {
> 1289 struct skb_shared_info *shinfo = skb_shinfo(skb);
> 1290 int nr_frags = shinfo->nr_frags;
> 1291 RING_IDX cons = np->rx.rsp_cons;
> 1292 struct sk_buff *nskb;
> 1293
> 1294 while ((nskb = __skb_dequeue(list))) {
> 1295 struct netif_rx_response *rx =
> 1296 RING_GET_RESPONSE(&np->rx, ++cons);
> 1297
> 1298
> 1299 if (nr_frags == MAX_SKB_FRAGS) {
> 1300 unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> 1301
> 1302 BUG_ON(pull_to <= skb_headlen(skb));
> 1303 __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> 1304 nr_frags = shinfo->nr_frags;
> 1305 }
> 1306 BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> 1307
> 1308 __skb_fill_page_desc(skb, nr_frags,
> 1309 skb_frag_page(skb_shinfo(nskb)->frags),
> 1310 rx->offset, rx->status);
>
> Can I conclude that nr_frags == MAX_SKB_FRAGS, pull_to <=
> skb_headlen(skb) and the panic happens before the next BUG_ON is reached ?
>
So the oops message below is from a kernel with Jan's patch applied,
right? The quoting level confuses me.
> > [ 717.568040] netfront: Too many frags
> > [ 723.768738] ------------[ cut here ]------------
> > [ 723.769226] kernel BUG at drivers/xen/netfront/netfront.c:1302!
>From you description and the line number I presume it is a kernel with
Jan's patch applied.
Looking at Jan's patch for upstream I discovered a problem that might
lead to this crash.
Wei.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-05 14:53 ` Wei Liu
2013-07-07 1:10 ` David Miller
@ 2013-07-07 1:10 ` David Miller
2013-07-08 9:59 ` Jan Beulich
2013-07-08 9:59 ` Jan Beulich
3 siblings, 0 replies; 85+ messages in thread
From: David Miller @ 2013-07-07 1:10 UTC (permalink / raw)
To: wei.liu2; +Cc: JBeulich, ian.campbell, g.w.kant, xen-devel, netdev, stable
From: Wei Liu <wei.liu2@citrix.com>
Date: Fri, 5 Jul 2013 15:53:19 +0100
> You seem to be altering the behavior of the original code, because in
> your patch the skb->len is incremented before use, while in the original
> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
> correctly set.
Indeed, I think you're right.
Because __pskb_pull_tail() decrements the number of bytes pulled
from skb->data_len, it won't be accounted for in skb->len
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-05 14:53 ` Wei Liu
@ 2013-07-07 1:10 ` David Miller
2013-07-07 1:10 ` David Miller
` (2 subsequent siblings)
3 siblings, 0 replies; 85+ messages in thread
From: David Miller @ 2013-07-07 1:10 UTC (permalink / raw)
To: wei.liu2; +Cc: ian.campbell, netdev, stable, xen-devel, JBeulich, g.w.kant
From: Wei Liu <wei.liu2@citrix.com>
Date: Fri, 5 Jul 2013 15:53:19 +0100
> You seem to be altering the behavior of the original code, because in
> your patch the skb->len is incremented before use, while in the original
> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
> correctly set.
Indeed, I think you're right.
Because __pskb_pull_tail() decrements the number of bytes pulled
from skb->data_len, it won't be accounted for in skb->len
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
2013-07-05 19:46 ` Dion Kant
2013-07-06 13:36 ` Wei Liu
@ 2013-07-08 8:25 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-08 8:25 UTC (permalink / raw)
To: Dion Kant; +Cc: Wei Liu, xen-devel
>>> On 05.07.13 at 21:46, Dion Kant <g.w.kant@hunenet.nl> wrote:
> After obtaining results with your patch
>
> 1285 static RING_IDX xennet_fill_frags(struct netfront_info *np,
> 1286 struct sk_buff *skb,
> 1287 struct sk_buff_head *list)
> 1288 {
> 1289 struct skb_shared_info *shinfo = skb_shinfo(skb);
> 1290 int nr_frags = shinfo->nr_frags;
> 1291 RING_IDX cons = np->rx.rsp_cons;
> 1292 struct sk_buff *nskb;
> 1293
> 1294 while ((nskb = __skb_dequeue(list))) {
> 1295 struct netif_rx_response *rx =
> 1296 RING_GET_RESPONSE(&np->rx, ++cons);
> 1297
> 1298
> 1299 if (nr_frags == MAX_SKB_FRAGS) {
> 1300 unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> 1301
> 1302 BUG_ON(pull_to <= skb_headlen(skb));
> 1303 __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> 1304 nr_frags = shinfo->nr_frags;
> 1305 }
> 1306 BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> 1307
> 1308 __skb_fill_page_desc(skb, nr_frags,
> 1309 skb_frag_page(skb_shinfo(nskb)->frags),
> 1310 rx->offset, rx->status);
>
> Can I conclude that nr_frags == MAX_SKB_FRAGS, pull_to <=
> skb_headlen(skb) and the panic happens before the next BUG_ON is reached ?
Yes, exactly. And Wei pointed at an issue with the patch, so I'll
have to see whether that might be an explanation for the crash
you observed.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-05 14:53 ` Wei Liu
2013-07-07 1:10 ` David Miller
2013-07-07 1:10 ` David Miller
@ 2013-07-08 9:59 ` Jan Beulich
2013-07-08 12:16 ` Dion Kant
` (2 more replies)
2013-07-08 9:59 ` Jan Beulich
3 siblings, 3 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-08 9:59 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Campbell, davem, Dion Kant, xen-devel, netdev, stable
>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
>> --- a/drivers/net/xen-netfront.c
>> +++ b/drivers/net/xen-netfront.c
>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
>> RING_GET_RESPONSE(&np->rx, ++cons);
>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>>
>> + if (nr_frags == MAX_SKB_FRAGS) {
>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>> +
>> + BUG_ON(pull_to <= skb_headlen(skb));
>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>
> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
> caller code:
>
> while loop {
> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> skb->data_len = rx->status;
>
> i = xennet_fill_frags(np, skb, &tmpq);
>
> /*
>
> * Truesize is the actual allocation size, even if the
>
> * allocation is only partially used.
>
> */
> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> skb->len += skb->data_len;
> }
>
> handle_incoming_packet();
>
> You seem to be altering the behavior of the original code, because in
> your patch the skb->len is incremented before use, while in the original
> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
> correctly set.
Right. So I basically need to keep skb->len up-to-date along with
->data_len. Just handed a patch to Dion with that done; I'll defer
sending a v2 for the upstream code until I know the change works
for our kernel.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-05 14:53 ` Wei Liu
` (2 preceding siblings ...)
2013-07-08 9:59 ` Jan Beulich
@ 2013-07-08 9:59 ` Jan Beulich
3 siblings, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-08 9:59 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
>> --- a/drivers/net/xen-netfront.c
>> +++ b/drivers/net/xen-netfront.c
>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
>> RING_GET_RESPONSE(&np->rx, ++cons);
>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>>
>> + if (nr_frags == MAX_SKB_FRAGS) {
>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>> +
>> + BUG_ON(pull_to <= skb_headlen(skb));
>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>
> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
> caller code:
>
> while loop {
> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> skb->data_len = rx->status;
>
> i = xennet_fill_frags(np, skb, &tmpq);
>
> /*
>
> * Truesize is the actual allocation size, even if the
>
> * allocation is only partially used.
>
> */
> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> skb->len += skb->data_len;
> }
>
> handle_incoming_packet();
>
> You seem to be altering the behavior of the original code, because in
> your patch the skb->len is incremented before use, while in the original
> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
> correctly set.
Right. So I basically need to keep skb->len up-to-date along with
->data_len. Just handed a patch to Dion with that done; I'll defer
sending a v2 for the upstream code until I know the change works
for our kernel.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 9:59 ` Jan Beulich
@ 2013-07-08 12:16 ` Dion Kant
2013-07-08 12:41 ` Jan Beulich
2013-07-08 12:41 ` Jan Beulich
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
2013-07-08 14:20 ` Jan Beulich
2 siblings, 2 replies; 85+ messages in thread
From: Dion Kant @ 2013-07-08 12:16 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, davem
On 07/08/2013 11:59 AM, Jan Beulich wrote:
>>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
>> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
>>> --- a/drivers/net/xen-netfront.c
>>> +++ b/drivers/net/xen-netfront.c
>>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
>>> RING_GET_RESPONSE(&np->rx, ++cons);
>>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>>>
>>> + if (nr_frags == MAX_SKB_FRAGS) {
>>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>>> +
>>> + BUG_ON(pull_to <= skb_headlen(skb));
>>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>>
>> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
>> caller code:
>>
>> while loop {
>> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
>> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
>> skb->data_len = rx->status;
>>
>> i = xennet_fill_frags(np, skb, &tmpq);
>>
>> /*
>>
>> * Truesize is the actual allocation size, even if the
>>
>> * allocation is only partially used.
>>
>> */
>> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
>> skb->len += skb->data_len;
>> }
>>
>> handle_incoming_packet();
>>
>> You seem to be altering the behavior of the original code, because in
>> your patch the skb->len is incremented before use, while in the original
>> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
>> correctly set.
>
> Right. So I basically need to keep skb->len up-to-date along with
> ->data_len. Just handed a patch to Dion with that done; I'll defer
> sending a v2 for the upstream code until I know the change works
> for our kernel.
>
> Jan
>
Jan,
I was wondering about the following.
In netif_poll(struct napi_struct *napi, int budget) the skb->cb is assigend,
but it may be clipped to RX_COPY_THRESHOLD
NETFRONT_SKB_CB(skb)->pull_to = rx->status;
if (NETFRONT_SKB_CB(skb)->pull_to > RX_COPY_THRESHOLD)
NETFRONT_SKB_CB(skb)->pull_to = RX_COPY_THRESHOLD;
How does this modification of NETFRONT_SKB_CB(skb)->pull_to propagates
or is this irrelevant?
Dion
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 12:16 ` Dion Kant
@ 2013-07-08 12:41 ` Jan Beulich
2013-07-08 12:41 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-08 12:41 UTC (permalink / raw)
To: Dion Kant; +Cc: Ian Campbell, Wei Liu, davem, xen-devel, netdev, stable
>>> On 08.07.13 at 14:16, Dion Kant <g.w.kant@hunenet.nl> wrote:
> I was wondering about the following.
>
> In netif_poll(struct napi_struct *napi, int budget) the skb->cb is assigend,
> but it may be clipped to RX_COPY_THRESHOLD
>
> NETFRONT_SKB_CB(skb)->pull_to = rx->status;
> if (NETFRONT_SKB_CB(skb)->pull_to > RX_COPY_THRESHOLD)
> NETFRONT_SKB_CB(skb)->pull_to = RX_COPY_THRESHOLD;
>
> How does this modification of NETFRONT_SKB_CB(skb)->pull_to propagates
> or is this irrelevant?
I'm not sure I understand what you're asking. pull_to is a private
(to netfront) field used to communicate how much of the skb
needs to be pulled. I don't see how the word "propagate" makes
any sense here, but as said I may simply not be getting what
you're asking about.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 12:16 ` Dion Kant
2013-07-08 12:41 ` Jan Beulich
@ 2013-07-08 12:41 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-08 12:41 UTC (permalink / raw)
To: Dion Kant; +Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, davem
>>> On 08.07.13 at 14:16, Dion Kant <g.w.kant@hunenet.nl> wrote:
> I was wondering about the following.
>
> In netif_poll(struct napi_struct *napi, int budget) the skb->cb is assigend,
> but it may be clipped to RX_COPY_THRESHOLD
>
> NETFRONT_SKB_CB(skb)->pull_to = rx->status;
> if (NETFRONT_SKB_CB(skb)->pull_to > RX_COPY_THRESHOLD)
> NETFRONT_SKB_CB(skb)->pull_to = RX_COPY_THRESHOLD;
>
> How does this modification of NETFRONT_SKB_CB(skb)->pull_to propagates
> or is this irrelevant?
I'm not sure I understand what you're asking. pull_to is a private
(to netfront) field used to communicate how much of the skb
needs to be pulled. I don't see how the word "propagate" makes
any sense here, but as said I may simply not be getting what
you're asking about.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 9:59 ` Jan Beulich
2013-07-08 12:16 ` Dion Kant
@ 2013-07-08 14:20 ` Jan Beulich
2013-07-08 15:22 ` Eric Dumazet
` (5 more replies)
2013-07-08 14:20 ` Jan Beulich
2 siblings, 6 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-08 14:20 UTC (permalink / raw)
To: Wei Liu, davem; +Cc: Ian Campbell, Dion Kant, xen-devel, netdev, stable
>>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
>>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
>> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
>>> --- a/drivers/net/xen-netfront.c
>>> +++ b/drivers/net/xen-netfront.c
>>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
>>> RING_GET_RESPONSE(&np->rx, ++cons);
>>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>>>
>>> + if (nr_frags == MAX_SKB_FRAGS) {
>>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>>> +
>>> + BUG_ON(pull_to <= skb_headlen(skb));
>>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>>
>> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
>> caller code:
>>
>> while loop {
>> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
>> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
>> skb->data_len = rx->status;
>>
>> i = xennet_fill_frags(np, skb, &tmpq);
>>
>> /*
>
>>
>> * Truesize is the actual allocation size, even if the
>
>>
>> * allocation is only partially used.
>
>>
>> */
>> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
>> skb->len += skb->data_len;
>> }
>>
>> handle_incoming_packet();
>>
>> You seem to be altering the behavior of the original code, because in
>> your patch the skb->len is incremented before use, while in the original
>> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
>> correctly set.
>
> Right. So I basically need to keep skb->len up-to-date along with
> ->data_len. Just handed a patch to Dion with that done; I'll defer
> sending a v2 for the upstream code until I know the change works
> for our kernel.
Okay, so with that done (see below) Dion is now seeing the
WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
course, with it having crashed before, it's hard to tell whether the
triggering now is an effect of the patch, or just got unmasked by it.
Looking over the ->truesize handling, I can't see how the change
here could break things: RX_COPY_THRESHOLD is already
accounted for by how alloc_skb() gets called, and the increment
right after the call to xennet_fill_frags() should now be even more
correct than before (since __pskb_pull_tail() can drop fragments,
which would then have made this an over-estimation afaict).
That all said with me knowing pretty little about the networking
code, so I'd appreciate if you could point out anything wrong with
my idea of how things work. Additionally - is my fundamental (for
this patch) assumption right that multiple __pskb_pull_tail() call
are cumulative (i.e. calling it twice with a delta of
pull_to - skb_headlen(skb) would indeed end up pulling up to
pull_to, provided there is enough data)?
Jan
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -831,10 +831,20 @@ static RING_IDX xennet_fill_frags(struct
RING_GET_RESPONSE(&np->rx, ++cons);
skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
+ if (nr_frags == MAX_SKB_FRAGS) {
+ unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
+
+ BUG_ON(pull_to <= skb_headlen(skb));
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ nr_frags = shinfo->nr_frags;
+ }
+ BUG_ON(nr_frags >= MAX_SKB_FRAGS);
+
__skb_fill_page_desc(skb, nr_frags,
skb_frag_page(nfrag),
rx->offset, rx->status);
+ skb->len += rx->status;
skb->data_len += rx->status;
skb_shinfo(nskb)->nr_frags = 0;
@@ -929,7 +939,8 @@ static int handle_incoming_queue(struct
while ((skb = __skb_dequeue(rxq)) != NULL) {
int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
- __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ if (pull_to > skb_headlen(skb))
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
/* Ethernet work: Delayed to here as it peeks the header. */
skb->protocol = eth_type_trans(skb, dev);
@@ -1014,7 +1025,7 @@ err:
skb_shinfo(skb)->frags[0].page_offset = rx->offset;
skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
- skb->data_len = rx->status;
+ skb->len = skb->data_len = rx->status;
i = xennet_fill_frags(np, skb, &tmpq);
@@ -1023,7 +1034,6 @@ err:
* allocation is only partially used.
*/
skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
- skb->len += skb->data_len;
if (rx->flags & XEN_NETRXF_csum_blank)
skb->ip_summed = CHECKSUM_PARTIAL;
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 9:59 ` Jan Beulich
2013-07-08 12:16 ` Dion Kant
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
@ 2013-07-08 14:20 ` Jan Beulich
2 siblings, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-08 14:20 UTC (permalink / raw)
To: Wei Liu, davem; +Cc: netdev, Dion Kant, Ian Campbell, stable, xen-devel
>>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
>>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
>> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
>>> --- a/drivers/net/xen-netfront.c
>>> +++ b/drivers/net/xen-netfront.c
>>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
>>> RING_GET_RESPONSE(&np->rx, ++cons);
>>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>>>
>>> + if (nr_frags == MAX_SKB_FRAGS) {
>>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>>> +
>>> + BUG_ON(pull_to <= skb_headlen(skb));
>>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>>
>> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
>> caller code:
>>
>> while loop {
>> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
>> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
>> skb->data_len = rx->status;
>>
>> i = xennet_fill_frags(np, skb, &tmpq);
>>
>> /*
>
>>
>> * Truesize is the actual allocation size, even if the
>
>>
>> * allocation is only partially used.
>
>>
>> */
>> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
>> skb->len += skb->data_len;
>> }
>>
>> handle_incoming_packet();
>>
>> You seem to be altering the behavior of the original code, because in
>> your patch the skb->len is incremented before use, while in the original
>> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
>> correctly set.
>
> Right. So I basically need to keep skb->len up-to-date along with
> ->data_len. Just handed a patch to Dion with that done; I'll defer
> sending a v2 for the upstream code until I know the change works
> for our kernel.
Okay, so with that done (see below) Dion is now seeing the
WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
course, with it having crashed before, it's hard to tell whether the
triggering now is an effect of the patch, or just got unmasked by it.
Looking over the ->truesize handling, I can't see how the change
here could break things: RX_COPY_THRESHOLD is already
accounted for by how alloc_skb() gets called, and the increment
right after the call to xennet_fill_frags() should now be even more
correct than before (since __pskb_pull_tail() can drop fragments,
which would then have made this an over-estimation afaict).
That all said with me knowing pretty little about the networking
code, so I'd appreciate if you could point out anything wrong with
my idea of how things work. Additionally - is my fundamental (for
this patch) assumption right that multiple __pskb_pull_tail() call
are cumulative (i.e. calling it twice with a delta of
pull_to - skb_headlen(skb) would indeed end up pulling up to
pull_to, provided there is enough data)?
Jan
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -831,10 +831,20 @@ static RING_IDX xennet_fill_frags(struct
RING_GET_RESPONSE(&np->rx, ++cons);
skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
+ if (nr_frags == MAX_SKB_FRAGS) {
+ unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
+
+ BUG_ON(pull_to <= skb_headlen(skb));
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ nr_frags = shinfo->nr_frags;
+ }
+ BUG_ON(nr_frags >= MAX_SKB_FRAGS);
+
__skb_fill_page_desc(skb, nr_frags,
skb_frag_page(nfrag),
rx->offset, rx->status);
+ skb->len += rx->status;
skb->data_len += rx->status;
skb_shinfo(nskb)->nr_frags = 0;
@@ -929,7 +939,8 @@ static int handle_incoming_queue(struct
while ((skb = __skb_dequeue(rxq)) != NULL) {
int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
- __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ if (pull_to > skb_headlen(skb))
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
/* Ethernet work: Delayed to here as it peeks the header. */
skb->protocol = eth_type_trans(skb, dev);
@@ -1014,7 +1025,7 @@ err:
skb_shinfo(skb)->frags[0].page_offset = rx->offset;
skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
- skb->data_len = rx->status;
+ skb->len = skb->data_len = rx->status;
i = xennet_fill_frags(np, skb, &tmpq);
@@ -1023,7 +1034,6 @@ err:
* allocation is only partially used.
*/
skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
- skb->len += skb->data_len;
if (rx->flags & XEN_NETRXF_csum_blank)
skb->ip_summed = CHECKSUM_PARTIAL;
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
2013-07-08 15:22 ` Eric Dumazet
@ 2013-07-08 15:22 ` Eric Dumazet
2013-07-09 7:47 ` Jan Beulich
2013-07-09 7:47 ` Jan Beulich
2013-07-08 15:48 ` [Xen-devel] " Wei Liu
` (3 subsequent siblings)
5 siblings, 2 replies; 85+ messages in thread
From: Eric Dumazet @ 2013-07-08 15:22 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, davem, Ian Campbell, Dion Kant, xen-devel, netdev, stable
On Mon, 2013-07-08 at 15:20 +0100, Jan Beulich wrote:
> Okay, so with that done (see below) Dion is now seeing the
> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
> course, with it having crashed before, it's hard to tell whether the
> triggering now is an effect of the patch, or just got unmasked by it.
Please note this warning is not the sign of an error.
It can be triggered because network stack (IP + TCP) had to reallocate
skb->head to hold all the headers.
pskb_may_pull() doesn't change skb->truesize, for various reasons, so we
can have a situation where the warning triggers.
For example, it can happen when drivers use a really small skb->head,
like 64 bytes, as its not big enough to hold ethernet+IP+TCP headers.
For example, I did a patch on a wifi driver (iwl3945: better skb
management in rx path) because of this sub optimal skb->head sizing.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
@ 2013-07-08 15:22 ` Eric Dumazet
2013-07-08 15:22 ` [Xen-devel] " Eric Dumazet
` (4 subsequent siblings)
5 siblings, 0 replies; 85+ messages in thread
From: Eric Dumazet @ 2013-07-08 15:22 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
On Mon, 2013-07-08 at 15:20 +0100, Jan Beulich wrote:
> Okay, so with that done (see below) Dion is now seeing the
> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
> course, with it having crashed before, it's hard to tell whether the
> triggering now is an effect of the patch, or just got unmasked by it.
Please note this warning is not the sign of an error.
It can be triggered because network stack (IP + TCP) had to reallocate
skb->head to hold all the headers.
pskb_may_pull() doesn't change skb->truesize, for various reasons, so we
can have a situation where the warning triggers.
For example, it can happen when drivers use a really small skb->head,
like 64 bytes, as its not big enough to hold ethernet+IP+TCP headers.
For example, I did a patch on a wifi driver (iwl3945: better skb
management in rx path) because of this sub optimal skb->head sizing.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
2013-07-08 15:22 ` Eric Dumazet
2013-07-08 15:22 ` [Xen-devel] " Eric Dumazet
@ 2013-07-08 15:48 ` Wei Liu
2013-07-09 6:52 ` Jan Beulich
2013-07-09 6:52 ` [Xen-devel] " Jan Beulich
2013-07-08 15:48 ` Wei Liu
` (2 subsequent siblings)
5 siblings, 2 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-08 15:48 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, davem, Ian Campbell, Dion Kant, xen-devel, netdev, stable
On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
> >>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
> >>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
> >> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
> >>> --- a/drivers/net/xen-netfront.c
> >>> +++ b/drivers/net/xen-netfront.c
> >>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
> >>> RING_GET_RESPONSE(&np->rx, ++cons);
> >>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
> >>>
> >>> + if (nr_frags == MAX_SKB_FRAGS) {
> >>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> >>> +
> >>> + BUG_ON(pull_to <= skb_headlen(skb));
> >>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> >>
> >> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
> >> caller code:
> >>
> >> while loop {
> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> >> skb->data_len = rx->status;
> >>
> >> i = xennet_fill_frags(np, skb, &tmpq);
> >>
> >> /*
> >
> >>
> >> * Truesize is the actual allocation size, even if the
> >
> >>
> >> * allocation is only partially used.
> >
> >>
> >> */
> >> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> >> skb->len += skb->data_len;
> >> }
> >>
> >> handle_incoming_packet();
> >>
> >> You seem to be altering the behavior of the original code, because in
> >> your patch the skb->len is incremented before use, while in the original
> >> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
> >> correctly set.
> >
> > Right. So I basically need to keep skb->len up-to-date along with
> > ->data_len. Just handed a patch to Dion with that done; I'll defer
> > sending a v2 for the upstream code until I know the change works
> > for our kernel.
>
> Okay, so with that done (see below) Dion is now seeing the
> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
> course, with it having crashed before, it's hard to tell whether the
> triggering now is an effect of the patch, or just got unmasked by it.
>
I think the previous crash was due to incorrect accounting.
BUG_ON(pull_to <= skb_headlen(skb))
skb_headlen(skb) = skb->len - skb->data_len
In your first patch, skb_headlen(skb) can return a very large number if
skb->len is smaller then skb->data_len (they are both unsigned), because
before calling skb_headlen() skb->data_len is incremented while skb->len
is not.
As for the WARN_ON_ONCE(delte < len), I haven't looked that far as I
discover a problem with your new patch. Please see below comments.
> Looking over the ->truesize handling, I can't see how the change
> here could break things: RX_COPY_THRESHOLD is already
> accounted for by how alloc_skb() gets called, and the increment
> right after the call to xennet_fill_frags() should now be even more
> correct than before (since __pskb_pull_tail() can drop fragments,
> which would then have made this an over-estimation afaict).
>
> That all said with me knowing pretty little about the networking
> code, so I'd appreciate if you could point out anything wrong with
> my idea of how things work. Additionally - is my fundamental (for
> this patch) assumption right that multiple __pskb_pull_tail() call
> are cumulative (i.e. calling it twice with a delta of
> pull_to - skb_headlen(skb) would indeed end up pulling up to
> pull_to, provided there is enough data)?
>
> Jan
>
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -831,10 +831,20 @@ static RING_IDX xennet_fill_frags(struct
> RING_GET_RESPONSE(&np->rx, ++cons);
> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>
> + if (nr_frags == MAX_SKB_FRAGS) {
> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> +
> + BUG_ON(pull_to <= skb_headlen(skb));
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + nr_frags = shinfo->nr_frags;
> + }
> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> +
> __skb_fill_page_desc(skb, nr_frags,
> skb_frag_page(nfrag),
> rx->offset, rx->status);
>
> + skb->len += rx->status;
> skb->data_len += rx->status;
>
> skb_shinfo(nskb)->nr_frags = 0;
> @@ -929,7 +939,8 @@ static int handle_incoming_queue(struct
> while ((skb = __skb_dequeue(rxq)) != NULL) {
> int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>
> - __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + if (pull_to > skb_headlen(skb))
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>
> /* Ethernet work: Delayed to here as it peeks the header. */
> skb->protocol = eth_type_trans(skb, dev);
> @@ -1014,7 +1025,7 @@ err:
>
> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> - skb->data_len = rx->status;
> + skb->len = skb->data_len = rx->status;
This is not correct. You should not be needing this. Now you lose count
of SKB head len. Try to go without the above line and see if it makes a
difference?
Wei.
>
> i = xennet_fill_frags(np, skb, &tmpq);
>
> @@ -1023,7 +1034,6 @@ err:
> * allocation is only partially used.
> */
> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> - skb->len += skb->data_len;
>
> if (rx->flags & XEN_NETRXF_csum_blank)
> skb->ip_summed = CHECKSUM_PARTIAL;
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
` (2 preceding siblings ...)
2013-07-08 15:48 ` [Xen-devel] " Wei Liu
@ 2013-07-08 15:48 ` Wei Liu
2013-07-12 8:32 ` [Xen-devel] " Wei Liu
2013-07-12 8:32 ` Wei Liu
5 siblings, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-08 15:48 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
> >>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
> >>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
> >> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
> >>> --- a/drivers/net/xen-netfront.c
> >>> +++ b/drivers/net/xen-netfront.c
> >>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
> >>> RING_GET_RESPONSE(&np->rx, ++cons);
> >>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
> >>>
> >>> + if (nr_frags == MAX_SKB_FRAGS) {
> >>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> >>> +
> >>> + BUG_ON(pull_to <= skb_headlen(skb));
> >>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> >>
> >> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
> >> caller code:
> >>
> >> while loop {
> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> >> skb->data_len = rx->status;
> >>
> >> i = xennet_fill_frags(np, skb, &tmpq);
> >>
> >> /*
> >
> >>
> >> * Truesize is the actual allocation size, even if the
> >
> >>
> >> * allocation is only partially used.
> >
> >>
> >> */
> >> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> >> skb->len += skb->data_len;
> >> }
> >>
> >> handle_incoming_packet();
> >>
> >> You seem to be altering the behavior of the original code, because in
> >> your patch the skb->len is incremented before use, while in the original
> >> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
> >> correctly set.
> >
> > Right. So I basically need to keep skb->len up-to-date along with
> > ->data_len. Just handed a patch to Dion with that done; I'll defer
> > sending a v2 for the upstream code until I know the change works
> > for our kernel.
>
> Okay, so with that done (see below) Dion is now seeing the
> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
> course, with it having crashed before, it's hard to tell whether the
> triggering now is an effect of the patch, or just got unmasked by it.
>
I think the previous crash was due to incorrect accounting.
BUG_ON(pull_to <= skb_headlen(skb))
skb_headlen(skb) = skb->len - skb->data_len
In your first patch, skb_headlen(skb) can return a very large number if
skb->len is smaller then skb->data_len (they are both unsigned), because
before calling skb_headlen() skb->data_len is incremented while skb->len
is not.
As for the WARN_ON_ONCE(delte < len), I haven't looked that far as I
discover a problem with your new patch. Please see below comments.
> Looking over the ->truesize handling, I can't see how the change
> here could break things: RX_COPY_THRESHOLD is already
> accounted for by how alloc_skb() gets called, and the increment
> right after the call to xennet_fill_frags() should now be even more
> correct than before (since __pskb_pull_tail() can drop fragments,
> which would then have made this an over-estimation afaict).
>
> That all said with me knowing pretty little about the networking
> code, so I'd appreciate if you could point out anything wrong with
> my idea of how things work. Additionally - is my fundamental (for
> this patch) assumption right that multiple __pskb_pull_tail() call
> are cumulative (i.e. calling it twice with a delta of
> pull_to - skb_headlen(skb) would indeed end up pulling up to
> pull_to, provided there is enough data)?
>
> Jan
>
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -831,10 +831,20 @@ static RING_IDX xennet_fill_frags(struct
> RING_GET_RESPONSE(&np->rx, ++cons);
> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>
> + if (nr_frags == MAX_SKB_FRAGS) {
> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> +
> + BUG_ON(pull_to <= skb_headlen(skb));
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + nr_frags = shinfo->nr_frags;
> + }
> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> +
> __skb_fill_page_desc(skb, nr_frags,
> skb_frag_page(nfrag),
> rx->offset, rx->status);
>
> + skb->len += rx->status;
> skb->data_len += rx->status;
>
> skb_shinfo(nskb)->nr_frags = 0;
> @@ -929,7 +939,8 @@ static int handle_incoming_queue(struct
> while ((skb = __skb_dequeue(rxq)) != NULL) {
> int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>
> - __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + if (pull_to > skb_headlen(skb))
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>
> /* Ethernet work: Delayed to here as it peeks the header. */
> skb->protocol = eth_type_trans(skb, dev);
> @@ -1014,7 +1025,7 @@ err:
>
> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> - skb->data_len = rx->status;
> + skb->len = skb->data_len = rx->status;
This is not correct. You should not be needing this. Now you lose count
of SKB head len. Try to go without the above line and see if it makes a
difference?
Wei.
>
> i = xennet_fill_frags(np, skb, &tmpq);
>
> @@ -1023,7 +1034,6 @@ err:
> * allocation is only partially used.
> */
> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> - skb->len += skb->data_len;
>
> if (rx->flags & XEN_NETRXF_csum_blank)
> skb->ip_summed = CHECKSUM_PARTIAL;
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 15:48 ` [Xen-devel] " Wei Liu
2013-07-09 6:52 ` Jan Beulich
@ 2013-07-09 6:52 ` Jan Beulich
2013-07-09 16:51 ` Wei Liu
2013-07-09 16:51 ` [Xen-devel] " Wei Liu
1 sibling, 2 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-09 6:52 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Campbell, davem, Dion Kant, xen-devel, netdev, stable
>>> On 08.07.13 at 17:48, Wei Liu <wei.liu2@citrix.com> wrote:
> On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
>> @@ -1014,7 +1025,7 @@ err:
>>
>> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
>> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
>> - skb->data_len = rx->status;
>> + skb->len = skb->data_len = rx->status;
>
> This is not correct. You should not be needing this. Now you lose count
> of SKB head len. Try to go without the above line and see if it makes a
> difference?
I don't follow - at this point, there's 0 bytes of head (this only
changes with the first call to __pskb_pull_tail()). Hence ->len ==
->data_len seems correct to me (and afaict pulling would do the
wrong thing if I dropped that change).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 15:48 ` [Xen-devel] " Wei Liu
@ 2013-07-09 6:52 ` Jan Beulich
2013-07-09 6:52 ` [Xen-devel] " Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-09 6:52 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
>>> On 08.07.13 at 17:48, Wei Liu <wei.liu2@citrix.com> wrote:
> On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
>> @@ -1014,7 +1025,7 @@ err:
>>
>> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
>> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
>> - skb->data_len = rx->status;
>> + skb->len = skb->data_len = rx->status;
>
> This is not correct. You should not be needing this. Now you lose count
> of SKB head len. Try to go without the above line and see if it makes a
> difference?
I don't follow - at this point, there's 0 bytes of head (this only
changes with the first call to __pskb_pull_tail()). Hence ->len ==
->data_len seems correct to me (and afaict pulling would do the
wrong thing if I dropped that change).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 15:22 ` [Xen-devel] " Eric Dumazet
@ 2013-07-09 7:47 ` Jan Beulich
2013-07-09 7:47 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-09 7:47 UTC (permalink / raw)
To: Eric Dumazet
Cc: Ian Campbell, Wei Liu, davem, Dion Kant, xen-devel, netdev, stable
>>> On 08.07.13 at 17:22, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Mon, 2013-07-08 at 15:20 +0100, Jan Beulich wrote:
>
>> Okay, so with that done (see below) Dion is now seeing the
>> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
>> course, with it having crashed before, it's hard to tell whether the
>> triggering now is an effect of the patch, or just got unmasked by it.
>
> Please note this warning is not the sign of an error.
Thanks for pointing this out.
> It can be triggered because network stack (IP + TCP) had to reallocate
> skb->head to hold all the headers.
>
> pskb_may_pull() doesn't change skb->truesize, for various reasons, so we
> can have a situation where the warning triggers.
>
> For example, it can happen when drivers use a really small skb->head,
> like 64 bytes, as its not big enough to hold ethernet+IP+TCP headers.
We know from the crashes prior to the patch here that this is
exactly what is happening.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 15:22 ` [Xen-devel] " Eric Dumazet
2013-07-09 7:47 ` Jan Beulich
@ 2013-07-09 7:47 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-09 7:47 UTC (permalink / raw)
To: Eric Dumazet
Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
>>> On 08.07.13 at 17:22, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Mon, 2013-07-08 at 15:20 +0100, Jan Beulich wrote:
>
>> Okay, so with that done (see below) Dion is now seeing the
>> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
>> course, with it having crashed before, it's hard to tell whether the
>> triggering now is an effect of the patch, or just got unmasked by it.
>
> Please note this warning is not the sign of an error.
Thanks for pointing this out.
> It can be triggered because network stack (IP + TCP) had to reallocate
> skb->head to hold all the headers.
>
> pskb_may_pull() doesn't change skb->truesize, for various reasons, so we
> can have a situation where the warning triggers.
>
> For example, it can happen when drivers use a really small skb->head,
> like 64 bytes, as its not big enough to hold ethernet+IP+TCP headers.
We know from the crashes prior to the patch here that this is
exactly what is happening.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-09 6:52 ` [Xen-devel] " Jan Beulich
2013-07-09 16:51 ` Wei Liu
@ 2013-07-09 16:51 ` Wei Liu
2013-07-10 6:58 ` Jan Beulich
2013-07-10 6:58 ` [Xen-devel] " Jan Beulich
1 sibling, 2 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-09 16:51 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, davem, Dion Kant, xen-devel, netdev, stable
On Tue, Jul 09, 2013 at 07:52:31AM +0100, Jan Beulich wrote:
> >>> On 08.07.13 at 17:48, Wei Liu <wei.liu2@citrix.com> wrote:
> > On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
> >> @@ -1014,7 +1025,7 @@ err:
> >>
> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> >> - skb->data_len = rx->status;
> >> + skb->len = skb->data_len = rx->status;
> >
> > This is not correct. You should not be needing this. Now you lose count
> > of SKB head len. Try to go without the above line and see if it makes a
> > difference?
>
> I don't follow - at this point, there's 0 bytes of head (this only
> changes with the first call to __pskb_pull_tail()). Hence ->len ==
> ->data_len seems correct to me (and afaict pulling would do the
> wrong thing if I dropped that change).
>
My bad, I suggested the wrong thing. :-(
But I would prefer skb->len += skb->data_len. In the case that skb->len
== 0 it's the same as your line while skb->len is not zero it would also
do the right thing.
As for the warning in skb_try_coalesce, I don't see any direct call to
it in netfront, I will need to think about it. It looks like it's really
something very deep in the stack.
Wei.
> Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-09 6:52 ` [Xen-devel] " Jan Beulich
@ 2013-07-09 16:51 ` Wei Liu
2013-07-09 16:51 ` [Xen-devel] " Wei Liu
1 sibling, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-09 16:51 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
On Tue, Jul 09, 2013 at 07:52:31AM +0100, Jan Beulich wrote:
> >>> On 08.07.13 at 17:48, Wei Liu <wei.liu2@citrix.com> wrote:
> > On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
> >> @@ -1014,7 +1025,7 @@ err:
> >>
> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> >> - skb->data_len = rx->status;
> >> + skb->len = skb->data_len = rx->status;
> >
> > This is not correct. You should not be needing this. Now you lose count
> > of SKB head len. Try to go without the above line and see if it makes a
> > difference?
>
> I don't follow - at this point, there's 0 bytes of head (this only
> changes with the first call to __pskb_pull_tail()). Hence ->len ==
> ->data_len seems correct to me (and afaict pulling would do the
> wrong thing if I dropped that change).
>
My bad, I suggested the wrong thing. :-(
But I would prefer skb->len += skb->data_len. In the case that skb->len
== 0 it's the same as your line while skb->len is not zero it would also
do the right thing.
As for the warning in skb_try_coalesce, I don't see any direct call to
it in netfront, I will need to think about it. It looks like it's really
something very deep in the stack.
Wei.
> Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-09 16:51 ` [Xen-devel] " Wei Liu
2013-07-10 6:58 ` Jan Beulich
@ 2013-07-10 6:58 ` Jan Beulich
2013-07-10 10:04 ` Wei Liu
` (3 more replies)
1 sibling, 4 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-10 6:58 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Campbell, davem, Dion Kant, xen-devel, netdev, stable
>>> On 09.07.13 at 18:51, Wei Liu <wei.liu2@citrix.com> wrote:
> On Tue, Jul 09, 2013 at 07:52:31AM +0100, Jan Beulich wrote:
>> >>> On 08.07.13 at 17:48, Wei Liu <wei.liu2@citrix.com> wrote:
>> > On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
>> >> @@ -1014,7 +1025,7 @@ err:
>> >>
>> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
>> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
>> >> - skb->data_len = rx->status;
>> >> + skb->len = skb->data_len = rx->status;
>> >
>> > This is not correct. You should not be needing this. Now you lose count
>> > of SKB head len. Try to go without the above line and see if it makes a
>> > difference?
>>
>> I don't follow - at this point, there's 0 bytes of head (this only
>> changes with the first call to __pskb_pull_tail()). Hence ->len ==
>> ->data_len seems correct to me (and afaict pulling would do the
>> wrong thing if I dropped that change).
>>
>
> My bad, I suggested the wrong thing. :-(
>
> But I would prefer skb->len += skb->data_len. In the case that skb->len
> == 0 it's the same as your line while skb->len is not zero it would also
> do the right thing.
I can do that, albeit I don't see how ->len could end up non-zero
here.
> As for the warning in skb_try_coalesce, I don't see any direct call to
> it in netfront, I will need to think about it. It looks like it's really
> something very deep in the stack.
Yes, as the call stack provided by Dion proves. The question
really is whether the patch somehow results in ->truesize to be
incorrect, or whether - as Eric points out - this is "normal" for
the sort of special SKBs here (having a rather small headlen). If
what he says is applicable here, it may hint at the pulling we do
still not being sufficient for the full TCP header to be in the linear
part (which iirc is the main [if not the only purpose] of us doing
the pull in the first place).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-09 16:51 ` [Xen-devel] " Wei Liu
@ 2013-07-10 6:58 ` Jan Beulich
2013-07-10 6:58 ` [Xen-devel] " Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-10 6:58 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
>>> On 09.07.13 at 18:51, Wei Liu <wei.liu2@citrix.com> wrote:
> On Tue, Jul 09, 2013 at 07:52:31AM +0100, Jan Beulich wrote:
>> >>> On 08.07.13 at 17:48, Wei Liu <wei.liu2@citrix.com> wrote:
>> > On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
>> >> @@ -1014,7 +1025,7 @@ err:
>> >>
>> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
>> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
>> >> - skb->data_len = rx->status;
>> >> + skb->len = skb->data_len = rx->status;
>> >
>> > This is not correct. You should not be needing this. Now you lose count
>> > of SKB head len. Try to go without the above line and see if it makes a
>> > difference?
>>
>> I don't follow - at this point, there's 0 bytes of head (this only
>> changes with the first call to __pskb_pull_tail()). Hence ->len ==
>> ->data_len seems correct to me (and afaict pulling would do the
>> wrong thing if I dropped that change).
>>
>
> My bad, I suggested the wrong thing. :-(
>
> But I would prefer skb->len += skb->data_len. In the case that skb->len
> == 0 it's the same as your line while skb->len is not zero it would also
> do the right thing.
I can do that, albeit I don't see how ->len could end up non-zero
here.
> As for the warning in skb_try_coalesce, I don't see any direct call to
> it in netfront, I will need to think about it. It looks like it's really
> something very deep in the stack.
Yes, as the call stack provided by Dion proves. The question
really is whether the patch somehow results in ->truesize to be
incorrect, or whether - as Eric points out - this is "normal" for
the sort of special SKBs here (having a rather small headlen). If
what he says is applicable here, it may hint at the pulling we do
still not being sufficient for the full TCP header to be in the linear
part (which iirc is the main [if not the only purpose] of us doing
the pull in the first place).
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 6:58 ` [Xen-devel] " Jan Beulich
2013-07-10 10:04 ` Wei Liu
@ 2013-07-10 10:04 ` Wei Liu
2013-07-10 10:46 ` Jan Beulich
2013-07-10 10:46 ` Jan Beulich
2013-07-10 13:19 ` [Xen-devel] " Eric Dumazet
2013-07-10 13:19 ` Eric Dumazet
3 siblings, 2 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-10 10:04 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, davem, Dion Kant, xen-devel, netdev, stable
On Wed, Jul 10, 2013 at 07:58:06AM +0100, Jan Beulich wrote:
> >>> On 09.07.13 at 18:51, Wei Liu <wei.liu2@citrix.com> wrote:
> > On Tue, Jul 09, 2013 at 07:52:31AM +0100, Jan Beulich wrote:
> >> >>> On 08.07.13 at 17:48, Wei Liu <wei.liu2@citrix.com> wrote:
> >> > On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
> >> >> @@ -1014,7 +1025,7 @@ err:
> >> >>
> >> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> >> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> >> >> - skb->data_len = rx->status;
> >> >> + skb->len = skb->data_len = rx->status;
> >> >
> >> > This is not correct. You should not be needing this. Now you lose count
> >> > of SKB head len. Try to go without the above line and see if it makes a
> >> > difference?
> >>
> >> I don't follow - at this point, there's 0 bytes of head (this only
> >> changes with the first call to __pskb_pull_tail()). Hence ->len ==
> >> ->data_len seems correct to me (and afaict pulling would do the
> >> wrong thing if I dropped that change).
> >>
> >
> > My bad, I suggested the wrong thing. :-(
> >
> > But I would prefer skb->len += skb->data_len. In the case that skb->len
> > == 0 it's the same as your line while skb->len is not zero it would also
> > do the right thing.
>
> I can do that, albeit I don't see how ->len could end up non-zero
> here.
>
> > As for the warning in skb_try_coalesce, I don't see any direct call to
> > it in netfront, I will need to think about it. It looks like it's really
> > something very deep in the stack.
>
> Yes, as the call stack provided by Dion proves. The question
> really is whether the patch somehow results in ->truesize to be
> incorrect, or whether - as Eric points out - this is "normal" for
> the sort of special SKBs here (having a rather small headlen). If
> what he says is applicable here, it may hint at the pulling we do
> still not being sufficient for the full TCP header to be in the linear
__netdev_alloc_skb in netfront is fed with RX_COPY_THRESHOLD+NET_IP_ALIGN.
RX_COPY_THRESHOLD is 256 while MAX_TCP_HEADER can be as larger as 256+48
depending on kernel configurations.
> part (which iirc is the main [if not the only purpose] of us doing
> the pull in the first place).
>
Ian, any comment on this?
Jan, looking at the commit log, the overrun issue in
xennet_get_responses was not introduced by __pskb_pull_tail. The call to
xennet_fill_frags has always been in the same place.
Now I start to think about the purpose of "max = MAX_SKB_FRAGS +
(rx->status <= RX_COPY_THRESHOLD)" in xennet_get_responses which queues
up to MAX_SKB_FRAGS+1 responeses. I'm not clear about the rationale
of that line.
Wei.
> Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 6:58 ` [Xen-devel] " Jan Beulich
@ 2013-07-10 10:04 ` Wei Liu
2013-07-10 10:04 ` [Xen-devel] " Wei Liu
` (2 subsequent siblings)
3 siblings, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-10 10:04 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
On Wed, Jul 10, 2013 at 07:58:06AM +0100, Jan Beulich wrote:
> >>> On 09.07.13 at 18:51, Wei Liu <wei.liu2@citrix.com> wrote:
> > On Tue, Jul 09, 2013 at 07:52:31AM +0100, Jan Beulich wrote:
> >> >>> On 08.07.13 at 17:48, Wei Liu <wei.liu2@citrix.com> wrote:
> >> > On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
> >> >> @@ -1014,7 +1025,7 @@ err:
> >> >>
> >> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> >> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> >> >> - skb->data_len = rx->status;
> >> >> + skb->len = skb->data_len = rx->status;
> >> >
> >> > This is not correct. You should not be needing this. Now you lose count
> >> > of SKB head len. Try to go without the above line and see if it makes a
> >> > difference?
> >>
> >> I don't follow - at this point, there's 0 bytes of head (this only
> >> changes with the first call to __pskb_pull_tail()). Hence ->len ==
> >> ->data_len seems correct to me (and afaict pulling would do the
> >> wrong thing if I dropped that change).
> >>
> >
> > My bad, I suggested the wrong thing. :-(
> >
> > But I would prefer skb->len += skb->data_len. In the case that skb->len
> > == 0 it's the same as your line while skb->len is not zero it would also
> > do the right thing.
>
> I can do that, albeit I don't see how ->len could end up non-zero
> here.
>
> > As for the warning in skb_try_coalesce, I don't see any direct call to
> > it in netfront, I will need to think about it. It looks like it's really
> > something very deep in the stack.
>
> Yes, as the call stack provided by Dion proves. The question
> really is whether the patch somehow results in ->truesize to be
> incorrect, or whether - as Eric points out - this is "normal" for
> the sort of special SKBs here (having a rather small headlen). If
> what he says is applicable here, it may hint at the pulling we do
> still not being sufficient for the full TCP header to be in the linear
__netdev_alloc_skb in netfront is fed with RX_COPY_THRESHOLD+NET_IP_ALIGN.
RX_COPY_THRESHOLD is 256 while MAX_TCP_HEADER can be as larger as 256+48
depending on kernel configurations.
> part (which iirc is the main [if not the only purpose] of us doing
> the pull in the first place).
>
Ian, any comment on this?
Jan, looking at the commit log, the overrun issue in
xennet_get_responses was not introduced by __pskb_pull_tail. The call to
xennet_fill_frags has always been in the same place.
Now I start to think about the purpose of "max = MAX_SKB_FRAGS +
(rx->status <= RX_COPY_THRESHOLD)" in xennet_get_responses which queues
up to MAX_SKB_FRAGS+1 responeses. I'm not clear about the rationale
of that line.
Wei.
> Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 10:04 ` [Xen-devel] " Wei Liu
@ 2013-07-10 10:46 ` Jan Beulich
2013-07-10 12:50 ` Ian Campbell
2013-07-10 12:50 ` Ian Campbell
2013-07-10 10:46 ` Jan Beulich
1 sibling, 2 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-10 10:46 UTC (permalink / raw)
To: Ian Campbell, Wei Liu; +Cc: davem, Dion Kant, xen-devel, netdev, stable
>>> On 10.07.13 at 12:04, Wei Liu <wei.liu2@citrix.com> wrote:
> Jan, looking at the commit log, the overrun issue in
> xennet_get_responses was not introduced by __pskb_pull_tail. The call to
> xennet_fill_frags has always been in the same place.
I'm convinced it was: Prior to that commit, if the first response slot
contained up to RX_COPY_THRESHOLD bytes, it got entirely
consumed into the linear portion of the SKB, leaving the number of
fragments available for filling at MAX_SKB_FRAGS. Said commit
dropped the early copying, leaving the fragment count at 1
unconditionally, and now accumulates all of the response slots into
fragments, only pulling after all of them got filled in. It neglected to
realize - due to the count now always being 1 at the beginning - that
this can lead to MAX_SKB_FRAGS + 1 frags getting filled, corrupting
memory.
Ian - I have to admit that I'm slightly irritated by you so far not
having participated at all in sorting out the fix for this bug that a
change of yours introduced.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 10:04 ` [Xen-devel] " Wei Liu
2013-07-10 10:46 ` Jan Beulich
@ 2013-07-10 10:46 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-10 10:46 UTC (permalink / raw)
To: Ian Campbell, Wei Liu; +Cc: netdev, Dion Kant, davem, stable, xen-devel
>>> On 10.07.13 at 12:04, Wei Liu <wei.liu2@citrix.com> wrote:
> Jan, looking at the commit log, the overrun issue in
> xennet_get_responses was not introduced by __pskb_pull_tail. The call to
> xennet_fill_frags has always been in the same place.
I'm convinced it was: Prior to that commit, if the first response slot
contained up to RX_COPY_THRESHOLD bytes, it got entirely
consumed into the linear portion of the SKB, leaving the number of
fragments available for filling at MAX_SKB_FRAGS. Said commit
dropped the early copying, leaving the fragment count at 1
unconditionally, and now accumulates all of the response slots into
fragments, only pulling after all of them got filled in. It neglected to
realize - due to the count now always being 1 at the beginning - that
this can lead to MAX_SKB_FRAGS + 1 frags getting filled, corrupting
memory.
Ian - I have to admit that I'm slightly irritated by you so far not
having participated at all in sorting out the fix for this bug that a
change of yours introduced.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 10:46 ` Jan Beulich
@ 2013-07-10 12:50 ` Ian Campbell
2013-07-10 12:53 ` Wei Liu
` (3 more replies)
2013-07-10 12:50 ` Ian Campbell
1 sibling, 4 replies; 85+ messages in thread
From: Ian Campbell @ 2013-07-10 12:50 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, davem, Dion Kant, xen-devel, netdev, stable
On Wed, 2013-07-10 at 11:46 +0100, Jan Beulich wrote:
> >>> On 10.07.13 at 12:04, Wei Liu <wei.liu2@citrix.com> wrote:
> > Jan, looking at the commit log, the overrun issue in
> > xennet_get_responses was not introduced by __pskb_pull_tail. The call to
> > xennet_fill_frags has always been in the same place.
>
> I'm convinced it was: Prior to that commit, if the first response slot
> contained up to RX_COPY_THRESHOLD bytes, it got entirely
> consumed into the linear portion of the SKB, leaving the number of
> fragments available for filling at MAX_SKB_FRAGS. Said commit
> dropped the early copying, leaving the fragment count at 1
> unconditionally, and now accumulates all of the response slots into
> fragments, only pulling after all of them got filled in. It neglected to
> realize - due to the count now always being 1 at the beginning - that
> this can lead to MAX_SKB_FRAGS + 1 frags getting filled, corrupting
> memory.
That argument makes sense to me.
Is it possible to hit a scenario where we need to pull more than
RX_COPY_THRESHOLD in order to fit all of the data in MAX_SKB_FRAGS ?
> Ian - I have to admit that I'm slightly irritated by you so far not
> having participated at all in sorting out the fix for this bug that a
> change of yours introduced.
Sorry I've been travelling and not following closely enough to realise
this was related to something I'd done.
Does this relate somehow to the patch Annie has sent out recently too?
Ian.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 10:46 ` Jan Beulich
2013-07-10 12:50 ` Ian Campbell
@ 2013-07-10 12:50 ` Ian Campbell
1 sibling, 0 replies; 85+ messages in thread
From: Ian Campbell @ 2013-07-10 12:50 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, netdev, stable, xen-devel, Dion Kant, davem
On Wed, 2013-07-10 at 11:46 +0100, Jan Beulich wrote:
> >>> On 10.07.13 at 12:04, Wei Liu <wei.liu2@citrix.com> wrote:
> > Jan, looking at the commit log, the overrun issue in
> > xennet_get_responses was not introduced by __pskb_pull_tail. The call to
> > xennet_fill_frags has always been in the same place.
>
> I'm convinced it was: Prior to that commit, if the first response slot
> contained up to RX_COPY_THRESHOLD bytes, it got entirely
> consumed into the linear portion of the SKB, leaving the number of
> fragments available for filling at MAX_SKB_FRAGS. Said commit
> dropped the early copying, leaving the fragment count at 1
> unconditionally, and now accumulates all of the response slots into
> fragments, only pulling after all of them got filled in. It neglected to
> realize - due to the count now always being 1 at the beginning - that
> this can lead to MAX_SKB_FRAGS + 1 frags getting filled, corrupting
> memory.
That argument makes sense to me.
Is it possible to hit a scenario where we need to pull more than
RX_COPY_THRESHOLD in order to fit all of the data in MAX_SKB_FRAGS ?
> Ian - I have to admit that I'm slightly irritated by you so far not
> having participated at all in sorting out the fix for this bug that a
> change of yours introduced.
Sorry I've been travelling and not following closely enough to realise
this was related to something I'd done.
Does this relate somehow to the patch Annie has sent out recently too?
Ian.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 12:50 ` Ian Campbell
@ 2013-07-10 12:53 ` Wei Liu
2013-07-10 12:53 ` Wei Liu
` (2 subsequent siblings)
3 siblings, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-10 12:53 UTC (permalink / raw)
To: Ian Campbell
Cc: Jan Beulich, Wei Liu, davem, Dion Kant, xen-devel, netdev, stable
On Wed, Jul 10, 2013 at 01:50:44PM +0100, Ian Campbell wrote:
> On Wed, 2013-07-10 at 11:46 +0100, Jan Beulich wrote:
> > >>> On 10.07.13 at 12:04, Wei Liu <wei.liu2@citrix.com> wrote:
> > > Jan, looking at the commit log, the overrun issue in
> > > xennet_get_responses was not introduced by __pskb_pull_tail. The call to
> > > xennet_fill_frags has always been in the same place.
> >
> > I'm convinced it was: Prior to that commit, if the first response slot
> > contained up to RX_COPY_THRESHOLD bytes, it got entirely
> > consumed into the linear portion of the SKB, leaving the number of
> > fragments available for filling at MAX_SKB_FRAGS. Said commit
> > dropped the early copying, leaving the fragment count at 1
> > unconditionally, and now accumulates all of the response slots into
> > fragments, only pulling after all of them got filled in. It neglected to
> > realize - due to the count now always being 1 at the beginning - that
> > this can lead to MAX_SKB_FRAGS + 1 frags getting filled, corrupting
> > memory.
>
> That argument makes sense to me.
>
> Is it possible to hit a scenario where we need to pull more than
> RX_COPY_THRESHOLD in order to fit all of the data in MAX_SKB_FRAGS ?
>
> > Ian - I have to admit that I'm slightly irritated by you so far not
> > having participated at all in sorting out the fix for this bug that a
> > change of yours introduced.
>
> Sorry I've been travelling and not following closely enough to realise
> this was related to something I'd done.
>
> Does this relate somehow to the patch Annie has sent out recently too?
>
No. That's not related.
Wei.
> Ian.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 12:50 ` Ian Campbell
2013-07-10 12:53 ` Wei Liu
@ 2013-07-10 12:53 ` Wei Liu
2013-07-10 13:58 ` Jan Beulich
2013-07-10 13:58 ` [Xen-devel] " Jan Beulich
3 siblings, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-10 12:53 UTC (permalink / raw)
To: Ian Campbell
Cc: Wei Liu, netdev, stable, xen-devel, Jan Beulich, Dion Kant, davem
On Wed, Jul 10, 2013 at 01:50:44PM +0100, Ian Campbell wrote:
> On Wed, 2013-07-10 at 11:46 +0100, Jan Beulich wrote:
> > >>> On 10.07.13 at 12:04, Wei Liu <wei.liu2@citrix.com> wrote:
> > > Jan, looking at the commit log, the overrun issue in
> > > xennet_get_responses was not introduced by __pskb_pull_tail. The call to
> > > xennet_fill_frags has always been in the same place.
> >
> > I'm convinced it was: Prior to that commit, if the first response slot
> > contained up to RX_COPY_THRESHOLD bytes, it got entirely
> > consumed into the linear portion of the SKB, leaving the number of
> > fragments available for filling at MAX_SKB_FRAGS. Said commit
> > dropped the early copying, leaving the fragment count at 1
> > unconditionally, and now accumulates all of the response slots into
> > fragments, only pulling after all of them got filled in. It neglected to
> > realize - due to the count now always being 1 at the beginning - that
> > this can lead to MAX_SKB_FRAGS + 1 frags getting filled, corrupting
> > memory.
>
> That argument makes sense to me.
>
> Is it possible to hit a scenario where we need to pull more than
> RX_COPY_THRESHOLD in order to fit all of the data in MAX_SKB_FRAGS ?
>
> > Ian - I have to admit that I'm slightly irritated by you so far not
> > having participated at all in sorting out the fix for this bug that a
> > change of yours introduced.
>
> Sorry I've been travelling and not following closely enough to realise
> this was related to something I'd done.
>
> Does this relate somehow to the patch Annie has sent out recently too?
>
No. That's not related.
Wei.
> Ian.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 6:58 ` [Xen-devel] " Jan Beulich
2013-07-10 10:04 ` Wei Liu
2013-07-10 10:04 ` [Xen-devel] " Wei Liu
@ 2013-07-10 13:19 ` Eric Dumazet
2013-07-10 13:19 ` Eric Dumazet
3 siblings, 0 replies; 85+ messages in thread
From: Eric Dumazet @ 2013-07-10 13:19 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, davem, Dion Kant, xen-devel, netdev, stable
On Wed, 2013-07-10 at 07:58 +0100, Jan Beulich wrote:
>
> Yes, as the call stack provided by Dion proves. The question
> really is whether the patch somehow results in ->truesize to be
> incorrect, or whether - as Eric points out - this is "normal" for
> the sort of special SKBs here (having a rather small headlen). If
> what he says is applicable here, it may hint at the pulling we do
> still not being sufficient for the full TCP header to be in the linear
> part (which iirc is the main [if not the only purpose] of us doing
> the pull in the first place).
I have not closely followed the thread (it seems it did not began on
netdev so I do not have the first messages), but if we cook an skb with
no tailroom (headroom is irrelevant), and IP stack or TCP stack has to
pull more bytes from the frags to skb->head, then skb->head must be
reallocated to get more headroom before pulling the headers.
This operation is _expensive_, as it involves copy of struct
skb_shared_info and eventually page refcounts games if original skb
was cloned (sniffer)
We have a one time WARN_ON_ONCE() in skb_try_coalesce(),
to detect such non optimal behavior, but not in the fast path, only in
the part used in TCP coalescing.
So the right thing would be to cook skbs so that there is enough
tailroom.
But there are 256 bytes of tailroom, it should be enough unless some
huge amount of headers are used.
If you believe it's fine and very unlikely, then ignore the warning,
because using 512 bytes of tailroom for all skbs only to cope with very
unlikely reallocations is not worth it.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 6:58 ` [Xen-devel] " Jan Beulich
` (2 preceding siblings ...)
2013-07-10 13:19 ` [Xen-devel] " Eric Dumazet
@ 2013-07-10 13:19 ` Eric Dumazet
3 siblings, 0 replies; 85+ messages in thread
From: Eric Dumazet @ 2013-07-10 13:19 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
On Wed, 2013-07-10 at 07:58 +0100, Jan Beulich wrote:
>
> Yes, as the call stack provided by Dion proves. The question
> really is whether the patch somehow results in ->truesize to be
> incorrect, or whether - as Eric points out - this is "normal" for
> the sort of special SKBs here (having a rather small headlen). If
> what he says is applicable here, it may hint at the pulling we do
> still not being sufficient for the full TCP header to be in the linear
> part (which iirc is the main [if not the only purpose] of us doing
> the pull in the first place).
I have not closely followed the thread (it seems it did not began on
netdev so I do not have the first messages), but if we cook an skb with
no tailroom (headroom is irrelevant), and IP stack or TCP stack has to
pull more bytes from the frags to skb->head, then skb->head must be
reallocated to get more headroom before pulling the headers.
This operation is _expensive_, as it involves copy of struct
skb_shared_info and eventually page refcounts games if original skb
was cloned (sniffer)
We have a one time WARN_ON_ONCE() in skb_try_coalesce(),
to detect such non optimal behavior, but not in the fast path, only in
the part used in TCP coalescing.
So the right thing would be to cook skbs so that there is enough
tailroom.
But there are 256 bytes of tailroom, it should be enough unless some
huge amount of headers are used.
If you believe it's fine and very unlikely, then ignore the warning,
because using 512 bytes of tailroom for all skbs only to cope with very
unlikely reallocations is not worth it.
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 12:50 ` Ian Campbell
` (2 preceding siblings ...)
2013-07-10 13:58 ` Jan Beulich
@ 2013-07-10 13:58 ` Jan Beulich
3 siblings, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-10 13:58 UTC (permalink / raw)
To: Ian Campbell; +Cc: Wei Liu, davem, Dion Kant, xen-devel, netdev, stable
>>> On 10.07.13 at 14:50, Ian Campbell <ian.campbell@citrix.com> wrote:
> On Wed, 2013-07-10 at 11:46 +0100, Jan Beulich wrote:
>> >>> On 10.07.13 at 12:04, Wei Liu <wei.liu2@citrix.com> wrote:
>> > Jan, looking at the commit log, the overrun issue in
>> > xennet_get_responses was not introduced by __pskb_pull_tail. The call to
>> > xennet_fill_frags has always been in the same place.
>>
>> I'm convinced it was: Prior to that commit, if the first response slot
>> contained up to RX_COPY_THRESHOLD bytes, it got entirely
>> consumed into the linear portion of the SKB, leaving the number of
>> fragments available for filling at MAX_SKB_FRAGS. Said commit
>> dropped the early copying, leaving the fragment count at 1
>> unconditionally, and now accumulates all of the response slots into
>> fragments, only pulling after all of them got filled in. It neglected to
>> realize - due to the count now always being 1 at the beginning - that
>> this can lead to MAX_SKB_FRAGS + 1 frags getting filled, corrupting
>> memory.
>
> That argument makes sense to me.
>
> Is it possible to hit a scenario where we need to pull more than
> RX_COPY_THRESHOLD in order to fit all of the data in MAX_SKB_FRAGS ?
I'm not aware of any, but I'm no expert here in any way.
> Does this relate somehow to the patch Annie has sent out recently too?
I don't think so.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-10 12:50 ` Ian Campbell
2013-07-10 12:53 ` Wei Liu
2013-07-10 12:53 ` Wei Liu
@ 2013-07-10 13:58 ` Jan Beulich
2013-07-10 13:58 ` [Xen-devel] " Jan Beulich
3 siblings, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-10 13:58 UTC (permalink / raw)
To: Ian Campbell; +Cc: Wei Liu, netdev, stable, xen-devel, Dion Kant, davem
>>> On 10.07.13 at 14:50, Ian Campbell <ian.campbell@citrix.com> wrote:
> On Wed, 2013-07-10 at 11:46 +0100, Jan Beulich wrote:
>> >>> On 10.07.13 at 12:04, Wei Liu <wei.liu2@citrix.com> wrote:
>> > Jan, looking at the commit log, the overrun issue in
>> > xennet_get_responses was not introduced by __pskb_pull_tail. The call to
>> > xennet_fill_frags has always been in the same place.
>>
>> I'm convinced it was: Prior to that commit, if the first response slot
>> contained up to RX_COPY_THRESHOLD bytes, it got entirely
>> consumed into the linear portion of the SKB, leaving the number of
>> fragments available for filling at MAX_SKB_FRAGS. Said commit
>> dropped the early copying, leaving the fragment count at 1
>> unconditionally, and now accumulates all of the response slots into
>> fragments, only pulling after all of them got filled in. It neglected to
>> realize - due to the count now always being 1 at the beginning - that
>> this can lead to MAX_SKB_FRAGS + 1 frags getting filled, corrupting
>> memory.
>
> That argument makes sense to me.
>
> Is it possible to hit a scenario where we need to pull more than
> RX_COPY_THRESHOLD in order to fit all of the data in MAX_SKB_FRAGS ?
I'm not aware of any, but I'm no expert here in any way.
> Does this relate somehow to the patch Annie has sent out recently too?
I don't think so.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
` (3 preceding siblings ...)
2013-07-08 15:48 ` Wei Liu
@ 2013-07-12 8:32 ` Wei Liu
2013-07-12 8:56 ` Jan Beulich
2013-07-12 8:56 ` Jan Beulich
2013-07-12 8:32 ` Wei Liu
5 siblings, 2 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-12 8:32 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, davem, Ian Campbell, Dion Kant, xen-devel, netdev, stable
On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
> >>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
> >>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
> >> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
> >>> --- a/drivers/net/xen-netfront.c
> >>> +++ b/drivers/net/xen-netfront.c
> >>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
> >>> RING_GET_RESPONSE(&np->rx, ++cons);
> >>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
> >>>
> >>> + if (nr_frags == MAX_SKB_FRAGS) {
> >>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> >>> +
> >>> + BUG_ON(pull_to <= skb_headlen(skb));
> >>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> >>
> >> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
> >> caller code:
> >>
> >> while loop {
> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> >> skb->data_len = rx->status;
> >>
> >> i = xennet_fill_frags(np, skb, &tmpq);
> >>
> >> /*
> >
> >>
> >> * Truesize is the actual allocation size, even if the
> >
> >>
> >> * allocation is only partially used.
> >
> >>
> >> */
> >> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> >> skb->len += skb->data_len;
> >> }
> >>
> >> handle_incoming_packet();
> >>
> >> You seem to be altering the behavior of the original code, because in
> >> your patch the skb->len is incremented before use, while in the original
> >> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
> >> correctly set.
> >
> > Right. So I basically need to keep skb->len up-to-date along with
> > ->data_len. Just handed a patch to Dion with that done; I'll defer
> > sending a v2 for the upstream code until I know the change works
> > for our kernel.
>
> Okay, so with that done (see below) Dion is now seeing the
> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
> course, with it having crashed before, it's hard to tell whether the
> triggering now is an effect of the patch, or just got unmasked by it.
>
I just ported your below patch to upstream kernel and I didn't see the
WARN_ON_ONCE. I only did iperf and netperf tests.
If the work load to trigger this bug is simple enough I can give it a
shot...
Wei.
> Looking over the ->truesize handling, I can't see how the change
> here could break things: RX_COPY_THRESHOLD is already
> accounted for by how alloc_skb() gets called, and the increment
> right after the call to xennet_fill_frags() should now be even more
> correct than before (since __pskb_pull_tail() can drop fragments,
> which would then have made this an over-estimation afaict).
>
> That all said with me knowing pretty little about the networking
> code, so I'd appreciate if you could point out anything wrong with
> my idea of how things work. Additionally - is my fundamental (for
> this patch) assumption right that multiple __pskb_pull_tail() call
> are cumulative (i.e. calling it twice with a delta of
> pull_to - skb_headlen(skb) would indeed end up pulling up to
> pull_to, provided there is enough data)?
>
> Jan
>
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -831,10 +831,20 @@ static RING_IDX xennet_fill_frags(struct
> RING_GET_RESPONSE(&np->rx, ++cons);
> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>
> + if (nr_frags == MAX_SKB_FRAGS) {
> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> +
> + BUG_ON(pull_to <= skb_headlen(skb));
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + nr_frags = shinfo->nr_frags;
> + }
> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> +
> __skb_fill_page_desc(skb, nr_frags,
> skb_frag_page(nfrag),
> rx->offset, rx->status);
>
> + skb->len += rx->status;
> skb->data_len += rx->status;
>
> skb_shinfo(nskb)->nr_frags = 0;
> @@ -929,7 +939,8 @@ static int handle_incoming_queue(struct
> while ((skb = __skb_dequeue(rxq)) != NULL) {
> int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>
> - __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + if (pull_to > skb_headlen(skb))
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>
> /* Ethernet work: Delayed to here as it peeks the header. */
> skb->protocol = eth_type_trans(skb, dev);
> @@ -1014,7 +1025,7 @@ err:
>
> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> - skb->data_len = rx->status;
> + skb->len = skb->data_len = rx->status;
>
> i = xennet_fill_frags(np, skb, &tmpq);
>
> @@ -1023,7 +1034,6 @@ err:
> * allocation is only partially used.
> */
> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> - skb->len += skb->data_len;
>
> if (rx->flags & XEN_NETRXF_csum_blank)
> skb->ip_summed = CHECKSUM_PARTIAL;
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
` (4 preceding siblings ...)
2013-07-12 8:32 ` [Xen-devel] " Wei Liu
@ 2013-07-12 8:32 ` Wei Liu
5 siblings, 0 replies; 85+ messages in thread
From: Wei Liu @ 2013-07-12 8:32 UTC (permalink / raw)
To: Jan Beulich
Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
> >>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
> >>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
> >> On Fri, Jul 05, 2013 at 10:32:41AM +0100, Jan Beulich wrote:
> >>> --- a/drivers/net/xen-netfront.c
> >>> +++ b/drivers/net/xen-netfront.c
> >>> @@ -831,6 +831,15 @@ static RING_IDX xennet_fill_frags(struct
> >>> RING_GET_RESPONSE(&np->rx, ++cons);
> >>> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
> >>>
> >>> + if (nr_frags == MAX_SKB_FRAGS) {
> >>> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> >>> +
> >>> + BUG_ON(pull_to <= skb_headlen(skb));
> >>> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> >>
> >> skb_headlen is in fact "skb->len - skb->data_len". Looking at the
> >> caller code:
> >>
> >> while loop {
> >> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> >> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> >> skb->data_len = rx->status;
> >>
> >> i = xennet_fill_frags(np, skb, &tmpq);
> >>
> >> /*
> >
> >>
> >> * Truesize is the actual allocation size, even if the
> >
> >>
> >> * allocation is only partially used.
> >
> >>
> >> */
> >> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> >> skb->len += skb->data_len;
> >> }
> >>
> >> handle_incoming_packet();
> >>
> >> You seem to be altering the behavior of the original code, because in
> >> your patch the skb->len is incremented before use, while in the original
> >> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
> >> correctly set.
> >
> > Right. So I basically need to keep skb->len up-to-date along with
> > ->data_len. Just handed a patch to Dion with that done; I'll defer
> > sending a v2 for the upstream code until I know the change works
> > for our kernel.
>
> Okay, so with that done (see below) Dion is now seeing the
> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
> course, with it having crashed before, it's hard to tell whether the
> triggering now is an effect of the patch, or just got unmasked by it.
>
I just ported your below patch to upstream kernel and I didn't see the
WARN_ON_ONCE. I only did iperf and netperf tests.
If the work load to trigger this bug is simple enough I can give it a
shot...
Wei.
> Looking over the ->truesize handling, I can't see how the change
> here could break things: RX_COPY_THRESHOLD is already
> accounted for by how alloc_skb() gets called, and the increment
> right after the call to xennet_fill_frags() should now be even more
> correct than before (since __pskb_pull_tail() can drop fragments,
> which would then have made this an over-estimation afaict).
>
> That all said with me knowing pretty little about the networking
> code, so I'd appreciate if you could point out anything wrong with
> my idea of how things work. Additionally - is my fundamental (for
> this patch) assumption right that multiple __pskb_pull_tail() call
> are cumulative (i.e. calling it twice with a delta of
> pull_to - skb_headlen(skb) would indeed end up pulling up to
> pull_to, provided there is enough data)?
>
> Jan
>
> --- a/drivers/net/xen-netfront.c
> +++ b/drivers/net/xen-netfront.c
> @@ -831,10 +831,20 @@ static RING_IDX xennet_fill_frags(struct
> RING_GET_RESPONSE(&np->rx, ++cons);
> skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
>
> + if (nr_frags == MAX_SKB_FRAGS) {
> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> +
> + BUG_ON(pull_to <= skb_headlen(skb));
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + nr_frags = shinfo->nr_frags;
> + }
> + BUG_ON(nr_frags >= MAX_SKB_FRAGS);
> +
> __skb_fill_page_desc(skb, nr_frags,
> skb_frag_page(nfrag),
> rx->offset, rx->status);
>
> + skb->len += rx->status;
> skb->data_len += rx->status;
>
> skb_shinfo(nskb)->nr_frags = 0;
> @@ -929,7 +939,8 @@ static int handle_incoming_queue(struct
> while ((skb = __skb_dequeue(rxq)) != NULL) {
> int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
>
> - __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + if (pull_to > skb_headlen(skb))
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
>
> /* Ethernet work: Delayed to here as it peeks the header. */
> skb->protocol = eth_type_trans(skb, dev);
> @@ -1014,7 +1025,7 @@ err:
>
> skb_shinfo(skb)->frags[0].page_offset = rx->offset;
> skb_frag_size_set(&skb_shinfo(skb)->frags[0], rx->status);
> - skb->data_len = rx->status;
> + skb->len = skb->data_len = rx->status;
>
> i = xennet_fill_frags(np, skb, &tmpq);
>
> @@ -1023,7 +1034,6 @@ err:
> * allocation is only partially used.
> */
> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
> - skb->len += skb->data_len;
>
> if (rx->flags & XEN_NETRXF_csum_blank)
> skb->ip_summed = CHECKSUM_PARTIAL;
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-12 8:32 ` [Xen-devel] " Wei Liu
@ 2013-07-12 8:56 ` Jan Beulich
2013-07-13 11:26 ` Dion Kant
2013-07-13 11:26 ` [Xen-devel] " Dion Kant
2013-07-12 8:56 ` Jan Beulich
1 sibling, 2 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-12 8:56 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Campbell, davem, Dion Kant, xen-devel, netdev, stable
>>> On 12.07.13 at 10:32, Wei Liu <wei.liu2@citrix.com> wrote:
> On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
>> >>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
>> >>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
>> >> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
>> >> skb->len += skb->data_len;
>> >> }
>> >>
>> >> handle_incoming_packet();
>> >>
>> >> You seem to be altering the behavior of the original code, because in
>> >> your patch the skb->len is incremented before use, while in the original
>> >> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
>> >> correctly set.
>> >
>> > Right. So I basically need to keep skb->len up-to-date along with
>> > ->data_len. Just handed a patch to Dion with that done; I'll defer
>> > sending a v2 for the upstream code until I know the change works
>> > for our kernel.
>>
>> Okay, so with that done (see below) Dion is now seeing the
>> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
>> course, with it having crashed before, it's hard to tell whether the
>> triggering now is an effect of the patch, or just got unmasked by it.
>>
>
> I just ported your below patch to upstream kernel and I didn't see the
> WARN_ON_ONCE. I only did iperf and netperf tests.
>
> If the work load to trigger this bug is simple enough I can give it a
> shot...
I'm meanwhile relatively convinced that the warning isn't an effect
of the patch (final verification pending); I intend to submit v2 as
soon as 3.11-rc1 is out.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-12 8:32 ` [Xen-devel] " Wei Liu
2013-07-12 8:56 ` Jan Beulich
@ 2013-07-12 8:56 ` Jan Beulich
1 sibling, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-12 8:56 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Campbell, netdev, stable, xen-devel, Dion Kant, davem
>>> On 12.07.13 at 10:32, Wei Liu <wei.liu2@citrix.com> wrote:
> On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
>> >>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
>> >>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
>> >> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
>> >> skb->len += skb->data_len;
>> >> }
>> >>
>> >> handle_incoming_packet();
>> >>
>> >> You seem to be altering the behavior of the original code, because in
>> >> your patch the skb->len is incremented before use, while in the original
>> >> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
>> >> correctly set.
>> >
>> > Right. So I basically need to keep skb->len up-to-date along with
>> > ->data_len. Just handed a patch to Dion with that done; I'll defer
>> > sending a v2 for the upstream code until I know the change works
>> > for our kernel.
>>
>> Okay, so with that done (see below) Dion is now seeing the
>> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
>> course, with it having crashed before, it's hard to tell whether the
>> triggering now is an effect of the patch, or just got unmasked by it.
>>
>
> I just ported your below patch to upstream kernel and I didn't see the
> WARN_ON_ONCE. I only did iperf and netperf tests.
>
> If the work load to trigger this bug is simple enough I can give it a
> shot...
I'm meanwhile relatively convinced that the warning isn't an effect
of the patch (final verification pending); I intend to submit v2 as
soon as 3.11-rc1 is out.
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [Xen-devel] [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-12 8:56 ` Jan Beulich
2013-07-13 11:26 ` Dion Kant
@ 2013-07-13 11:26 ` Dion Kant
2013-07-13 20:17 ` Dion Kant
1 sibling, 1 reply; 85+ messages in thread
From: Dion Kant @ 2013-07-13 11:26 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, Ian Campbell, davem, xen-devel, netdev, stable
On 07/12/2013 10:56 AM, Jan Beulich wrote:
>>>> On 12.07.13 at 10:32, Wei Liu <wei.liu2@citrix.com> wrote:
>> On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
>>>>>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
>>>>>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
>>>>> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
>>>>> skb->len += skb->data_len;
>>>>> }
>>>>>
>>>>> handle_incoming_packet();
>>>>>
>>>>> You seem to be altering the behavior of the original code, because in
>>>>> your patch the skb->len is incremented before use, while in the original
>>>>> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
>>>>> correctly set.
>>>>
>>>> Right. So I basically need to keep skb->len up-to-date along with
>>>> ->data_len. Just handed a patch to Dion with that done; I'll defer
>>>> sending a v2 for the upstream code until I know the change works
>>>> for our kernel.
>>>
>>> Okay, so with that done (see below) Dion is now seeing the
>>> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
>>> course, with it having crashed before, it's hard to tell whether the
>>> triggering now is an effect of the patch, or just got unmasked by it.
>>>
>>
>> I just ported your below patch to upstream kernel and I didn't see the
>> WARN_ON_ONCE. I only did iperf and netperf tests.
>>
>> If the work load to trigger this bug is simple enough I can give it a
>> shot...
>
> I'm meanwhile relatively convinced that the warning isn't an effect
> of the patch (final verification pending); I intend to submit v2 as
> soon as 3.11-rc1 is out.
>
> Jan
>
I have been doing some testing and focussed on xennet_fill_frags. I
enhanced a bit the detection of the skb->data_len > nr_frags * PAGE_SIZE
condition (as suggested by Jan) by storing rx->status and nr_frags as
function of each dequeued skb in the while loop.
There is something wrong with the part
+if (nr_frags == MAX_SKB_FRAGS) {
+ unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
+
+ BUG_ON(pull_to <= skb_headlen(skb));
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ nr_frags = shinfo->nr_frags;
+}
+BUG_ON(nr_frags >= MAX_SKB_FRAGS);
When nr_frags reaches MAX_SKB_FRAGS (and this happens), nr_frags is
"reset" to shinfo->nr_frags. In fact I see the nr_frags set to 0 the
first time nr_frags reaches MAX_SKB_FRAGS.
The first problem with this is of course that the "skb->truesize +=
PAGE_SIZE * skb_shinfo(skb)->nr_frags" calculation following the "i =
xennet_fill_frags(np, skb, &tmpq)" in neif_poll leads to a wrong
result. At the end the skb has ->truesize way smaller than ->len.
The second problem is that the BUG_ON(nr_frags >= MAX_SKB_FRAGS) shall
not occur, since "shinfo->nr_frags" afaict, is not updated in between.
Furthermore, I cannot figure out why, when control enters
xennet_fill_frags(), shinfo->nr_frags equals 1, and a little later when
nr_frags reaches MAX_SKB_FRAGS, it is 0.
Here is the (printk) result of a trial:
> 1 2013-07-13T04:27:30.757241+02:00 doma kernel: [ 41.759528] netfront: Too many frags
> 2 2013-07-13T04:27:39.629261+02:00 doma kernel: [ 50.628482] 17 == 17 (shinfo->nr_frags=0)
> 3 2013-07-13T04:27:39.629285+02:00 doma kernel: [ 50.628486] j=0,rx_trace[j]=48,nr_frags_trace[j]=1
> 4 2013-07-13T04:27:39.629287+02:00 doma kernel: [ 50.628488] j=1,rx_trace[j]=1000,nr_frags_trace[j]=2
> 5 2013-07-13T04:27:39.629289+02:00 doma kernel: [ 50.628489] j=2,rx_trace[j]=1000,nr_frags_trace[j]=3
> 6 2013-07-13T04:27:39.629291+02:00 doma kernel: [ 50.628491] j=3,rx_trace[j]=1000,nr_frags_trace[j]=4
> 7 2013-07-13T04:27:39.629293+02:00 doma kernel: [ 50.628492] j=4,rx_trace[j]=1000,nr_frags_trace[j]=5
> 8 2013-07-13T04:27:39.629294+02:00 doma kernel: [ 50.628493] j=5,rx_trace[j]=1000,nr_frags_trace[j]=6
> 9 2013-07-13T04:27:39.629296+02:00 doma kernel: [ 50.628494] j=6,rx_trace[j]=1000,nr_frags_trace[j]=7
> 10 2013-07-13T04:27:39.629298+02:00 doma kernel: [ 50.628496] j=7,rx_trace[j]=1000,nr_frags_trace[j]=8
> 11 2013-07-13T04:27:39.629300+02:00 doma kernel: [ 50.628497] j=8,rx_trace[j]=1000,nr_frags_trace[j]=9
> 12 2013-07-13T04:27:39.629302+02:00 doma kernel: [ 50.628498] j=9,rx_trace[j]=1000,nr_frags_trace[j]=10
> 13 2013-07-13T04:27:39.629304+02:00 doma kernel: [ 50.628499] j=10,rx_trace[j]=1000,nr_frags_trace[j]=11
> 14 2013-07-13T04:27:39.629305+02:00 doma kernel: [ 50.628503] j=11,rx_trace[j]=1000,nr_frags_trace[j]=12
> 15 2013-07-13T04:27:39.629307+02:00 doma kernel: [ 50.628504] j=12,rx_trace[j]=1000,nr_frags_trace[j]=13
> 16 2013-07-13T04:27:39.629328+02:00 doma kernel: [ 50.628506] j=13,rx_trace[j]=1000,nr_frags_trace[j]=14
> 17 2013-07-13T04:27:39.629330+02:00 doma kernel: [ 50.628507] j=14,rx_trace[j]=1000,nr_frags_trace[j]=15
> 18 2013-07-13T04:27:39.629332+02:00 doma kernel: [ 50.628508] j=15,rx_trace[j]=1000,nr_frags_trace[j]=16
> 19 2013-07-13T04:27:39.629335+02:00 doma kernel: [ 50.628510] j=16,rx_trace[j]=e40,nr_frags_trace[j]=0
> 20 2013-07-13T04:27:39.629336+02:00 doma kernel: [ 50.628511] xennet::xennet_fill_frags (fe88,feca,500,1,e40,42,42)
line 2 comes from
if (nr_frags == MAX_SKB_FRAGS) {
unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
BUG_ON(pull_to <= skb_headlen(skb));
__pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ printk("%u == %lu (shinfo->nr_frags=%u)\n",//temp
+ nr_frags,MAX_SKB_FRAGS, shinfo->nr_frags);//tmp
nr_frags = shinfo->nr_frags;
}
Lines 3 to 19 show the dumping I added at the end of the while loop
+if (skb->data_len > nr_frags * PAGE_SIZE) {//tmp
+ for (j=0; j<i; ++j)//tmp
+ printk("j=%u,rx_trace[j]=%x,nr_frags_trace[j]=%u\n", //tmp
+ j,rx_trace[j],nr_frags_trace[j]);//tmp
+ printk("xennet::xennet_fill_frags (%x,%x,%x,%u,%x,%x,%x)\n",//temp
+ skb->data_len, skb->len, skb->truesize, nr_frags,//tmp
+ rx->status, len0, data_len0);//temp
+}
For gathering the data I added a couple of variables on the stack
struct skb_shared_info *shinfo = skb_shinfo(skb);
int nr_frags = shinfo->nr_frags;
RING_IDX cons = np->rx.rsp_cons;
struct sk_buff *nskb;
+unsigned rx_trace[MAX_SKB_FRAGS+1], i=0,j, len0=0, data_len0=0;//tmp
+int nr_frags_trace[MAX_SKB_FRAGS+1];//tmp
and in the while loop
__skb_fill_page_desc(skb, nr_frags,
skb_frag_page(skb_shinfo(nskb)->frags),
rx->offset, rx->status);
+if (i==0) {//tmp
+ len0=skb->len;//tmp
+ data_len0=skb->data_len;//tmp
+}//tmp
+rx_trace[i] = rx->status;//tmp
skb->len += rx->status;
skb->data_len += rx->status;
skb_shinfo(nskb)->nr_frags = 0;
kfree_skb(nskb);
+nr_frags_trace[i++] = nr_frags;//tmp
nr_frags++;
As a coincidence I noticed that there has to be a subtle condition
related to speed of the while loop required to reach the state where
nr_frags becomes MAX_SKB_FRAGS.
A bit more delay in the while and it does not occur anymore.
E.g. first I had the gathering ordered like
+if (i==0) {//tmp
+ len0=skb->len;//tmp
+ data_len0=skb->data_len;//tmp
+}//tmp
+rx_trace[i] = rx->status;//tmp
+nr_frags_trace[i++] = nr_frags;//tmp
skb->len += rx->status;
skb->data_len += rx->status;
skb_shinfo(nskb)->nr_frags = 0;
kfree_skb(nskb);
nr_frags++;
and I was unable to find nr_frags reaching MAX_SKB_FRAGS within a finite
testing time. I thought this may be worthwhile to share this.
As a workload to trigger this I use simply
dom0-t:~ # scp bigrand.bin doma:/dev/null
and with this on my system the transfer rate settles around 45 MB/s
limited by a saturated core which runs the user land ssh process.
Dion
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-12 8:56 ` Jan Beulich
@ 2013-07-13 11:26 ` Dion Kant
2013-07-13 11:26 ` [Xen-devel] " Dion Kant
1 sibling, 0 replies; 85+ messages in thread
From: Dion Kant @ 2013-07-13 11:26 UTC (permalink / raw)
To: Jan Beulich; +Cc: Wei Liu, Ian Campbell, netdev, stable, xen-devel, davem
On 07/12/2013 10:56 AM, Jan Beulich wrote:
>>>> On 12.07.13 at 10:32, Wei Liu <wei.liu2@citrix.com> wrote:
>> On Mon, Jul 08, 2013 at 03:20:26PM +0100, Jan Beulich wrote:
>>>>>> On 08.07.13 at 11:59, "Jan Beulich" <JBeulich@suse.com> wrote:
>>>>>>> On 05.07.13 at 16:53, Wei Liu <wei.liu2@citrix.com> wrote:
>>>>> skb->truesize += PAGE_SIZE * skb_shinfo(skb)->nr_frags;
>>>>> skb->len += skb->data_len;
>>>>> }
>>>>>
>>>>> handle_incoming_packet();
>>>>>
>>>>> You seem to be altering the behavior of the original code, because in
>>>>> your patch the skb->len is incremented before use, while in the original
>>>>> code (which calls skb_headlen in handle_incoming_packet) the skb->len is
>>>>> correctly set.
>>>>
>>>> Right. So I basically need to keep skb->len up-to-date along with
>>>> ->data_len. Just handed a patch to Dion with that done; I'll defer
>>>> sending a v2 for the upstream code until I know the change works
>>>> for our kernel.
>>>
>>> Okay, so with that done (see below) Dion is now seeing the
>>> WARN_ON_ONCE(delta < len) in skb_try_coalesce() triggering. Of
>>> course, with it having crashed before, it's hard to tell whether the
>>> triggering now is an effect of the patch, or just got unmasked by it.
>>>
>>
>> I just ported your below patch to upstream kernel and I didn't see the
>> WARN_ON_ONCE. I only did iperf and netperf tests.
>>
>> If the work load to trigger this bug is simple enough I can give it a
>> shot...
>
> I'm meanwhile relatively convinced that the warning isn't an effect
> of the patch (final verification pending); I intend to submit v2 as
> soon as 3.11-rc1 is out.
>
> Jan
>
I have been doing some testing and focussed on xennet_fill_frags. I
enhanced a bit the detection of the skb->data_len > nr_frags * PAGE_SIZE
condition (as suggested by Jan) by storing rx->status and nr_frags as
function of each dequeued skb in the while loop.
There is something wrong with the part
+if (nr_frags == MAX_SKB_FRAGS) {
+ unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
+
+ BUG_ON(pull_to <= skb_headlen(skb));
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ nr_frags = shinfo->nr_frags;
+}
+BUG_ON(nr_frags >= MAX_SKB_FRAGS);
When nr_frags reaches MAX_SKB_FRAGS (and this happens), nr_frags is
"reset" to shinfo->nr_frags. In fact I see the nr_frags set to 0 the
first time nr_frags reaches MAX_SKB_FRAGS.
The first problem with this is of course that the "skb->truesize +=
PAGE_SIZE * skb_shinfo(skb)->nr_frags" calculation following the "i =
xennet_fill_frags(np, skb, &tmpq)" in neif_poll leads to a wrong
result. At the end the skb has ->truesize way smaller than ->len.
The second problem is that the BUG_ON(nr_frags >= MAX_SKB_FRAGS) shall
not occur, since "shinfo->nr_frags" afaict, is not updated in between.
Furthermore, I cannot figure out why, when control enters
xennet_fill_frags(), shinfo->nr_frags equals 1, and a little later when
nr_frags reaches MAX_SKB_FRAGS, it is 0.
Here is the (printk) result of a trial:
> 1 2013-07-13T04:27:30.757241+02:00 doma kernel: [ 41.759528] netfront: Too many frags
> 2 2013-07-13T04:27:39.629261+02:00 doma kernel: [ 50.628482] 17 == 17 (shinfo->nr_frags=0)
> 3 2013-07-13T04:27:39.629285+02:00 doma kernel: [ 50.628486] j=0,rx_trace[j]=48,nr_frags_trace[j]=1
> 4 2013-07-13T04:27:39.629287+02:00 doma kernel: [ 50.628488] j=1,rx_trace[j]=1000,nr_frags_trace[j]=2
> 5 2013-07-13T04:27:39.629289+02:00 doma kernel: [ 50.628489] j=2,rx_trace[j]=1000,nr_frags_trace[j]=3
> 6 2013-07-13T04:27:39.629291+02:00 doma kernel: [ 50.628491] j=3,rx_trace[j]=1000,nr_frags_trace[j]=4
> 7 2013-07-13T04:27:39.629293+02:00 doma kernel: [ 50.628492] j=4,rx_trace[j]=1000,nr_frags_trace[j]=5
> 8 2013-07-13T04:27:39.629294+02:00 doma kernel: [ 50.628493] j=5,rx_trace[j]=1000,nr_frags_trace[j]=6
> 9 2013-07-13T04:27:39.629296+02:00 doma kernel: [ 50.628494] j=6,rx_trace[j]=1000,nr_frags_trace[j]=7
> 10 2013-07-13T04:27:39.629298+02:00 doma kernel: [ 50.628496] j=7,rx_trace[j]=1000,nr_frags_trace[j]=8
> 11 2013-07-13T04:27:39.629300+02:00 doma kernel: [ 50.628497] j=8,rx_trace[j]=1000,nr_frags_trace[j]=9
> 12 2013-07-13T04:27:39.629302+02:00 doma kernel: [ 50.628498] j=9,rx_trace[j]=1000,nr_frags_trace[j]=10
> 13 2013-07-13T04:27:39.629304+02:00 doma kernel: [ 50.628499] j=10,rx_trace[j]=1000,nr_frags_trace[j]=11
> 14 2013-07-13T04:27:39.629305+02:00 doma kernel: [ 50.628503] j=11,rx_trace[j]=1000,nr_frags_trace[j]=12
> 15 2013-07-13T04:27:39.629307+02:00 doma kernel: [ 50.628504] j=12,rx_trace[j]=1000,nr_frags_trace[j]=13
> 16 2013-07-13T04:27:39.629328+02:00 doma kernel: [ 50.628506] j=13,rx_trace[j]=1000,nr_frags_trace[j]=14
> 17 2013-07-13T04:27:39.629330+02:00 doma kernel: [ 50.628507] j=14,rx_trace[j]=1000,nr_frags_trace[j]=15
> 18 2013-07-13T04:27:39.629332+02:00 doma kernel: [ 50.628508] j=15,rx_trace[j]=1000,nr_frags_trace[j]=16
> 19 2013-07-13T04:27:39.629335+02:00 doma kernel: [ 50.628510] j=16,rx_trace[j]=e40,nr_frags_trace[j]=0
> 20 2013-07-13T04:27:39.629336+02:00 doma kernel: [ 50.628511] xennet::xennet_fill_frags (fe88,feca,500,1,e40,42,42)
line 2 comes from
if (nr_frags == MAX_SKB_FRAGS) {
unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
BUG_ON(pull_to <= skb_headlen(skb));
__pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ printk("%u == %lu (shinfo->nr_frags=%u)\n",//temp
+ nr_frags,MAX_SKB_FRAGS, shinfo->nr_frags);//tmp
nr_frags = shinfo->nr_frags;
}
Lines 3 to 19 show the dumping I added at the end of the while loop
+if (skb->data_len > nr_frags * PAGE_SIZE) {//tmp
+ for (j=0; j<i; ++j)//tmp
+ printk("j=%u,rx_trace[j]=%x,nr_frags_trace[j]=%u\n", //tmp
+ j,rx_trace[j],nr_frags_trace[j]);//tmp
+ printk("xennet::xennet_fill_frags (%x,%x,%x,%u,%x,%x,%x)\n",//temp
+ skb->data_len, skb->len, skb->truesize, nr_frags,//tmp
+ rx->status, len0, data_len0);//temp
+}
For gathering the data I added a couple of variables on the stack
struct skb_shared_info *shinfo = skb_shinfo(skb);
int nr_frags = shinfo->nr_frags;
RING_IDX cons = np->rx.rsp_cons;
struct sk_buff *nskb;
+unsigned rx_trace[MAX_SKB_FRAGS+1], i=0,j, len0=0, data_len0=0;//tmp
+int nr_frags_trace[MAX_SKB_FRAGS+1];//tmp
and in the while loop
__skb_fill_page_desc(skb, nr_frags,
skb_frag_page(skb_shinfo(nskb)->frags),
rx->offset, rx->status);
+if (i==0) {//tmp
+ len0=skb->len;//tmp
+ data_len0=skb->data_len;//tmp
+}//tmp
+rx_trace[i] = rx->status;//tmp
skb->len += rx->status;
skb->data_len += rx->status;
skb_shinfo(nskb)->nr_frags = 0;
kfree_skb(nskb);
+nr_frags_trace[i++] = nr_frags;//tmp
nr_frags++;
As a coincidence I noticed that there has to be a subtle condition
related to speed of the while loop required to reach the state where
nr_frags becomes MAX_SKB_FRAGS.
A bit more delay in the while and it does not occur anymore.
E.g. first I had the gathering ordered like
+if (i==0) {//tmp
+ len0=skb->len;//tmp
+ data_len0=skb->data_len;//tmp
+}//tmp
+rx_trace[i] = rx->status;//tmp
+nr_frags_trace[i++] = nr_frags;//tmp
skb->len += rx->status;
skb->data_len += rx->status;
skb_shinfo(nskb)->nr_frags = 0;
kfree_skb(nskb);
nr_frags++;
and I was unable to find nr_frags reaching MAX_SKB_FRAGS within a finite
testing time. I thought this may be worthwhile to share this.
As a workload to trigger this I use simply
dom0-t:~ # scp bigrand.bin doma:/dev/null
and with this on my system the transfer rate settles around 45 MB/s
limited by a saturated core which runs the user land ssh process.
Dion
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-13 11:26 ` [Xen-devel] " Dion Kant
@ 2013-07-13 20:17 ` Dion Kant
2013-07-15 6:37 ` Jan Beulich
0 siblings, 1 reply; 85+ messages in thread
From: Dion Kant @ 2013-07-13 20:17 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel
On 07/13/2013 01:26 PM, Dion Kant wrote:
> There is something wrong with the part
>
> +if (nr_frags == MAX_SKB_FRAGS) {
> + unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
> +
> + BUG_ON(pull_to <= skb_headlen(skb));
> + __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
> + nr_frags = shinfo->nr_frags;
> +}
> +BUG_ON(nr_frags >= MAX_SKB_FRAGS);
>
> When nr_frags reaches MAX_SKB_FRAGS (and this happens), nr_frags is
> "reset" to shinfo->nr_frags. In fact I see the nr_frags set to 0 the
> first time nr_frags reaches MAX_SKB_FRAGS.
>
> The first problem with this is of course that the "skb->truesize +=
> PAGE_SIZE * skb_shinfo(skb)->nr_frags" calculation following the "i =
> xennet_fill_frags(np, skb, &tmpq)" in neif_poll leads to a wrong
> result. At the end the skb has ->truesize way smaller than ->len.
>
> The second problem is that the BUG_ON(nr_frags >= MAX_SKB_FRAGS) shall
> not occur, since "shinfo->nr_frags" afaict, is not updated in between.
>
> Furthermore, I cannot figure out why, when control enters
> xennet_fill_frags(), shinfo->nr_frags equals 1, and a little later when
> nr_frags reaches MAX_SKB_FRAGS, it is 0.
Jan,
After some reading I now understand, it is __pskb_pull_tail(skb, pull_to
- skb_headlen(skb)) which lowers shinfo->nr_frags by one.
Must we then not first set shinfo->nr_frags to its correct value before
the call to __pskb_pull_tail(skb, pull_to - skb_headlen(skb)), like below?
+if (nr_frags == MAX_SKB_FRAGS) {
+ unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
+
+ BUG_ON(pull_to <= skb_headlen(skb));
+ shinfo->nr_frags = nr_frags;
+ __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ nr_frags = shinfo->nr_frags;
+}
+BUG_ON(nr_frags >= MAX_SKB_FRAGS);
I'll do some testing with this.
Dion
^ permalink raw reply [flat|nested] 85+ messages in thread
* Re: [PATCH] xen-netfront: pull on receive skb may need to happen earlier
2013-07-13 20:17 ` Dion Kant
@ 2013-07-15 6:37 ` Jan Beulich
0 siblings, 0 replies; 85+ messages in thread
From: Jan Beulich @ 2013-07-15 6:37 UTC (permalink / raw)
To: Dion Kant; +Cc: xen-devel
>>> On 13.07.13 at 22:17, Dion Kant <g.w.kant@hunenet.nl> wrote:
> After some reading I now understand, it is __pskb_pull_tail(skb, pull_to
> - skb_headlen(skb)) which lowers shinfo->nr_frags by one.
>
> Must we then not first set shinfo->nr_frags to its correct value before
> the call to __pskb_pull_tail(skb, pull_to - skb_headlen(skb)), like below?
Yes, absolutely - your previous mail already made that clear.
Part of the issue here is that I think the code really should be
using skb_add_rx_frag(), which would take care of keeping the
field up-to-date. I'll probably re-write the patch to that effect.
Thanks a lot!
Jan
^ permalink raw reply [flat|nested] 85+ messages in thread
end of thread, other threads:[~2013-07-15 6:37 UTC | newest]
Thread overview: 85+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <8511913.uMAmUdIO30@eistomin.edss.local>
2013-05-17 8:59 ` [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Wei Liu
2013-05-17 9:08 ` Jan Beulich
2013-05-17 9:37 ` Eugene Istomin
2013-05-17 9:40 ` Eugene Istomin
2013-05-17 10:45 ` Jan Beulich
2013-05-17 12:03 ` Eugene Istomin
2013-05-17 12:25 ` Jan Beulich
2013-05-17 12:30 ` Eugene Istomin
2013-05-17 12:36 ` Jan Beulich
2013-05-17 12:52 ` Eugene Istomin
2013-05-17 13:00 ` Eugene Istomin
2013-05-18 10:06 ` Eugene Istomin
2013-05-20 6:52 ` Eugene Istomin
2013-05-20 8:58 ` Ian Campbell
2013-05-20 9:06 ` Wei Liu
2013-05-20 9:17 ` Eugene Istomin
2013-05-20 9:24 ` Wei Liu
2013-05-21 9:55 ` Jan Beulich
2013-05-21 11:09 ` Eugene Istomin
2013-05-21 11:49 ` Jan Beulich
2013-05-21 16:19 ` Eugene Istomin
2013-05-31 3:37 ` Eugene Istomin
2013-05-31 6:19 ` Jan Beulich
2013-05-31 6:22 ` Eugene Istomin
2013-05-31 8:20 ` Wei Liu
2013-05-21 15:28 ` Wei Liu
2013-05-31 15:47 ` Wei Liu
2013-05-31 16:00 ` Jan Beulich
2013-05-31 16:05 ` Wei Liu
2013-07-04 13:43 ` Dion Kant
2013-07-04 14:18 ` Jan Beulich
2013-07-04 15:01 ` Wei Liu
2013-07-05 9:32 ` [PATCH] xen-netfront: pull on receive skb may need to happen earlier Jan Beulich
2013-07-05 14:53 ` Wei Liu
2013-07-05 14:53 ` Wei Liu
2013-07-07 1:10 ` David Miller
2013-07-07 1:10 ` David Miller
2013-07-08 9:59 ` Jan Beulich
2013-07-08 12:16 ` Dion Kant
2013-07-08 12:41 ` Jan Beulich
2013-07-08 12:41 ` Jan Beulich
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
2013-07-08 15:22 ` Eric Dumazet
2013-07-08 15:22 ` [Xen-devel] " Eric Dumazet
2013-07-09 7:47 ` Jan Beulich
2013-07-09 7:47 ` Jan Beulich
2013-07-08 15:48 ` [Xen-devel] " Wei Liu
2013-07-09 6:52 ` Jan Beulich
2013-07-09 6:52 ` [Xen-devel] " Jan Beulich
2013-07-09 16:51 ` Wei Liu
2013-07-09 16:51 ` [Xen-devel] " Wei Liu
2013-07-10 6:58 ` Jan Beulich
2013-07-10 6:58 ` [Xen-devel] " Jan Beulich
2013-07-10 10:04 ` Wei Liu
2013-07-10 10:04 ` [Xen-devel] " Wei Liu
2013-07-10 10:46 ` Jan Beulich
2013-07-10 12:50 ` Ian Campbell
2013-07-10 12:53 ` Wei Liu
2013-07-10 12:53 ` Wei Liu
2013-07-10 13:58 ` Jan Beulich
2013-07-10 13:58 ` [Xen-devel] " Jan Beulich
2013-07-10 12:50 ` Ian Campbell
2013-07-10 10:46 ` Jan Beulich
2013-07-10 13:19 ` [Xen-devel] " Eric Dumazet
2013-07-10 13:19 ` Eric Dumazet
2013-07-08 15:48 ` Wei Liu
2013-07-12 8:32 ` [Xen-devel] " Wei Liu
2013-07-12 8:56 ` Jan Beulich
2013-07-13 11:26 ` Dion Kant
2013-07-13 11:26 ` [Xen-devel] " Dion Kant
2013-07-13 20:17 ` Dion Kant
2013-07-15 6:37 ` Jan Beulich
2013-07-12 8:56 ` Jan Beulich
2013-07-12 8:32 ` Wei Liu
2013-07-08 14:20 ` Jan Beulich
2013-07-08 9:59 ` Jan Beulich
2013-07-05 9:32 ` Jan Beulich
2013-07-05 9:36 ` kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Jan Beulich
2013-07-05 10:40 ` [Xen-users] " Dion Kant
2013-07-05 10:54 ` Wei Liu
2013-07-05 19:11 ` Dion Kant
2013-07-05 10:56 ` Jan Beulich
2013-07-05 19:46 ` Dion Kant
2013-07-06 13:36 ` Wei Liu
2013-07-08 8:25 ` Jan Beulich
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.