From mboxrd@z Thu Jan 1 00:00:00 1970 From: annie li Subject: Re: Fwd: Re: xennet: skb rides the rocket: 20 slots Date: Fri, 15 Mar 2013 13:14:47 +0800 Message-ID: <5142AE47.40107@oracle.com> References: <72958707.20130104172854@eikelenboom.it> <1357556115.7989.13.camel@zakaz.uk.xensource.com> <50EB8091.90705@oracle.com> <323202711.20130108215503@eikelenboom.it> <50ED1800.1080208@oracle.com> <20130109150850.GI18395@phenom.dumpdata.com> <50EEA46E.7000604@oracle.com> <1357820816.9456.15.camel@zakaz.uk.xensource.com> <50EFC079.1000601@oracle.com> <1357898207.9456.112.camel@zakaz.uk.xensource.com> <291EDFCB1E9E224A99088639C4762022012CCFEB82DE@LONPMAILBOX01.citrite.net> <1357899363.9456.116.camel@zakaz.uk.xensource.com> <50F3D269.6030601@oracle.com> <19010312768.20130124094542@eikelenboom.it> <422733537.20130309135735@eikelenboom.it> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2838854136689817486==" Return-path: In-Reply-To: <422733537.20130309135735@eikelenboom.it> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Sander Eikelenboom Cc: 'Steven Haigh' , Ian Campbell , "Palagummi, Siva" , Konrad Rzeszutek Wilk , "xen-devel@lists.xen.org" , msw@amazon.com, Wei Liu , Jacek Milewicz List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============2838854136689817486== Content-Type: multipart/alternative; boundary="------------010307030603070408040000" This is a multi-part message in MIME format. --------------010307030603070408040000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Sander, I checked this patch, and am thinking skb_gso_segment may return NULL. (see description of skb_gso_segment). And maybe this cause the NULL pointer deference issue. BTW, how did you send out those skbs with 64454 size? only copying files or doing iperf/netperf test? I hope I can verify it locally. Thanks Annie On 2013-3-9 20:57, Sander Eikelenboom wrote: > This is a forwarded message > From: Sander Eikelenboom > To: ANNIE LI > Date: Thursday, January 24, 2013, 9:45:42 AM > Subject: [Xen-devel] xennet: skb rides the rocket: 20 slots > > > Resend because xen-devel wasn't copied on the original ... > > ===8<==============Original message text=============== > > Monday, January 14, 2013, 10:39:53 AM, you wrote: > >> Hi >> I created a patch for this, but I failed to reproduce this issue and >> verify it. The patch was attached, > Hi Annie, > > I finally had time to seriously test the patch. > I put in some more warn's and made the bracing a bit more explicit (i hope i did the bracing right). > > Problem is the current code crashes with: > > [ 4189.815911] nf_conntrack: automatic helper assignment is deprecated and it will be removed soon. Use the iptables CT target to attach helpers instead. > [29601.932324] xennet: xennet_xmit_skb err_end: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:0 offset:106 skb_headlen:54 skb->len:64454, skb->data_len:0 skb->truesize:65168 nr_frags:0 page_size:4096 prot:0800 gso:1 linearize:0 gso_segs:46 dev:eth0 transp:0006 > [29601.932426] BUG: unable to handle kernel NULL pointer dereference at (null) > [29601.932461] IP: [] xennet_xmit_skb+0x204/0x370 > [29601.932498] PGD 2d497067 PUD 2dd94067 PMD 0 > [29601.932526] Oops: 0000 [#1] PREEMPT SMP > [29601.932549] Modules linked in: > [29601.932566] CPU 0 > [29601.932581] Pid: 2948, comm: deluged Not tainted 3.8.0-rc4-20130123-netpatched-rocketscience-radeon-qmax-new-a #1 > [29601.932615] RIP: e030:[] [] xennet_xmit_skb+0x204/0x370 > [29601.932650] RSP: e02b:ffff88002cd95698 EFLAGS: 00010207 > [29601.932669] RAX: 0000000000000000 RBX: ffff8800049574e8 RCX: 0000000000000036 > [29601.932691] RDX: ffff88000398006a RSI: ffff88002ce4b8c0 RDI: 0000000000000000 > [29601.932712] RBP: ffff88002cd95758 R08: 0000000000000000 R09: 0000000000000000 > [29601.932734] R10: 0000000000000000 R11: 0000000000000002 R12: ffff88002cc98000 > [29601.932755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000036 > [29601.932786] FS: 00007fa9ca911700(0000) GS:ffff88002fc00000(0000) knlGS:0000000000000000 > [29601.932811] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b > [29601.932830] CR2: 0000000000000000 CR3: 000000002cf0d000 CR4: 0000000000000660 > [29601.932853] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [29601.932877] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [29601.932900] Process deluged (pid: 2948, threadinfo ffff88002cd94000, task ffff88002ce4b150) > [29601.932923] Stack: > [29601.932934] ffff880000000036 000000000000fbc6 ffff880000000000 24e008890000fe90 > [29601.932970] ffff880000000000 0000000000001000 ffff880000000800 ffff880000000001 > [29601.933004] 0000000000000000 ffff88000000002e ffff88002cc98000 ffff880000000006 > [29601.933795] Call Trace: > [29601.933795] [] dev_hard_start_xmit+0x219/0x480 > [29601.933795] [] sch_direct_xmit+0xf6/0x290 > [29601.933795] [] dev_queue_xmit+0x1a6/0x5a0 > [29601.933795] [] ? dev_hard_start_xmit+0x480/0x480 > [29601.933795] [] ? trace_softirqs_off+0x85/0x1b0 > [29601.933795] [] ip_finish_output+0x226/0x530 > [29601.933795] [] ? ip_finish_output+0xcd/0x530 > [29601.933795] [] ip_output+0x59/0xe0 > [29601.933795] [] ip_local_out+0x28/0x90 > [29601.933795] [] ip_queue_xmit+0x17f/0x490 > [29601.933795] [] ? ip_send_unicast_reply+0x330/0x330 > [29601.933795] [] ? getnstimeofday+0x47/0xe0 > [29601.933795] [] ? __skb_clone+0x29/0x120 > [29601.933795] [] tcp_transmit_skb+0x3fd/0x8d0 > [29601.933795] [] tcp_write_xmit+0x22a/0xa80 > [29601.933795] [] ? alloc_pages_current+0xde/0x1c0 > [29601.933795] [] tcp_push_one+0x2b/0x40 > [29601.933795] [] tcp_sendmsg+0x8d4/0xe10 > [29601.933795] [] inet_sendmsg+0xa6/0x100 > [29601.933795] [] ? inet_autobind+0x60/0x60 > [29601.933795] [] sock_sendmsg+0x82/0xb0 > [29601.933795] [] ? lock_release+0x117/0x250 > [29601.933795] [] ? might_fault+0x84/0x90 > [29601.933795] [] ? might_fault+0x3b/0x90 > [29601.933795] [] ? __kmalloc+0xfb/0x160 > [29601.933795] [] ? verify_iovec+0x7d/0xf0 > [29601.933795] [] __sys_sendmsg+0x393/0x3a0 > [29601.933795] [] ? _raw_spin_unlock_irqrestore+0x75/0xa0 > [29601.933795] [] ? lock_acquire+0xd8/0x100 > [29601.933795] [] ? lock_release+0x117/0x250 > [29601.933795] [] ? fget_light+0xd7/0x140 > [29601.933795] [] ? fget_light+0x3a/0x140 > [29601.933795] [] sys_sendmsg+0x44/0x80 > [29601.933795] [] system_call_fastpath+0x16/0x1b > [29601.933795] Code: e9 ca fe ff ff 49 8b b4 24 80 00 00 00 48 89 df 45 31 ed e8 1f 16 20 00 48 3d 00 f0 ff ff 48 89 c7 76 08 e9 17 01 00 00 4c 89 f7 <4c> 8b 37 4c 89 e6 48 c7 07 00 00 00 00 41 ff c5 e8 b7 f3 ff ff > [29601.933795] RIP [] xennet_xmit_skb+0x204/0x370 > [29601.933795] RSP > [29601.933795] CR2: 0000000000000000 > [29602.018741] ---[ end trace 5ec54203e8f81a1b ]--- > [29602.018747] Kernel panic - not syncing: Fatal exception in interrupt > > > > Which accoring to addr2line is: > > segs = segs->next; > > I have attached the resulting patch > > -- > Sander > >> Thanks >> Annie >> On 2013-1-11 18:16, Ian Campbell wrote: >>> On Fri, 2013-01-11 at 10:09 +0000, Paul Durrant wrote: >>>>> -----Original Message----- >>>>> Without GSO I don't think you should be seeing packets larger than the MTU, >>>>> which would normally be either ~1500 or ~9000 and fit easily within any >>>>> sensible negotiation for the max frags. I don't think you should worry unduly >>>>> about this case. >>>>> >>>> A stack could still send down a packet with one byte per frag though, >>>> right? A copy-and-coalesce path would still be needed in this case. >>> True. In that case skb_linearise would probably do the job on Linux. >>> >>> Ian >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel > ===8<===========End of original message text=========== > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel --------------010307030603070408040000 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi Sander,

I checked this patch, and am thinking skb_gso_segment may return NULL. (see description of skb_gso_segment). And maybe this cause the NULL pointer deference issue.

BTW, how did you send out those skbs with 64454 size? only copying files or doing iperf/netperf test? I hope I can verify it locally.

Thanks
Annie

On 2013-3-9 20:57, Sander Eikelenboom wrote:

This is a forwarded message
From: Sander Eikelenboom <linux@eikelenboom.it>
To: ANNIE LI <annie.li@oracle.com>
Date: Thursday, January 24, 2013, 9:45:42 AM
Subject: [Xen-devel] xennet: skb rides the rocket: 20 slots


Resend because xen-devel wasn't copied on the original ...

===8<==============Original message text===============

Monday, January 14, 2013, 10:39:53 AM, you wrote:

Hi

I created a patch for this, but I failed to reproduce this issue and 
verify it. The patch was attached,

Hi Annie,

I finally had time to seriously test the patch.
I put in some more warn's and made the bracing a bit more explicit (i hope i did the bracing right).

Problem is the current code crashes with:

[ 4189.815911] nf_conntrack: automatic helper assignment is deprecated and it will be removed soon. Use the iptables CT target to attach helpers instead.
[29601.932324] xennet:  xennet_xmit_skb err_end: 19 slots MAX_SKB_FRAGS: 17 div_roundup:1 xennet_count_skb_frag_slots:0 offset:106 skb_headlen:54 skb->len:64454, skb->data_len:0 skb->truesize:65168 nr_frags:0 page_size:4096 prot:0800 gso:1 linearize:0 gso_segs:46 dev:eth0 transp:0006
[29601.932426] BUG: unable to handle kernel NULL pointer dereference at           (null)
[29601.932461] IP: [<ffffffff816551f4>] xennet_xmit_skb+0x204/0x370
[29601.932498] PGD 2d497067 PUD 2dd94067 PMD 0
[29601.932526] Oops: 0000 [#1] PREEMPT SMP
[29601.932549] Modules linked in:
[29601.932566] CPU 0
[29601.932581] Pid: 2948, comm: deluged Not tainted 3.8.0-rc4-20130123-netpatched-rocketscience-radeon-qmax-new-a #1
[29601.932615] RIP: e030:[<ffffffff816551f4>]  [<ffffffff816551f4>] xennet_xmit_skb+0x204/0x370
[29601.932650] RSP: e02b:ffff88002cd95698  EFLAGS: 00010207
[29601.932669] RAX: 0000000000000000 RBX: ffff8800049574e8 RCX: 0000000000000036
[29601.932691] RDX: ffff88000398006a RSI: ffff88002ce4b8c0 RDI: 0000000000000000
[29601.932712] RBP: ffff88002cd95758 R08: 0000000000000000 R09: 0000000000000000
[29601.932734] R10: 0000000000000000 R11: 0000000000000002 R12: ffff88002cc98000
[29601.932755] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000036
[29601.932786] FS:  00007fa9ca911700(0000) GS:ffff88002fc00000(0000) knlGS:0000000000000000
[29601.932811] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[29601.932830] CR2: 0000000000000000 CR3: 000000002cf0d000 CR4: 0000000000000660
[29601.932853] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[29601.932877] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[29601.932900] Process deluged (pid: 2948, threadinfo ffff88002cd94000, task ffff88002ce4b150)
[29601.932923] Stack:
[29601.932934]  ffff880000000036 000000000000fbc6 ffff880000000000 24e008890000fe90
[29601.932970]  ffff880000000000 0000000000001000 ffff880000000800 ffff880000000001
[29601.933004]  0000000000000000 ffff88000000002e ffff88002cc98000 ffff880000000006
[29601.933795] Call Trace:
[29601.933795]  [<ffffffff818570b9>] dev_hard_start_xmit+0x219/0x480
[29601.933795]  [<ffffffff81873856>] sch_direct_xmit+0xf6/0x290
[29601.933795]  [<ffffffff818574c6>] dev_queue_xmit+0x1a6/0x5a0
[29601.933795]  [<ffffffff81857320>] ? dev_hard_start_xmit+0x480/0x480
[29601.933795]  [<ffffffff810af255>] ? trace_softirqs_off+0x85/0x1b0
[29601.933795]  [<ffffffff818f2596>] ip_finish_output+0x226/0x530
[29601.933795]  [<ffffffff818f243d>] ? ip_finish_output+0xcd/0x530
[29601.933795]  [<ffffffff818f28f9>] ip_output+0x59/0xe0
[29601.933795]  [<ffffffff818f1438>] ip_local_out+0x28/0x90
[29601.933795]  [<ffffffff818f19df>] ip_queue_xmit+0x17f/0x490
[29601.933795]  [<ffffffff818f1860>] ? ip_send_unicast_reply+0x330/0x330
[29601.933795]  [<ffffffff810a55f7>] ? getnstimeofday+0x47/0xe0
[29601.933795]  [<ffffffff818471e9>] ? __skb_clone+0x29/0x120
[29601.933795]  [<ffffffff81907c2d>] tcp_transmit_skb+0x3fd/0x8d0
[29601.933795]  [<ffffffff8190ac9a>] tcp_write_xmit+0x22a/0xa80
[29601.933795]  [<ffffffff81137ebe>] ? alloc_pages_current+0xde/0x1c0
[29601.933795]  [<ffffffff8190b51b>] tcp_push_one+0x2b/0x40
[29601.933795]  [<ffffffff818fbdc4>] tcp_sendmsg+0x8d4/0xe10
[29601.933795]  [<ffffffff819225d6>] inet_sendmsg+0xa6/0x100
[29601.933795]  [<ffffffff81922530>] ? inet_autobind+0x60/0x60
[29601.933795]  [<ffffffff8183eb52>] sock_sendmsg+0x82/0xb0
[29601.933795]  [<ffffffff810b5d87>] ? lock_release+0x117/0x250
[29601.933795]  [<ffffffff81118a34>] ? might_fault+0x84/0x90
[29601.933795]  [<ffffffff811189eb>] ? might_fault+0x3b/0x90
[29601.933795]  [<ffffffff8114082b>] ? __kmalloc+0xfb/0x160
[29601.933795]  [<ffffffff8184c9ad>] ? verify_iovec+0x7d/0xf0
[29601.933795]  [<ffffffff8183fb73>] __sys_sendmsg+0x393/0x3a0
[29601.933795]  [<ffffffff819bafc5>] ? _raw_spin_unlock_irqrestore+0x75/0xa0
[29601.933795]  [<ffffffff810b58f8>] ? lock_acquire+0xd8/0x100
[29601.933795]  [<ffffffff810b5d87>] ? lock_release+0x117/0x250
[29601.933795]  [<ffffffff81168777>] ? fget_light+0xd7/0x140
[29601.933795]  [<ffffffff811686da>] ? fget_light+0x3a/0x140
[29601.933795]  [<ffffffff8183fd34>] sys_sendmsg+0x44/0x80
[29601.933795]  [<ffffffff819bbe29>] system_call_fastpath+0x16/0x1b
[29601.933795] Code: e9 ca fe ff ff 49 8b b4 24 80 00 00 00 48 89 df 45 31 ed e8 1f 16 20 00 48 3d 00 f0 ff ff 48 89 c7 76 08 e9 17 01 00 00 4c 89 f7 <4c> 8b 37 4c 89 e6 48 c7 07 00 00 00 00 41 ff c5 e8 b7 f3 ff ff
[29601.933795] RIP  [<ffffffff816551f4>] xennet_xmit_skb+0x204/0x370
[29601.933795]  RSP <ffff88002cd95698>
[29601.933795] CR2: 0000000000000000
[29602.018741] ---[ end trace 5ec54203e8f81a1b ]---
[29602.018747] Kernel panic - not syncing: Fatal exception in interrupt



Which accoring to addr2line is:

segs = segs->next;

I have attached the resulting patch

--
Sander

Thanks
Annie

On 2013-1-11 18:16, Ian Campbell wrote:

On Fri, 2013-01-11 at 10:09 +0000, Paul Durrant wrote:

-----Original Message-----
Without GSO I don't think you should be seeing packets larger than the MTU,
which would normally be either ~1500 or ~9000 and fit easily within any
sensible negotiation for the max frags. I don't think you should worry unduly
about this case.

A stack could still send down a packet with one byte per frag though,
right? A copy-and-coalesce path would still be needed in this case.

True. In that case skb_linearise would probably do the job on Linux.

Ian


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

===8<===========End of original message text===========

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

--------------010307030603070408040000-- --===============2838854136689817486== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2838854136689817486==--