* GRE+XFRM+GSO crashes
@ 2013-05-20 6:41 Timo Teras
2013-05-20 17:58 ` Pravin Shelar
0 siblings, 1 reply; 7+ messages in thread
From: Timo Teras @ 2013-05-20 6:41 UTC (permalink / raw)
To: Pravin B Shelar, netdev
Since upgrade from 3.8 to 3.9 I've started getting the below mentioned
BUG crashes. One of the few relevant changes seems to be GSO support in
GRE driver.
Turning off SG (and GSO) seems to make these crashes disappear.
The basic setup is:
- gre1 is an NBMA tunnel (no explicit destination, nor bound
target interface; opennhrp daemon creates neigh mappings)
- IPsec policy to encrypt all GRE traffic in transport mode
- VIA Padlock hardware for AES acceleration
- GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso off
Incidentally, when I tried exact same setup ran as virtualized, I was
unable to reproduce this crash. I suspect it depends on the target NIC
acceleration capabilities.
This is from vanilla 3.9.2 kernel:
BUG: unable to handle kernel NULL pointer dereference at 00000010
IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4
*pde = 00000000
Oops: 0000 [#1] SMP
Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
deflate zlib_deflate ctr twofish_i586 twofish_generic twofish_common
camellia_generic serpent_sse2_i586 xts lrw gf128mul glue_helper
ablk_helper cryptd serpent_generic blowfish_generic blowfish_common
cast5_generic cast_common des_generic cbc xcbc rmd160 sha512_generic
hmac crypto_null af_key xfrm_algo ip_gre ipt_REJECT iptable_filter
ipt_MASQUERADE xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat ip_tables ip6t_REJECT xt_LOG xt_limit xt_recent
xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
nf_conntrack xt_multiport ip6table_filter ip6_tables x_tables ipv6
af_packet padlock_sha padlock_aes via_cputemp hwmon hwmon_vid
serio_raw psmouse pcspkr shpchp pci_hotplug i2c_viapro i2c_core
via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus
snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169
firmware_class mii evdev fan parport_pc parport thermal button
acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat
sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi
libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop
Pid: 1794, comm: opennhrp Not tainted 3.9.2 #2-Alpine /CN700-8237
EIP: 0060:[<c123e5b8>] EFLAGS: 00010246 CPU: 0
EIP is at xfrm_output_resume+0x63/0x2a4
EAX: 00000000 EBX: f0572500 ECX: f5ffc8b8 EDX: 00000064
ESI: f0496400 EDI: 00000000 EBP: f0499bf8 ESP: f0499be8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0 task.ti=f0498000)
Stack:
f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14 c123e89b
f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c c1236865 f0572500
0000003c f0499c38 c12016e7 f0572500 f0499cb0 f84175fb f0572a00 f04bb400
Call Trace:
[<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
[<c123e806>] xfrm_output2+0xd/0xf
[<c123e89b>] xfrm_output+0x93/0xa0
[<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
[<c1236837>] xfrm4_output_finish+0x27/0x29
[<c1236865>] xfrm4_output+0x2c/0x63
[<c12016e7>] ip_local_out+0x1b/0x1e
[<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre]
[<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c
[<c11de379>] dev_queue_xmit+0x289/0x31a
[<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet]
[<c1040200>] ? ttwu_do_wakeup+0xe/0xa8
[<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre]
[<c11cd5a9>] sock_sendmsg+0x79/0x94
[<c10b33d4>] ? __pollwait+0xa4/0xa4
[<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3
[<c10367c5>] ? free_pid+0x99/0x9f
[<c106b92a>] ? call_rcu_sched+0xf/0x12
[<c1026e15>] ? release_task+0x36d/0x37d
[<c10390e8>] ? remove_wait_queue+0x31/0x36
[<c1027a1d>] ? do_wait+0x1a8/0x1b5
[<c11cef10>] sys_sendmsg+0x2b/0x46
[<c11cf394>] sys_socketcall+0x145/0x19f
[<c1257888>] syscall_call+0x7/0xb
Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94 c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0 fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53
EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP 0068:f0499be8
CR2: 0000000000000010
---[ end trace ac96c6b6b1a4992f ]---
Kernel panic - not syncing: Fatal exception in interrupt
Based on the disassembly, it seems the crash happens in
net/xfrm/xfrm_output.c:
int xfrm_output_resume(struct sk_buff *skb, int err)
{
while (likely((err = xfrm_output_one(skb, err)) == 0)) {
nf_reset(skb);
err = skb_dst(skb)->ops->local_out(skb);
^^^^^^^^^^^^ is NULL
if (unlikely(err != 1))
goto out;
if (!skb_dst(skb)->xfrm)
return dst_output(skb);
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: GRE+XFRM+GSO crashes 2013-05-20 6:41 GRE+XFRM+GSO crashes Timo Teras @ 2013-05-20 17:58 ` Pravin Shelar 2013-05-21 8:01 ` Timo Teras 0 siblings, 1 reply; 7+ messages in thread From: Pravin Shelar @ 2013-05-20 17:58 UTC (permalink / raw) To: Timo Teras; +Cc: netdev On Sun, May 19, 2013 at 11:41 PM, Timo Teras <timo.teras@iki.fi> wrote: > Since upgrade from 3.8 to 3.9 I've started getting the below mentioned > BUG crashes. One of the few relevant changes seems to be GSO support in > GRE driver. > > Turning off SG (and GSO) seems to make these crashes disappear. > > The basic setup is: > - gre1 is an NBMA tunnel (no explicit destination, nor bound > target interface; opennhrp daemon creates neigh mappings) > - IPsec policy to encrypt all GRE traffic in transport mode > - VIA Padlock hardware for AES acceleration > - GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso off > > Incidentally, when I tried exact same setup ran as virtualized, I was > unable to reproduce this crash. I suspect it depends on the target NIC > acceleration capabilities. > I do not have access to this hardware, can you tell me what are target device capabilities? Thanks, Pravin. > This is from vanilla 3.9.2 kernel: > BUG: unable to handle kernel NULL pointer dereference at 00000010 > IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 > *pde = 00000000 > Oops: 0000 [#1] SMP > Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport > deflate zlib_deflate ctr twofish_i586 twofish_generic twofish_common > camellia_generic serpent_sse2_i586 xts lrw gf128mul glue_helper > ablk_helper cryptd serpent_generic blowfish_generic blowfish_common > cast5_generic cast_common des_generic cbc xcbc rmd160 sha512_generic > hmac crypto_null af_key xfrm_algo ip_gre ipt_REJECT iptable_filter > ipt_MASQUERADE xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 > nf_nat_ipv4 nf_nat ip_tables ip6t_REJECT xt_LOG xt_limit xt_recent > xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack xt_multiport ip6table_filter ip6_tables x_tables ipv6 > af_packet padlock_sha padlock_aes via_cputemp hwmon hwmon_vid > serio_raw psmouse pcspkr shpchp pci_hotplug i2c_viapro i2c_core > via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus > snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd > soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169 > firmware_class mii evdev fan parport_pc parport thermal button > acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat > sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi > libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop > Pid: 1794, comm: opennhrp Not tainted 3.9.2 #2-Alpine /CN700-8237 > EIP: 0060:[<c123e5b8>] EFLAGS: 00010246 CPU: 0 > EIP is at xfrm_output_resume+0x63/0x2a4 > EAX: 00000000 EBX: f0572500 ECX: f5ffc8b8 EDX: 00000064 > ESI: f0496400 EDI: 00000000 EBP: f0499bf8 ESP: f0499be8 > DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 > CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690 > DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 > DR6: ffff0ff0 DR7: 00000400 > Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0 task.ti=f0498000) > Stack: > f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14 c123e89b > f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c c1236865 f0572500 > 0000003c f0499c38 c12016e7 f0572500 f0499cb0 f84175fb f0572a00 f04bb400 > Call Trace: > [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c > [<c123e806>] xfrm_output2+0xd/0xf > [<c123e89b>] xfrm_output+0x93/0xa0 > [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c > [<c1236837>] xfrm4_output_finish+0x27/0x29 > [<c1236865>] xfrm4_output+0x2c/0x63 > [<c12016e7>] ip_local_out+0x1b/0x1e > [<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre] > [<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c > [<c11de379>] dev_queue_xmit+0x289/0x31a > [<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet] > [<c1040200>] ? ttwu_do_wakeup+0xe/0xa8 > [<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre] > [<c11cd5a9>] sock_sendmsg+0x79/0x94 > [<c10b33d4>] ? __pollwait+0xa4/0xa4 > [<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3 > [<c10367c5>] ? free_pid+0x99/0x9f > [<c106b92a>] ? call_rcu_sched+0xf/0x12 > [<c1026e15>] ? release_task+0x36d/0x37d > [<c10390e8>] ? remove_wait_queue+0x31/0x36 > [<c1027a1d>] ? do_wait+0x1a8/0x1b5 > [<c11cef10>] sys_sendmsg+0x2b/0x46 > [<c11cf394>] sys_socketcall+0x145/0x19f > [<c1257888>] syscall_call+0x7/0xb > Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94 c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0 fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53 > EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP 0068:f0499be8 > CR2: 0000000000000010 > ---[ end trace ac96c6b6b1a4992f ]--- > Kernel panic - not syncing: Fatal exception in interrupt > > > Based on the disassembly, it seems the crash happens in > net/xfrm/xfrm_output.c: > > int xfrm_output_resume(struct sk_buff *skb, int err) > { > while (likely((err = xfrm_output_one(skb, err)) == 0)) { > nf_reset(skb); > > err = skb_dst(skb)->ops->local_out(skb); > ^^^^^^^^^^^^ is NULL > if (unlikely(err != 1)) > goto out; > > if (!skb_dst(skb)->xfrm) > return dst_output(skb); ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: GRE+XFRM+GSO crashes 2013-05-20 17:58 ` Pravin Shelar @ 2013-05-21 8:01 ` Timo Teras 2013-05-21 22:32 ` Pravin Shelar 0 siblings, 1 reply; 7+ messages in thread From: Timo Teras @ 2013-05-21 8:01 UTC (permalink / raw) To: Pravin Shelar; +Cc: netdev On Mon, 20 May 2013 10:58:03 -0700 Pravin Shelar <pshelar@nicira.com> wrote: > On Sun, May 19, 2013 at 11:41 PM, Timo Teras <timo.teras@iki.fi> > wrote: > > Since upgrade from 3.8 to 3.9 I've started getting the below > > mentioned BUG crashes. One of the few relevant changes seems to be > > GSO support in GRE driver. > > > > Turning off SG (and GSO) seems to make these crashes disappear. > > > > The basic setup is: > > - gre1 is an NBMA tunnel (no explicit destination, nor bound > > target interface; opennhrp daemon creates neigh mappings) > > - IPsec policy to encrypt all GRE traffic in transport mode > > - VIA Padlock hardware for AES acceleration > > - GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso > > off > > > > Incidentally, when I tried exact same setup ran as virtualized, I > > was unable to reproduce this crash. I suspect it depends on the > > target NIC acceleration capabilities. > > > I do not have access to this hardware, can you tell me what are target > device capabilities? The physical hardware from which the OOPS is from: r8169 0000:00:09.0 eth0: RTL8169sc/8110sc at 0xf801c000, 00:30:18:a8:14:ac, XID 18000000 IRQ 18 r8169 0000:00:09.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok] (defaults after boot) Offload parameters for eth0: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: off When running virtualized in which I was unable to reproduce this OOPS: e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 52:54:00:12:34:56 e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection Offload parameters for eth0: rx-checksumming: off tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: off > > Thanks, > Pravin. > > This is from vanilla 3.9.2 kernel: > > BUG: unable to handle kernel NULL pointer dereference at 00000010 > > IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 > > *pde = 00000000 > > Oops: 0000 [#1] SMP > > Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport > > deflate zlib_deflate ctr twofish_i586 twofish_generic > > twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul > > glue_helper ablk_helper cryptd serpent_generic blowfish_generic > > blowfish_common cast5_generic cast_common des_generic cbc xcbc > > rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre > > ipt_REJECT iptable_filter ipt_MASQUERADE xt_nat iptable_nat > > nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables > > ip6t_REJECT xt_LOG xt_limit xt_recent xt_policy xt_tcpudp > > nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack xt_multiport > > ip6table_filter ip6_tables x_tables ipv6 af_packet padlock_sha > > padlock_aes via_cputemp hwmon hwmon_vid serio_raw psmouse pcspkr > > shpchp pci_hotplug i2c_viapro i2c_core via_rhine snd_via82xx > > snd_ac97_codec snd_pcm snd_timer ac97_bus snd_page_alloc > > snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore > > firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169 > > firmware_class mii evdev fan parport_pc parport thermal button > > acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat > > sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi > > libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop > > Pid: 1794, comm: opennhrp Not tainted 3.9.2 > > #2-Alpine /CN700-8237 EIP: 0060:[<c123e5b8>] EFLAGS: 00010246 > > CPU: 0 EIP is at xfrm_output_resume+0x63/0x2a4 EAX: 00000000 EBX: > > f0572500 ECX: f5ffc8b8 EDX: 00000064 ESI: f0496400 EDI: 00000000 > > EBP: f0499bf8 ESP: f0499be8 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: > > 0068 CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690 > > DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 > > DR6: ffff0ff0 DR7: 00000400 > > Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0 > > task.ti=f0498000) Stack: > > f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14 > > c123e89b f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c > > c1236865 f0572500 0000003c f0499c38 c12016e7 f0572500 f0499cb0 > > f84175fb f0572a00 f04bb400 Call Trace: > > [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c > > [<c123e806>] xfrm_output2+0xd/0xf > > [<c123e89b>] xfrm_output+0x93/0xa0 > > [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c > > [<c1236837>] xfrm4_output_finish+0x27/0x29 > > [<c1236865>] xfrm4_output+0x2c/0x63 > > [<c12016e7>] ip_local_out+0x1b/0x1e > > [<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre] > > [<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c > > [<c11de379>] dev_queue_xmit+0x289/0x31a > > [<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet] > > [<c1040200>] ? ttwu_do_wakeup+0xe/0xa8 > > [<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre] > > [<c11cd5a9>] sock_sendmsg+0x79/0x94 > > [<c10b33d4>] ? __pollwait+0xa4/0xa4 > > [<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3 > > [<c10367c5>] ? free_pid+0x99/0x9f > > [<c106b92a>] ? call_rcu_sched+0xf/0x12 > > [<c1026e15>] ? release_task+0x36d/0x37d > > [<c10390e8>] ? remove_wait_queue+0x31/0x36 > > [<c1027a1d>] ? do_wait+0x1a8/0x1b5 > > [<c11cef10>] sys_sendmsg+0x2b/0x46 > > [<c11cf394>] sys_socketcall+0x145/0x19f > > [<c1257888>] syscall_call+0x7/0xb > > Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94 > > c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0 > > fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53 > > EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP > > 0068:f0499be8 CR2: 0000000000000010 ---[ end trace ac96c6b6b1a4992f > > ]--- Kernel panic - not syncing: Fatal exception in interrupt > > > > > > Based on the disassembly, it seems the crash happens in > > net/xfrm/xfrm_output.c: > > > > int xfrm_output_resume(struct sk_buff *skb, int err) > > { > > while (likely((err = xfrm_output_one(skb, err)) == 0)) { > > nf_reset(skb); > > > > err = skb_dst(skb)->ops->local_out(skb); > > ^^^^^^^^^^^^ is NULL > > if (unlikely(err != 1)) > > goto out; > > > > if (!skb_dst(skb)->xfrm) > > return dst_output(skb); ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: GRE+XFRM+GSO crashes 2013-05-21 8:01 ` Timo Teras @ 2013-05-21 22:32 ` Pravin Shelar 2013-05-21 22:46 ` Eric Dumazet 2013-05-22 11:09 ` Timo Teras 0 siblings, 2 replies; 7+ messages in thread From: Pravin Shelar @ 2013-05-21 22:32 UTC (permalink / raw) To: Timo Teras; +Cc: netdev [-- Attachment #1: Type: text/plain, Size: 6968 bytes --] On Tue, May 21, 2013 at 1:01 AM, Timo Teras <timo.teras@iki.fi> wrote: > On Mon, 20 May 2013 10:58:03 -0700 > Pravin Shelar <pshelar@nicira.com> wrote: > >> On Sun, May 19, 2013 at 11:41 PM, Timo Teras <timo.teras@iki.fi> >> wrote: >> > Since upgrade from 3.8 to 3.9 I've started getting the below >> > mentioned BUG crashes. One of the few relevant changes seems to be >> > GSO support in GRE driver. >> > >> > Turning off SG (and GSO) seems to make these crashes disappear. >> > >> > The basic setup is: >> > - gre1 is an NBMA tunnel (no explicit destination, nor bound >> > target interface; opennhrp daemon creates neigh mappings) >> > - IPsec policy to encrypt all GRE traffic in transport mode >> > - VIA Padlock hardware for AES acceleration >> > - GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso >> > off >> > >> > Incidentally, when I tried exact same setup ran as virtualized, I >> > was unable to reproduce this crash. I suspect it depends on the >> > target NIC acceleration capabilities. >> > >> I do not have access to this hardware, can you tell me what are target >> device capabilities? > > The physical hardware from which the OOPS is from: > > r8169 0000:00:09.0 eth0: RTL8169sc/8110sc at 0xf801c000, 00:30:18:a8:14:ac, XID 18000000 IRQ 18 > r8169 0000:00:09.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok] > > (defaults after boot) > Offload parameters for eth0: > rx-checksumming: on > tx-checksumming: off > scatter-gather: off > tcp-segmentation-offload: off > udp-fragmentation-offload: off > generic-segmentation-offload: off > generic-receive-offload: on > large-receive-offload: off > rx-vlan-offload: on > tx-vlan-offload: on > ntuple-filters: off > receive-hashing: off > > When running virtualized in which I was unable to reproduce this OOPS: > > e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 52:54:00:12:34:56 > e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection > > Offload parameters for eth0: > rx-checksumming: off > tx-checksumming: on > scatter-gather: on > tcp-segmentation-offload: on > udp-fragmentation-offload: off > generic-segmentation-offload: on > generic-receive-offload: on > large-receive-offload: off > rx-vlan-offload: on > tx-vlan-offload: on > ntuple-filters: off > receive-hashing: off > OK. I am still not able to reproduce it, Can you try attached patch? if that works, I will send out proper patch. >> >> Thanks, >> Pravin. >> > This is from vanilla 3.9.2 kernel: >> > BUG: unable to handle kernel NULL pointer dereference at 00000010 >> > IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 >> > *pde = 00000000 >> > Oops: 0000 [#1] SMP >> > Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport >> > deflate zlib_deflate ctr twofish_i586 twofish_generic >> > twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul >> > glue_helper ablk_helper cryptd serpent_generic blowfish_generic >> > blowfish_common cast5_generic cast_common des_generic cbc xcbc >> > rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre >> > ipt_REJECT iptable_filter ipt_MASQUERADE xt_nat iptable_nat >> > nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables >> > ip6t_REJECT xt_LOG xt_limit xt_recent xt_policy xt_tcpudp >> > nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack xt_multiport >> > ip6table_filter ip6_tables x_tables ipv6 af_packet padlock_sha >> > padlock_aes via_cputemp hwmon hwmon_vid serio_raw psmouse pcspkr >> > shpchp pci_hotplug i2c_viapro i2c_core via_rhine snd_via82xx >> > snd_ac97_codec snd_pcm snd_timer ac97_bus snd_page_alloc >> > snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore >> > firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169 >> > firmware_class mii evdev fan parport_pc parport thermal button >> > acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat >> > sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi >> > libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop >> > Pid: 1794, comm: opennhrp Not tainted 3.9.2 >> > #2-Alpine /CN700-8237 EIP: 0060:[<c123e5b8>] EFLAGS: 00010246 >> > CPU: 0 EIP is at xfrm_output_resume+0x63/0x2a4 EAX: 00000000 EBX: >> > f0572500 ECX: f5ffc8b8 EDX: 00000064 ESI: f0496400 EDI: 00000000 >> > EBP: f0499bf8 ESP: f0499be8 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: >> > 0068 CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690 >> > DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 >> > DR6: ffff0ff0 DR7: 00000400 >> > Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0 >> > task.ti=f0498000) Stack: >> > f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14 >> > c123e89b f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c >> > c1236865 f0572500 0000003c f0499c38 c12016e7 f0572500 f0499cb0 >> > f84175fb f0572a00 f04bb400 Call Trace: >> > [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c >> > [<c123e806>] xfrm_output2+0xd/0xf >> > [<c123e89b>] xfrm_output+0x93/0xa0 >> > [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c >> > [<c1236837>] xfrm4_output_finish+0x27/0x29 >> > [<c1236865>] xfrm4_output+0x2c/0x63 >> > [<c12016e7>] ip_local_out+0x1b/0x1e >> > [<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre] >> > [<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c >> > [<c11de379>] dev_queue_xmit+0x289/0x31a >> > [<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet] >> > [<c1040200>] ? ttwu_do_wakeup+0xe/0xa8 >> > [<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre] >> > [<c11cd5a9>] sock_sendmsg+0x79/0x94 >> > [<c10b33d4>] ? __pollwait+0xa4/0xa4 >> > [<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3 >> > [<c10367c5>] ? free_pid+0x99/0x9f >> > [<c106b92a>] ? call_rcu_sched+0xf/0x12 >> > [<c1026e15>] ? release_task+0x36d/0x37d >> > [<c10390e8>] ? remove_wait_queue+0x31/0x36 >> > [<c1027a1d>] ? do_wait+0x1a8/0x1b5 >> > [<c11cef10>] sys_sendmsg+0x2b/0x46 >> > [<c11cf394>] sys_socketcall+0x145/0x19f >> > [<c1257888>] syscall_call+0x7/0xb >> > Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94 >> > c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0 >> > fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53 >> > EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP >> > 0068:f0499be8 CR2: 0000000000000010 ---[ end trace ac96c6b6b1a4992f >> > ]--- Kernel panic - not syncing: Fatal exception in interrupt >> > >> > >> > Based on the disassembly, it seems the crash happens in >> > net/xfrm/xfrm_output.c: >> > >> > int xfrm_output_resume(struct sk_buff *skb, int err) >> > { >> > while (likely((err = xfrm_output_one(skb, err)) == 0)) { >> > nf_reset(skb); >> > >> > err = skb_dst(skb)->ops->local_out(skb); >> > ^^^^^^^^^^^^ is NULL >> > if (unlikely(err != 1)) >> > goto out; >> > >> > if (!skb_dst(skb)->xfrm) >> > return dst_output(skb); > [-- Attachment #2: 0001-xfrm-Restore-skb-cb-after-segmenting-it.patch --] [-- Type: application/octet-stream, Size: 1091 bytes --] From e437b201925c9cf7d326498785be4847ec07f1cf Mon Sep 17 00:00:00 2001 From: Pravin B Shelar <pshelar@nicira.com> Date: Tue, 21 May 2013 15:23:04 -0700 Subject: [PATCH] xfrm: Restore skb-cb after segmenting it. Signed-off-by: Pravin B Shelar <pshelar@nicira.com> --- net/xfrm/xfrm_output.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c index bcfda89..8c39951 100644 --- a/net/xfrm/xfrm_output.c +++ b/net/xfrm/xfrm_output.c @@ -152,7 +152,12 @@ static int xfrm_output2(struct sk_buff *skb) static int xfrm_output_gso(struct sk_buff *skb) { struct sk_buff *segs; + struct inet_skb_parm cb; + /* GSO code make use of skb control block, save it locally, + * so that we can restore it back to individual segments. + **/ + cb = *IPCB(skb); segs = skb_gso_segment(skb, 0); kfree_skb(skb); if (IS_ERR(segs)) @@ -163,6 +168,8 @@ static int xfrm_output_gso(struct sk_buff *skb) int err; segs->next = NULL; + *IPCB(segs) = cb; + err = xfrm_output2(segs); if (unlikely(err)) { -- 1.7.1 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: GRE+XFRM+GSO crashes 2013-05-21 22:32 ` Pravin Shelar @ 2013-05-21 22:46 ` Eric Dumazet 2013-05-22 4:56 ` Pravin Shelar 2013-05-22 11:09 ` Timo Teras 1 sibling, 1 reply; 7+ messages in thread From: Eric Dumazet @ 2013-05-21 22:46 UTC (permalink / raw) To: Pravin Shelar; +Cc: Timo Teras, netdev On Tue, 2013-05-21 at 15:32 -0700, Pravin Shelar wrote: > OK. I am still not able to reproduce it, Can you try attached patch? > if that works, I will send out proper patch. strange, dev_gso_segment() doesnt change skb->cb[] only dev_gso_segment() does. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: GRE+XFRM+GSO crashes 2013-05-21 22:46 ` Eric Dumazet @ 2013-05-22 4:56 ` Pravin Shelar 0 siblings, 0 replies; 7+ messages in thread From: Pravin Shelar @ 2013-05-22 4:56 UTC (permalink / raw) To: Eric Dumazet; +Cc: Timo Teras, netdev On Tue, May 21, 2013 at 3:46 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > On Tue, 2013-05-21 at 15:32 -0700, Pravin Shelar wrote: > >> OK. I am still not able to reproduce it, Can you try attached patch? >> if that works, I will send out proper patch. > > strange, dev_gso_segment() doesnt change skb->cb[] > only dev_gso_segment() does. > The patch is about SKB_GSO_CB not DEV_GSO_CB. > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: GRE+XFRM+GSO crashes 2013-05-21 22:32 ` Pravin Shelar 2013-05-21 22:46 ` Eric Dumazet @ 2013-05-22 11:09 ` Timo Teras 1 sibling, 0 replies; 7+ messages in thread From: Timo Teras @ 2013-05-22 11:09 UTC (permalink / raw) To: Pravin Shelar; +Cc: netdev On Tue, 21 May 2013 15:32:35 -0700 Pravin Shelar <pshelar@nicira.com> wrote: > On Tue, May 21, 2013 at 1:01 AM, Timo Teras <timo.teras@iki.fi> wrote: > > On Mon, 20 May 2013 10:58:03 -0700 > > Pravin Shelar <pshelar@nicira.com> wrote: > > > >> On Sun, May 19, 2013 at 11:41 PM, Timo Teras <timo.teras@iki.fi> > >> wrote: > >> > Since upgrade from 3.8 to 3.9 I've started getting the below > >> > mentioned BUG crashes. One of the few relevant changes seems to > >> > be GSO support in GRE driver. > >> > > >> > Turning off SG (and GSO) seems to make these crashes disappear. > >> > > >> > The basic setup is: > >> > - gre1 is an NBMA tunnel (no explicit destination, nor bound > >> > target interface; opennhrp daemon creates neigh mappings) > >> > - IPsec policy to encrypt all GRE traffic in transport mode > >> > - VIA Padlock hardware for AES acceleration > >> > - GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, > >> > gso off > >> > > >> > Incidentally, when I tried exact same setup ran as virtualized, I > >> > was unable to reproduce this crash. I suspect it depends on the > >> > target NIC acceleration capabilities. > >> > > >> I do not have access to this hardware, can you tell me what are > >> target device capabilities? > > > > The physical hardware from which the OOPS is from: > >[snip] > > OK. I am still not able to reproduce it, Can you try attached patch? > if that works, I will send out proper patch. No. Seems this is not GSO related at all, after all. I was unable to reliably reproduce this oops originally and for some weird reason it happened only with GSO and this specific hardware. I finally pinpointed a way to trigger this reliably, and it does happen without GSO too. I'm pretty sure I've pinpointed the commit breaking things, and have a fix already building. Will post it after testing. Thanks for the help anyways. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-05-22 11:08 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-05-20 6:41 GRE+XFRM+GSO crashes Timo Teras 2013-05-20 17:58 ` Pravin Shelar 2013-05-21 8:01 ` Timo Teras 2013-05-21 22:32 ` Pravin Shelar 2013-05-21 22:46 ` Eric Dumazet 2013-05-22 4:56 ` Pravin Shelar 2013-05-22 11:09 ` Timo Teras
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.