GRE+XFRM+GSO crashes

GRE+XFRM+GSO crashes

[Timo Teras]

Since upgrade from 3.8 to 3.9 I've started getting the below mentioned
BUG crashes. One of the few relevant changes seems to be GSO support in
GRE driver.

Turning off SG (and GSO) seems to make these crashes disappear.

The basic setup is:
- gre1 is an NBMA tunnel (no explicit destination, nor bound
  target interface; opennhrp daemon creates neigh mappings)
- IPsec policy to encrypt all GRE traffic in transport mode
- VIA Padlock hardware for AES acceleration
- GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso off

Incidentally, when I tried exact same setup ran as virtualized, I was
unable to reproduce this crash. I suspect it depends on the target NIC
acceleration capabilities.

This is from vanilla 3.9.2 kernel:
 BUG: unable to handle kernel NULL pointer dereference at 00000010
 IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4
 *pde = 00000000 
 Oops: 0000 [#1] SMP 
 Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
 deflate zlib_deflate ctr twofish_i586 twofish_generic twofish_common
 camellia_generic serpent_sse2_i586 xts lrw gf128mul glue_helper
 ablk_helper cryptd serpent_generic blowfish_generic blowfish_common
 cast5_generic cast_common des_generic cbc xcbc rmd160 sha512_generic
 hmac crypto_null af_key xfrm_algo ip_gre ipt_REJECT iptable_filter
 ipt_MASQUERADE xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
 nf_nat_ipv4 nf_nat ip_tables ip6t_REJECT xt_LOG xt_limit xt_recent
 xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
 nf_conntrack xt_multiport ip6table_filter ip6_tables x_tables ipv6
 af_packet padlock_sha padlock_aes via_cputemp hwmon hwmon_vid
 serio_raw psmouse pcspkr shpchp pci_hotplug i2c_viapro i2c_core
 via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus
 snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd
 soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169
 firmware_class mii evdev fan parport_pc parport thermal button
 acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat
 sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi
 libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop
 Pid: 1794, comm: opennhrp Not tainted 3.9.2 #2-Alpine    /CN700-8237
 EIP: 0060:[<c123e5b8>] EFLAGS: 00010246 CPU: 0
 EIP is at xfrm_output_resume+0x63/0x2a4
 EAX: 00000000 EBX: f0572500 ECX: f5ffc8b8 EDX: 00000064
 ESI: f0496400 EDI: 00000000 EBP: f0499bf8 ESP: f0499be8
  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
 CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690
 DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
 DR6: ffff0ff0 DR7: 00000400
 Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0 task.ti=f0498000)
 Stack:
  f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14 c123e89b
  f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c c1236865 f0572500
  0000003c f0499c38 c12016e7 f0572500 f0499cb0 f84175fb f0572a00 f04bb400
 Call Trace:
  [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
  [<c123e806>] xfrm_output2+0xd/0xf
  [<c123e89b>] xfrm_output+0x93/0xa0
  [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
  [<c1236837>] xfrm4_output_finish+0x27/0x29
  [<c1236865>] xfrm4_output+0x2c/0x63
  [<c12016e7>] ip_local_out+0x1b/0x1e
  [<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre]
  [<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c
  [<c11de379>] dev_queue_xmit+0x289/0x31a
  [<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet]
  [<c1040200>] ? ttwu_do_wakeup+0xe/0xa8
  [<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre]
  [<c11cd5a9>] sock_sendmsg+0x79/0x94
  [<c10b33d4>] ? __pollwait+0xa4/0xa4
  [<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3
  [<c10367c5>] ? free_pid+0x99/0x9f
  [<c106b92a>] ? call_rcu_sched+0xf/0x12
  [<c1026e15>] ? release_task+0x36d/0x37d
  [<c10390e8>] ? remove_wait_queue+0x31/0x36
  [<c1027a1d>] ? do_wait+0x1a8/0x1b5
  [<c11cef10>] sys_sendmsg+0x2b/0x46
  [<c11cf394>] sys_socketcall+0x145/0x19f
  [<c1257888>] syscall_call+0x7/0xb
 Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94 c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0 fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53
 EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP 0068:f0499be8
 CR2: 0000000000000010
 ---[ end trace ac96c6b6b1a4992f ]---
 Kernel panic - not syncing: Fatal exception in interrupt


Based on the disassembly, it seems the crash happens in
net/xfrm/xfrm_output.c:

int xfrm_output_resume(struct sk_buff *skb, int err)
{
        while (likely((err = xfrm_output_one(skb, err)) == 0)) {
                nf_reset(skb);

                err = skb_dst(skb)->ops->local_out(skb);
		      ^^^^^^^^^^^^ is NULL
                if (unlikely(err != 1))
                        goto out;

                if (!skb_dst(skb)->xfrm)
                        return dst_output(skb);

Re: GRE+XFRM+GSO crashes

[Pravin Shelar]

On Sun, May 19, 2013 at 11:41 PM, Timo Teras <timo.teras@iki.fi> wrote:
> Since upgrade from 3.8 to 3.9 I've started getting the below mentioned
> BUG crashes. One of the few relevant changes seems to be GSO support in
> GRE driver.
>
> Turning off SG (and GSO) seems to make these crashes disappear.
>
> The basic setup is:
> - gre1 is an NBMA tunnel (no explicit destination, nor bound
>   target interface; opennhrp daemon creates neigh mappings)
> - IPsec policy to encrypt all GRE traffic in transport mode
> - VIA Padlock hardware for AES acceleration
> - GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso off
>
> Incidentally, when I tried exact same setup ran as virtualized, I was
> unable to reproduce this crash. I suspect it depends on the target NIC
> acceleration capabilities.
>
I do not have access to this hardware, can you tell me what are target
device capabilities?

Thanks,
Pravin.
> This is from vanilla 3.9.2 kernel:
>  BUG: unable to handle kernel NULL pointer dereference at 00000010
>  IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4
>  *pde = 00000000
>  Oops: 0000 [#1] SMP
>  Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
>  deflate zlib_deflate ctr twofish_i586 twofish_generic twofish_common
>  camellia_generic serpent_sse2_i586 xts lrw gf128mul glue_helper
>  ablk_helper cryptd serpent_generic blowfish_generic blowfish_common
>  cast5_generic cast_common des_generic cbc xcbc rmd160 sha512_generic
>  hmac crypto_null af_key xfrm_algo ip_gre ipt_REJECT iptable_filter
>  ipt_MASQUERADE xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
>  nf_nat_ipv4 nf_nat ip_tables ip6t_REJECT xt_LOG xt_limit xt_recent
>  xt_policy xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
>  nf_conntrack xt_multiport ip6table_filter ip6_tables x_tables ipv6
>  af_packet padlock_sha padlock_aes via_cputemp hwmon hwmon_vid
>  serio_raw psmouse pcspkr shpchp pci_hotplug i2c_viapro i2c_core
>  via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus
>  snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd
>  soundcore firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169
>  firmware_class mii evdev fan parport_pc parport thermal button
>  acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat
>  sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi
>  libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop
>  Pid: 1794, comm: opennhrp Not tainted 3.9.2 #2-Alpine    /CN700-8237
>  EIP: 0060:[<c123e5b8>] EFLAGS: 00010246 CPU: 0
>  EIP is at xfrm_output_resume+0x63/0x2a4
>  EAX: 00000000 EBX: f0572500 ECX: f5ffc8b8 EDX: 00000064
>  ESI: f0496400 EDI: 00000000 EBP: f0499bf8 ESP: f0499be8
>   DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
>  CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690
>  DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
>  DR6: ffff0ff0 DR7: 00000400
>  Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0 task.ti=f0498000)
>  Stack:
>   f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14 c123e89b
>   f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c c1236865 f0572500
>   0000003c f0499c38 c12016e7 f0572500 f0499cb0 f84175fb f0572a00 f04bb400
>  Call Trace:
>   [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
>   [<c123e806>] xfrm_output2+0xd/0xf
>   [<c123e89b>] xfrm_output+0x93/0xa0
>   [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
>   [<c1236837>] xfrm4_output_finish+0x27/0x29
>   [<c1236865>] xfrm4_output+0x2c/0x63
>   [<c12016e7>] ip_local_out+0x1b/0x1e
>   [<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre]
>   [<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c
>   [<c11de379>] dev_queue_xmit+0x289/0x31a
>   [<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet]
>   [<c1040200>] ? ttwu_do_wakeup+0xe/0xa8
>   [<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre]
>   [<c11cd5a9>] sock_sendmsg+0x79/0x94
>   [<c10b33d4>] ? __pollwait+0xa4/0xa4
>   [<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3
>   [<c10367c5>] ? free_pid+0x99/0x9f
>   [<c106b92a>] ? call_rcu_sched+0xf/0x12
>   [<c1026e15>] ? release_task+0x36d/0x37d
>   [<c10390e8>] ? remove_wait_queue+0x31/0x36
>   [<c1027a1d>] ? do_wait+0x1a8/0x1b5
>   [<c11cef10>] sys_sendmsg+0x2b/0x46
>   [<c11cf394>] sys_socketcall+0x145/0x19f
>   [<c1257888>] syscall_call+0x7/0xb
>  Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94 c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0 fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53
>  EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP 0068:f0499be8
>  CR2: 0000000000000010
>  ---[ end trace ac96c6b6b1a4992f ]---
>  Kernel panic - not syncing: Fatal exception in interrupt
>
>
> Based on the disassembly, it seems the crash happens in
> net/xfrm/xfrm_output.c:
>
> int xfrm_output_resume(struct sk_buff *skb, int err)
> {
>         while (likely((err = xfrm_output_one(skb, err)) == 0)) {
>                 nf_reset(skb);
>
>                 err = skb_dst(skb)->ops->local_out(skb);
>                       ^^^^^^^^^^^^ is NULL
>                 if (unlikely(err != 1))
>                         goto out;
>
>                 if (!skb_dst(skb)->xfrm)
>                         return dst_output(skb);

Re: GRE+XFRM+GSO crashes

[Timo Teras]

On Mon, 20 May 2013 10:58:03 -0700
Pravin Shelar <pshelar@nicira.com> wrote:

> On Sun, May 19, 2013 at 11:41 PM, Timo Teras <timo.teras@iki.fi>
> wrote:
> > Since upgrade from 3.8 to 3.9 I've started getting the below
> > mentioned BUG crashes. One of the few relevant changes seems to be
> > GSO support in GRE driver.
> >
> > Turning off SG (and GSO) seems to make these crashes disappear.
> >
> > The basic setup is:
> > - gre1 is an NBMA tunnel (no explicit destination, nor bound
> >   target interface; opennhrp daemon creates neigh mappings)
> > - IPsec policy to encrypt all GRE traffic in transport mode
> > - VIA Padlock hardware for AES acceleration
> > - GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso
> > off
> >
> > Incidentally, when I tried exact same setup ran as virtualized, I
> > was unable to reproduce this crash. I suspect it depends on the
> > target NIC acceleration capabilities.
> >
> I do not have access to this hardware, can you tell me what are target
> device capabilities?

The physical hardware from which the OOPS is from:

r8169 0000:00:09.0 eth0: RTL8169sc/8110sc at 0xf801c000, 00:30:18:a8:14:ac, XID 18000000 IRQ 18
r8169 0000:00:09.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]

(defaults after boot)
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

When running virtualized in which I was unable to reproduce this OOPS:

e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 52:54:00:12:34:56
e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection

Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

> 
> Thanks,
> Pravin.
> > This is from vanilla 3.9.2 kernel:
> >  BUG: unable to handle kernel NULL pointer dereference at 00000010
> >  IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4
> >  *pde = 00000000
> >  Oops: 0000 [#1] SMP
> >  Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
> >  deflate zlib_deflate ctr twofish_i586 twofish_generic
> > twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul
> > glue_helper ablk_helper cryptd serpent_generic blowfish_generic
> > blowfish_common cast5_generic cast_common des_generic cbc xcbc
> > rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre
> > ipt_REJECT iptable_filter ipt_MASQUERADE xt_nat iptable_nat
> > nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables
> > ip6t_REJECT xt_LOG xt_limit xt_recent xt_policy xt_tcpudp
> > nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack xt_multiport
> > ip6table_filter ip6_tables x_tables ipv6 af_packet padlock_sha
> > padlock_aes via_cputemp hwmon hwmon_vid serio_raw psmouse pcspkr
> > shpchp pci_hotplug i2c_viapro i2c_core via_rhine snd_via82xx
> > snd_ac97_codec snd_pcm snd_timer ac97_bus snd_page_alloc
> > snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore
> > firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169
> > firmware_class mii evdev fan parport_pc parport thermal button
> > acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat
> > sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi
> > libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop
> > Pid: 1794, comm: opennhrp Not tainted 3.9.2
> > #2-Alpine    /CN700-8237 EIP: 0060:[<c123e5b8>] EFLAGS: 00010246
> > CPU: 0 EIP is at xfrm_output_resume+0x63/0x2a4 EAX: 00000000 EBX:
> > f0572500 ECX: f5ffc8b8 EDX: 00000064 ESI: f0496400 EDI: 00000000
> > EBP: f0499bf8 ESP: f0499be8 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS:
> > 0068 CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690
> >  DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> >  DR6: ffff0ff0 DR7: 00000400
> >  Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0
> > task.ti=f0498000) Stack:
> >   f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14
> > c123e89b f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c
> > c1236865 f0572500 0000003c f0499c38 c12016e7 f0572500 f0499cb0
> > f84175fb f0572a00 f04bb400 Call Trace:
> >   [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
> >   [<c123e806>] xfrm_output2+0xd/0xf
> >   [<c123e89b>] xfrm_output+0x93/0xa0
> >   [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
> >   [<c1236837>] xfrm4_output_finish+0x27/0x29
> >   [<c1236865>] xfrm4_output+0x2c/0x63
> >   [<c12016e7>] ip_local_out+0x1b/0x1e
> >   [<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre]
> >   [<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c
> >   [<c11de379>] dev_queue_xmit+0x289/0x31a
> >   [<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet]
> >   [<c1040200>] ? ttwu_do_wakeup+0xe/0xa8
> >   [<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre]
> >   [<c11cd5a9>] sock_sendmsg+0x79/0x94
> >   [<c10b33d4>] ? __pollwait+0xa4/0xa4
> >   [<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3
> >   [<c10367c5>] ? free_pid+0x99/0x9f
> >   [<c106b92a>] ? call_rcu_sched+0xf/0x12
> >   [<c1026e15>] ? release_task+0x36d/0x37d
> >   [<c10390e8>] ? remove_wait_queue+0x31/0x36
> >   [<c1027a1d>] ? do_wait+0x1a8/0x1b5
> >   [<c11cef10>] sys_sendmsg+0x2b/0x46
> >   [<c11cf394>] sys_socketcall+0x145/0x19f
> >   [<c1257888>] syscall_call+0x7/0xb
> >  Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94
> > c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0
> > fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53
> > EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP
> > 0068:f0499be8 CR2: 0000000000000010 ---[ end trace ac96c6b6b1a4992f
> > ]--- Kernel panic - not syncing: Fatal exception in interrupt
> >
> >
> > Based on the disassembly, it seems the crash happens in
> > net/xfrm/xfrm_output.c:
> >
> > int xfrm_output_resume(struct sk_buff *skb, int err)
> > {
> >         while (likely((err = xfrm_output_one(skb, err)) == 0)) {
> >                 nf_reset(skb);
> >
> >                 err = skb_dst(skb)->ops->local_out(skb);
> >                       ^^^^^^^^^^^^ is NULL
> >                 if (unlikely(err != 1))
> >                         goto out;
> >
> >                 if (!skb_dst(skb)->xfrm)
> >                         return dst_output(skb);

Re: GRE+XFRM+GSO crashes

[Pravin Shelar]

[-- Attachment #1: Type: text/plain, Size: 6968 bytes --]

On Tue, May 21, 2013 at 1:01 AM, Timo Teras <timo.teras@iki.fi> wrote:
> On Mon, 20 May 2013 10:58:03 -0700
> Pravin Shelar <pshelar@nicira.com> wrote:
>
>> On Sun, May 19, 2013 at 11:41 PM, Timo Teras <timo.teras@iki.fi>
>> wrote:
>> > Since upgrade from 3.8 to 3.9 I've started getting the below
>> > mentioned BUG crashes. One of the few relevant changes seems to be
>> > GSO support in GRE driver.
>> >
>> > Turning off SG (and GSO) seems to make these crashes disappear.
>> >
>> > The basic setup is:
>> > - gre1 is an NBMA tunnel (no explicit destination, nor bound
>> >   target interface; opennhrp daemon creates neigh mappings)
>> > - IPsec policy to encrypt all GRE traffic in transport mode
>> > - VIA Padlock hardware for AES acceleration
>> > - GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off, gso
>> > off
>> >
>> > Incidentally, when I tried exact same setup ran as virtualized, I
>> > was unable to reproduce this crash. I suspect it depends on the
>> > target NIC acceleration capabilities.
>> >
>> I do not have access to this hardware, can you tell me what are target
>> device capabilities?
>
> The physical hardware from which the OOPS is from:
>
> r8169 0000:00:09.0 eth0: RTL8169sc/8110sc at 0xf801c000, 00:30:18:a8:14:ac, XID 18000000 IRQ 18
> r8169 0000:00:09.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]
>
> (defaults after boot)
> Offload parameters for eth0:
> rx-checksumming: on
> tx-checksumming: off
> scatter-gather: off
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off
> receive-hashing: off
>
> When running virtualized in which I was unable to reproduce this OOPS:
>
> e1000 0000:00:03.0 eth0: (PCI:33MHz:32-bit) 52:54:00:12:34:56
> e1000 0000:00:03.0 eth0: Intel(R) PRO/1000 Network Connection
>
> Offload parameters for eth0:
> rx-checksumming: off
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: off
> receive-hashing: off
>

OK. I am still not able to reproduce it, Can you try attached patch?
if that works, I will send out proper patch.

>>
>> Thanks,
>> Pravin.
>> > This is from vanilla 3.9.2 kernel:
>> >  BUG: unable to handle kernel NULL pointer dereference at 00000010
>> >  IP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4
>> >  *pde = 00000000
>> >  Oops: 0000 [#1] SMP
>> >  Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
>> >  deflate zlib_deflate ctr twofish_i586 twofish_generic
>> > twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul
>> > glue_helper ablk_helper cryptd serpent_generic blowfish_generic
>> > blowfish_common cast5_generic cast_common des_generic cbc xcbc
>> > rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre
>> > ipt_REJECT iptable_filter ipt_MASQUERADE xt_nat iptable_nat
>> > nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_tables
>> > ip6t_REJECT xt_LOG xt_limit xt_recent xt_policy xt_tcpudp
>> > nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack xt_multiport
>> > ip6table_filter ip6_tables x_tables ipv6 af_packet padlock_sha
>> > padlock_aes via_cputemp hwmon hwmon_vid serio_raw psmouse pcspkr
>> > shpchp pci_hotplug i2c_viapro i2c_core via_rhine snd_via82xx
>> > snd_ac97_codec snd_pcm snd_timer ac97_bus snd_page_alloc
>> > snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore
>> > firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169
>> > firmware_class mii evdev fan parport_pc parport thermal button
>> > acpi_cpufreq freq_table mperf processor nls_utf8 nls_cp437 vfat fat
>> > sata_via ehci_pci ehci_hcd uhci_hcd ata_generic pata_via pata_acpi
>> > libata usb_storage usbcore usb_common sd_mod scsi_mod squashfs loop
>> > Pid: 1794, comm: opennhrp Not tainted 3.9.2
>> > #2-Alpine    /CN700-8237 EIP: 0060:[<c123e5b8>] EFLAGS: 00010246
>> > CPU: 0 EIP is at xfrm_output_resume+0x63/0x2a4 EAX: 00000000 EBX:
>> > f0572500 ECX: f5ffc8b8 EDX: 00000064 ESI: f0496400 EDI: 00000000
>> > EBP: f0499bf8 ESP: f0499be8 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS:
>> > 0068 CR0: 80050033 CR2: 00000010 CR3: 3104c000 CR4: 00000690
>> >  DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
>> >  DR6: ffff0ff0 DR7: 00000400
>> >  Process opennhrp (pid: 1794, ti=f0498000 task=f670eff0
>> > task.ti=f0498000) Stack:
>> >   f0496420 f0572500 c1236810 f0572528 f0499c00 c123e806 f0499c14
>> > c123e89b f0572500 c1236810 f0572528 f0499c1c c1236837 f0499c2c
>> > c1236865 f0572500 0000003c f0499c38 c12016e7 f0572500 f0499cb0
>> > f84175fb f0572a00 f04bb400 Call Trace:
>> >   [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
>> >   [<c123e806>] xfrm_output2+0xd/0xf
>> >   [<c123e89b>] xfrm_output+0x93/0xa0
>> >   [<c1236810>] ? xfrm4_extract_output+0x8c/0x8c
>> >   [<c1236837>] xfrm4_output_finish+0x27/0x29
>> >   [<c1236865>] xfrm4_output+0x2c/0x63
>> >   [<c12016e7>] ip_local_out+0x1b/0x1e
>> >   [<f84175fb>] ipgre_tunnel_xmit+0x80a/0x892 [ip_gre]
>> >   [<c11ddff1>] dev_hard_start_xmit+0x27d/0x37c
>> >   [<c11de379>] dev_queue_xmit+0x289/0x31a
>> >   [<f8332308>] packet_sendmsg+0x933/0x9d0 [af_packet]
>> >   [<c1040200>] ? ttwu_do_wakeup+0xe/0xa8
>> >   [<f8416874>] ? ipgre_header_parse+0x15/0x15 [ip_gre]
>> >   [<c11cd5a9>] sock_sendmsg+0x79/0x94
>> >   [<c10b33d4>] ? __pollwait+0xa4/0xa4
>> >   [<c11cd7e3>] __sys_sendmsg+0x16e/0x1f3
>> >   [<c10367c5>] ? free_pid+0x99/0x9f
>> >   [<c106b92a>] ? call_rcu_sched+0xf/0x12
>> >   [<c1026e15>] ? release_task+0x36d/0x37d
>> >   [<c10390e8>] ? remove_wait_queue+0x31/0x36
>> >   [<c1027a1d>] ? do_wait+0x1a8/0x1b5
>> >   [<c11cef10>] sys_sendmsg+0x2b/0x46
>> >   [<c11cf394>] sys_socketcall+0x145/0x19f
>> >   [<c1257888>] syscall_call+0x7/0xb
>> >  Code: ff 8b 43 74 c7 43 70 00 00 00 00 85 c0 74 0f 3e ff 08 0f 94
>> > c2 84 d2 74 05 e8 35 17 e6 ff 8b 43 48 c7 43 74 00 00 00 00 83 e0
>> > fe <8b> 50 10 89 d8 ff 52 34 83 f8 01 89 c7 0f 85 24 02 00 00 8b 53
>> > EIP: [<c123e5b8>] xfrm_output_resume+0x63/0x2a4 SS:ESP
>> > 0068:f0499be8 CR2: 0000000000000010 ---[ end trace ac96c6b6b1a4992f
>> > ]--- Kernel panic - not syncing: Fatal exception in interrupt
>> >
>> >
>> > Based on the disassembly, it seems the crash happens in
>> > net/xfrm/xfrm_output.c:
>> >
>> > int xfrm_output_resume(struct sk_buff *skb, int err)
>> > {
>> >         while (likely((err = xfrm_output_one(skb, err)) == 0)) {
>> >                 nf_reset(skb);
>> >
>> >                 err = skb_dst(skb)->ops->local_out(skb);
>> >                       ^^^^^^^^^^^^ is NULL
>> >                 if (unlikely(err != 1))
>> >                         goto out;
>> >
>> >                 if (!skb_dst(skb)->xfrm)
>> >                         return dst_output(skb);
>

[-- Attachment #2: 0001-xfrm-Restore-skb-cb-after-segmenting-it.patch --]
[-- Type: application/octet-stream, Size: 1091 bytes --]

From e437b201925c9cf7d326498785be4847ec07f1cf Mon Sep 17 00:00:00 2001
From: Pravin B Shelar <pshelar@nicira.com>
Date: Tue, 21 May 2013 15:23:04 -0700
Subject: [PATCH] xfrm: Restore skb-cb after segmenting it.

Signed-off-by: Pravin B Shelar <pshelar@nicira.com>
---
 net/xfrm/xfrm_output.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c
index bcfda89..8c39951 100644
--- a/net/xfrm/xfrm_output.c
+++ b/net/xfrm/xfrm_output.c
@@ -152,7 +152,12 @@ static int xfrm_output2(struct sk_buff *skb)
 static int xfrm_output_gso(struct sk_buff *skb)
 {
 	struct sk_buff *segs;
+	struct inet_skb_parm cb;
 
+	/* GSO code make use of skb control block, save it locally,
+	 * so that we can restore it back to individual segments.
+	 **/
+	cb = *IPCB(skb);
 	segs = skb_gso_segment(skb, 0);
 	kfree_skb(skb);
 	if (IS_ERR(segs))
@@ -163,6 +168,8 @@ static int xfrm_output_gso(struct sk_buff *skb)
 		int err;
 
 		segs->next = NULL;
+		*IPCB(segs) = cb;
+
 		err = xfrm_output2(segs);
 
 		if (unlikely(err)) {
-- 
1.7.1

Re: GRE+XFRM+GSO crashes

[Eric Dumazet]

On Tue, 2013-05-21 at 15:32 -0700, Pravin Shelar wrote:

> OK. I am still not able to reproduce it, Can you try attached patch?
> if that works, I will send out proper patch.

strange, dev_gso_segment() doesnt change skb->cb[]
only dev_gso_segment() does.

Re: GRE+XFRM+GSO crashes

[Pravin Shelar]

On Tue, May 21, 2013 at 3:46 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> On Tue, 2013-05-21 at 15:32 -0700, Pravin Shelar wrote:
>
>> OK. I am still not able to reproduce it, Can you try attached patch?
>> if that works, I will send out proper patch.
>
> strange, dev_gso_segment() doesnt change skb->cb[]
> only dev_gso_segment() does.
>
The patch is about SKB_GSO_CB not DEV_GSO_CB.

>
>

Re: GRE+XFRM+GSO crashes

[Timo Teras]

On Tue, 21 May 2013 15:32:35 -0700
Pravin Shelar <pshelar@nicira.com> wrote:

> On Tue, May 21, 2013 at 1:01 AM, Timo Teras <timo.teras@iki.fi> wrote:
> > On Mon, 20 May 2013 10:58:03 -0700
> > Pravin Shelar <pshelar@nicira.com> wrote:
> >
> >> On Sun, May 19, 2013 at 11:41 PM, Timo Teras <timo.teras@iki.fi>
> >> wrote:
> >> > Since upgrade from 3.8 to 3.9 I've started getting the below
> >> > mentioned BUG crashes. One of the few relevant changes seems to
> >> > be GSO support in GRE driver.
> >> >
> >> > Turning off SG (and GSO) seems to make these crashes disappear.
> >> >
> >> > The basic setup is:
> >> > - gre1 is an NBMA tunnel (no explicit destination, nor bound
> >> >   target interface; opennhrp daemon creates neigh mappings)
> >> > - IPsec policy to encrypt all GRE traffic in transport mode
> >> > - VIA Padlock hardware for AES acceleration
> >> > - GRE traffic goes to r8169 NIC; rx on, gro on, tx off, sg off,
> >> > gso off
> >> >
> >> > Incidentally, when I tried exact same setup ran as virtualized, I
> >> > was unable to reproduce this crash. I suspect it depends on the
> >> > target NIC acceleration capabilities.
> >> >
> >> I do not have access to this hardware, can you tell me what are
> >> target device capabilities?
> >
> > The physical hardware from which the OOPS is from:
> >[snip]
> 
> OK. I am still not able to reproduce it, Can you try attached patch?
> if that works, I will send out proper patch.

No. Seems this is not GSO related at all, after all. I was unable to
reliably reproduce this oops originally and for some weird reason it
happened only with GSO and this specific hardware. I finally
pinpointed a way to trigger this reliably, and it does happen without
GSO too.

I'm pretty sure I've pinpointed the commit breaking things, and have a
fix already building. Will post it after testing.

Thanks for the help anyways.