Linux kernel error stack

Linux kernel error stack

[Satish Patel]

Folks,

I am installing openstack and as you know i have lots of bridges and
vlan interface on my Linux CentOS 7.5

I was getting following error stack on 3.10 kernel and found this is
kernel bug which required kernel upgrade so now i have upgraded my
kernel to 4.17.12 but i am still seeing same kernel stack error on my
dmesg

I have disable TSO, LRO, SG & GSO on my NIC but still getting error
just wanted to understand what is this and why it popping up


[ 8177.718745] ------------[ cut here ]------------
[ 8177.718748] lxcbr0: caps=(0x00000801ffc070a8, 0x0000000000000000)
len=7292 data_len=7240 gso_size=1448 gso_type=1 ip_summed=3
[ 8177.718756] WARNING: CPU: 21 PID: 139 at net/core/dev.c:2662
skb_warn_bad_offload+0xd8/0xef
[ 8177.718757] Modules linked in: ebtable_filter ebtable_nat tcp_diag
udp_diag inet_diag unix_diag btrfs xor zstd_decompress zstd_compress
xxhash raid6_pq loop iptable_raw veth bonding ip6table_mangle
xt_CHECKSUM rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser ib_srpt
target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_uverbs
ib_umad rdma_cm ib_cm iw_cm sb_edac x86_pkg_temp_thermal
intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd cryptd
glue_helper intel_cstate intel_rapl_perf iTCO_wdt iTCO_vendor_support
ocrdma ib_core joydev input_leds pcspkr ipmi_si hpwdt be2iscsi lpc_ich
ipmi_devintf sg hpilo mfd_core ipmi_msghandler video acpi_power_meter
iscsi_boot_sysfs ioatdma shpchp dca pcc_cpufreq xfs sd_mod mgag200
[ 8177.718807]  i2c_algo_bit serio_raw drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm hpsa scsi_transport_sas
be2net dm_mirror dm_region_hash dm_log vhost_net tun tap vhost
iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_vs iptable_nat
nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_mangle
iptable_filter ipt_REJECT nf_reject_ipv4 ipt_MASQUERADE
nf_nat_masquerade_ipv4 nf_nat nf_conntrack libcrc32c crc32c_intel
ip6table_filter ip6_tables ebtables dm_snapshot dm_bufio dm_multipath
dm_mod 8021q garp mrp ip_tables
[ 8177.718833] CPU: 21 PID: 139 Comm: ksoftirqd/21 Tainted: G        W
        4.17.12-1.el7.elrepo.x86_64 #1
[ 8177.718833] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 06/01/2015
[ 8177.718835] RIP: 0010:skb_warn_bad_offload+0xd8/0xef
[ 8177.718836] RSP: 0000:ffffc90003797470 EFLAGS: 00010282
[ 8177.718837] RAX: 0000000000000000 RBX: ffff8807aaa09c00 RCX: 0000000000000006
[ 8177.718838] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff88042f8d69d0
[ 8177.718840] RBP: ffffc900037974a0 R08: 0000000000000000 R09: 0000000000000d96
[ 8177.718840] R10: 0000000000000003 R11: 0000000000000d95 R12: ffff88082a035000
[ 8177.718841] R13: 0000000000000003 R14: ffff8807aaa09c00 R15: ffffffffa0004618
[ 8177.718843] FS:  0000000000000000(0000) GS:ffff88042f8c0000(0000)
knlGS:0000000000000000
[ 8177.718844] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8177.718845] CR2: 000055ac69aea158 CR3: 000000000220a001 CR4: 00000000001606e0
[ 8177.718846] Call Trace:
[ 8177.718849]  skb_checksum_help+0x1a5/0x1c0
[ 8177.718851]  checksum_tg+0x22/0x29 [xt_CHECKSUM]
[ 8177.718854]  ipt_do_table+0x2e7/0x630 [ip_tables]
[ 8177.718858]  ? br_dev_queue_push_xmit+0x170/0x170
[ 8177.718860]  ? br_forward+0x92/0xa0
[ 8177.718862]  ? br_dev_xmit+0x23f/0x3c0
[ 8177.718865]  iptable_mangle_hook+0x37/0xf0 [iptable_mangle]
[ 8177.718868]  nf_hook_slow+0x42/0xc0
[ 8177.718870]  ip_output+0xd0/0xe0
[ 8177.718873]  ? ip_fragment.constprop.49+0x80/0x80
[ 8177.718875]  ip_forward_finish+0x49/0x70
[ 8177.718877]  ip_forward+0x388/0x470
[ 8177.718880]  ? pskb_trim_rcsum.part.11+0x4c/0x4c
[ 8177.718882]  ip_rcv_finish+0x120/0x400
[ 8177.718885]  ip_sabotage_in+0x35/0x40
[ 8177.718888]  nf_hook_slow+0x42/0xc0
[ 8177.718890]  ip_rcv+0x32f/0x410
[ 8177.718892]  ? inet_del_offload+0x40/0x40
[ 8177.718895]  __netif_receive_skb_core+0x401/0xb10
[ 8177.718896]  ? poll_idle+0x90/0x9a
[ 8177.718898]  ? ebt_do_table+0x663/0x73a [ebtables]
[ 8177.718901]  __netif_receive_skb+0x18/0x60
[ 8177.718903]  netif_receive_skb_internal+0x45/0xf0
[ 8177.718904]  netif_receive_skb+0x1c/0x70
[ 8177.718906]  br_netif_receive_skb+0x28/0x60
[ 8177.718909]  br_pass_frame_up+0xcd/0x160
[ 8177.718913]  ? __br_handle_local_finish+0xa0/0xa0
[ 8177.718915]  br_handle_frame_finish+0x23a/0x450
[ 8177.718917]  ? br_pass_frame_up+0x160/0x160
[ 8177.718919]  br_nf_hook_thresh+0xdf/0xf0
[ 8177.718922]  ? br_pass_frame_up+0x160/0x160
[ 8177.718924]  br_nf_pre_routing_finish+0x1a1/0x380
[ 8177.718926]  ? br_pass_frame_up+0x160/0x160
[ 8177.718928]  ? nf_nat_ipv4_in+0x2e/0x80 [nf_nat_ipv4]
[ 8177.718930]  ? iptable_nat_ipv4_in+0x15/0x20 [iptable_nat]
[ 8177.718932]  br_nf_pre_routing+0x2d6/0x3f0
[ 8177.718934]  ? br_nf_forward_arp+0x270/0x270
[ 8177.718936]  nf_hook_slow+0x42/0xc0
[ 8177.718939]  br_handle_frame+0x1ef/0x300
[ 8177.718941]  ? br_pass_frame_up+0x160/0x160
[ 8177.718945]  __netif_receive_skb_core+0x20d/0xb10
[ 8177.718947]  ? dev_gro_receive+0x2ca/0x5a0
[ 8177.718949]  __netif_receive_skb+0x18/0x60
[ 8177.718950]  netif_receive_skb_internal+0x45/0xf0
[ 8177.718953]  ? tcp4_gro_complete+0x86/0x90
[ 8177.718954]  napi_gro_complete+0x73/0x90
[ 8177.718955]  napi_gro_flush+0x5f/0x80
[ 8177.718957]  napi_complete_done+0x96/0xe0
[ 8177.718960]  be_poll+0x1b3/0x320 [be2net]
[ 8177.718962]  net_rx_action+0x289/0x3f0

Re: Linux kernel error stack

[Florian Westphal]

Satish Patel <satish.txt@gmail.com> wrote:
> I am installing openstack and as you know i have lots of bridges and
> vlan interface on my Linux CentOS 7.5
> 
> I was getting following error stack on 3.10 kernel and found this is
> kernel bug which required kernel upgrade so now i have upgraded my
> kernel to 4.17.12 but i am still seeing same kernel stack error on my
> dmesg
> 
> I have disable TSO, LRO, SG & GSO on my NIC but still getting error
> just wanted to understand what is this and why it popping up

Get rid of CHECKSUM target in the iptables rules.
This thing was added 8 years ago to work around dhcp bugs, I don't
think its use is needed anymore.
Try removing it and see that all VMs can still retrieve IP address
via DHCP.

I'm curious as to the rules, normally CHECKSUM target should be
limited to -p udp --dport bootp; its bad idea to feed it normal
packets, its expensive to do this in software rather than have device
do the checksumming.

As for fix, I'm tempted to send patch to make checksum target
eval a no-op & add deprecation warning on init...

Other "fix" is to

diff --git a/net/netfilter/xt_CHECKSUM.c b/net/netfilter/xt_CHECKSUM.c
index 9f4151ec3e06..23a17dda604d 100644
--- a/net/netfilter/xt_CHECKSUM.c
+++ b/net/netfilter/xt_CHECKSUM.c
@@ -25,8 +25,12 @@ MODULE_ALIAS("ip6t_CHECKSUM");
 static unsigned int
 checksum_tg(struct sk_buff *skb, const struct xt_action_param *par)
 {
-	if (skb->ip_summed == CHECKSUM_PARTIAL)
-		skb_checksum_help(skb);
+	if (skb->ip_summed == CHECKSUM_PARTIAL) {
+		if (skb_shinfo(skb)->gso_size)
+			skb->ip_summed = CHECKSUM_NONE;
+		else
+			skb_checksum_help(skb);
+	}
 
 	return XT_CONTINUE;
 }

(unfortunately, the target isn't restricted to POSTROUTING, sigh).

Re: Linux kernel error stack

[Satish Patel]

Thanks Florian,

FYI, I don't have any CHECKSUM configure in my iptables, i have
following rules, also do you think this kernel WARNNING is just
warning and not impacting my performance, based on that i have to
decided criticality of this issue.

[root@ostack-infra-02 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

On Sun, Aug 5, 2018 at 5:48 AM, Florian Westphal <fw@strlen.de> wrote:
> Satish Patel <satish.txt@gmail.com> wrote:
>> I am installing openstack and as you know i have lots of bridges and
>> vlan interface on my Linux CentOS 7.5
>>
>> I was getting following error stack on 3.10 kernel and found this is
>> kernel bug which required kernel upgrade so now i have upgraded my
>> kernel to 4.17.12 but i am still seeing same kernel stack error on my
>> dmesg
>>
>> I have disable TSO, LRO, SG & GSO on my NIC but still getting error
>> just wanted to understand what is this and why it popping up
>
> Get rid of CHECKSUM target in the iptables rules.
> This thing was added 8 years ago to work around dhcp bugs, I don't
> think its use is needed anymore.
> Try removing it and see that all VMs can still retrieve IP address
> via DHCP.
>
> I'm curious as to the rules, normally CHECKSUM target should be
> limited to -p udp --dport bootp; its bad idea to feed it normal
> packets, its expensive to do this in software rather than have device
> do the checksumming.
>
> As for fix, I'm tempted to send patch to make checksum target
> eval a no-op & add deprecation warning on init...
>
> Other "fix" is to
>
> diff --git a/net/netfilter/xt_CHECKSUM.c b/net/netfilter/xt_CHECKSUM.c
> index 9f4151ec3e06..23a17dda604d 100644
> --- a/net/netfilter/xt_CHECKSUM.c
> +++ b/net/netfilter/xt_CHECKSUM.c
> @@ -25,8 +25,12 @@ MODULE_ALIAS("ip6t_CHECKSUM");
>  static unsigned int
>  checksum_tg(struct sk_buff *skb, const struct xt_action_param *par)
>  {
> -       if (skb->ip_summed == CHECKSUM_PARTIAL)
> -               skb_checksum_help(skb);
> +       if (skb->ip_summed == CHECKSUM_PARTIAL) {
> +               if (skb_shinfo(skb)->gso_size)
> +                       skb->ip_summed = CHECKSUM_NONE;
> +               else
> +                       skb_checksum_help(skb);
> +       }
>
>         return XT_CONTINUE;
>  }
>
> (unfortunately, the target isn't restricted to POSTROUTING, sigh).

Re: Linux kernel error stack

[Florian Westphal]

Satish Patel <satish.txt@gmail.com> wrote:
> Thanks Florian,
> 
> FYI, I don't have any CHECKSUM configure in my iptables,

You have, according to WARN stacktrace you provided.

iptables-save -c
ip6tables-save -c

Re: Linux kernel error stack

[Satish Patel]

I have following kernel logging set.

[root@ostack-infra-02 tools]# cat /proc/sys/kernel/printk
3 4 1 3


iptables output

[root@ostack-infra-02 tools]# iptables-save -c
# Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
*raw
:PREROUTING ACCEPT [42284573:38782693391]
:OUTPUT ACCEPT [219668:42675751]
COMMIT
# Completed on Sun Aug  5 15:52:56 2018
# Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
*nat
:PREROUTING ACCEPT [656413:38450886]
:INPUT ACCEPT [5524:564390]
:OUTPUT ACCEPT [3217:212903]
:POSTROUTING ACCEPT [559058:34193539]
[5197:311820] -A POSTROUTING -s 10.0.3.0/24 ! -d 10.0.3.0/24 -j MASQUERADE
COMMIT
# Completed on Sun Aug  5 15:52:56 2018
# Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
*mangle
:PREROUTING ACCEPT [42209568:38687818558]
:INPUT ACCEPT [170634:197796840]
:FORWARD ACCEPT [42939243:38526139954]
:OUTPUT ACCEPT [198956:41288636]
:POSTROUTING ACCEPT [43138199:38567428590]
[84166:59495417] -A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM
--checksum-fill
[68739:5153476] -A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM
--checksum-fill
[755:275452] -A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp
--dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Sun Aug  5 15:52:56 2018
# Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
*filter
:INPUT ACCEPT [164954:197217701]
:FORWARD ACCEPT [42801701:38377075499]
:OUTPUT ACCEPT [198963:41289860]
[0:0] -A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT
[4932:328918] -A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT
[0:0] -A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT
[755:250585] -A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT
[80745:145594351] -A FORWARD -o lxcbr0 -j ACCEPT
[56801:3471569] -A FORWARD -i lxcbr0 -j ACCEPT
COMMIT
# Completed on Sun Aug  5 15:52:56 2018

On Sun, Aug 5, 2018 at 3:42 PM, Florian Westphal <fw@strlen.de> wrote:
> Satish Patel <satish.txt@gmail.com> wrote:
>> Thanks Florian,
>>
>> FYI, I don't have any CHECKSUM configure in my iptables,
>
> You have, according to WARN stacktrace you provided.
>
> iptables-save -c
> ip6tables-save -c

Re: Linux kernel error stack

[Satish Patel]

Oh wait!! I think you are right there i didn't notice that iptables
has following setting.

"CHECKSUM --checksum-fill"

On Sun, Aug 5, 2018 at 3:54 PM, Satish Patel <satish.txt@gmail.com> wrote:
> I have following kernel logging set.
>
> [root@ostack-infra-02 tools]# cat /proc/sys/kernel/printk
> 3 4 1 3
>
>
> iptables output
>
> [root@ostack-infra-02 tools]# iptables-save -c
> # Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
> *raw
> :PREROUTING ACCEPT [42284573:38782693391]
> :OUTPUT ACCEPT [219668:42675751]
> COMMIT
> # Completed on Sun Aug  5 15:52:56 2018
> # Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
> *nat
> :PREROUTING ACCEPT [656413:38450886]
> :INPUT ACCEPT [5524:564390]
> :OUTPUT ACCEPT [3217:212903]
> :POSTROUTING ACCEPT [559058:34193539]
> [5197:311820] -A POSTROUTING -s 10.0.3.0/24 ! -d 10.0.3.0/24 -j MASQUERADE
> COMMIT
> # Completed on Sun Aug  5 15:52:56 2018
> # Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
> *mangle
> :PREROUTING ACCEPT [42209568:38687818558]
> :INPUT ACCEPT [170634:197796840]
> :FORWARD ACCEPT [42939243:38526139954]
> :OUTPUT ACCEPT [198956:41288636]
> :POSTROUTING ACCEPT [43138199:38567428590]
> [84166:59495417] -A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM
> --checksum-fill
> [68739:5153476] -A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM
> --checksum-fill
> [755:275452] -A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp
> --dport 68 -j CHECKSUM --checksum-fill
> COMMIT
> # Completed on Sun Aug  5 15:52:56 2018
> # Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
> *filter
> :INPUT ACCEPT [164954:197217701]
> :FORWARD ACCEPT [42801701:38377075499]
> :OUTPUT ACCEPT [198963:41289860]
> [0:0] -A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT
> [4932:328918] -A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT
> [0:0] -A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT
> [755:250585] -A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT
> [80745:145594351] -A FORWARD -o lxcbr0 -j ACCEPT
> [56801:3471569] -A FORWARD -i lxcbr0 -j ACCEPT
> COMMIT
> # Completed on Sun Aug  5 15:52:56 2018
>
> On Sun, Aug 5, 2018 at 3:42 PM, Florian Westphal <fw@strlen.de> wrote:
>> Satish Patel <satish.txt@gmail.com> wrote:
>>> Thanks Florian,
>>>
>>> FYI, I don't have any CHECKSUM configure in my iptables,
>>
>> You have, according to WARN stacktrace you provided.
>>
>> iptables-save -c
>> ip6tables-save -c

Re: Linux kernel error stack

[Satish Patel]

I am configured where are those rules stored why iptables -L -n -t nat
not showing them?

On Sun, Aug 5, 2018 at 3:56 PM, Satish Patel <satish.txt@gmail.com> wrote:
> Oh wait!! I think you are right there i didn't notice that iptables
> has following setting.
>
> "CHECKSUM --checksum-fill"
>
> On Sun, Aug 5, 2018 at 3:54 PM, Satish Patel <satish.txt@gmail.com> wrote:
>> I have following kernel logging set.
>>
>> [root@ostack-infra-02 tools]# cat /proc/sys/kernel/printk
>> 3 4 1 3
>>
>>
>> iptables output
>>
>> [root@ostack-infra-02 tools]# iptables-save -c
>> # Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
>> *raw
>> :PREROUTING ACCEPT [42284573:38782693391]
>> :OUTPUT ACCEPT [219668:42675751]
>> COMMIT
>> # Completed on Sun Aug  5 15:52:56 2018
>> # Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
>> *nat
>> :PREROUTING ACCEPT [656413:38450886]
>> :INPUT ACCEPT [5524:564390]
>> :OUTPUT ACCEPT [3217:212903]
>> :POSTROUTING ACCEPT [559058:34193539]
>> [5197:311820] -A POSTROUTING -s 10.0.3.0/24 ! -d 10.0.3.0/24 -j MASQUERADE
>> COMMIT
>> # Completed on Sun Aug  5 15:52:56 2018
>> # Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
>> *mangle
>> :PREROUTING ACCEPT [42209568:38687818558]
>> :INPUT ACCEPT [170634:197796840]
>> :FORWARD ACCEPT [42939243:38526139954]
>> :OUTPUT ACCEPT [198956:41288636]
>> :POSTROUTING ACCEPT [43138199:38567428590]
>> [84166:59495417] -A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM
>> --checksum-fill
>> [68739:5153476] -A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM
>> --checksum-fill
>> [755:275452] -A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp
>> --dport 68 -j CHECKSUM --checksum-fill
>> COMMIT
>> # Completed on Sun Aug  5 15:52:56 2018
>> # Generated by iptables-save v1.4.21 on Sun Aug  5 15:52:56 2018
>> *filter
>> :INPUT ACCEPT [164954:197217701]
>> :FORWARD ACCEPT [42801701:38377075499]
>> :OUTPUT ACCEPT [198963:41289860]
>> [0:0] -A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT
>> [4932:328918] -A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT
>> [0:0] -A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT
>> [755:250585] -A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT
>> [80745:145594351] -A FORWARD -o lxcbr0 -j ACCEPT
>> [56801:3471569] -A FORWARD -i lxcbr0 -j ACCEPT
>> COMMIT
>> # Completed on Sun Aug  5 15:52:56 2018
>>
>> On Sun, Aug 5, 2018 at 3:42 PM, Florian Westphal <fw@strlen.de> wrote:
>>> Satish Patel <satish.txt@gmail.com> wrote:
>>>> Thanks Florian,
>>>>
>>>> FYI, I don't have any CHECKSUM configure in my iptables,
>>>
>>> You have, according to WARN stacktrace you provided.
>>>
>>> iptables-save -c
>>> ip6tables-save -c

Re: Linux kernel error stack

[Florian Westphal]

Satish Patel <satish.txt@gmail.com> wrote:
> > [84166:59495417] -A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM
> > --checksum-fill
> > [68739:5153476] -A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM
> > --checksum-fill

These rules make no sense to me, and are also source of your backtrace.
Who set this up?

If this is coming from openstack, I suggest asking openstack developers
WTH this is supposed to do.

> > [755:275452] -A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp
> > --dport 68 -j CHECKSUM --checksum-fill

This was needed to work around dhcpd issues w. checksum offloading but I
guess that DCHCP will work fine without this rule too nowadays.

So I suggest you simply get rid of these rules.

Re: Linux kernel error stack

[Satish Patel]

Yes this is openstack-ansible deployment tool which set them up. I am
wondering where are these rules saved? I believe openstack-ansible use
LXC container to deploy services so must be part of LXC startup
scripts.

I have checked there is no firewalld and iptables service running on system..

You think i should get rid of all CHEKSUM option in iptables rules? Am i right?


On Sun, Aug 5, 2018 at 4:02 PM, Florian Westphal <fw@strlen.de> wrote:
> Satish Patel <satish.txt@gmail.com> wrote:
>> > [84166:59495417] -A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM
>> > --checksum-fill
>> > [68739:5153476] -A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM
>> > --checksum-fill
>
> These rules make no sense to me, and are also source of your backtrace.
> Who set this up?
>
> If this is coming from openstack, I suggest asking openstack developers
> WTH this is supposed to do.
>
>> > [755:275452] -A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp
>> > --dport 68 -j CHECKSUM --checksum-fill
>
> This was needed to work around dhcpd issues w. checksum offloading but I
> guess that DCHCP will work fine without this rule too nowadays.
>
> So I suggest you simply get rid of these rules.

Re: Linux kernel error stack

[Satish Patel]

Florian,

It seems those rules coming from here
https://github.com/openstack/openstack-ansible-os_neutron/blob/master/files/post-up-metadata-checksum

On Sun, Aug 5, 2018 at 4:09 PM, Satish Patel <satish.txt@gmail.com> wrote:
> Yes this is openstack-ansible deployment tool which set them up. I am
> wondering where are these rules saved? I believe openstack-ansible use
> LXC container to deploy services so must be part of LXC startup
> scripts.
>
> I have checked there is no firewalld and iptables service running on system..
>
> You think i should get rid of all CHEKSUM option in iptables rules? Am i right?
>
>
> On Sun, Aug 5, 2018 at 4:02 PM, Florian Westphal <fw@strlen.de> wrote:
>> Satish Patel <satish.txt@gmail.com> wrote:
>>> > [84166:59495417] -A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM
>>> > --checksum-fill
>>> > [68739:5153476] -A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM
>>> > --checksum-fill
>>
>> These rules make no sense to me, and are also source of your backtrace.
>> Who set this up?
>>
>> If this is coming from openstack, I suggest asking openstack developers
>> WTH this is supposed to do.
>>
>>> > [755:275452] -A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp
>>> > --dport 68 -j CHECKSUM --checksum-fill
>>
>> This was needed to work around dhcpd issues w. checksum offloading but I
>> guess that DCHCP will work fine without this rule too nowadays.
>>
>> So I suggest you simply get rid of these rules.

Re: Linux kernel error stack

[Satish Patel]

Florian,

I have removed those port 80 CHECKSUM rules and everything looks good
i didn't see kernel WARN mesg.

Thank you so much! You just nailed it :)

On Sun, Aug 5, 2018 at 4:15 PM, Satish Patel <satish.txt@gmail.com> wrote:
> Florian,
>
> It seems those rules coming from here
> https://github.com/openstack/openstack-ansible-os_neutron/blob/master/files/post-up-metadata-checksum
>
> On Sun, Aug 5, 2018 at 4:09 PM, Satish Patel <satish.txt@gmail.com> wrote:
>> Yes this is openstack-ansible deployment tool which set them up. I am
>> wondering where are these rules saved? I believe openstack-ansible use
>> LXC container to deploy services so must be part of LXC startup
>> scripts.
>>
>> I have checked there is no firewalld and iptables service running on system..
>>
>> You think i should get rid of all CHEKSUM option in iptables rules? Am i right?
>>
>>
>> On Sun, Aug 5, 2018 at 4:02 PM, Florian Westphal <fw@strlen.de> wrote:
>>> Satish Patel <satish.txt@gmail.com> wrote:
>>>> > [84166:59495417] -A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM
>>>> > --checksum-fill
>>>> > [68739:5153476] -A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM
>>>> > --checksum-fill
>>>
>>> These rules make no sense to me, and are also source of your backtrace.
>>> Who set this up?
>>>
>>> If this is coming from openstack, I suggest asking openstack developers
>>> WTH this is supposed to do.
>>>
>>>> > [755:275452] -A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp
>>>> > --dport 68 -j CHECKSUM --checksum-fill
>>>
>>> This was needed to work around dhcpd issues w. checksum offloading but I
>>> guess that DCHCP will work fine without this rule too nowadays.
>>>
>>> So I suggest you simply get rid of these rules.

Re: Linux kernel error stack

[Satish Patel]

After reading further related DHCP checksum issue, it seems we need
that rules when you running DHCP on same host machine where your guest
using host DHCP service, in that case virtual nic won't do checksum.
If your DHCP running on different host then your physical nic perform
checksum.

On Sun, Aug 5, 2018 at 4:39 PM, Satish Patel <satish.txt@gmail.com> wrote:
> Florian,
>
> I have removed those port 80 CHECKSUM rules and everything looks good
> i didn't see kernel WARN mesg.
>
> Thank you so much! You just nailed it :)
>
> On Sun, Aug 5, 2018 at 4:15 PM, Satish Patel <satish.txt@gmail.com> wrote:
>> Florian,
>>
>> It seems those rules coming from here
>> https://github.com/openstack/openstack-ansible-os_neutron/blob/master/files/post-up-metadata-checksum
>>
>> On Sun, Aug 5, 2018 at 4:09 PM, Satish Patel <satish.txt@gmail.com> wrote:
>>> Yes this is openstack-ansible deployment tool which set them up. I am
>>> wondering where are these rules saved? I believe openstack-ansible use
>>> LXC container to deploy services so must be part of LXC startup
>>> scripts.
>>>
>>> I have checked there is no firewalld and iptables service running on system..
>>>
>>> You think i should get rid of all CHEKSUM option in iptables rules? Am i right?
>>>
>>>
>>> On Sun, Aug 5, 2018 at 4:02 PM, Florian Westphal <fw@strlen.de> wrote:
>>>> Satish Patel <satish.txt@gmail.com> wrote:
>>>>> > [84166:59495417] -A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM
>>>>> > --checksum-fill
>>>>> > [68739:5153476] -A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM
>>>>> > --checksum-fill
>>>>
>>>> These rules make no sense to me, and are also source of your backtrace.
>>>> Who set this up?
>>>>
>>>> If this is coming from openstack, I suggest asking openstack developers
>>>> WTH this is supposed to do.
>>>>
>>>>> > [755:275452] -A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp
>>>>> > --dport 68 -j CHECKSUM --checksum-fill
>>>>
>>>> This was needed to work around dhcpd issues w. checksum offloading but I
>>>> guess that DCHCP will work fine without this rule too nowadays.
>>>>
>>>> So I suggest you simply get rid of these rules.

Re: Linux kernel error stack

[Florian Westphal]

Satish Patel <satish.txt@gmail.com> wrote:
> After reading further related DHCP checksum issue, it seems we need
> that rules when you running DHCP on same host machine where your guest
> using host DHCP service, in that case virtual nic won't do checksum.
> If your DHCP running on different host then your physical nic perform
> checksum.

Partially right, its only needed for older dhcp software that relies
on packet-header checksum correctness (metadata contains info when
checksum will be filled in by nic later, modern servers will
use that and ignore the packet-header checksums in this case).

If you remove the rule and your VMs still get leases via DHCP then
the CHECKSUM workaround isn't needed.

Re: Linux kernel error stack

[Florian Westphal]

Satish Patel <satish.txt@gmail.com> wrote:
> Florian,
> 
> It seems those rules coming from here
> https://github.com/openstack/openstack-ansible-os_neutron/blob/master/files/post-up-metadata-checksum

This is crazy, and, as you found, it doesn't even do what they seem to
think it does.

I see no reason for these rules at all.

Re: Linux kernel error stack

[Michal Kubecek]

On Sun, Aug 05, 2018 at 11:03:42PM +0200, Florian Westphal wrote:
> Satish Patel <satish.txt@gmail.com> wrote:
> > Florian,
> > 
> > It seems those rules coming from here
> > https://github.com/openstack/openstack-ansible-os_neutron/blob/master/files/post-up-metadata-checksum
> 
> This is crazy, and, as you found, it doesn't even do what they seem to
> think it does.
> 
> I see no reason for these rules at all.

Oops, exactly this issue was already discussed almost a year ago:

  http://lkml.kernel.org/r/20170824104824.2C318A0F3A@unicorn.suse.cz

But something more urgent came and I forgot to get back to it. :-(

Michal Kubecek

Re: Linux kernel error stack

[Florian Westphal]

Michal Kubecek <mkubecek@suse.cz> wrote:
> Oops, exactly this issue was already discussed almost a year ago:
> 
>   http://lkml.kernel.org/r/20170824104824.2C318A0F3A@unicorn.suse.cz
> 
> But something more urgent came and I forgot to get back to it. :-(

I did not even remeber, thanks for the pointer.
So I think best course of action is to update man page to clearly
say this only works in postrouting and with udp, and is ONLY
intended for working around old dhcp software.

>From kernel, have checkentry emit a warning
when this is used for protocols other than udp,
and make this a no-op for everything else.

Same for postrouting: warn from checkentry if the hook is something
else, and have target function not do anything for !postrouting.

Does that make sense?

I can work on a patch later this week if needed.

Re: Linux kernel error stack

[Michal Kubecek]

On Mon, Aug 06, 2018 at 01:15:37AM +0200, Florian Westphal wrote:
> Michal Kubecek <mkubecek@suse.cz> wrote:
> > Oops, exactly this issue was already discussed almost a year ago:
> > 
> >   http://lkml.kernel.org/r/20170824104824.2C318A0F3A@unicorn.suse.cz
> > 
> > But something more urgent came and I forgot to get back to it. :-(
> 
> I did not even remeber, thanks for the pointer.
> So I think best course of action is to update man page to clearly
> say this only works in postrouting and with udp, and is ONLY
> intended for working around old dhcp software.

As GSO for UDP is on its way to mainline, one might get into trouble
even with UDP if the rule is not specific enough.
 
> From kernel, have checkentry emit a warning
> when this is used for protocols other than udp,
> and make this a no-op for everything else.
> 
> Same for postrouting: warn from checkentry if the hook is something
> else, and have target function not do anything for !postrouting.
> 
> Does that make sense?

It sounds reasonable to me.

> I can work on a patch later this week if needed.

I'll be on vacation until Sunday so that I certainly won't get to it
earlier than next week.

Michal Kubecek

Re: Linux kernel error stack

[Florian Westphal]

Michal Kubecek <mkubecek@suse.cz> wrote:
> On Mon, Aug 06, 2018 at 01:15:37AM +0200, Florian Westphal wrote:
> > Michal Kubecek <mkubecek@suse.cz> wrote:
> > > Oops, exactly this issue was already discussed almost a year ago:
> > > 
> > >   http://lkml.kernel.org/r/20170824104824.2C318A0F3A@unicorn.suse.cz
> > > 
> > > But something more urgent came and I forgot to get back to it. :-(
> > 
> > I did not even remeber, thanks for the pointer.
> > So I think best course of action is to update man page to clearly
> > say this only works in postrouting and with udp, and is ONLY
> > intended for working around old dhcp software.
> 
> As GSO for UDP is on its way to mainline, one might get into trouble
> even with UDP if the rule is not specific enough.

Yes, we still need a fix to ignore GSO too.