Kernel panic on kernel-3.10.0-693.21.1.el7 in ndisc.h

Kernel panic on kernel-3.10.0-693.21.1.el7 in ndisc.h

[Roman Makhov]

Hello,

We have a problem with Kernel panic after upgrade from CentOS 7.3
(kernel-3.10.0-514.el7) to CentOS 7.4 (kernel-3.10.0-693.21.1.el7).
It occurs when we have the incoming traffic from other nodes and we
are performing the re-configuration of IPv6 interfaces.

It is high-availability system without 802.15.4 support.

The log of crash:
=========================================================
#10 [ffff88043fc03cf0] async_page_fault at ffffffff816b7798
    [exception RIP: ndisc_send_rs+238]
    RIP: ffffffff8166575e  RSP: ffff88043fc03da8  RFLAGS: 00010202
    RAX: 0000000000000002  RBX: ffff88042caa9000  RCX: 0000000000000001
    RDX: 0000000000000000  RSI: 0000000000000200  RDI: ffffffff816534f7
    RBP: ffff88043fc03dd0   R8: 0000000000000000   R9: ffffffff81e9f1c0
    R10: 0000000000000002  R11: ffff88043fc03da8  R12: 0000000000000008
    R13: 0000000000000006  R14: ffff88043fc03de0  R15: ffffffff81772410
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#11 [ffff88043fc03da0] ndisc_send_rs at ffffffff81665704
=========================================================

I see that crash points on ndisc.h, it is ndisc_ops_opt_addr_space()
in function:
=========================================================
crash> kmem ffffffff8166575e
ffffffff8166575e (T) ndisc_send_rs+238
/usr/src/debug/kernel-3.10.0-693.21.1.el7/linux-3.10.0-693.21.1.el7.x86_64/include/net/ndisc.h:
251

      PAGE        PHYSICAL      MAPPING       INDEX CNT FLAGS
ffffea0000059940   1665000                0        0  1 1fffff00000400 reserved
crash>
=========================================================

I checked the difference between 514 and 693 kernels is in the patch
https://patchwork.kernel.org/patch/9179229/ .

Any suggesions about what I am doing wrong are welcome.

Thanks!
Roman Makhov

Fwd: Kernel panic on kernel-3.10.0-693.21.1.el7 in ndisc.h

[Roman Makhov]

Hello,

We have a problem with Kernel panic after upgrade from CentOS 7.3
(kernel-3.10.0-514.el7) to CentOS 7.4 (kernel-3.10.0-693.21.1.el7).
It occurs when we have the incoming traffic from other nodes and we
are performing the re-configuration of IPv6 interfaces.

It is high-availability system without 802.15.4 support.

The log of crash:
=========================================================
#10 [ffff88043fc03cf0] async_page_fault at ffffffff816b7798
    [exception RIP: ndisc_send_rs+238]
    RIP: ffffffff8166575e  RSP: ffff88043fc03da8  RFLAGS: 00010202
    RAX: 0000000000000002  RBX: ffff88042caa9000  RCX: 0000000000000001
    RDX: 0000000000000000  RSI: 0000000000000200  RDI: ffffffff816534f7
    RBP: ffff88043fc03dd0   R8: 0000000000000000   R9: ffffffff81e9f1c0
    R10: 0000000000000002  R11: ffff88043fc03da8  R12: 0000000000000008
    R13: 0000000000000006  R14: ffff88043fc03de0  R15: ffffffff81772410
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#11 [ffff88043fc03da0] ndisc_send_rs at ffffffff81665704
=========================================================

I see that crash points on ndisc.h, it is ndisc_ops_opt_addr_space()
in function:
=========================================================
crash> kmem ffffffff8166575e
ffffffff8166575e (T) ndisc_send_rs+238
/usr/src/debug/kernel-3.10.0-693.21.1.el7/linux-3.10.0-693.21.1.el7.x86_64/include/net/ndisc.h:
251

      PAGE        PHYSICAL      MAPPING       INDEX CNT FLAGS
ffffea0000059940   1665000                0        0  1 1fffff00000400 reserved
crash>
=========================================================

I checked the difference between 514 and 693 kernels is in the patch
https://patchwork.kernel.org/patch/9179229/ .

Any suggesions about what I am doing wrong are welcome.

Thanks!
Roman Makhov

Re: Kernel panic on kernel-3.10.0-693.21.1.el7 in ndisc.h

[Alexander Aring]

Hi,

On Sun, May 13, 2018 at 02:35:07PM +0300, Roman Makhov wrote:
> Hello,
> 
> We have a problem with Kernel panic after upgrade from CentOS 7.3
> (kernel-3.10.0-514.el7) to CentOS 7.4 (kernel-3.10.0-693.21.1.el7).
> It occurs when we have the incoming traffic from other nodes and we
> are performing the re-configuration of IPv6 interfaces.
> 
> It is high-availability system without 802.15.4 support.
> 
> The log of crash:
> =========================================================
> #10 [ffff88043fc03cf0] async_page_fault at ffffffff816b7798
>     [exception RIP: ndisc_send_rs+238]
>     RIP: ffffffff8166575e  RSP: ffff88043fc03da8  RFLAGS: 00010202
>     RAX: 0000000000000002  RBX: ffff88042caa9000  RCX: 0000000000000001
>     RDX: 0000000000000000  RSI: 0000000000000200  RDI: ffffffff816534f7
>     RBP: ffff88043fc03dd0   R8: 0000000000000000   R9: ffffffff81e9f1c0
>     R10: 0000000000000002  R11: ffff88043fc03da8  R12: 0000000000000008
>     R13: 0000000000000006  R14: ffff88043fc03de0  R15: ffffffff81772410
>     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
> #11 [ffff88043fc03da0] ndisc_send_rs at ffffffff81665704
> =========================================================
> 
> I see that crash points on ndisc.h, it is ndisc_ops_opt_addr_space()
> in function:
> =========================================================
> crash> kmem ffffffff8166575e
> ffffffff8166575e (T) ndisc_send_rs+238
> /usr/src/debug/kernel-3.10.0-693.21.1.el7/linux-3.10.0-693.21.1.el7.x86_64/include/net/ndisc.h:
> 251
> 
>       PAGE        PHYSICAL      MAPPING       INDEX CNT FLAGS
> ffffea0000059940   1665000                0        0  1 1fffff00000400 reserved
> crash>
> =========================================================
> 
> I checked the difference between 514 and 693 kernels is in the patch
> https://patchwork.kernel.org/patch/9179229/ .
> 
> Any suggesions about what I am doing wrong are welcome.
> 

Me as original author of this patch,

I cannot help you with such a dinosaurs kernel. Please try it with the
latest one and check if the problem still exists.

- Alex

Re: Kernel panic on kernel-3.10.0-693.21.1.el7 in ndisc.h

[Alexander Aring]

Hi,

On Mon, May 14, 2018 at 11:40:02AM -0400, Alexander Aring wrote:
...
> I cannot help you with such a dinosaurs kernel. Please try it with the
> latest one and check if the problem still exists.
> 

and please don't write me private e-mails, always use the mailingst.

Thanks.

- Alex

Re: Kernel panic on kernel-3.10.0-693.21.1.el7 in ndisc.h

[Roman Makhov]

Hi Alexander,

Thank you for the answer.
Unfortunately CentOS goes with these dinosaurs.
So we will try to debug the problem in the current one and try to
reproduce on the latest kernel.

Thanks,
Roman.

2018-05-14 18:40 GMT+03:00 Alexander Aring <aring@mojatatu.com>:
> Hi,
>
> On Sun, May 13, 2018 at 02:35:07PM +0300, Roman Makhov wrote:
>> Hello,
>>
>> We have a problem with Kernel panic after upgrade from CentOS 7.3
>> (kernel-3.10.0-514.el7) to CentOS 7.4 (kernel-3.10.0-693.21.1.el7).
>> It occurs when we have the incoming traffic from other nodes and we
>> are performing the re-configuration of IPv6 interfaces.
>>
>> It is high-availability system without 802.15.4 support.
>>
>> The log of crash:
>> =========================================================
>> #10 [ffff88043fc03cf0] async_page_fault at ffffffff816b7798
>>     [exception RIP: ndisc_send_rs+238]
>>     RIP: ffffffff8166575e  RSP: ffff88043fc03da8  RFLAGS: 00010202
>>     RAX: 0000000000000002  RBX: ffff88042caa9000  RCX: 0000000000000001
>>     RDX: 0000000000000000  RSI: 0000000000000200  RDI: ffffffff816534f7
>>     RBP: ffff88043fc03dd0   R8: 0000000000000000   R9: ffffffff81e9f1c0
>>     R10: 0000000000000002  R11: ffff88043fc03da8  R12: 0000000000000008
>>     R13: 0000000000000006  R14: ffff88043fc03de0  R15: ffffffff81772410
>>     ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
>> #11 [ffff88043fc03da0] ndisc_send_rs at ffffffff81665704
>> =========================================================
>>
>> I see that crash points on ndisc.h, it is ndisc_ops_opt_addr_space()
>> in function:
>> =========================================================
>> crash> kmem ffffffff8166575e
>> ffffffff8166575e (T) ndisc_send_rs+238
>> /usr/src/debug/kernel-3.10.0-693.21.1.el7/linux-3.10.0-693.21.1.el7.x86_64/include/net/ndisc.h:
>> 251
>>
>>       PAGE        PHYSICAL      MAPPING       INDEX CNT FLAGS
>> ffffea0000059940   1665000                0        0  1 1fffff00000400 reserved
>> crash>
>> =========================================================
>>
>> I checked the difference between 514 and 693 kernels is in the patch
>> https://patchwork.kernel.org/patch/9179229/ .
>>
>> Any suggesions about what I am doing wrong are welcome.
>>
>
> Me as original author of this patch,
>
> I cannot help you with such a dinosaurs kernel. Please try it with the
> latest one and check if the problem still exists.
>
> - Alex

Re: Kernel panic on kernel-3.10.0-693.21.1.el7 in ndisc.h

[Stephen Hemminger]

On Mon, 14 May 2018 21:29:03 +0300
Roman Makhov <roman.makhov@gmail.com> wrote:

> Hi Alexander,
> 
> Thank you for the answer.
> Unfortunately CentOS goes with these dinosaurs.
> So we will try to debug the problem in the current one and try to
> reproduce on the latest kernel.
> 
> Thanks,
> Roman.

If you are stuck in old kernels, please bug the CentOs maintainers not upstream developers.

Re: Kernel panic on kernel-3.10.0-693.21.1.el7 in ndisc.h

[Michal Kubecek]

On Mon, May 14, 2018 at 02:06:11PM -0700, Stephen Hemminger wrote:
> On Mon, 14 May 2018 21:29:03 +0300 Roman Makhov <roman.makhov@gmail.com> wrote:
> > Thank you for the answer.
> > Unfortunately CentOS goes with these dinosaurs.
> > So we will try to debug the problem in the current one and try to
> > reproduce on the latest kernel.
> 
> If you are stuck in old kernels, please bug the CentOs maintainers not
> upstream developers.

Actually, it's not even the dinosaur. Kernels from RHEL (and therefore
CentOS) or SLES differ from the original mainline version they are based
on quite a lot. It's possible that this bug didn't really exist in the
old 3.10 and is related to one of the backports added to CentOS (or
rather RHEL in this case) kernel. People outside of the distribution
have little idea what was backported and why so unless the issue can be
reproduced with mainline kernel they cannot be expected to help.

Michal Kubecek