[4.10][regression][PM] Oops pci_restore_msi_state() e1000e?

[4.10][regression][PM] Oops pci_restore_msi_state() e1000e?

[Shawn Starr]

Hello kernel devs,

Hibernation appears broken for me in 4.10, I notice the e1000e does not resume on boot, and kernel shows why..

This kernel contains agd5f's 4.11/4.12-drm-next branches merged in.

I am unsure if this is a result of the AMDGPU performing a bad hibernation (since it hangs when it finishes dumping pages to disk and doesn't reboot system when /sys/power/disk is set for 'reboot' mode.) otherwise this bug has been persistent in 4.10-rcX.

Thanks,
Shawn

[  912.256010] BUG: unable to handle kernel NULL pointer dereference at 000000000000003c
[  912.257227] IP: pci_restore_msi_state+0x1bd/0x270
[  912.258218] PGD 0

[  912.260182] Oops: 0000 [#1] SMP
[  912.261168] Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables bridge stp llc toshiba_acpi industrialio toshiba_haps hp_accel lis3lv02d input_polldev intel_rapl arc4 iwlmvm x86_pkg_temp_thermal intel_powerclamp dell_wmi coretemp iTCO_wdt vfat dell_laptop sparse_keymap kvm_intel fat mac80211 iTCO_vendor_support dell_smbios mei_wdt dcdbas dell_smm_hwmon ppdev kvm uvcvideo videobuf2_vmalloc snd_hda_codec_realtek snd_hda_codec_generic snd_usb_audio videobuf2_memops videobuf2_v4l2 irqbypass videobuf2_core snd_hda_codec_hdmi crct10dif_pclmul snd_soc_rt5640 crc32_pclmul snd_soc_rl6231 ghash_clmulni_intel snd_hda_intel snd_soc_core snd_hda_codec snd_usbmidi_lib snd_hda_core snd_compress snd_pcm_dmaengine ac97_bus iwlwifi intel_cstate intel_uncore videodev snd_hwdep intel_rapl_perf snd_seq
[  912.265654]  media snd_rawmidi snd_seq_device cfg80211 snd_pcm snd_timer pcspkr ie31200_edac joydev snd video snd_soc_sst_acpi soundcore mei_me parport_pc edac_core elan_i2c mei parport snd_soc_sst_match i2c_designware_platform shpchp spi_pxa2xx_platform tpm_tis dw_dmac lpc_ich i2c_designware_core tpm_tis_core i2c_i801 tpm dell_smo8800 dell_rbtn rfkill wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc amdkfd amd_iommu_v2 amdgpu i2c_algo_bit drm_kms_helper ttm drm netconsole sdhci_pci sdhci_acpi sdhci crc32c_intel mmc_core serio_raw fjes i2c_hid e1000e ptp pps_core
[  912.269609] CPU: 2 PID: 867 Comm: NetworkManager Tainted: G        W      4.10.0+ #1
[  912.270919] Hardware name: Dell Inc. Precision M6800/05NG6V, BIOS A16 12/01/2015
[  912.272242] task: ffff8f3fb7629d80 task.stack: ffffb79243cec000
[  912.273576] RIP: 0010:pci_restore_msi_state+0x1bd/0x270
[  912.274916] RSP: 0018:ffffb79243cef540 EFLAGS: 00010286
[  912.276262] RAX: 0000000000000000 RBX: ffff8f3fe97f9000 RCX: 0000000000000000
[  912.277623] RDX: 0000000000000080 RSI: 0000000000000246 RDI: 0000000000000246
[  912.278990] RBP: ffffb79243cef560 R08: 0000000000000002 R09: ffffb79243cef50c
[  912.280359] R10: 00000000000000c8 R11: ffff8f3fecc00028 R12: 0000000000000000
[  912.281735] R13: 00000000fffffffc R14: ffff8f3fe97f97e0 R15: ffff8f3fe97f90a0
[  912.283114] FS:  00007fb468d04940(0000) GS:ffff8f3fedc80000(0000) knlGS:0000000000000000
[  912.284506] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  912.286248] CR2: 000000000000003c CR3: 00000007f5881000 CR4: 00000000001406a0
[  912.287652] Call Trace:
[  912.289049]  pci_restore_state.part.35+0x119/0x240
[  912.290449]  ? pci_restore_standard_config+0x40/0x40
[  912.291840]  pci_restore_state+0x18/0x20
[  912.293224]  pci_restore_standard_config+0x37/0x40
[  912.294599]  pci_pm_runtime_resume+0x46/0xa0
[  912.295969]  __rpm_callback+0xc2/0x200
[  912.297337]  ? ipv6_mc_down+0x6d/0x80
[  912.298703]  rpm_callback+0x5f/0x80
[  912.300063]  ? pci_restore_standard_config+0x40/0x40
[  912.301423]  rpm_resume+0x4a4/0x6b0
[  912.302778]  ? inetdev_event+0x3d/0x4e0
[  912.304133]  ? skb_dequeue+0x59/0x70
[  912.305487]  __pm_runtime_resume+0x4e/0x80
[  912.306839]  e1000e_open+0x59/0x450 [e1000e]
[  912.308175]  __dev_open+0xc6/0x140
[  912.309494]  ? _raw_spin_unlock_bh+0x1a/0x20
[  912.310801]  __dev_change_flags+0x9d/0x160
[  912.312103]  dev_change_flags+0x29/0x60
[  912.313391]  do_setlink+0x338/0xc90
[  912.314643]  ? __nla_reserve+0x41/0x50
[  912.315876]  ? __nla_put+0x20/0x30
[  912.317094]  ? _raw_read_unlock_bh+0x20/0x30
[  912.318296]  ? inet6_fill_ifla6_attrs+0x494/0x4b0
[  912.319481]  ? inet6_fill_link_af+0x1a/0x30
[  912.320651]  ? rtnl_fill_ifinfo+0xb62/0x10c0
[  912.321817]  ? nla_parse+0x31/0x110
[  912.323566]  rtnl_newlink+0x7ac/0x8e0
[  912.325432]  ? security_capset+0x60/0x80
[  912.327301]  ? ns_capable_common+0x68/0x80
[  912.329151]  ? ns_capable+0x13/0x20
[  912.330949]  rtnetlink_rcv_msg+0xe6/0x210
[  912.332712]  ? __d_lookup+0x116/0x130
[  912.334455]  ? __kmalloc_node_track_caller+0x1f6/0x2b0
[  912.336182]  ? __alloc_skb+0x87/0x1c0
[  912.337858]  ? rtnl_newlink+0x8e0/0x8e0
[  912.339504]  netlink_rcv_skb+0xa4/0xc0
[  912.341130]  rtnetlink_rcv+0x28/0x30
[  912.342719]  netlink_unicast+0x18c/0x220
[  912.344262]  netlink_sendmsg+0x2f7/0x3b0
[  912.345800]  sock_sendmsg+0x38/0x50
[  912.347304]  ___sys_sendmsg+0x29d/0x2b0
[  912.348765]  ? __do_proc_dointvec+0x309/0x3c0
[  912.350196]  ? proc_dointvec+0x38/0x40
[  912.351623]  ? sysctl_head_finish+0x38/0x40
[  912.353060]  ? proc_sys_call_handler+0xa1/0x110
[  912.354502]  ? lockref_put_or_lock+0x5e/0x80
[  912.355880]  ? dput+0x189/0x260
[  912.357189]  ? mntput+0x24/0x40
[  912.358429]  ? __fget_light+0x25/0x60
[  912.359601]  __sys_sendmsg+0x54/0x90
[  912.360707]  SyS_sendmsg+0x12/0x20
[  912.361745]  entry_SYSCALL_64_fastpath+0x1a/0xa9
[  912.362723] RIP: 0033:0x7fb4675b33b0
[  912.363678] RSP: 002b:00007ffe1dd61eb0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
[  912.364645] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb4675b33b0
[  912.365609] RDX: 0000000000000000 RSI: 00007ffe1dd61f10 RDI: 000000000000000c
[  912.366572] RBP: 00007ffe1dd62390 R08: 0000000000000000 R09: 0000000000000000
[  912.367534] R10: 000055a22f626100 R11: 0000000000000293 R12: 0000000000000001
[  912.368485] R13: 0000000000000002 R14: 0000000000000013 R15: 00007ffe1dd62060
[  912.369438] Code: 66 89 4d e6 0f b7 c9 e8 22 9b fd ff 48 89 df e8 0a 07 bb ff 0f b6 53 4d 8b 73 38 48 8d 4d e6 48 8b 7b 10 83 c2 02 e8 63 87 fd ff <41> 0f b6 4c 24 3c 41 8b 54 24 38 be ff ff ff ff c0 e9 04 83 e1
[  912.370519] RIP: pci_restore_msi_state+0x1bd/0x270 RSP: ffffb79243cef540
[  912.371567] CR2: 000000000000003c
[  912.383128] ---[ end trace 64a23c1a0597e90d ]---

Re: [4.10][regression][PM] Oops pci_restore_msi_state() e1000e? (Re: [BUG] 4.11.0-rc1 panic on shutdown X61s)

[Shawn Starr]

This isn't just Lenovo,

My Dell laptop experiences same problem, there is bugzilla:

https://bugzilla.kernel.org/show_bug.cgi?id=194801

Thanks,
Shawn

On 02/25/2017 04:53 AM, Shawn Starr wrote:
> Hello kernel devs,
>
> Hibernation appears broken for me in 4.10, I notice the e1000e does not resume on boot, and kernel shows why..
>
> This kernel contains agd5f's 4.11/4.12-drm-next branches merged in.
>
> I am unsure if this is a result of the AMDGPU performing a bad hibernation (since it hangs when it finishes dumping pages to disk and doesn't reboot system when /sys/power/disk is set for 'reboot' mode.) otherwise this bug has been persistent in 4.10-rcX.
>
> Thanks,
> Shawn
>
> [  912.256010] BUG: unable to handle kernel NULL pointer dereference at 000000000000003c
> [  912.257227] IP: pci_restore_msi_state+0x1bd/0x270
> [  912.258218] PGD 0
>
> [  912.260182] Oops: 0000 [#1] SMP
> [  912.261168] Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables bridge stp llc toshiba_acpi industrialio toshiba_haps hp_accel lis3lv02d input_polldev intel_rapl arc4 iwlmvm x86_pkg_temp_thermal intel_powerclamp dell_wmi coretemp iTCO_wdt vfat dell_laptop sparse_keymap kvm_intel fat mac80211 iTCO_vendor_support dell_smbios mei_wdt dcdbas dell_smm_hwmon ppdev kvm uvcvideo videobuf2_vmalloc snd_hda_codec_realtek snd_hda_codec_generic snd_usb_audio videobuf2_memops videobuf2_v4l2 irqbypass videobuf2_core snd_hda_codec_hdmi crct10dif_pclmul snd_soc_rt5640 crc32_pclmul snd_soc_rl6231 ghash_clmulni_intel snd_hda_intel snd_soc_core snd_hda_codec snd_usbmidi_lib snd_hda_core snd_compress snd_pcm_dmaengine ac97_bus iwlwifi intel_cstate intel_uncore videodev snd_hwdep intel_rapl_perf snd_seq
> [  912.265654]  media snd_rawmidi snd_seq_device cfg80211 snd_pcm snd_timer pcspkr ie31200_edac joydev snd video snd_soc_sst_acpi soundcore mei_me parport_pc edac_core elan_i2c mei parport snd_soc_sst_match i2c_designware_platform shpchp spi_pxa2xx_platform tpm_tis dw_dmac lpc_ich i2c_designware_core tpm_tis_core i2c_i801 tpm dell_smo8800 dell_rbtn rfkill wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc amdkfd amd_iommu_v2 amdgpu i2c_algo_bit drm_kms_helper ttm drm netconsole sdhci_pci sdhci_acpi sdhci crc32c_intel mmc_core serio_raw fjes i2c_hid e1000e ptp pps_core
> [  912.269609] CPU: 2 PID: 867 Comm: NetworkManager Tainted: G        W      4.10.0+ #1
> [  912.270919] Hardware name: Dell Inc. Precision M6800/05NG6V, BIOS A16 12/01/2015
> [  912.272242] task: ffff8f3fb7629d80 task.stack: ffffb79243cec000
> [  912.273576] RIP: 0010:pci_restore_msi_state+0x1bd/0x270
> [  912.274916] RSP: 0018:ffffb79243cef540 EFLAGS: 00010286
> [  912.276262] RAX: 0000000000000000 RBX: ffff8f3fe97f9000 RCX: 0000000000000000
> [  912.277623] RDX: 0000000000000080 RSI: 0000000000000246 RDI: 0000000000000246
> [  912.278990] RBP: ffffb79243cef560 R08: 0000000000000002 R09: ffffb79243cef50c
> [  912.280359] R10: 00000000000000c8 R11: ffff8f3fecc00028 R12: 0000000000000000
> [  912.281735] R13: 00000000fffffffc R14: ffff8f3fe97f97e0 R15: ffff8f3fe97f90a0
> [  912.283114] FS:  00007fb468d04940(0000) GS:ffff8f3fedc80000(0000) knlGS:0000000000000000
> [  912.284506] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  912.286248] CR2: 000000000000003c CR3: 00000007f5881000 CR4: 00000000001406a0
> [  912.287652] Call Trace:
> [  912.289049]  pci_restore_state.part.35+0x119/0x240
> [  912.290449]  ? pci_restore_standard_config+0x40/0x40
> [  912.291840]  pci_restore_state+0x18/0x20
> [  912.293224]  pci_restore_standard_config+0x37/0x40
> [  912.294599]  pci_pm_runtime_resume+0x46/0xa0
> [  912.295969]  __rpm_callback+0xc2/0x200
> [  912.297337]  ? ipv6_mc_down+0x6d/0x80
> [  912.298703]  rpm_callback+0x5f/0x80
> [  912.300063]  ? pci_restore_standard_config+0x40/0x40
> [  912.301423]  rpm_resume+0x4a4/0x6b0
> [  912.302778]  ? inetdev_event+0x3d/0x4e0
> [  912.304133]  ? skb_dequeue+0x59/0x70
> [  912.305487]  __pm_runtime_resume+0x4e/0x80
> [  912.306839]  e1000e_open+0x59/0x450 [e1000e]
> [  912.308175]  __dev_open+0xc6/0x140
> [  912.309494]  ? _raw_spin_unlock_bh+0x1a/0x20
> [  912.310801]  __dev_change_flags+0x9d/0x160
> [  912.312103]  dev_change_flags+0x29/0x60
> [  912.313391]  do_setlink+0x338/0xc90
> [  912.314643]  ? __nla_reserve+0x41/0x50
> [  912.315876]  ? __nla_put+0x20/0x30
> [  912.317094]  ? _raw_read_unlock_bh+0x20/0x30
> [  912.318296]  ? inet6_fill_ifla6_attrs+0x494/0x4b0
> [  912.319481]  ? inet6_fill_link_af+0x1a/0x30
> [  912.320651]  ? rtnl_fill_ifinfo+0xb62/0x10c0
> [  912.321817]  ? nla_parse+0x31/0x110
> [  912.323566]  rtnl_newlink+0x7ac/0x8e0
> [  912.325432]  ? security_capset+0x60/0x80
> [  912.327301]  ? ns_capable_common+0x68/0x80
> [  912.329151]  ? ns_capable+0x13/0x20
> [  912.330949]  rtnetlink_rcv_msg+0xe6/0x210
> [  912.332712]  ? __d_lookup+0x116/0x130
> [  912.334455]  ? __kmalloc_node_track_caller+0x1f6/0x2b0
> [  912.336182]  ? __alloc_skb+0x87/0x1c0
> [  912.337858]  ? rtnl_newlink+0x8e0/0x8e0
> [  912.339504]  netlink_rcv_skb+0xa4/0xc0
> [  912.341130]  rtnetlink_rcv+0x28/0x30
> [  912.342719]  netlink_unicast+0x18c/0x220
> [  912.344262]  netlink_sendmsg+0x2f7/0x3b0
> [  912.345800]  sock_sendmsg+0x38/0x50
> [  912.347304]  ___sys_sendmsg+0x29d/0x2b0
> [  912.348765]  ? __do_proc_dointvec+0x309/0x3c0
> [  912.350196]  ? proc_dointvec+0x38/0x40
> [  912.351623]  ? sysctl_head_finish+0x38/0x40
> [  912.353060]  ? proc_sys_call_handler+0xa1/0x110
> [  912.354502]  ? lockref_put_or_lock+0x5e/0x80
> [  912.355880]  ? dput+0x189/0x260
> [  912.357189]  ? mntput+0x24/0x40
> [  912.358429]  ? __fget_light+0x25/0x60
> [  912.359601]  __sys_sendmsg+0x54/0x90
> [  912.360707]  SyS_sendmsg+0x12/0x20
> [  912.361745]  entry_SYSCALL_64_fastpath+0x1a/0xa9
> [  912.362723] RIP: 0033:0x7fb4675b33b0
> [  912.363678] RSP: 002b:00007ffe1dd61eb0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
> [  912.364645] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fb4675b33b0
> [  912.365609] RDX: 0000000000000000 RSI: 00007ffe1dd61f10 RDI: 000000000000000c
> [  912.366572] RBP: 00007ffe1dd62390 R08: 0000000000000000 R09: 0000000000000000
> [  912.367534] R10: 000055a22f626100 R11: 0000000000000293 R12: 0000000000000001
> [  912.368485] R13: 0000000000000002 R14: 0000000000000013 R15: 00007ffe1dd62060
> [  912.369438] Code: 66 89 4d e6 0f b7 c9 e8 22 9b fd ff 48 89 df e8 0a 07 bb ff 0f b6 53 4d 8b 73 38 48 8d 4d e6 48 8b 7b 10 83 c2 02 e8 63 87 fd ff <41> 0f b6 4c 24 3c 41 8b 54 24 38 be ff ff ff ff c0 e9 04 83 e1
> [  912.370519] RIP: pci_restore_msi_state+0x1bd/0x270 RSP: ffffb79243cef540
> [  912.371567] CR2: 000000000000003c
> [  912.383128] ---[ end trace 64a23c1a0597e90d ]---
>

RE: [4.10][regression][PM] Oops pci_restore_msi_state() e1000e? (Re: [BUG] 4.11.0-rc1 panic on shutdown X61s)

[Brown, Aaron F]

> From: Shawn Starr [mailto:shawn.starr@rogers.com]
> Sent: Tuesday, March 14, 2017 1:43 PM
> To: linux-kernel@vger.kernel.org
> Cc: Brown, Aaron F <aaron.f.brown@intel.com>
> Subject: Re: [4.10][regression][PM] Oops pci_restore_msi_state() e1000e?
> (Re: [BUG] 4.11.0-rc1 panic on shutdown X61s)
> 
> This isn't just Lenovo,
> 
> My Dell laptop experiences same problem, there is bugzilla:
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=194801

Yes, I had noticed a Dell laptop in one of the reports in addition to the Thinkpads.  As it turns out one of the systems I have been trying to reproduce it on is a Lenova with an 82677LM, a newer chipset than the one Vito Caputo is seeing the problem on (82566) and a slightly older one than the one Boris Petkov reported (82579LM.)  I also have been trying with a SuperMicro that has an 82579LM that matches Boris's lspci output very closely and still have not managed to trigger it.  I used the config Boris provided to build a kernel and tried it on those systems (and a number of other ones) and still have not succeeded in making it break.  Given I've tried with the same config as one that's breaking and the range of chipsets it is appearing on I'm suspicious it's more to do with the system environment than the exact kernel config or e1000e chipset, but really don't know.  If I try to pursue it more I guess I'll try installing a Debian build on some of the systems.

Regardless, Sasha Neftin weighed in on it agreeing it needs to be reverted and the upstream people were included, so hopefully it will get reverted in one of the next dot releases.

-Aaron